dense node format

[heffergm]

Hi all,
I was wondering if I could get your thoughts on what sort of level of effort it would require to allow for extracts in dense node format. I see that it's mentioned in the readme, and the question derives from an interest we've got in starting to build and run an on demand extract system. I know we'd discussed this a while back, and it got derailed, but there's interest here in picking it up again.

Right now, in the format you're outputting, it looks like there's limited usefulness for a lot of standardized tooling that expects dense nodes. We're still doing a POC, but if it gets accepted I think we could contribute some code to make this happen if you think it's something that makes sense.

Ultimately we'd be looking to run an API based on vex, behind an api key based system (only for the purposes of rate limiting and being able to get in touch with users when necessary), as well as a front end for ease of use (poc: http://hkrishna.github.io/ode/#12/40.7261/-73.9807).

Let me know your thoughts...

[abyrd]

A few weeks back I completed minutely update support in our Java version of Vanilla Extract. After some tweaks to the inner workings the performance of the Java version is completely acceptable, and it will be much less challenging to maintain and extend. Therefore, at this point all development will focus on the Java version.

Note that while Vanilla Extract keeps all imported OSM data, its indexing focuses entirely on ways. Geographic extracts contain only nodes that are referenced by ways. More so than lack of dense nodes in PBF output, this makes the extracts unsuitable for most people.

The reason for this is that it is focused on routing. VEX could probably be extended to track “lone” nodes that are not in any way without too much difficulty, but these lone nodes were just not a high priority for our use cases.

I increasingly dislike the PBF format and hope to see it replaced in the future, but it is of course the current standard and needs to be supported. It would be straightforward to produce dense-nodes output in the Java version of Vanilla Extract, it’s just a matter of setting aside some time to add it to Conveyal’s osm-library.