Add RAWR data fetcher #244
Conversation
…r RAWR DataFetcher.
This is now in `common.py`, making it easier to see where the shared code is. Also factored out an interface, implemented by OsmFixtureLookup, which is used to make queries about relationships between elements; mainly station relation complexes and highway routes, etc...
… to an expanded box.
Previously, the code looped over layers and looked up features in a per-layer index. This caused a problem for some types of processing. For example, for a feature which might appear in `pois`, `landuse` and `buildings` layers, that only one of those features is named. This is considerably harder to do when the feature is extracted from several different indexes. Another issue was that the properties were copied for each feature and in each layer to add the `min_zoom` property, which meant that there was a lot more memory usage and GC pressure. Since the size of RAWR tiles is a concern, it would be better to keep the memory usage down where possible. The new code has a single index, and keeps `min_zoom` for each layer in a separate `dict`. This means that the code which iterates over the layers has to do a second check for the zoom level, and the feature is included at the minimum zoom level it appears in any layer. A side effect of this is that each `_Feature` object is unique, and so we can use `id()` to check equality, which helps when de-duplicating across tiles.
…oms are precomputed for all layers.
There was a problem hiding this comment.
LGTM!
About the data fetcher itself, it looks like we're going to be creating a new one for each tile pyramid. Are we going to introduce a higher level concept, like a data service abstraction, that manages this given any tile? Who will fetch the data from s3 and hand that to the data fetcher?
|
|
||
| def tile(self): | ||
| from raw_tiles.tile import Tile | ||
| return Tile(self.z, self.x, self.y) |
There was a problem hiding this comment.
I wonder if we should at some point just create our own repo to hold these kinds of objects and related utility functions?
| @staticmethod | ||
| def lookup(typ): | ||
| "turn the enum into a string" | ||
| return ShapeType._LOOKUP[typ-1] |
There was a problem hiding this comment.
What do you think of using the enum package here?
There was a problem hiding this comment.
Good point. At first, it didn't seem necessary to bring that in for something which, at least initially, was very simple. After adding the lookup function, it's starting to look like it would be cleaner with the enum package. Fixed in #247.
This adds a RAWR compatible data fetcher, although it isn't hooked up to actually download tiles at the moment. The only use of it is from the test suite. The next step will be to hook it into the rendering pipeline and allow configuring it.
Several issues with the current approach (shared with the fixture data fetcher):
vector-datasource. Hopefully once the code is working, we can pull those out and put them wherever would be more appropriate.