Can we move _map output out of the main Zarr tree?

[j08lue]

It is currently hard-coded that with each Zarr array of an indicator, the code writes a _map version of the same data, which has been reprojected to Web Mercator.

First of all, I think the creation of this copy should be optional - many will not need it, e.g. when using the indicators as input to Physrisk.

Secondly, the _map folders inside the Zarr tree can confuse clients to think that they are arrays in the same group. I think they should be separated at a higher level, perhaps a completely separate Zarr store, since they have a different projection than the main data and will in many ways not be compatible.

What do the maintainers think?

[joemoorhouse]

Hi @j08lue,

Yes, agree it should be separated at a higher level and optional as:

• it has a different projection
• only some users may want this
  I reached the same conclusion and starting prefixing with 'maps' so it is separated at a higher level.
  See e.g.
  https://github.com/os-climate/hazard/blob/main/src/hazard/onboard/flopros_flood.py
  where the pack 'maps/inundation/flopros_{path_component(k)}/v1/flood_sop_map' is prefixed with 'maps'.
  Thanks,
  Joe

[j08lue]

Ok, so the behaviour around maps is currently implemented differently between indicators. My comment above was about degree_days. I suggest to

1. Adjust the behaviour of degree_days to match that of flopros_flood
2. Introduce a global CLI flag to disable maps production

Please note that different projections and multiscale (optimized for dynamic data visualization e.g. on web maps) will be part of the upcoming GeoZarr spec - we hope to see some good advancements on that in the next few months, btw. Perhaps the maps production in this code here should be reconsidered, once that spec is out.

[j08lue]

1. Adjust the behaviour of degree_days to match that of flopros_flood

Something like this: developmentseed@92ee4f1

Although it will fail because we have source="map_array" and I see the function requires source="map_array_pyramid" - no idea what that means, though.

[joemoorhouse]

Yes, definitely we will want to reconsider once spec is out. source="map_array_pyramid is probably now the recommended approach for visualization. This prepares the map tiles into a Zarr format ready for XYZ Web Mercator tiles.