Updating existing data in Zarr storage

[vietnguyengit]

Hi guys, apologise if this question has been asked, I couldn't find so I ask here.

In my scenario, I have 200000 NetCDF files storing marine data converted to Zarr (let's call it Big Zarr). It's common that after a few weeks/months, the data providers (where I received NetCDF files from) make some changes to the NetCDF files they sent before, not all 200k files, but just... for example 100 NetCDF files produced from a particular device due to new calibration enhancement.

When that happens, I want to just overwrite a specific region of Big Zarr holding data of the revised NetCDF files. Of course, I don't want to rewrite the whole 200k files to Zarr, I also don't want to append new revised NetCDF files to the end of current Big Zarr storage.

My question are, will that be doable? And if so, what should I need to look at ?

Thank you very much.

[davidbrochart]

Hi @vietnguyengit,

It's definitely possible to write to a subset of your Zarr array, for instance:

z[row0:row1, col0:col1] = data

But your data may have been stored using xarray's to_zarr, so you might want to access it using a coordinate system instead of raw indices.

[vietnguyengit]

Hi @davidbrochart , thanks for your rely. I tested out today after exploring a few docs, and yes it's doable. I identify the region to be replaced then use to_zarr() with mode r+ and the region. Works great!

My next question, I think it's worth to ask, instead of replacing the region with new data, after identifying the region, I want to remove it out from current Zarr storage, is this doable as well? And what should I need to look at?

Thank you.

[davidbrochart]

It depends if by removing data you mean just removing chunk files, while keeping the same coordinates. In this case yes, you can delete chunks and they will be initialized on read with the fill_value. You might also want to look at the docs for empty chunks.

[vietnguyengit]

Perfect thanks @davidbrochart

[vietnguyengit]

Hi, @davidbrochart hope you're doing well, I come across this topic again and realise that when "deleting" chunks, write_empty_chunks=True not going to work with to_zarr(mode='r+'), not sure if it's a Zarr restriction or I just don't know the answer.

The reason I need to use r+ is that I want to update only a region of the Zarr store where a source NetCDF file was ingested. E.g ingested 26 files to the Zarr store, but then the data provider tells me to remove the 5th file, I want to fill the region of that 5th file from the existing Zarr store with empty values, not appending (a mode), not rewriting (w mode).

Currently, I try to create an "empty" xarray dataset and do it the same way when the pipeline receives a revised NetCDF file with to_zarr(mode='r+', region=dict())

Can you please advise? Thanks, David.