Design doc - Publish Dandisets that contain Zarr archives

[kabilar]

• Define user requirements
  • Update design doc based on 2024-01-26 discussion @kabilar
• Define technical specifications @waxlamp @kabilar
• Prototype and evaluate tradeoffs of potential solutions
  • Pure DANDI side based on zarr manifest files
    • Create manifest files with paths and version IDs for each chunk for a specific version of the Zarr archive @yarikoptic
      • Implemented in this repo: https://github.com/dandi/zarr-manifests/ with more complete copy at https://datasets.datalad.org/?dir=/dandi/zarr-manifests/zarr-manifests-v2-sorted
    • Demonstrate use in dandidav: planned in Provide zarrs/ top level folder with support for experimental zarr manifests dandidav#43
    • Work out design for support in dandi-archive to adopt such an approach
  • Explore Arraylake @waxlamp

[kabilar]

cc @aaronkanzer

[waxlamp]

Leaving this review as a comment because it seems this is still WIP. I will also throw the switch to convert this PR to a draft.

[waxlamp]

Suggested change

When a blob asset is updated, a new version (i.e. a copy) is uploaded to the S3 bucket. Zarr archives are too large so multiple copies should not be created. A Zarr archive is uploaded once and it is updated in place. This design means that the Zarr archive is immutable once the Dandiset is published, so that the published Dandiset is immutable. Currently, a Dandiset cannot be published if it contains a Zarr asset. For more details, see the [zarr-support-3 design doc](https://github.com/dandi/dandi-archive/blob/master/doc/design/zarr-support-3.md).

When a non-Zarr asset blob is updated, a new copy of that file is uploaded to the S3 bucket. Zarr archives are too large so multiple copies should not be created. A Zarr archive is uploaded once and it is updated in place. This design means that the Zarr archive is immutable once the Dandiset is published, so that the published Dandiset is immutable. Currently, a Dandiset cannot be published if it contains a Zarr asset. For more details, see the [zarr-support-3 design doc](https://github.com/dandi/dandi-archive/blob/master/doc/design/zarr-support-3.md).

Updating to use more accurate domain language (there isn't such a thing as a "blob asset", and the word "version" has a specific meaning in the context of DANDI).

[waxlamp]

Essentially, we disallow publishing of Zarr-containing Dandisets because we don't want to create copies of Zarrs if and when they are updated, and we're seeking a design that would allow us to do so. You might be able to condense this paragraph down to express that more directly; the sentence in the middle ("this design means...") seems a bit out of place in particular.

[satra]

i would add that there are a two requirements.

• updating/publishing in a dandiset
• adding the asset to another dandiset
• being able to publish both

for implementation:

• we currently don't have a readonly mode for an asset and any modification of zarr added to another dandiset should fail, unless control is handed over (more complicated).
• adding should have either a link (readonly) or copy option. on the blob side, we have copy on write, but for assets this could get complicated for large trees.

[waxlamp]

This step seems out of place in the use case description. We can already do this part (update a Zarr that is in a draft version), and the important novelty is in step 3 and beyond.

[kabilar]

I reordered steps 2 and 3, which would reflect functionality that is currently not in place (i.e. publishing a Dandiset with a Zarr archive and subsequently updating the Draft version). Please let me know if I misunderstood your comment.

[kabilar]

Closing this pull request as I am re-forking dandi/dandi-archive directly to kabilar. Will reopen a new pull request.