Skip to content

Commit

Permalink
Update requirements doc
Browse files Browse the repository at this point in the history
  • Loading branch information
kabilar committed Jan 26, 2024
1 parent dd40a75 commit 2987b81
Showing 1 changed file with 12 additions and 10 deletions.
22 changes: 12 additions & 10 deletions doc/design/zarr-publish-1.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,31 @@
# Publish Dandisets with Zarr archives
# Support updates to Zarr archives after publishing the corresponding Dandiset

This document describes the current implementation of publishing Dandisets with Zarr archives, a desired use case, and the associated requirements of this use case.

## Current implementation

Currently when a blob asset is updated, a new version (i.e. a copy) is uploaded to S3. Zarr archives are too large so multiple copies should not be created. A Zarr archive is uploaded once and it is edited in place. This design means that the Zarr archive is immutable once the Dandiset is published. For more details, see the [zarr-support-3 design doc](https://github.com/dandi/dandi-archive/blob/master/doc/design/zarr-support-3.md).
When a blob asset is updated, a new version (i.e. a copy) is uploaded to the S3 bucket. Zarr archives are too large so multiple copies should not be created. A Zarr archive is uploaded once and it is updated in place. This design means that the Zarr archive is immutable once the Dandiset is published, so that the published Dandiset is immutable. For more details, see the [zarr-support-3 design doc](https://github.com/dandi/dandi-archive/blob/master/doc/design/zarr-support-3.md).

## Use case 1

Publish Dandisets with Zarr archive(s) following the general publishing procedure described in the [publish-1 design doc](https://github.com/dandi/dandi-archive/blob/master/doc/design/publish-1.md). A modified publishing procedure that includes Zarr archive(s) is summarized below.
Publish a Dandiset containing a Zarr archive(s), and subsequently update the Zarr archive(s).

1. User uploads a new Dandiset which includes Zarr archive(s).
2. User uploads an updated Zarr archive in the `Draft` version of the Dandiset.
3. User publishes the Dandiset and thereby creates a new version of the Dandiset.
The publishing procedure would follow the description found in the [publish-1 design doc](https://github.com/dandi/dandi-archive/blob/master/doc/design/publish-1.md). A modified publishing procedure that includes Zarr archive(s) is summarized below.

1. User uploads a new Dandiset which includes a Zarr archive(s).
2. User uploads an updated Zarr archive(s) to the `Draft` version of the Dandiset.
3. User publishes the Dandiset and thereby creates a new immutable version of the Dandiset.
4. User repeats steps 2 and 3.

## MVP Requirements (Target date: April 30, 2024)

1. Permit versioning of Zarr archives without creating a copy of the entire Zarr archive
1. Permit publishing of Dandisets with Zarr archives
1. Minimize storage costs in the design
1. Support versioning of Zarr archives without creating a copy of the entire Zarr archive.
1. Support publishing immutable versions of Dandisets with Zarr archives, where the Zarr archives are potentially updated between versions.
1. Minimize storage costs in the design.

## MVP+1 Requirements

1. Permit linking of a Zarr asset to multiple Dandisets - [dandi-archive/issues/1792](https://github.com/dandi/dandi-archive/issues/1792)
1. Support linking of a Zarr asset to multiple Dandisets - [dandi-archive/issues/1792](https://github.com/dandi/dandi-archive/issues/1792)



0 comments on commit 2987b81

Please sign in to comment.