Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update About text #74

Merged
merged 14 commits into from
Oct 28, 2024
57 changes: 34 additions & 23 deletions ome2024-ngff-challenge/src/About.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -10,57 +10,68 @@

<ul>
<li>
Collect examples of OME-Zarr from multiple modalities, use cases, and
institutions, available from a single location.
Advancing the status of OME-Zarr specification proposals —
<a href="https://ngff.openmicroscopy.org/rfc/">Requests for Comment (RFC)</a> —
especially the adoption of the Zarr V3 specification
</li>
<li>
Test the utility (and lack thereof) of existing OME-Zarr metadata and
image viewers.
Collecting the URLs of converted examples of OME-Zarr from multiple modalities, use cases, and
institutions, all hosted by each participant themselves
</li>
<li>
Advance the adoption of the Zarr V3 specification, and also existing
OME-Zarr RFCs.
Testing the level of maturity of OME-Zarr conversion tools, image viewers,
and metadata in general
</li>
</ul>
<p>
The inspiration for the Challenge came from presentations at the OME Meeting
that demonstrated the level of adoption of OME-Zarr, but highlighted the
poor findability of the data.
that demonstrated the level of adoption of OME-Zarr and the relative ease of
conversion for some community members, but highlighted the often poor findability of the data
and the need for driving the OME-Zarr specification towards version 1.0.
<p>
A lofty goal of 1 Petabyte (PB) of data was set, mostly to make clear the scale of what <em>could be</em>.
</p>
<p>
Work on the challenge started in earnest in July 2024. All progress is
tracked in the <a href="https://github.com/ome/ome2024-ngff-challenge">ome2024-ngff-challenge</a>
repo. It was agreed that the results
of the OME-NGFF Challenge would be presented at the
repository. It was agreed that the deadline for submission would be such that the results
of the OME-NGFF Challenge could be presented at the
<a href="https://globalbioimaging.org/exchange-of-experience/exchange-of-experience-ix">
2024 Global BioImaging Meeting</a>.
</p>

<h2>Process</h2>
<p>
The Challenge was run via a series of virtual meetings coordinated on the
The Challenge was run via a series of virtual meetings open to all which were coordinated on the
Image.sc Forums, with all
<a href="https://hackmd.io/3emKqKQsT_2U35vLepzDEQ">notes and lists of participants</a>
available.
available. Initial meetings including that at OME2024 focused on defining the scope of
changes that would be made to the OME-Zarr format. A Python-based tool to convert data
to this format was built. Submissions were then collected in simple CSV files with a Zarr
URL per row.
</p>

<h2>Results</h2>
<p>
At the outset, we weren’t sure how much OME-Zarr data was available, if it
could be converted to Zarr V3, or how many organisations would participate.
In the end, we have far exceeded our expectations, with more than 0.5 PByte
of OME-Zarr made available, across a wide range of modalities.
At the outset, it was unclear how much OME-Zarr data was available, if it
could be converted to Zarr V3, and how many organisations would participate.
In the end, we are pleased with the more than 500 Terabytes of data which
have been made available, across a wide range of modalities.
</p>
<p>
To make the assembly as accessible as possible, we built the the OME-NGFF
Challenge Viewer [URL?}, which incorporates a CSV with locations of
datasets, and connects datasets to the OME NGFF Validator, for metadata
validation and viewing.
To make the collection as accessible as possible, the
<a href="https://ome.github.io/ome2024-ngff-challenge">OME-NGFF Challenge Viewer</a>,
was built which uses a CSV with locations of
datasets to enable collection statistics, browsing of thumbnails, and opening
the datasets with the OME NGFF Validator, for metadata
validation and viewing. Please let us know on image.sc or GitHub if you have any issues
or ideas.
joshmoore marked this conversation as resolved.
Show resolved Hide resolved
</p>
<p>
Perhaps the most important outcome of the Challenge is the establishment of
a federated bioimage data system based on OME-Zarr. To our knowledge, this
is the largest federated bioimage data collection ever assembled.
Looking back, perhaps the most rewording outcome of the Challenge is that with a remarkably
joshmoore marked this conversation as resolved.
Show resolved Hide resolved
modest investment of time and cloud resources, we have almost inadvertently prototyped
a federated bioimage data system based on OME-Zarr, the largest one we know of.
We hope future challenges will continue to push the state-of-the-art forward.
</p>
</div>

Expand Down
7 changes: 5 additions & 2 deletions ome2024-ngff-challenge/src/Main.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,11 @@
<div class="summary">
<h3 style="text-align:center">
{#if csvUrl == SAMPLES_HOME}
In the summer of 2024, the OME-NGFF project generated
<strong style="font-weight:600">{filesizeformat(totalBytes)}</strong> of data in Zarr v3 format
This site makes
<strong style="font-weight:600">{filesizeformat(totalBytes)}</strong>
of public bioimage data available in the Zarr v3 format.
The OME-NGFF community converted and now hosts
this federated collection of the next version OME-Zarr.
{/if}
<div style="font-size: 90%">
Showing Collection:
Expand Down