OME-NGFF and Deep Learning: current OME specs, DL use cases and integration with Bioimage.io ecosystem

[jdeschamps]

Hi!

Background

OME-NGFF maintains specifications for bioimages stored in next generation file formats (hdf5, zarr), in particular with the aim to have cloud based hosting. The prototype implementation is ome-zarr-py. Zarr allows storing large datasets in chunks and support parallel and random access, pyramidal representations and metadata at different level of the Zarr hierarchy.

Obviously, Zarr, and NGFF in general, are an attractive file format for DL: possibility to store multiple images or image types in one file, cloud storage, compatibility with very large datasets etc.

At the OME-NGFF hackathon, we had a small working group focused on loading OME-Zarr in deep learning pipelines (PyTorch). In parallel, we had several discussions on DL use cases for OME-Zarr.

In this discussion, I want to highlight these use cases and discuss intersections between the OME-NGFF efforts, the DL community work and AI4Life's mission.

Aim

As I see it, there are two potential outcomes from such a discussion:

• Propose use cases to OME-NGFF to assess the compatibility of the current specs and foster a discussion on possible evolutions (as in User stories transformations ome/ngff#84 for example)
• Come up with AI4Life recommendations for dataset organization, and corresponding specs (DL-NGFF) that are compatible with OME-NGFF specs (e.g. Bioimage.io dataset subspecs as an extension of OME-NGFF) for the more specific use cases (since OME-NGFF specs will unavoidably be more general)

In particular, the latter one would allow the following:

• Use of Ai4Life DL-NGFF in the BIA (which already uses OME-NGFF)
• Add data loaders for DL-NGFF to the Bioimage.io-core in order to load compatible Zarr datasets
• Use DL-NGFF via the core in the BioEngine or Containerized notebooks

Use Cases

The OME-NGFF specs detail the internal organization of OME-NGFF files. In particular, the specs specify where labels should be stored: in a group at the same level than the raw image multiscales.

In this section, I describe DL use cases that were compiled during the Hackathon (with @jo-mueller and @lorenzocerrone, after discussions with @joshmoore and @d-v-b among others).

Here we differentiate cases with labels (pixel-annotation with integer value, as understood from OME-NGFF specs) from others cases with target images (e.g. scalar values). Keep in kind that OME-NGFF need to also maintain a multi-resolution hierarchy.

Story 1: semantic/instance segmentation

• Dataset contains multiple raw images, of potentially different sizesbut same dimensionality.
• Each raw image in the dataset has corresponding labels, saved in the same Zarr.
• The label values represent the different classes/instances.
• Label images have same (Z)YX dimensions as the raw images.

Pain points:

• labels are child of multiscale in OME-NGFF (see "standalone" label images  ome/ngff#179)

Story 2: semantic/instance segmentation with partial labeling

• Dataset contains multiple raw images, of potentially different sizes but same dimensionality.
• Each raw image in the dataset might have corresponding labels, saved in the same Zarr.
• The label values represent the different classes/instances.
• Label images are crops of the raw images, with potentially multiple crops per image.

Note: CellMap project in Janelia doesn't use the labels from OME-NGFF, they have they own structure for the multiclass crops (https://janelia-cellmap.github.io/cellmap-schemas/api/annotation/)

Pain points:

• See story 1
• Can labels have different dimensions than the raw images? (crop)
• Need transformation (see Coordinate systems and new coordinate transformations proposal ome/ngff#138) to map crops in the raw image.

Story 3: semantic segmentation with multiple labels

• Dataset contains multiple raw images, of potentially different sizes but same dimensionality.
• Each raw image in the dataset might have corresponding labels, saved in the same Zarr.
• The label values represent the different classes/instances.
• There might be multiple label images for a raw image

Note: OME-Zarr allows multiple label images.

Paint points: see stories 1 and 2

Story 4: panoptic segmentation

• Dataset contains multiple raw images, of potentially different sizes but same dimensionality.
• Each raw image in the dataset has corresponding labels, saved in the same Zarr.
• The labels have a dimension representing multiple classes and each pixel value represents an instance.
• Label images might have same (Z)YX dimension as the raw images or be cropped.

Note: OME-Zarr allows multiple label images (classes), which can then each accommodate their own instances (pixel values).

Pain points:

• see stories 1 and 2
• If we want fast access across the classes, then they cannot be stored in different labels

Story 5: supervised training with target image (compatible with labels)

• Dataset contains multiple raw images, of potentially different sizes but same dimensionality.
• Each raw image in the dataset might have a corresponding target image, saved in the same Zarr.
• Target images might have different dimensions than the raw images:
  • different spatial dimensions (e.g. Z projection)
  • different number of channels (e.g. channel splitting)
  • more examples?
  • etc.

Pain points:

• Need to be able to relate an image to another through the metadata, along with transformation (origin, scaling etc.)

Story 6: object detection

• Dataset contains multiple raw images, of potentially different sizes but same dimensionality.
• Each raw image in the dataset might have a corresponding list of targets.
• Targets can be represented with a certain number of parameters:
  • points
  • bounding boxes
  • polygons
  • more examples? (WKT/GeoJSON/etc.)
  • etc.

Pain points:

• OME-NGFF currently does not support tables (Table spec proposal ome/ngff#64)

Story 7: store predictions

• Dataset contains multiple raw images, alongside their targets (used for training) if applicable
• Predictions are saved next to the raw images, potentially multiple per image
  • denoised images
  • each step of a diffusion model
  • semantic/instance/panoptic segmentation
  • list (e.g. detection)
  • more examples?
  • etc.

Pain points: see story 5

Story 8: store DL-ready dataset

• Dataset is organized in train, validation and test patches
• Each patch has same dimension and might have a corresponding target (label or scalar)
• All data saved in one file

Note: Can be useful for reproducibility.

Pain points:

• Can the higher level of an OME-NGFF be groups of images (rather than images directly)?

Potentially useful metadata

• Whether the target image is a label or a scalar
• Image physical dimension
• Sparsity of annotations (dense vs sparse)
• Annotation quality index

Current OME-NGFF specs status

As far as I know, the development of OME-NGFF is currently a bit slow and a new governance model is going to emerge soon. The next version (v0.5.0) should contain transformations (ome/ngff#138), which is useful for many of the current use cases.

Finally, in the long term future, there might be a way to extend the specs. A spec extension could be the best way to define DL specific specs.

What now?

I would love to hear what you think of these use cases, and whether you can come up with new ones. My aim is the following:

• Improve current use cases with your feedback
• Open an issue on OME-NGFF to share these use cases
• Discuss Zarr file organization for the various use cases with the aim to agree on a standard (BIA, Bioimage.io specs and core) within AI4Life (while maintaining OME-NGFF compatibility)

Finally, we could organize a meeting soon to share experience working with Zarr (Teresa and Kola from EBI, Craig, Wei, Esti etc.) and go through this discussion.

@bioimage-io/spec-dev

[d-v-b]

great writeup @jdeschamps.

My takeaway from these stories, and from my experience working with segmentations / annotations with cellmap, is that people do lots of different things with label images. People might have one layout of images for annotation, another layout for training, and another one for final segmentations, etc. Ultimately, I don't think it's possible for OME-NGFF to capture this variability in a way that works for everyone.

So I think OME-NGFF should adopt a more narrow scope for label data. Specifically, I don't think OME-NGFF should attempt to describe a standard layout for labelled images, because real world usage of labelled images entails many different layouts, and that's OK. In concrete terms, this would mean removing the labels metadata from the spec, and remove references to a label-specific layout. It was never clear to me what problem the labels metadata was actually trying to solve, and it fails to solve the very real problem of "how to arrange annotations / segmentations for ML applications"

Instead, I would prefer that OME-NGFF focus on standards for just images, not standards for collections of images, and let people arrange their OME-NGFF images in ways that make immediate sense for their particular use case. At the same time, we should invest in tools that make these application- or community-specific layouts easy to create, validate, document, etc, so people can a) use layouts that are useful to them and b) express these layouts in machine / human friendly ways. Maybe eventually all the ML people agree on a single layout that solves all their problems and that can go into the spec. But I don't think this is happening soon.

So if we reframe the focus of NGFF to standards for images, one immediate problem that the spec could solve is "how can an image convey to an application that it contains labels, and not scalar intensities". See ome/ngff#203 for one idea to address this -- encode the "labelness" of an image by describing the units of the elements of the image. There are probably many other per-image metadata problems specific to label images / ML applications that I'm not thinking of.

[constantinpape]

Thanks for the write-up @jdeschamps. This is a good summary of many of the scenarios that could be useful for storing segmentations or other deep learning results. I also agree with @d-v-b that I don't find the current labels in OME-NGFF very useful and that there are many different use-cases for how to store and represent segmentations or other deep learning analysis results (as also indicated by the many user stories here).

So I also think that labels in OME-NGFF should be re-thought (I am not sure if fully removing is an option since it's part of the spec, cc @joshmoore, maybe it could be deprecated). I also agree with @d-v-b that it's probably not feasible to create specs for all the different kinds of segmentation representations etc. However, it would be good to standardize these representations to some degree. A potential approach would be to have "conventions" or "extensions" that build on OME-NGFF and can be used to define the layout for a given segmentation (or whatever) layout, but that would not need to go through the full spec process.

[jdeschamps]

Thanks for your inputs @d-v-b and @constantinpape!

I fully agree, the OME-NGFF simply cannot be tailored to all use cases and will remain general. I also find the labels quite confusing in general. Opening an issue on the OME-NGFF repo would be a good way to continue a discussion touching on the place of the labels within the specs and communicate the use cases for future reference. More so than trying to force the NGFF specs to say something about deep-learning.

I will open the issue in the next few weeks, once I feel everybody in Ai4Life had a chance to contribute or raise their opinion. I'll make sure to mention the points raised here regarding potential evolutions of the labels in NGFF.

Now, I obviously very much like the idea of a "convention" or "extension" for DL (hence this discussion), and it fits AI4LIfe mission perfectly. While it is unlikely that the whole field would agree on a particular file organization, a good starting point it to come up with our own recommendations. In turn, these would allow easy read/write in the Bioimage.io core.

Alternatively, we could thinking of some more hacky way to point to images (e.g. inputs/outputs during model validation) in a Zarr file, for instance with relative paths within the Zarr file. My preference stays with having OME-NGFF specs "extensions" as this has much broader impact that the just a core dataloader, but that would mean being incompatible with other conventions.

[joshmoore]

Very short feedback since I find GH discussions very hard to keep up with and track ("what have I already read?!")

• 💯 for the write-up
• 👍 for whatever cross-project discussion mechanism works
• No objections to deprecating labels (there are in fact already a few issues discussing this)
• I'm not personally convinced that there's not a place in NGFF-land for defining relationships between images (and specifically, their labels) but I definitely agree that there can't be just one. It might well be, though, that only simple cases can have sufficiently well-defined descriptions to be implemented in multiple clients, but I do think that the "click on a Zarr and see an image with your labels"-style use as is currently implemented is useful for many.

[TorecLuik]

Instead, I would prefer that OME-NGFF focus on standards for just images, not standards for collections of images, and let people arrange their OME-NGFF images in ways that make immediate sense for their particular use case.

Doesn't OME-NGFF already focus on standards for "collections" of images, like screens, well plates, et cetera?
I don't think there is such a big difference between these metadata structures, except that one is "biological" (actual physical plates put under a microscope) and the other is "computational" ( 'metadata' of the original image captured in a label-image).

It is clear that you would at least want to be able to read/find/store such label (images) in all software compatible with some specification X, and the goal of NGFF seems to be to have 1 specification for 'metadata' instead of 100.

I also agree that AI4Life / BioImage.io would be a good entity to focus on this metadata structure.

[d-v-b]

Doesn't OME-NGFF already focus on standards for "collections" of images, like screens, well plates, et cetera?

Yes, but these collections were, in my opinion, defined with too much of a top-down approach. I am advocating for a bottom-up approach, where we try to make higher-level things like collections expressible as simple combinations of lower-level things (images), but without restricting people too much.

[normanrz]

Thanks everybody for capturing the discussion and adding your thoughts. I just wanted to link a discussion we had on image.sc, here.

Also, there is an upcoming community call on the topic of labels on April 3rd: https://forum.image.sc/t/ome-ngff-community-call-labels-and-other-collections/93815