Better frame selection

[manthey]

Currently, on a multiframe image, we show a simple control for selecting the frame:

In many instance we have more information and know that the frames have further information: Z (z-slice), T (time), XY (xy location, also called view), and C (channel) values that each have a range. When this is the case, instead of having a single slider for frame, it would be nicer to have multiple sliders.

Further, for single-channel frames where the is a known C range, it would be good to have an option to composite multiple C frames together using false color. This could be done, for instance, by having a radio button picking between the slider and a drop-down list of channels that could be checked on or off. As a further refinement, either as a whole or individually, channels could have composition options. These would include:

• range to map to the false color channel, which could be:
  • full-range
  • non-zero histogram range
  • extended non-zero histogram range (where the histogram excluding a small amount at either end is ranged to pixel values 1-254 so that the extremes fall on 0 and 255 even if they are far from the rest of the range)
  • manually specified
• color to map the channel to

[manthey]

This should be written so the controls can be reused in other backbone views.

[manthey]

This would be something vaguely like

Where the first row would be the current frame control. The selection is either "by frame" (just the frame control) or "by axis" (with better words).
Z, Time, XY would be much like frames in that a single value can be picked OR max-merge (and possibly some future other composite values)
Channels can either be a single value OR can be auto-composite, or composite individually. For compositing, each channel can be turned on or off, have a color, show a histogram (based on the current other settings) and have a high and low value specified. Auto-scale can be picked along with a "closeness" value (0 picks the high and low from the histogram, 0.002 would exclude the highest and lowest 0.2% of values). There would be controls to show all or auto-scale all.
If there is more than 1 band, channels might only have a single-value slider. Bands can be treated like channels (especially when there are more than 3 non-opacity bands).

[naglepuff]

What's the difference between a frame, channel, and band?

Currently in imageViewerSelectWidget.js it looks like the control is only visible if the image metadata contains a frames attribute. Would the changes here include supporting additional metadata shapes, i.e. looking for metadata.bands or metadata.channels?

[manthey]

frames is the abstraction that there are multiple images within a file and will be present with a list of length more than 1 if there are any type of multiple frames.

bands are different samples with in the same image (e.g., R, G, B, and alpha channels for many images). For geospatial (especially), bands can be much longer list (e.g., various infrared channels, panchromatic, etc.). These aren't separate frames -- they are just multiple samples in the same image. Some libraries refer to this as Samples or the S axis. If the image has anything other than L, LA, RGB, or RGBA, there will be a bands entry in the metadata which is a dictionary of known bands.

If we have any semantic information about multiple frames, there will be IndexRange and IndexStride entries in the metadata, with IndexRange a dictionary of Index<axis> keys and values for how many frames are along that axis. Currently, we've seen axes of C (channels), Z (vertical z stack), T (time), XY (different stage positions, some libraries call this P). This will probably be extended in the future to support arbitrary axes. Within the frames list, we populate Index<axis> keys with the 0-based index along that axis.

Channels are conceptually like bands -- but are stored in different images and mean something different. Rather than imaging different wavelengths of light like bands, they often imply different sampling conditions (different light sources, different surface treatments, etc.). When we know what the name of the channels are, we'll have a channels array of the names of the channels, and a channelmap with the channel names as keys and the first index where that name occurs as a value.

For instance, some metadata might be like so:

{'IndexRange': {'IndexC': 5, 'IndexZ': 29},
 'IndexStride': {'IndexC': 1, 'IndexZ': 5},
 'channelmap': {'A594': 2, 'Brightfield': 0, 'CY3': 1, 'CY5': 3, 'DAPI': 4},
 'channels': ['Brightfield', 'CY3', 'A594', 'CY5', 'DAPI'],
 'frames': [{'Channel': 'Brightfield',
             'Frame': 0,
             'Index': 0,
             'IndexC': 0,
             'IndexT': 0,
             'IndexZ': 0},
            {'Channel': 'CY3',
             'Frame': 1,
             'Index': 0,
             'IndexC': 1,
             'IndexT': 0,
             'IndexZ': 0},
...
            {'Channel': 'DAPI',
             'Frame': 144,
             'Index': 28,
             'IndexC': 4,
             'IndexT': 0,
             'IndexZ': 28}],
 'levels': 3,
 'magnification': 45.645426328281914,
 'mm_x': 0.00021907999999999997,
 'mm_y': 0.00021907999999999997,
 'sizeX': 1024,
 'sizeY': 1022,
 'tileHeight': 256,
 'tileWidth': 1024}

This image has 145 frames, so the current slider will range from 0 - 145. There are 5 channels and 29 Z levels. 5 * 29 = 145.

[manthey]

And, as an example, a bands entry might look like this:

{'bands': {1: {'interpretation': 'red',
               'max': 427.0,
               'mean': 206.73541757128385,
               'min': 184.0,
               'stdev': 21.171334580515904},
           2: {'interpretation': 'green',
               'max': 817.0,
               'mean': 281.38261687653784,
               'min': 170.0,
               'stdev': 55.19638494265467},
           3: {'interpretation': 'blue',
               'max': 1209.0,
               'mean': 300.15487045882185,
               'min': 165.0,
               'stdev': 104.52763870219812},
           4: {'interpretation': 0,
               'max': 837.0,
               'mean': 153.5612968591692,
               'min': 71.0,
               'stdev': 79.11189385014731},
           5: {'interpretation': 0,
               'max': 1578.0,
               'mean': 225.66833116225214,
               'min': 1.0,
               'stdev': 144.91035778520907},
           6: {'interpretation': 0,
               'max': 891.0,
               'mean': 180.366189028803,
               'min': 46.0,
               'stdev': 110.34713875874439},
           7: {'interpretation': 0,
               'max': 1916.0,
               'mean': 416.96540743957155,
               'min': 44.0,
               'stdev': 304.05037413071625},
           8: {'interpretation': 0,
               'max': 711.0,
               'mean': 156.6503111883051,
               'min': 13.0,
               'stdev': 117.87815117899153}},

We don't promise that we have the statistics for the bands.

And, I wish we had better interpretation values for bands -- maybe we can extract better data from gdal when it is the source.

[manthey]

Maybe we should expand on the explanation that is here: https://girder.github.io/large_image/example_usage.html#images-with-multiple-frames

[naglepuff]

This is definitely very helpful. Can an image have both an array of frames and bands?

Also, could you point me towards images to use for testing that include multiple indices and images that have a bands entry?

[manthey]

Yes, an image can have bands AND frames.

If you have run tox in large_image (e.g., tox core-py39), you'll have a build/tox/externaldata directory with a bunch of images. See 04091217_ruc.nc for a lot of bands, DDX58_AXL_EGFR_well2_XY01.ome.tif for channels and z axes, landcover_sample_1000.tif for a sample where bands is categoric rather than continuous (e.g., pixel values are categories),.

Sadly, I don't think we have a sample with bands and channels. We could expand the test tile source to generate such for testing. In general, the geospatial sources report bands.

[manthey]

With #935, you can create a test source with bands and channels both populated. You can use the multi source yaml file to access such a source from Girder.

[manthey]

Once these controls exist, we will probably additional options of "make this configuration the default" and "name this configuration" and then pick from (and manage) named configurations. And a reset-all button, too, probably.

[manthey]

This has a series of follow on issues:

• Frame viewer as an embeddable component Investigate frame viewer as a embeddable component (jupyter, trame, girder, etc.) #1107, max-merge option on axis in frame selector #1118
• Histogram Histogram in frame selector #1109
• More min/max options More min/max options in frame controls #1117
• max-merge option on axis max-merge option on axis in frame selector #1118
• Band compositing should default to only a few bands Band compositing should default to only a few bands #1120
• keep values when changing mode Frame controls should keep values when changing mode #1122
• Presets Frame control presets #1123
• Internal fields Frame control internal fields #1124
• Default band names Frame control default band names #1125
• Documentation Better documentation explaining frames and bands #1126
• Frame control hot keys Frame control hot keys #1134