Image segmenter of overlayed images and functions for segmenting interactively #266

wgottwald · 2023-03-04T15:55:42Z

Motivation

I wanted to be able to segment an image that has been overlayed by a second one (i.e. something like an anatomical image overlayed by a heatmap of some sort like in PET imaging or hyperpolarized MRI (for example like figure 4 from this publication: https://doi.org/10.1038/ncomms15126).

List of changes

generic.py

So i added a second image_segmenter class that reads a secondary image --> image_segmenter_overlayed
Also I wanted to be able to interactively change the slice (for a 3D dataset) and segment multiple regions simultaneously.
First the image-segmenter objects for each slice needed to be saved into a list --> get_segmenter_list
This list then gets passed on to the function that actually performs the segmenting ---> draw_masks

I thought that the segmented regions should also have the possibility to be named, in contrast to the numbering that happens per default for a image-segmenter object. This is given to draw_masks

After drawing the masks their values need to be saved as a dictionary from the segmenter list--> get_masks
The keys are either the default numbers or the given names.
Finally I found it useful to retrieve the contours of each mask, as its sometimes good to have those values afterwards for plots. This is accomplished by ---> get_mask_contours, as suggested by @ianhi.

I am open to improvements, just let me know. I was unsure if a second image-segmenter class makes more sense than just modifying the main image-segmenter class, but I thought its a bit more foolproof.

setup.cfg

I also added scikit-image as its needed by the get_mask_contoursfunction.

pre-commit-config.yaml

The pre-commit-config.yaml file was updated as there were issues when commiting to this fork, the reasons are listed in the Issue here.

docs/examples/image-segmentation-multiple-images.ipynb and docs/index.md

An example notebook was also made, which shows how the new functions work. I had some issues getting this notebook running reliably, but I think that could depend on the plotting backend. Let me know if thats also an issue for you.

Gitignore

I also added the .idea/ folder to the gitignore for everyone working with Pycharm.

Other files that were changed

It seems that the formatter changed some of the notebooks and files, and would only let me commit then. I am not sure what happenend there but maybe someone else does.

Conclusion

I marked this as a draft pull request as a lot of changes are made and this is still somewhat work in progress.

newer MacOs systems use zsh istead of bash and therefore the input is slightly different.

Update contributing.md for zsh use

added support for zsh

…ns to display segmenter data Added a image_segmenter_overlayed class which reads a secondary image thats overlayed over the primary. This is useful for medical data for example PET where one has an anatomical reference image and then a heatmap overlayed and wants to segment based on both. Also added some functions to go along with it as well as a example notebook. Also had some issues with commiting, these are describes in an issue here: mpl-extensions#265. This changed the pre-commit-config.

review-notebook-app · 2023-03-04T15:55:46Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

wgottwald · 2023-03-06T08:35:06Z

Short follow up:

It would be incredibly useful to us if the image_segmenter_overlayed class would soon be merged into main, since we can use it within our group more easily. As far as I can see it does not pose large changes, as its just a second class with a different name.
The other functions and changes are nice to have but not as urgent.

ianhi · 2023-03-06T19:03:24Z

@wgottwald can I ask why you closed this? There's a bunch of nice work here.

I didn't respond previously as I managed to stay off my laptop for the whole weekend :O

wgottwald · 2023-03-06T21:56:27Z

I didn't get to comment about it yet but I found some bugs today regarding the segmenter overlay with different extents and the mask dictionary. I would like to fix them before committing again 😊 what do you think about a new class or do you want to include it in the main segmenter class ?

wgottwald · 2023-03-06T21:57:01Z

I will commit again as soon as it's fixed, probably tomorrow.

ianhi · 2023-03-07T04:37:29Z

Ok some brief thoughts (busy day and now it's late)

I really like several of the additions, the overlayed images, and the ability to work through stacks of images are both tings I've wanted in the past and just never bothered with. I also really like the ability to have multiple (possibly named) ROIs that's a really nice addition.

That said I think with a bit more work we can come up with a better API for expressing these capabilities. As it stands a few functions should be moved on to the object, and I think that we can fold the overlay capability into the existing segmenter

For the stacks of images we should find a way to generalize the data model the segmenter so it can work for single images, or stacks of them. Then which image is currently displayed can be controlled by a property (like i do here: https://github.com/ianhi/mpl-image-labeller/blob/8f3aed4232022871cc2e4f52b3e3f3c8479f822c/mpl_image_labeller/_labeller.py#L302-L326) which can optionally be linked to a slider.

Maybe an api could look like this:

Image stacks

stack_segmenter = image_segmenter(image_stack)

stack_segmenter.mask # returns something with the same shape as image_stack

image overlay

image_segmenter([img1, img2], overlay_image_stack=True)

Multiple ROIs
with an implementation inspired by this: https://github.com/ianhi/mpl-point-clicker/blob/cf254d56a5ffe7ec2cc201f67d0d326f1006694e/mpl_point_clicker/_clicker.py#L50-L53

 image_segmenter(image_stack, rois = ['Tumor','Kidney','Vessel'])

# alternatively
 image_segmenter(image_stack, rois = 3)

how does that sound?

Docs

Thank you so much for taking the time to add a docs page - I was going to ask for it eventually and it made this much easier to understand what was going on.

Finally:

I've really been meaning to pull image_segmenter out into it's own pypi package - perhaps mpl-image-segmenter as it doesn't really align with the core functionality of this package. So heads up that I may make that soon and we'll need to transfer this PR over to that.

wgottwald · 2023-03-07T07:41:39Z

Sounds good! I will take a look at that. Thanks for all the suggestions.
One issue I had so far with overlays is that the images can be of different extents (something like a 256x256x10 anatomical image which has a heatmap of 12x12x10 overlayed). In that case one wants to get a mask for the heatmap, as thats the data that is later used. So its a bit difficult working that out, but I think something like a reference image parameter with

either its own extent
a function that calculates the extents to be the same as on the segmented image
would be useful. The version I have implemented above does not fully provide that - which was the reason why I initially closed this issue before someone could push it into main.

I also think that the segmenter should be its own pypi package, that probably makes more sense. Let me know when that happens.

ianhi · 2023-03-07T15:52:38Z

I also think that the segmenter should be its own pypi package, that probably makes more sense. Let me know when that happens.

done!
github: https://github.com/ianhi/mpl-image-segmenter
pypi: https://pypi.org/project/mpl-image-segmenter/

wgottwald · 2023-03-07T16:03:54Z

Very cool! Thanks for doing that!
I will fork that repository and make a branch where I transport my changes so far over from this. This pull request can then be moved over - right?

ianhi · 2023-03-07T16:04:39Z

I will fork that repository and make a branch where I transport my changes so far over from this. This pull request can then be moved over - right?

We'll have to close this one, and you open a new one over there

ianhi

woops forgot to post these line level comments

ianhi · 2023-03-07T03:37:22Z

mpl_interactions/generic.py

+    if roi_names:
+        class_selector = widgets.Dropdown(options=roi_names, description="ROI name")
+    else:
+        class_selector = widgets.Dropdown(
+            options=list(range(1, n_rois + 1)), description="ROI number"
+        )
+
+    erasing_button = widgets.Checkbox(value=False, description="Erasing")
+    # create interactive slider for echoes
+
+    slice_slider = widgets.IntSlider(
+        value=n_slices // 2, min=0, max=n_slices - 1, description="Slice: "
+    )


If this is going to live in this package then we can't assume that ipywidgets is available and have to always have a fallback to the same functionality using matplotlib widgets.

ianhi · 2023-03-07T03:52:56Z

mpl_interactions/generic.py

+        self.mask_alpha = mask_alpha
+
+        if mask_colors is None:
+            # this will break if there are more than 10 classes
+            if nclasses <= 10:
+                self.mask_colors = to_rgba_array(list(TABLEAU_COLORS)[:nclasses])
+            else:
+                # up to 949 classes. Hopefully that is always enough....
+                self.mask_colors = to_rgba_array(list(XKCD_COLORS)[:nclasses])
+        else:
+            self.mask_colors = to_rgba_array(np.atleast_1d(mask_colors))
+            # should probably check the shape here
+        self.mask_colors[:, -1] = self.mask_alpha


If going with two classes then using inheritance is the way to go. It would remove a bunch of the duplicated code like this chunk.

ianhi · 2023-03-07T03:54:20Z

setup.cfg

@@ -70,6 +70,7 @@ test =
    requests
    scipy
    xarray
+    scikit-image


This need to be optionally imported in the get_mask_contours function - too heavy of a dependency to add.

actually better yet. I'd like to use https://contourpy.readthedocs.io/en/stable/

it doesn't actually add a dependency because matplotlib depends on it.

ianhi · 2023-03-07T04:12:02Z

mpl_interactions/generic.py

+        img,
+        second_img,


While 2 images is probably the most useful I think there's no reason this shouldn't be extended to an arbitrary number of overlayed images.

Maybe a better solution here is to change the img argument of image_segmeneter from accepting just a single array to accepting an iterable of arrays.

ianhi · 2023-03-07T04:13:22Z

mpl_interactions/generic.py

+def get_mask_contours(segmenter_list):
+    """
+    Extract the contours of a mask segmented with mpl_interactions image_segmenter_overlayed


Rather than live indepdently this function (or some version of it) should live on the image_segmenter object

wgottwald added 6 commits March 3, 2023 14:18

Update contributing.md for zsh use

3dd5225

newer MacOs systems use zsh istead of bash and therefore the input is slightly different.

Merge pull request #1 from wgottwald/wgottwald-updated-contributing-file

1d0eebf

Update contributing.md for zsh use

fixed visualization typo - updated contributing for MacOs

14e2d3d

Updated contributing.md

66a63c5

added support for zsh

Update image-segmentation-multiple-images.ipynb

4a4e25b

wgottwald marked this pull request as ready for review March 6, 2023 08:33

wgottwald closed this Mar 6, 2023

wgottwald reopened this Mar 7, 2023

ianhi mentioned this pull request Mar 7, 2023

Updating API mpl-extensions/mpl-image-segmenter#3

Open

ianhi reviewed Mar 7, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image segmenter of overlayed images and functions for segmenting interactively #266

Image segmenter of overlayed images and functions for segmenting interactively #266

wgottwald commented Mar 4, 2023

review-notebook-app bot commented Mar 4, 2023

wgottwald commented Mar 6, 2023

ianhi commented Mar 6, 2023

wgottwald commented Mar 6, 2023

wgottwald commented Mar 6, 2023

ianhi commented Mar 7, 2023

wgottwald commented Mar 7, 2023

ianhi commented Mar 7, 2023

wgottwald commented Mar 7, 2023

ianhi commented Mar 7, 2023

ianhi left a comment

ianhi Mar 7, 2023

ianhi Mar 7, 2023

ianhi Mar 7, 2023

ianhi Mar 7, 2023

ianhi Mar 7, 2023

ianhi Mar 7, 2023

Image segmenter of overlayed images and functions for segmenting interactively #266

Are you sure you want to change the base?

Image segmenter of overlayed images and functions for segmenting interactively #266

Conversation

wgottwald commented Mar 4, 2023

Motivation

List of changes

generic.py

setup.cfg

pre-commit-config.yaml

docs/examples/image-segmentation-multiple-images.ipynb and docs/index.md

Gitignore

Other files that were changed

Conclusion

review-notebook-app bot commented Mar 4, 2023

wgottwald commented Mar 6, 2023

Short follow up:

ianhi commented Mar 6, 2023

wgottwald commented Mar 6, 2023

wgottwald commented Mar 6, 2023

ianhi commented Mar 7, 2023

wgottwald commented Mar 7, 2023

ianhi commented Mar 7, 2023

wgottwald commented Mar 7, 2023

ianhi commented Mar 7, 2023

ianhi left a comment

Choose a reason for hiding this comment

ianhi Mar 7, 2023

Choose a reason for hiding this comment

ianhi Mar 7, 2023

Choose a reason for hiding this comment

ianhi Mar 7, 2023

Choose a reason for hiding this comment

ianhi Mar 7, 2023

Choose a reason for hiding this comment

ianhi Mar 7, 2023

Choose a reason for hiding this comment

ianhi Mar 7, 2023

Choose a reason for hiding this comment