Support HyperSpy 2.0

[hakonanes]

HyperSpy 2.0 is slated for release around mid-September according to hyperspy/hyperspy#2996 (comment). What does it take for kikuchipy to be compatible with this release?

An example of kikuchipy's failing test suite run with HyperSpy 2.0: https://github.com/hyperspy/hyperspy-extensions-list/actions/runs/5563155043/job/15068218388.

Current incompatibilities that I'm aware of are related to HyperSpy's IO moving to RosettaSciIO, which we should use. Features that must be handled:

• Saving to HSPY/ZSPY and image formats.
• Internal use of the function overwrite_dataset() for overwriting. We can import this from the hspy plugin in HyperSpy 1.7.5, but not from the same plugin in RosettaSciIO 0.1.
• Use of rgb_tools. We can import this from HyperSpy 1.7.5, but not from RosettaSciIO 0.1.

Other incompatibilities:

• Our creation of markers from geometrical EBSD simulations may will not work with the new marker collections in Marker Collection hyperspy/hyperspy#3148. Our markers are updated in Update use of markers to HyperSpy v2 #695.

[hakonanes]

• Use of rgb_tools. We can import this from HyperSpy 1.7.5, but not from RosettaSciIO 0.1.

I have to correct myself. This is not part of the listed public API, but it can be imported as from rsciio.utils import rgb_tools.

• Internal use of the function overwrite_dataset() for overwriting. We can import this from the hspy plugin in HyperSpy 1.7.5, but not from the same plugin in RosettaSciIO 0.1.

This function, however, can only be imported as from rsciio.hspy._api import overwrite_dataset. Thus, it is not part of the public API nor can it be imported from a "public looking" module.

[CSSFrancis]

@hakonanes assuming that the markers PR gets merged that will also have to be changed.

I was planning on making a PR for that once everything starts to get fixed there.

It might be worthwhile to think if there is anything that better annotation might help with. I've been thinking about overlaying phase information on the navigator. That or it might be nice to make something like arrange multiple navigators using an abstraction of hyperspy/hyperspy#3140.

It might also be good to let hyperspy plot colored plots for navigators as well.

[hakonanes]

The use of markers must be changed as well, of course. I didn't think of this. Thanks. A PR with the necessary changes would be great!

We navigate EBSD signals using phase and orientation color maps in kikuchipy already (see e.g. our visualization tutorial). It is very useful. We have a convenience function for this, get_rgb_navigator(), which takes an array of shape (n rows, n columns, 3) and returns a suitable signal with rgb8 or rgb16 data type. This might be a good utility function to move to HyperSpy?

Linking multiple navigators would be useful, indeed. This might already be possible with the current events functionality? I haven't tried it. A convenience function would be useful, accepting a signal and multiple suitable arrays or signals to use as navigators.

[hakonanes]

This work will be pushed to a v0.10.0 release as the new release of PyEBSDIndex v0.2.0 breaks kikuchipy from conda-forge (not from PyPI). See the progress of that work at #652.

[CSSFrancis]

@hakonanes I was going to see about changing the marker implementation but I'm not sure I fully understand how the io plugins work in kikuchipy.

[hakonanes]

Thank you for showing interest in using HyperSpy 2.0 in kikuchipy! I'm almost done repurposing our IO plugins to use RosettaSciIO in #651. It might be easier to make kikuchipy compatible with HS 2.0 after that.

I don't think I'll finish that PR the coming week, though. I plan to use my spare time on orix instead. I'll get back to you once the PR is done!

[CSSFrancis]

@hakonanes any movement on this? It would be good to make a hyperspy-bundle release relatively soon.

[hakonanes]

None since mid December. I plan to finish this in the latter half of February. A minor release will be made soon after that. If the bundle can't wait, I guess releasing one without kikuchipy is the only option.

[CSSFrancis]

@hakonanes I don't think there is any major rush on the bundle, these things just take time. At the very latest it would be good to make a bundle release before the next hyperspy/hyperspy affiliated training session. (I'm not sure exactly when that is)

Let me know if you need some help! I realize that starting a new job etc. means that you might not have as much to time to devote to some of these things.

[hakonanes]

Let me know if you need some help!

Thank you, that is very much appreciated.

I realize that starting a new job etc. means that you might not have as much to time to devote to some of these things.

Yes, I naturally have to reduce my commitment to adding new features to softwares I help maintain. But I will help maintain them (discuss here, fix issues and the like).

[hakonanes]

I've restarted this work as I tried to do all compatibility changes at once in #651. Instead, I've created a hyperspy2-support branch in this repo. Smaller PRs towards v2 compatiblity goes into this branch. This is hopefully more manageable.

[CSSFrancis]

@hakonanes Do you need any help with this?

[ericpre]

The difficult part may be with updating to the markers API. In addition to @CSSFrancis's offer help, I would happy to help too, because it would be good to make a release compatible with hyperspy 2.0 sooner than later - it will start to become a significant issue for users to stay on hyperspy 1.x.
Regarding the IO code, updating to the rsciio structure could be done later, it shouldn't be a requirement to be compatible with hyperspy, and there should be only a couple of import to update.

[hakonanes]

Thank you both for offering to help. I've opened a draft PR updating kikuchipy's IO to align with RosettaSciIO's design in #694. I'll ping you for a review (how detailed you want) once it's ready.

[hakonanes]

Changing to HyperSpy v2's marker API is done in #695, just need to look over the docs 🤞🏻.

@ericpre and @CSSFrancis, this means we can release a v0.11.0 as soon as HyperSpy 2.2 is out (need src layout extension support).

[hakonanes]

Haven't published to PyPI yet, since we're encountering this issue in our publish workflow: pypa/gh-action-pypi-publish#283. Apparently, it's considered less safe to build and publish to TestPyPI and PyPI in the same workflow. However, a fix in twine soon to be released will ensure we don't have to change anything...

So, waiting.

[hakonanes]

Nevermind, disabling attestations on the TestPyPI publish step solved the problem.