Document napari plugin

.github/workflows/test_and_deploy.yml

@@ -41,7 +41,7 @@ jobs:
 
     steps:
       # these libraries enable testing on Qt on linux
-      - uses: pyvista/setup-headless-display-action@v2
+      - uses: pyvista/setup-headless-display-action@v3
         with:
           qt: true
       - name: Cache Test Data

docs/source/community/index.md

@@ -5,6 +5,7 @@ develop a new feature, or improve the documentation.
 To help you get started, we have prepared a statement on the project's [mission and scope](target-mission),
 a [roadmap](target-roadmaps) outlining our current priorities, and a detailed [contributing guide](target-contributing).
 
+(target-get-in-touch)=
 ```{include} ../snippets/get-in-touch.md
 ```
 

docs/source/user_guide/index.md

@@ -8,7 +8,7 @@ Before you dive deeper, we highly recommend reading about the structure
 and usage of [movement datasets](movement_dataset.md), which are a central
 concept in the package.
 
-::::{grid} 1 2 2 3
+::::{grid} 1 1 2 2
 :gutter: 3
 
 :::{grid-item-card} {fas}`wrench;sd-text-primary` Installation
@@ -32,6 +32,13 @@ Load and save tracking data.
 Learn about our data structures.
 :::
 
+:::{grid-item-card} {fas}`line-chart;sd-text-primary` Viewing data in napari
+:link: napari_plugin
+:link-type: doc
+
+Use our `napari` plugin to interactively explore your data.
+:::
+
 ::::
 
 
@@ -42,4 +49,5 @@ Learn about our data structures.
 installation
 input_output
 movement_dataset
+napari_plugin
 ```

docs/source/user_guide/napari_plugin.md

@@ -0,0 +1,173 @@
+(target-napari-plugin)=
+# Viewing data in napari
+
+You can visualise `movement` motion tracks in [napari](napari:) using our
+plugin, which extends `napari`'s core functionality and provides a widget
+that can be docked in the `napari` window. Currently, you can use it to
+visualise 2D [poses datasets](target-poses-and-bboxes-dataset)
+as points overlaid on video frames.
+
+:::{warning}
+This plugin is still in early stages of development but we are working on
+ironing out the kinks. [Get in touch](target-get-in-touch)
+if you find any bugs or have suggestions for improvements!
+:::
+
+## Installation
+
+The `napari` plugin is shipped with the `movement` package starting from
+version `0.1.0`. If you install `movement` via `conda`, `napari` will already
+be available. If you use the `pip` installer, make sure to
+install `movement` with the `[napari]` extra:
+
+```sh
+pip install movement[napari]
+```
+
+## Launch napari
+
+Type the following command in your terminal:
+
+```sh
+napari -w movement
+```
+
+This will open the `napari` window with the `movement` widget docked on the
+right-hand side, as in the the [screenshot](target-widget-screenshot) below.
+
+In `napari`, data is typically loaded into [layers](napari:guides/layers.html),
+which can be reordered and toggled for visibility in the layers list panel.
+For example, keypoint data can be added as a
+[points layer](napari:howtos/layers/points.html),
+while image stacks (including videos) can be added as
+[image layers](napari:howtos/layers/image.html).
+Below, we'll explain how to do this.
+
+## Load a background layer
+
+Though this is not strictly necessary, it is usually informative to
+view the keypoints overlaid on a background that provides
+some spatial context. You can either [load the video](target-load-video)
+corresponding to the poses dataset, or a [single image](target-load-frame),
+e.g., a still frame derived from that video. In the following sections,
+we will show you how to do both and discuss some limitations.
+
+(target-load-video)=
+### Load a video
+
+To load a video, drag and drop the video file onto the `napari` window.
+You will see a pop-up dialog asking you to select the reader.
+Choose the `video` reader—corresponding to the
+[`napari-video`](https://github.com/janclemenslab/napari-video)
+plugin—and click `OK`. You can optionally select to remember this reader
+for all files with the same extension.
+
+`napari-video` will load the video as an image stack with a slider
+at the bottom that you can use to navigate through frames.
+You may also use the left and right arrow keys to navigate
+frame-by-frame.
+
+Clicking on the play button will start the video playback at a default
+rate of 10 frames per second. You can adjust that by right-clicking on the
+play button or by opening the `napari > Preferences` menu and changing
+the `Playback frames per second` setting.
+
+:::{admonition} Video playback limitations
+:class: warning
+
+- The video playback may freeze or stutter if you click on the slider to jump
+  to a specific frame. We recommended pausing the playback before such jumps.
+- `napari-video` may struggle to play videos at a high frame rate, depending
+  on your hardware, the video resolution and codec. If you experience
+  performance issues, such as the video freezing or skipping frames,
+  try reducing the playback frames per second or fall back to
+  using a [single image](target-load-frame) as a background.
+:::
+
+
+(target-load-frame)=
+### Load an image
+
+This usually means using a still frame extracted from the video, but in theory
+you could use any image that's in the same coordinate system as the
+tracking data. For example, you could use a schematic diagram of the arena,
+as long as it has the same width and height as the video and is
+properly aligned with the tracking data.
+
+::: {dropdown} Extracting a still frame from a video
+:color: info
+:icon: info
+
+You can use the command line tool [`ffmpeg`](https://www.ffmpeg.org/)
+to extract a still frame from a video.
+
+To extract the first frame of a video:
+
+```sh
+ffmpeg -i video.mp4 -frames:v 1 first-frame.png
+```
+
+To extract a frame at a specific time stamp (e.g. at 2 seconds):
+
+```sh
+ffmpeg -i video.mp4 -ss 00:00:02 -frames:v 1 frame-2sec.png
+```
+:::
+
+To load any image into `napari`, simply drag and drop the image file into
+the napari window. Alternatively, you can use the `File > Open File(s)` menu
+option and select the file from the file dialog.
+In any case, the image will be loaded as a single 2D frame without a slider.
+
+## Load the poses dataset
+
+Now you are ready to load some pose tracks over your chosen background layer.
+
+On the right-hand side of the window you should see
+an expanded `Load poses` menu. To load some pose data in napari:
+1. Select the `source software` from the dropdown menu.
+2. Set the `fps`  (frames per second) of the video the pose data refers to. Note this will only affect the units of the time variable shown when hovering over a keypoint. If the fps is not known, you can set it to 1, which will effectively make the time variable equal to the frame number.
+3. Select the file containing the predicted poses. The path can be directly pasted or you can use the file browser button.
+4. Click `Load`.
+
+The data should be loaded into the viewer as a
+[points layer](napari:howtos/layers/points.html).
+By default, it is added at the top of the layer list.
+
+::: {note}
+See [supported formats](target-supported-formats) for more information on
+the expected software and file formats.
+:::
+
+
+You will see a view similar to the one below:
+
+(target-widget-screenshot)=
+
+![napari widget with poses dataset loaded](../_static/napari_plugin_with_poses_as_points.png)
+
+The predicted keypoints are represented as points, colour-coded by
+keypoint ID for single-individual datasets, or by individual ID for
+multi-individual datasets. These IDs can be also displayed as text
+next to the points by enabling the `display text` option from the
+layer controls panel.
+
+Hovering with your mouse over a point
+(with the points layer selected) will
+bring up a tooltip containing the names of the individual and keypoint,
+the point-wise confidence score (as predicted by the source software),
+and the time in seconds (this is calculated from the frame number and
+the `fps` value you provided).
+
+Using the slider at the bottom of the window, you can move through
+the frames of the dataset, and the points and video will update
+in sync.
+
+::: {admonition} Stay tuned
+Though the display style of the points layer is currently fixed, we are
+working on adding more customisation options in future releases, such as
+enabling you to change the point size, colour, or shape.
+
+We are also working on enabling the visualisation of
+[bounding boxes datasets](target-poses-and-bboxes-dataset) in the plugin.
+:::

movement/napari/_loader_widgets.py

@@ -8,12 +8,12 @@
 from napari.viewer import Viewer
 from qtpy.QtWidgets import (
     QComboBox,
+    QDoubleSpinBox,
     QFileDialog,
     QFormLayout,
     QHBoxLayout,
     QLineEdit,
     QPushButton,
-    QSpinBox,
     QWidget,
 )
 
@@ -56,11 +56,20 @@ def _create_source_software_widget(self):
 
     def _create_fps_widget(self):
         """Create a spinbox for selecting the frames per second (fps)."""
-        self.fps_spinbox = QSpinBox()
+        self.fps_spinbox = QDoubleSpinBox()
         self.fps_spinbox.setObjectName("fps_spinbox")
-        self.fps_spinbox.setMinimum(1)
-        self.fps_spinbox.setMaximum(1000)
-        self.fps_spinbox.setValue(30)
+        self.fps_spinbox.setMinimum(0.1)
+        self.fps_spinbox.setMaximum(1000.0)
+        self.fps_spinbox.setValue(1.0)
+        self.fps_spinbox.setDecimals(2)
+        # How much we increment/decrement when the user clicks the arrows
+        self.fps_spinbox.setSingleStep(1)
+        # Add a tooltip
+        self.fps_spinbox.setToolTip(
+            "Set the frames per second of the tracking data.\n"
+            "This just affects the displayed time when hovering over a point\n"
+            "(it doesn't set the playback speed)."
+        )
         self.layout().addRow("fps:", self.fps_spinbox)
 
     def _create_file_path_widget(self):
@@ -124,9 +133,6 @@ def _on_load_clicked(self):
         self.file_name = Path(file_path).name
         self._add_points_layer()
 
-        self._set_playback_fps(fps)
-        logger.debug(f"Set napari playback speed to {fps} fps.")
-
     def _add_points_layer(self):
         """Add the predicted poses to the viewer as a Points layer."""
         # Style properties for the napari Points layer
@@ -144,12 +150,6 @@ def _add_points_layer(self):
         self.viewer.add_points(self.data[:, 1:], **points_style.as_kwargs())
         logger.info("Added poses dataset as a napari Points layer.")
 
-    @staticmethod
-    def _set_playback_fps(fps: int):
-        """Set the playback speed for the napari viewer."""
-        settings = get_settings()
-        settings.application.playback_fps = fps
-
     @staticmethod
     def _enable_layer_tooltips():
         """Toggle on tooltip visibility for napari layers.

pyproject.toml

@@ -48,7 +48,9 @@ entry-points."napari.manifest".movement = "movement.napari:napari.yaml"
 [project.optional-dependencies]
 napari = [
   "napari[all]>=0.5.0",
-  "brainglobe-utils[qt]>=0.6"  # needed for collapsible widgets
+  "brainglobe-utils[qt]>=0.6",  # needed for collapsible widgets
+  "napari-video",
+  "pyvideoreader>=0.5.3",  # since switching to depend on openCV-headless
 ]
 dev = [
   "pytest",

tests/test_unit/test_napari_plugin/test_poses_loader_widget.py

@@ -8,7 +8,7 @@
 import pytest
 from napari.settings import get_settings
 from pytest import DATA_PATHS
-from qtpy.QtWidgets import QComboBox, QLineEdit, QPushButton, QSpinBox
+from qtpy.QtWidgets import QComboBox, QDoubleSpinBox, QLineEdit, QPushButton
 
 from movement.napari._loader_widgets import PosesLoader
 
@@ -25,7 +25,7 @@ def test_poses_loader_widget_instantiation(make_napari_viewer_proxy):
     # Check that the expected widgets are present in the layout
     expected_widgets = [
         (QComboBox, "source_software_combo"),
-        (QSpinBox, "fps_spinbox"),
+        (QDoubleSpinBox, "fps_spinbox"),
         (QLineEdit, "file_path_edit"),
         (QPushButton, "load_button"),
         (QPushButton, "browse_button"),
@@ -160,14 +160,10 @@ def test_on_load_clicked_with_valid_file_path(
         "Converted poses dataset to a napari Tracks array.",
         "Tracks array shape: (2170, 4)",
         "Added poses dataset as a napari Points layer.",
-        "Set napari playback speed to 60 fps.",
     }
     log_messages = {record.getMessage() for record in caplog.records}
     assert expected_log_messages <= log_messages
 
     # Check that a Points layer was added to the viewer
     points_layer = poses_loader_widget.viewer.layers[0]
     assert points_layer.name == f"poses: {file_path.name}"
-
-    # Check that the playback fps was set correctly
-    assert get_settings().application.playback_fps == 60