Merge branch 'master' into add-zarrarray-as-mag

.github/workflows/ci.yml

@@ -295,7 +295,8 @@ jobs:
 
       - name: Build Docs
         run: |
-          docs/generate.sh --persist
+          cd docs
+          ./generate.sh --persist
 
       - name: Push docs (for branch)
         if: github.ref_type == 'branch'
@@ -324,7 +325,7 @@ jobs:
           SLACK_HOOK: ${{ secrets.LINK_CHECKER_SLACK_HOOK }}
         run: |
           cd docs
-          poetry run linkchecker https://docs.webknossos.org > link_status || \
+          poetry run linkchecker --config linkcheckerrc https://docs.webknossos.org > link_status || \
             curl -X POST --data-urlencode "payload={\"text\": \":warning: Broken Links on doc.webknossos.org :warning:\n"'```'"\n$(cat link_status)\n"'```"}' \
               "$SLACK_HOOK"
 

.github/workflows/publish_docs.yml

@@ -27,7 +27,8 @@ jobs:
 
     - name: Build Docs
       run: |
-        docs/generate.sh --persist
+        cd docs
+        ./generate.sh --persist
 
     - name: Push docs
       env:
@@ -44,6 +45,6 @@ jobs:
         SLACK_HOOK: ${{ secrets.LINK_CHECKER_SLACK_HOOK }}
       run: |
         cd docs
-        poetry run linkchecker https://docs.webknossos.org > link_status || \
+        poetry run linkchecker --config linkcheckerrc https://docs.webknossos.org > link_status || \
           curl -X POST --data-urlencode "payload={\"text\": \":warning: Broken Links on doc.webknossos.org :warning:\n"'```'"\n$(cat link_status)\n"'```"}' \
             "$SLACK_HOOK"

CONTRIBUTING.md

@@ -140,7 +140,7 @@ The `webknossos` folder contains examples, which are not part of the package, bu
 The tests also contain functionality for the WEBKNOSSOS client. There a two modes to run the tests:
 
 1. `./test.sh --refresh-snapshots`, sending network requests to a WEBKNOSSOS instance:
-  This expects a local WEBKNOSSOS setup with specific test data, which is shipped with WEBKNOSSOS. If you're starting and running WEBKNOSSOS manually, please use port 9000 (the default) and run the `tools/postgres/dbtool.js prepare-test-db` script in the WEBKNOSSOS repository (⚠️ this overwrites your local WEBKNOSSOS database). Alternatively, a docker-compose setup is started automatically for the tests, see `./test.sh` and `tests/docker-compose.yml` for details. The network requests & response are recorded as "cassettes" by [vcr.py](https://vcrpy.readthedocs.io), see next point:
+  This expects a local WEBKNOSSOS setup with specific test data, which is shipped with WEBKNOSSOS. If you're starting and running WEBKNOSSOS manually, please use port 9000 (the default) and run the `tools/postgres/dbtool.js prepare-test-db` script in the WEBKNOSSOS repository (⚠️ this overwrites your local WEBKNOSSOS database). Alternatively, a `docker compose` setup is started automatically for the tests, see `./test.sh` and `tests/docker-compose.yml` for details. The network requests & response are recorded as "cassettes" by [vcr.py](https://vcrpy.readthedocs.io), see next point:
 2. `./test.sh` replays responses from previous network snapshots using [vcr.py](https://vcrpy.readthedocs.io) via [pytest-recording](https://github.com/kiwicom/pytest-recording). No additional network requests are allowed in this mode.
 
 `./test.sh --store-durations` updates the durations for
@@ -149,7 +149,7 @@ which is used in the CI to split the tests for different runners.
 
 #### `cluster_tools` package
 
-For testing the `slurm` setup a docker-compose setup is available. Please see the [respective Readme](https://github.com/scalableminds/webknossos-libs/blob/master/cluster_tools/README.md) for details.
+For testing the `slurm` setup a `docker compose` setup is available. Please see the [respective Readme](https://github.com/scalableminds/webknossos-libs/blob/master/cluster_tools/README.md) for details.
 
 For testing the `kubernetes` setup, we recommend a [Kubernetes-in-Docker setup](https://kind.sigs.k8s.io/).
 

cluster_tools/Changelog.md

@@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/) `MAJOR.MIN
 For upgrade instructions, please check the respective *Breaking Changes* sections.
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.1...HEAD)
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.2...HEAD)
 
 ### Breaking Changes
 
@@ -18,6 +18,10 @@ For upgrade instructions, please check the respective *Breaking Changes* section
 ### Fixed
 
 
+## [0.15.2](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.15.2) - 2024-09-05
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.1...v0.15.2)
+
+
 ## [0.15.1](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.15.1) - 2024-08-13
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.0...v0.15.1)
 

cluster_tools/README.md

@@ -76,7 +76,7 @@ If you would like to configure these limits independently, you can do so by sett
 ```
 # See ./dockered-slurm/README.md for troubleshooting
 cd dockered-slurm
-docker-compose up -d
+docker compose up -d
 docker exec -it slurmctld bash
 docker exec -it c1 bash
 ```

cluster_tools/dockered-slurm/README.md

@@ -2,7 +2,7 @@
 
 **Note: Refer to cluster_tools/README.md for how to use this container in the context of testing the cluster_tools**
 
-This is a multi-container Slurm cluster using docker-compose.  The compose file
+This is a multi-container Slurm cluster using `docker compose`.  The compose file
 creates named volumes for persistent storage of MySQL data files as well as
 Slurm state and log directories.
 
@@ -26,10 +26,10 @@ The compose file will create the following named volumes:
 
 ## Starting the Cluster
 
-Run `docker-compose` to instantiate the cluster:
+Run `docker compose` to instantiate the cluster:
 
 ```console
-$ docker-compose up -d
+$ docker compose up -d
 ```
 
 > Note: If you encounter permission errors (`Failed to check keyfile "/etc/munge/munge.key": Permission denied`), follow the steps from the "Deleting the Cluster" section and run the previous command again.
@@ -47,10 +47,10 @@ $ ./register_cluster.sh
 > ready before registering the cluster.  Otherwise, you may get an error such
 > as **sacctmgr: error: Problem talking to the database: Connection refused**.
 >
-> You can check the status of the cluster by viewing the logs: `docker-compose
+> You can check the status of the cluster by viewing the logs: `docker compose
 > logs -f`
 
-> Note: If you encounter an error that the daemon is not running (`Error response from daemon: Container <...> is not running`), the start of the containers was not successful. Check the logs using `docker-compose logs -f` and revisit the last step.
+> Note: If you encounter an error that the daemon is not running (`Error response from daemon: Container <...> is not running`), the start of the containers was not successful. Check the logs using `docker compose logs -f` and revisit the last step.
 
 ## Accessing the Cluster
 
@@ -85,18 +85,18 @@ slurm-2.out
 ## Stopping and Restarting the Cluster
 
 ```console
-$ docker-compose stop
+$ docker compose stop
 ```
 
 ```console
-$ docker-compose start
+$ docker compose start
 ```
 
 ## Deleting the Cluster
 
 To remove all containers and volumes, run:
 
 ```console
-$ docker-compose rm -sf
+$ docker compose rm -sf
 $ docker volume rm dockered-slurm_etc_munge dockered-slurm_etc_slurm dockered-slurm_slurm_jobdir dockered-slurm_var_lib_mysql dockered-slurm_var_log_slurm
 ```

cluster_tools/dockered-slurm/register_cluster.sh

@@ -2,4 +2,4 @@
 set -e
 
 docker exec slurmctld bash -c "/usr/bin/sacctmgr --immediate add cluster name=linux" && \
-docker-compose restart slurmdbd slurmctld
+docker compose restart slurmdbd slurmctld

docs/.gitignore

@@ -137,6 +137,6 @@ dmypy.json
 # Cython debug symbols
 cython_debug/
 
-/src/api
+/src/api/webknossos
 /wk-repo
 /out

docs/README.md

@@ -1,17 +1,17 @@
 # Documentation
 
-Run `docs/generate.sh` to open a server rendering the documentation.
-To update docstrings restart the server, manually written pages in `src` are auto-reloaded.
+## Development
+Run `./generate.sh` to open a live-reloading server rendering the documentation.
 
-To get a live-reloading server for the docstrings, run `docs/generate.sh --api`. This opens pdoc, which looks differently than the final result, but the actual contents are the same.
+## Production
 
-To produce the html in `out`, run `docs/generate.sh --persist`.
+Run `./generate.sh --persist` to produce the production website/HTML in `out`. Use GitHub Actions for building and deploying the website.
 
 
 ## Further links
 
 * https://www.mkdocs.org
-* https://pdoc.dev
 * https://squidfunk.github.io/mkdocs-material
 * https://facelessuser.github.io/pymdown-extensions
 * https://python-markdown.github.io/extensions/#officially-supported-extensions
+* https://mkdocstrings.github.io/

docs/generate.sh

@@ -1,11 +1,7 @@
 #! /usr/bin/env bash
 set -Eeo pipefail
 
-PROJECT_DIR="$(dirname "$(dirname "$0")")"
-
-
-cd "$PROJECT_DIR/docs"
-poetry install
+poetry install --no-root
 
 if [ ! -d "wk-repo" ]; then
     echo
@@ -14,25 +10,11 @@ if [ ! -d "wk-repo" ]; then
     echo 'git clone --depth 1 [email protected]:scalableminds/webknossos.git docs/wk-repo'
     exit 1
 fi
+rm -rf src/api/webknossos
+PYTHONPATH=$PYTHONPATH poetry run python generate_api_doc_pages.py
 
-export PDOC_CLASS_MODULES="$(poetry run python get_keyword_mapping.py)"
-if [ $# -eq 1 ] && [ "$1" = "--api" ]; then
-    poetry run pdoc ../webknossos/webknossos !webknossos.dataset._utils -t pdoc_templates/pure_pdoc -h 0.0.0.0 -p 8196
+if [ $# -eq 1 ] && [ "$1" = "--persist" ]; then
+    PYTHONPATH=$PYTHONPATH poetry run mkdocs build
 else
-    rm -rf src/api
-    poetry run pdoc ../webknossos/webknossos !webknossos.dataset._utils -t pdoc_templates/with_mkdocs -o src/api
-    # rename .html files to .md
-    find src/api -iname "*.html" -exec sh -c 'mv "$0" "${0%.html}.md"' {} \;
-    # assert that API docs are written
-    webknossos_files="$(find src/api/webknossos -type f -name "*.md" | wc -l)"
-    if ! [ "$webknossos_files" -gt "50" ]; then
-       echo "Error: There are too few ($webknossos_files, expected > 80) files in src/api/webknossos,"
-       echo "probably there was an error with pdoc before!"
-       exit 1
-    fi
-    if [ $# -eq 1 ] && [ "$1" = "--persist" ]; then
-        PYTHONPATH=$PYTHONPATH:. poetry run mkdocs build
-    else
-        PYTHONPATH=$PYTHONPATH:. poetry run mkdocs serve -a localhost:8197 --watch-theme
-    fi
+    PYTHONPATH=$PYTHONPATH poetry run mkdocs serve -a localhost:8197 --watch-theme
 fi

docs/generate_api_doc_pages.py

@@ -0,0 +1,63 @@
+import inspect
+from logging import getLogger
+from pathlib import Path
+
+import webknossos
+
+"""
+This script generates a mapping of all classes in webknossos to their
+corresponding MkDocs pages. It is used to generate the API reference.
+"""
+
+logger = getLogger(__name__)
+
+OUT_PATH = Path("src/api")
+
+# Initialize a dictionary to store submodules and their classes
+submodules_classes = {}
+
+# Iterate through all the members of the module
+for name, member in inspect.getmembers(webknossos):
+    # Check if the member is a module (submodule)
+    if inspect.ismodule(member) and member.__name__.startswith("webknossos"):
+        submodule_name = member.__name__
+
+        # List classes in the submodule
+        classes = inspect.getmembers(member, inspect.isclass)
+
+        # Filter classes that are defined in this specific submodule
+        defined_classes = [
+            cls for cls_name, cls in classes if cls.__module__ == submodule_name
+        ]
+
+        # If there are classes defined in this submodule, add them to the dictionary
+        if defined_classes:
+            submodules_classes[submodule_name] = defined_classes
+
+# Print the submodules and their classes
+for submodule, classes in submodules_classes.items():
+    # The file content uses a special syntax for MkDocs to render the
+    # docstrings as Markdown. The syntax is:
+    # ::: module.submodule.class
+    # See https://mkdocstrings.github.io/python/
+    classes_string = "\n".join(
+        [
+            f"            - {c.__name__}"
+            for c in classes
+            if not c.__name__.startswith("_")
+        ]
+    )
+    file_content = f"""::: {submodule}
+    options:
+        members:\n{classes_string}\n"""
+
+    submodule_parts = submodule.split(".")
+    submodule_name = submodule_parts[-1]
+    file_name = Path(f"{submodule_name}.md")
+    file_path = "/".join(submodule_parts[:-1])
+
+    out_path = OUT_PATH.joinpath(file_path)
+    out_path.mkdir(exist_ok=True, parents=True)
+
+    logger.debug(f"Writing API docs to {out_path.joinpath(file_name)}")
+    out_path.joinpath(file_name).write_text(file_content)

docs/linkcheckerrc

@@ -0,0 +1,2 @@
+[filtering]
+ignorewarnings=http-redirected