Merge branch 'develop' into backport-upstream

README.md

@@ -20,6 +20,7 @@ documentation:
 * [Debugging pull requests on BlueBrain5](bluebrain/documentation/debug_bb5.md)
 * [Continuous Integration with Spack on BlueBrain5](bluebrain/documentation/ci_bb5.md)
 * [Reproducing a failed CI build](bluebrain/documentation/reproduce_ci.md)
+* [Upstreaming a package](bluebrain/documentation/upstreaming.md)
 
 # <img src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo.svg" width="64" valign="middle" alt="Spack"/> Spack
 

bluebrain/bin/upstream.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+
+if [ -z ${1} ]
+then
+    echo "Call with upstreamed package name as first argument"
+    echo "It will check your current branch, checkout and update upstream/develop, then go back to your original branch."
+    echo "After that it will checkout the package folder from upstream/develop, remove the patches or bluebrain version from both git and filesystem"
+    echo ""
+    echo "If you want to update your upstream-develop branch from upstream/develop, pass 'update' as the second argument"
+    echo ""
+    echo "WARNING - IF YOU HAVE ANY WORK ON upstream-develop IT WILL BE LOST"
+    exit 0
+fi
+
+
+
+set -e
+
+upstreamed=$1
+update_upstream=$2
+
+UPSTREAM_REMOTE=$(git remote -v | awk '/github.com:spack\/spack/ {print $1}' | head -n 1)
+
+if [ -z "${UPSTREAM_REMOTE}" ]
+then
+    echo "upstream remote not found - adding"
+    git remote add upstream github.com:spack/spack
+    UPSTREAM_REMOTE=upstream
+fi
+
+if [ "${update_upstream}" == "update" ]
+then
+    CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
+    echo "Current branch is ${CURRENT_BRANCH} - updating ${UPSTREAM_REMOTE}/develop"
+    git checkout -B upstream-develop --track ${UPSTREAM_REMOTE}/develop
+    git pull
+    git checkout ${CURRENT_BRANCH}
+fi
+
+echo "Checking out ${upstreamed} from ${UPSTREAM_REMOTE}/develop "
+git checkout ${UPSTREAM_REMOTE}/develop -- var/spack/repos/builtin/packages/${upstreamed}/
+
+ORIGINAL=""
+if [ -e bluebrain/repo-patches/packages/${upstreamed} ]
+then
+    ORIGINAL=bluebrain/repo-patches/packages/${upstreamed}
+elif [ -e bluebrain/repo-bluebrain/packages/${upstreamed} ]
+then
+    ORIGINAL+=" bluebrain/repo-bluebrain/packages/${upstreamed}"
+fi
+if [ -z ${ORIGINAL} ]
+then
+    echo "Original already removed"
+else
+    echo "Original package was in ${ORIGINAL}"
+    set +e
+    git rm -fr ${ORIGINAL}
+    set -e
+    rm -fr ${ORIGINAL}
+fi
+
+echo "All done"

bluebrain/deployment/environments/applications_viz.yaml

@@ -13,10 +13,12 @@ spack:
   specs:
     - brayns
     - [email protected]
+    - [email protected]
     - py-bcsb
     - [email protected]
     - [email protected]
     - py-brayns
     - [email protected]
+    - [email protected]
     - brion +python
     - ultraliser ^hdf5~mpi

bluebrain/documentation/upstreaming.md

@@ -0,0 +1,42 @@
+Upstreaming packages
+====================
+
+Getting your packages to upstream
+---------------------------------
+
+The easiest way to do this is to create your own fork of [Spack](https://github.com/spack/spack). Create a new branch starting from the latest `develop` branch and copy your changes there. Your package recipe and any patches should go under `var/spack/repos/builtin/packages/`.
+
+You'll need to:
+  * make sure a recent (preferably the latest) version is present in the recipe
+  * if specifying a version linked to a git branch, ensure the version name and branch name are the same
+  * make sure dependencies are correct and complete
+    * give them the correct type. For Python dependencies `type=("build", "run")` in most cases
+    * for Python packages: no need to specify `py-wheel` or `py-pip`, those come with the parent class
+    * when adding multiple versions, keep in mind that different versions can have different dependencies, versions of dependencies, ...
+  * ensure your code is properly formatted (black) and the header is correct (including the copyright year)
+  * an upstreamed package should not depend on anything internal - it should install from your spack fork without any bluebrain-specific modifications
+
+Create your pull request from your fork to the Spack `develop` branch and go through the review process.
+
+Backporting upstreamed changes
+------------------------------
+
+Once your pull request has been merged, you'll need to backport your changes to our own spack repository. In order to do this, there's a script available: `bluebrain/bin/upstream.sh`.
+If you call it without arguments, it will explain how to use it.
+
+Prepare to run by creating the branch you'd like to commit your changes to.
+
+---
+**WARNING**: if you have a local branch called `upstream-develop` it **will** be reset to match upstream/develop
+---
+
+The first argument is the name of the package you've upstreamed
+The optional second argument can be 'update' to update your `upstream-develop` branch from `upstream/develop`. Typically you'll update upstream only on the first package in your batch.
+
+The script will:
+  * if necessary, create the upstream remote pointing to github.com:spack/spack
+  * if asked (through the 'update' argument), checkout upstream-develop tracking upstream/develop, `git pull` and go back to the branch you were on when you started
+  * `git checkout` the upstream/develop version of your upstreamed package (the first argument) in var/spack/repos/builtin/packages
+  * for whatever exists of your package under `bluebrain/repo-{patches,bluebrain}/packages/`: `git rm` and `rm -rf`
+
+Now you can commit, push and file a pull request to [our own Spack repository](https://github.com/bluebrain/spack).

bluebrain/repo-bluebrain/packages/brayns/package.py

@@ -17,6 +17,7 @@ class Brayns(CMakePackage):
     version("develop", branch="develop")
     version("3.2.2", tag="3.2.2")
     version("3.3.0", tag="3.3.0")
+    version("3.4.0", tag="3.4.0")
 
     depends_on("[email protected]:", type="build")
     depends_on("[email protected]:", type="build")

bluebrain/repo-bluebrain/packages/nmodl/package.py

@@ -15,6 +15,7 @@ class Nmodl(CMakePackage):
 
     version("develop", branch="master", submodules=True)
     version("llvm", branch="llvm", submodules=True)
+    version("0.7.a2", commit="6f6db3b6f25e066db46d8a5fc85a1697363b995c")
     version("0.7.a1", commit="2ce4a2b91dfcfe6356b6a5003c4e99b8711564ee")
     version("0.6.0", tag="0.6")
     version("0.5.0", tag="0.5")

bluebrain/repo-bluebrain/packages/py-astrovascpy/package.py

@@ -0,0 +1,56 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack.package import *
+
+
+class PyAstrovascpy(PythonPackage):
+    """
+    Vasculature blood flow computation and impact of astrocytic
+    endfeet on vessels
+    """
+
+    homepage = "https://github.com/BlueBrain/AstroVascPy"
+
+    url = "https://github.com/BlueBrain/AstroVascPy/releases/tag/0.1"
+
+    maintainers("tristan0x")
+
+    version("0.1", sha256="1e9b2036e9c1ce4cb0650612c0be2e7e7b142754e1550dfbfb0b757cee4f8bdd")
+
+    variant("vtk", default=False, description="add VTK support (mainly for visualization)")
+
+    depends_on("py-setuptools", type=("build", "run"))
+    depends_on("py-setuptools-scm", type="build")
+
+    depends_on("py-click", type=("build", "run"))
+    depends_on("py-cached-property", type=("build", "run"))
+    depends_on("py-coverage", type=("build", "run"))
+    depends_on("py-cython", type=("build", "run"))
+    depends_on("py-h5py", type=("build", "run"))
+    depends_on("py-libsonata", type=("build", "run"))
+    depends_on("py-matplotlib", type=("build", "run"))
+    depends_on("py-morphio", type=("build", "run"))
+    depends_on("py-networkx", type=("build", "run"))
+    depends_on("py-numpy", type=("build", "run"))
+    depends_on("py-pandas", type=("build", "run"))
+    depends_on("py-petsc4py", type=("build", "run"))
+    depends_on("py-psutil", type=("build", "run"))
+    depends_on("py-pyyaml", type=("build", "run"))
+    depends_on("py-scipy", type=("build", "run"))
+    depends_on("py-seaborn", type=("build", "run"))
+    depends_on("py-tables", type=("build", "run"))
+    depends_on("py-tqdm", type=("build", "run"))
+    depends_on("py-trimesh", type=("build", "run"))
+    depends_on("py-vascpy", type=("build", "run"))
+    depends_on("vtk+python", type=("build", "run"), when="+vtk")
+
+    depends_on("py-pytest", type="test")
+    depends_on("py-pytest-mpi", type="test")
+
+    @run_after("install")
+    @on_package_attributes(run_tests=True)
+    def test_install(self):
+        python("-m", "pytest", "tests")

bluebrain/repo-bluebrain/packages/py-brayns/package.py

@@ -20,6 +20,7 @@ class PyBrayns(PythonPackage):
     version("develop", branch="develop")
     version("3.2.2", tag="3.2.2")
     version("3.3.0", tag="3.3.0")
+    version("3.4.0", tag="3.4.0")
 
     depends_on("py-setuptools", type=("build"))
     depends_on("[email protected]:", type=("build", "run"))

bluebrain/repo-bluebrain/packages/py-cwl-registry/package.py

@@ -10,7 +10,7 @@ class PyCwlRegistry(PythonPackage):
     git = "ssh://[email protected]/nse/cwl-registry.git"
 
     version("develop", branch="main")
-    version("0.4.7", tag="cwl-registry-v0.4.7")
+    version("0.4.9", tag="cwl-registry-v0.4.9")
     version("0.3.14", tag="cwl-registry-v0.3.14")
     version("0.3.12", tag="cwl-registry-v0.3.12")
     version("0.3.10", tag="cwl-registry-v0.3.10")

bluebrain/repo-bluebrain/packages/py-neurodamus/package.py

@@ -12,6 +12,7 @@ class PyNeurodamus(PythonPackage):
     git = "https://github.com/BlueBrain/neurodamus.git"
 
     version("develop", branch="main")
+    version("2.16.5", tag="2.16.5")
     version("2.16.4", tag="2.16.4")
     version("2.16.3", tag="2.16.3")
     version("2.16.2", tag="2.16.2")

bluebrain/repo-patches/packages/highfive/package.py

@@ -15,6 +15,7 @@ class Highfive(CMakePackage):
     git = "https://github.com/BlueBrain/HighFive.git"
 
     version("master", branch="master")
+    version("2.8.0", sha256="cd2502cae61bfb00e32dd18c9dc75289e09ad1db5c2a46d3b0eefd32e0df983b")
     version("2.7.1", sha256="25b4c51a94d1e670dc93b9b73f51e79b65d8ff49bcd6e5d5582d5ecd2789a249")
     version("2.7.0", sha256="8e05672ddf81a59ce014b1d065bd9a8c5034dbd91a5c2578e805ef880afa5907")
     version("2.6.2", sha256="ab51b9fbb49e877dd1aa7b53b4b26875f41e4e0b8ee0fc2f1d735e0d1e43d708")

bluebrain/repo-patches/packages/neuron/package.py

@@ -32,6 +32,7 @@ class Neuron(CMakePackage):
     patch("patch-v800-cmake-nvhpc.patch", when="@8.0.0%nvhpc^[email protected]:")
 
     version("develop", branch="master")
+    version("9.0.a13", commit="3bbdd8da")
     version("9.0.a12", commit="6004512f")
     version("9.0.a11", commit="b38dd11b")
     version("9.0.a10", commit="af51e51e")

var/spack/repos/builtin/packages/py-bluepyemodel/package.py

@@ -12,6 +12,8 @@ class PyBluepyemodel(PythonPackage):
     homepage = "https://github.com/BlueBrain/BluePyEModel"
     pypi = "bluepyemodel/bluepyemodel-0.0.46.tar.gz"
 
+    version("0.0.59", sha256="5e8869522d82e719f9775c2d95cfe953cedc66bc44355765a6f406289baf6791")
+    version("0.0.58", sha256="327de9d2c49e7ff83cc77850873293299d4eacf95b3cf33716e5a8501685f08c")
     version("0.0.57", sha256="0b91e39e5066ab4a996bd932577b49648169e549c5f05bb3f93e345b4b186093")
     version("0.0.46", sha256="ad4c125e491f3337fcc341a4f389b8a616d883ce50fd77d9fb0ea6e13be5da61")