Switch to Rye, update installation instructions (#38)

* Update installation instructions Signed-off-by: Fabrice Normandin <[email protected]> * Add missing `mila_setup.sh` script Signed-off-by: Fabrice Normandin <[email protected]> * Make the script executable Signed-off-by: Fabrice Normandin <[email protected]> * Add content to `mila_setup.sh` script Signed-off-by: Fabrice Normandin <[email protected]> * Switch to [Rye](https://rye.astral.sh/) Signed-off-by: Fabrice Normandin <[email protected]> * Add instructions at the end of mila_setup.sh Signed-off-by: Fabrice Normandin <[email protected]> * Update devcontainer file Signed-off-by: Fabrice Normandin <[email protected]> * Update the build script Signed-off-by: Fabrice Normandin <[email protected]> * Update the docs workflow Signed-off-by: Fabrice Normandin <[email protected]> * Fix python-requires to >= 3.10 Signed-off-by: Fabrice Normandin <[email protected]> * Tweak the devcontainer so it works Signed-off-by: Fabrice Normandin <[email protected]> * Tweak the devcontainer file Signed-off-by: Fabrice Normandin <[email protected]> * Re-add the .python-version file Signed-off-by: Fabrice Normandin <[email protected]> * Update requirement files for py3.10 Signed-off-by: Fabrice Normandin <[email protected]> * Make requirement files universal (all platforms?) Signed-off-by: Fabrice Normandin <[email protected]> * Update the installation instructions Signed-off-by: Fabrice Normandin <[email protected]> --------- Signed-off-by: Fabrice Normandin <[email protected]>
mila-iqia · Aug 19, 2024 · 4631c1b · 4631c1b
1 parent 1a635ca
commit 4631c1b
Show file tree

Hide file tree

Showing 14 changed files with 1,106 additions and 3,957 deletions.
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -3,19 +3,17 @@
 {
 	"name": "Research Template",
 	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
-	"image": "mcr.microsoft.com/devcontainers/python:1-3.10-bullseye",
+	"image": "mcr.microsoft.com/devcontainers/python:1-3.12-bullseye",
 	// Features to add to the dev container. More info: https://containers.dev/features.
 	"features": {
-		"ghcr.io/devcontainers-contrib/features/pdm:2": {},
 		"ghcr.io/devcontainers-contrib/features/pre-commit:2": {},
 		"ghcr.io/devcontainers-contrib/features/apt-get-packages": {
 			"packages": "vim"
-		}
+		},
+		"ghcr.io/e-gineering/devcontainer-features/rye:1": {}
 	},
-
 	// Use 'forwardPorts' to make a list of ports inside the container available locally.
 	// "forwardPorts": [],
-
 	// Configure tool-specific properties.
 	"customizations": {
 		// TODO: look into this one.
@@ -39,12 +37,21 @@
 					"**/CVS": true,
 					"**/.DS_Store": true,
 					"**/Thumbs.db": true,
+					// ADDED:
+					"**/__pycache__": true,
+					".mypy_cache": true,
 					".venv": true,
-					".pdm.toml": true,
-					".pdm-python": true,
-					".pdm-build": true,
+					".pytest_cache": true,
+					".benchmarks": true,
 					".ruff_cache": true
-				}
+				},
+				"python.testing.unittestEnabled": false,
+				"python.testing.pytestEnabled": true,
+				"python.experiments.optOutFrom": [
+					"pythonTerminalEnvVarActivation"
+				],
+				"python.terminal.activateEnvInCurrentTerminal": false,
+				"python.terminal.activateEnvironment": true
 			},
 			// Add the IDs of extensions you want installed when the container is created.
 			"extensions": [
@@ -61,15 +68,16 @@
 	"containerEnv": {
 		"SCRATCH": "/home/vscode/scratch",
 		"SLURM_TMPDIR": "/tmp",
-		"NETWORK_DIR": "/network"
+		"NETWORK_DIR": "/network",
+		// used by rye (fixed issues with symlinked .cache dir to $SCRATCH)
+		"UV_LINK_MODE": "symlink"
 	},
 	"mounts": [
 		// https://code.visualstudio.com/remote/advancedcontainers/add-local-file-mount
 		// Mount a directory which will contain the pdm installation cache (shared with the host machine).
 		// This will use $SCRATCH/.cache/pdm, otherwise
 		// Mount a "$SCRATCH" directory in the host to ~/scratch in the container.
 		"source=${localEnv:SCRATCH},target=/home/vscode/scratch,type=bind,consistency=cached",
-		"source=${localEnv:SCRATCH}/.cache/pdm,target=/home/vscode/.pdm_install_cache,type=bind,consistency=cached",
 		// Mount a /network to match the /network directory on the host.
 		// FIXME: This assumes that either the NETWORK_DIR environment variable is set on the host, or
 		// that the /network directory exists.
@@ -90,32 +98,19 @@
 	// create the pdm cache dir on the host machine if it doesn exist yet so the mount above
 	// doesn't fail.
 	"initializeCommand": {
-		"create pdm install cache": "mkdir -p ${SCRATCH?need the SCRATCH environment variable to be set.}/.cache/pdm",      // todo: put this on $SCRATCH on the host (e.g. compute node)
-		"create fake SLURM_TMPDIR": "mkdir -p ${SLURM_TMPDIR:-/tmp/slurm_tmpdir}",  // this is fine on compute nodes
+		"create fake SLURM_TMPDIR": "mkdir -p ${SLURM_TMPDIR:-/tmp/slurm_tmpdir}", // this is fine on compute nodes
 		"create ssh cache dir": "mkdir -p ~/.cache/ssh"
 	},
-	// NOTE: Getting some permission issues with the .cache dir if mounting .cache/pdm to
-	// .cache/pdm in the container. Therefore, here I'm making a symlink from ~/.cache/pdm to
-	// ~/.pdm_install_cache so the ~/.cache directory is writeable by the container.
 	"onCreateCommand": {
-		"setup_pdm_install_cache": "mkdir -p ~/.cache && ln -s /home/vscode/.pdm_install_cache /home/vscode/.cache/pdm",
-		"pre-commit": "pre-commit install --install-hooks",
-		"setup_pdm_config": "pdm config install.cache true && pdm config venv.with_pip true && pdm config venv.in_project false"
+		"pre-commit": "pre-commit install --install-hooks"
 	},
 	"updateContentCommand": {
-		"pdm_install": "pdm install"
+		// BUG: doesn't work atm. Perhaps some permission issues?
+		// "Sync dependencies": "rye sync --all-features"
 	},
-
 	// Use 'postCreateCommand' to run commands after the container is created.
-	// "postCreateCommand": "pip3 install --user -r requirements.txt",
 	"postCreateCommand": {
-		// NOTE: This sets up the python interpreter correctly, but the shell doesn't show the
-		// environment name, which isn't a big deal.
-		"setup_venv_shell": "pdm venv activate >> ~/.bash_aliases && chmod +x ~/.bash_aliases",
-		// "pdm-pep-582": "pdm --pep582 >> ~/.bash_profile",
-		// "pdm": "pdm config install.cache false && pdm config venv.in_project false && pdm install",
 		// "pre-commit": "pre-commit install --install-hooks"
-		// "activate_venv": "source .venv/bin/activate"
 	}
 	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
 	// "remoteUser": "root"

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -44,21 +44,25 @@ jobs:
         python-version: ['3.10']
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v5
+    - name: Setup Rye with caching of venvs
+      uses: eifinger/setup-rye@v4
+      id: setup-rye
       with:
-        python-version: ${{ matrix.python-version }}
-    - run: pip install pdm
+        enable-cache: true
+        github-token: ${{ secrets.GITHUB_TOKEN }}
+        cache-prefix: ${{ matrix.python-version }}
+    - name: Pin python-version ${{ matrix.python-version }}
+      run: rye pin ${{ matrix.python-version }}
     - name: Install dependencies
-      run: pdm install
+      run: rye sync --no-lock
     - name: Test with pytest (very fast)
       env:
         JAX_PLATFORMS: cpu
-      run: pdm run pytest -v --shorter-than=1.0 --cov=project --cov-report=xml --cov-append --skip-if-files-missing
+      run: rye run pytest -v --shorter-than=1.0 --cov=project --cov-report=xml --cov-append --skip-if-files-missing
     - name: Test with pytest (fast)
       env:
         JAX_PLATFORMS: cpu
-      run: pdm run pytest -v --cov=project --cov-report=xml --cov-append --skip-if-files-missing
+      run: rye run pytest -v --cov=project --cov-report=xml --cov-append --skip-if-files-missing
 
     - name: Store coverage report as an artifact
       uses: actions/upload-artifact@v4
@@ -75,16 +79,21 @@ jobs:
         python-version: ['3.10']
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v5
+    - name: Setup Rye with caching of venvs
+      uses: eifinger/setup-rye@v4
+      id: setup-rye
       with:
-        python-version: ${{ matrix.python-version }}
-    - run: pip install pdm
+        enable-cache: true
+        github-token: ${{ secrets.GITHUB_TOKEN }}
+        cache-prefix: ${{ matrix.python-version }}
+    - name: Pin python-version ${{ matrix.python-version }}
+      run: rye pin ${{ matrix.python-version }}
+
     - name: Install dependencies
-      run: pdm config install.cache true && pdm install
+      run: rye sync --no-lock
 
     - name: Test with pytest
-      run: pdm run pytest -v --cov=project --cov-report=xml --cov-append --skip-if-files-missing
+      run: rye run pytest -v --cov=project --cov-report=xml --cov-append --skip-if-files-missing
     # TODO: this is taking too long to run, and is failing consistently. Need to debug this before making it part of the CI again.
     # - name: Test with pytest (only slow tests)
     #   run: pdm run pytest -v -m slow --slow --cov=project --cov-report=xml --cov-append
@@ -132,16 +141,12 @@ jobs:
         cluster: ['mila']
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python 3.10
-      uses: actions/setup-python@v5
-      with:
-        python-version: "3.10"
-    - run: pip install pdm
+    - name: Set up the repo using the setup script
+      run: scripts/mila_setup.sh
     - name: Install dependencies
-      run: pdm install
-
+      run: rye sync --no-lock
     - name: Test with pytest
-      run: pdm run pytest -v --cov=project --cov-report=xml --cov-append --skip-if-files-missing
+      run: rye run pytest -v --cov=project --cov-report=xml --cov-append --skip-if-files-missing
 
     # TODO: Re-enable this later
     # - name: Test with pytest (only slow tests)

diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -11,17 +11,20 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
-      - name: Set up Python 3.12
-        uses: actions/setup-python@v5
+      - name: Setup Rye with caching of venvs
+        uses: eifinger/setup-rye@v4
+        id: setup-rye
         with:
-          python-version: 3.12
-      - name: Install pdm
-        run: pip install pdm
+          enable-cache: true
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Pin python-version
+        run: rye pin 3.10
+
       - name: Install dependencies
-        run: pdm install
+        run: rye sync --no-lock --features docs
 
       - name: Deploy docs
-        run: pdm run mkdocs gh-deploy --force
+        run: rye run mkdocs gh-deploy --force
         # note: Checking if we really need the one below:
         # uses: mhausenblas/[email protected]
         # # Or use mhausenblas/mkdocs-deploy-gh-pages@nomaterial to build without the mkdocs-material theme

diff --git a/.gitignore b/.gitignore
@@ -162,9 +162,10 @@ cython_debug/
 #.idea/
 # ADDED:
 .vscode
-.pdm-python
 wandb
 logs
 lightning_logs
 .vscode
 **.npz
+# .python-version
+.testmondata*
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -32,7 +32,7 @@ repos:
 
   - repo: https://github.com/charliermarsh/ruff-pre-commit
     # Ruff version.
-    rev: "v0.5.1"
+    rev: "v0.5.7"
     hooks:
       - id: ruff
         args: ['--line-length', '99', '--fix']
@@ -62,14 +62,6 @@ repos:
       - id: nbstripout
         require_serial: true
 
-  # Dependency management
-  - repo: https://github.com/pdm-project/pdm
-    rev: 2.16.1
-    hooks:
-      - id: pdm-lock-check
-        require_serial: true
-
-
   # md formatting
   - repo: https://github.com/executablebooks/mdformat
     rev: 0.7.17

diff --git a/.python-version b/.python-version
@@ -0,0 +1 @@
+3.10.14
diff --git a/docs/getting_started/install.md b/docs/getting_started/install.md
@@ -5,47 +5,37 @@ There are two ways to install this project
 1. Using Conda (recommended for newcomers)
 2. Using a development container (recommended if you are able to install Docker on your machine)
 
-## Using Conda and pip
+## Installation
 
-### Prerequisites
-
-You need to have [Conda](https://docs.conda.io/en/latest/) installed on your machine.
-
-### Installation
-
-1. Clone the repository and navigate to the root directory:
+1. Clone the repository:
 
     ```bash
     git clone https://www.github.com/mila-iqia/ResearchTemplate
     cd ResearchTemplate
     ```
 
-2. Create a conda environment
+2. Installing dependencies
 
-    ```bash
-    conda create -n research_template python=3.12
-    conda activate research_template
-    ```
+    You can install the package using `pip install -e .`, although we recommend using the [Rye](https://rye.astral.sh/)
+    package manager. This makes it easier to switch python versions and to add or change the dependencies later on.
 
-    Notes:
+    1. On your machine:
 
-    - If you don't Conda installed, you can download it from [here](https://docs.conda.io/en/latest/miniconda.html).
-    - If you'd rather use a virtual environment instead of Conda, you can totally do so, as long as you have a version of Python >= 3.12.
+        ```console
+        curl -sSf https://rye.astral.sh/get | bash
+        rye sync       # Creates a virtual environment and installs dependencies in it.
+        ```
 
-    <!-- TODO: - If you're on the `mila` cluster, you can run this setup script: (...) -->
+    2. On the Mila cluster:
 
-3. Install the package using pip:
+        If you're on the `mila` cluster, you can run this setup script (on a *compute* node):
 
-    ```bash
-    pip install -e .
-    ```
-
-    Optionally, you can also install the package using [PDM](https://pdm-project.org/en/latest/). This makes it easier to add or change the dependencies later on:
-
-    ```bash
-    pip install pdm
-    pdm install
-    ```
+        ```console
+        # Get a compute node to run an interactive job:
+        salloc --gres=gpu:1 --cpus-per-task=4 --mem=16G --time=1:00:00
+        # Run the installation script.
+        scripts/mila_setup.sh
+        ```
 
 ## Using a development container