running jupyter notebooks on farm

[ctb]

Start Jupyter Notebook on farm

Titus Brown, [email protected]

June 19, 2023

To edit: / latest on github

(Adapted from Nick Ulle's tutorial!)

Instructions

First, create a mamba environment with jupyter notebook installed:

module load mamba
mamba create -y -n jup notebook nb_conda_kernels

Activate it:

mamba activate jup

Set a password:

jupyter notebook password

Create a script run-jupyter-on-node.sh containing:

#! /bin/bash -eu

# initialize conda
conda_base=$(conda info --base)
. ${conda_base}/etc/profile.d/conda.sh

# activate relevant environment - CUSTOMIZE ME
conda activate jup

### pick a port
read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range

port="$(shuf -i $LOWERPORT-$UPPERPORT -n 1)"
while :; do
  netstat -taln | egrep ":$port.*LISTEN" || break
  port="$(shuf -i $LOWERPORT-$UPPERPORT -n 1)"
done

### print out stuff
cat <<EOF

You need to use SSH tunneling to port forward from your desktop to Jupyter Lab.

Run the following command in a new terminal on your machine:

ssh -L${port}:${HOSTNAME}:${port} [email protected] -N

Then point your browser to http://localhost:${port}

EOF

### run me!
jupyter notebook --no-browser --ip=${HOSTNAME} --port=${port}

and then run with srun:

srun -p high2 --time=3:00:00 --nodes=1 \
            --cpus-per-task 1 --mem 5GB \
            --pty bash ./run-jupyter-on-node.sh

[ctb]

ref #44 rstudio server

[ctb]

A more automated solution based on #44 rstudio-server.

hackmd: https://hackmd.io/9Uk2wAOiQs6hnnaV_39DtQ?view

Two or three things to note and/or change:

• there's no guard against running this on the head node, which we might want to change!
• I chose to instruct people to set a single password with jupyter notebook password rather than generating a new one each time with xkcdpass
• we might want to redirect jupyter notebook output somewhere. right now it goes to stdout which is distracting.

hackmd copy/pasted below:

Start Jupyter Notebook from head node on farm

The below uses the mambaforge installation of conda.

First, create a mamba environment containing jupyter notebook:

mamba create -y -n jup notebook nb_conda_kernels

Activate it:

mamba activate jup

Set a password:

jupyter notebook password

Create a script run-jupyter-on-node.sh containing:

#! /bin/bash -eu

# initialize conda
conda_base=$(conda info --base)
. ${conda_base}/etc/profile.d/conda.sh

# activate relevant environment - CUSTOMIZE ME
conda activate jup

### pick a port
read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range

port="$(shuf -i $LOWERPORT-$UPPERPORT -n 1)"
while :; do
  netstat -taln | egrep ":$port.*LISTEN" || break
  port="$(shuf -i $LOWERPORT-$UPPERPORT -n 1)"
done

### print out stuff
cat <<EOF

You need to use SSH tunneling to port forward from your desktop to Jupyter Lab.

Run the following command in a new terminal on your machine:

ssh -L${port}:${HOSTNAME}:${port} [email protected] -N

Then point your browser to http://localhost:${port}

EOF

### run me!
jupyter notebook --no-browser --ip=${HOSTNAME} --port=${port}

and then run with srun:

srun -p high2 --time=3:00:00 --nodes=1 \
            --cpus-per-task 1 --mem 5GB \
            --pty bash ./run-jupyter-on-node.sh

[ccbaumler]

This has worked for me on the first try. However, the original comment's link is broken.

[ctb]

now at https://nick-ulle.github.io/bag-of-tricks/chapters/remote-computing.html#running-jupyter-rstudio-remotely - fixing link in top issue

[ctb]

updated for june 19, 2023