Multi course multi grader setup using nginx and Shibboleth

Kick-off

Sources:

• jhub_shibboleth_auth
• jhub-shibboleth-user-authenticator
• Using nbgrader with JupyterHub
• Advanced setup for automated nginx-proxy with separate containers

Pre-requisites:

• properly registered DNS record for the JupyterHub server (replace string FQDN.YOUR.SERV.ER in the followings, accordingly)
• available SSL port (443) - tried to use other ports, too, but no luck (Shibboleth requires 443 or 8443, AFAIK)
• registered Shibboleth SP (if jhub-shibboleth-user-authenticator is going to be used)
• SSL certificates, e.g. via Let's Encrypt
• (debian bullseye) linux distro on server with
• docker.io, docker-compose pkgs.
  (jupyterhub/jupyterhub:3.1.0 image seems to require docker.io v20.10.5, which can be found in bullseye)

Setup nginx / shibboleth / JupyterHub

Get docker images:

docker image pull gesiscss/nginx-shibboleth
docker image pull jupyterhub/jupyterhub

Create directories for docker-compose:

mkdir -p /var/lib/docker-jhub/{nginx,jupyterhub}
cd /var/lib/docker-jhub

Nginx and Shibboleth configuration

Patched nginx/nginx_shibboleth.conf:

map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
}

server {
    listen 443 ssl; # Shibboleth requires 443 or 8443
    server_name FQDN.YOUR.SERV.ER

    ssl_certificate     /etc/letsencrypt/live/FQDN.YOUR.SERV.ER/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/FQDN.YOUR.SERV.ER/privkey.pem;

    #FastCGI authorizer for Auth Request module
    location = /shibauthorizer {
        internal;
        include fastcgi_params;
        fastcgi_pass unix:/var/run/shibboleth/shibauthorizer.sock;
    }

    #FastCGI responder
    location /Shibboleth.sso {
        include fastcgi_params;
        fastcgi_pass unix:/var/run/shibboleth/shibresponder.sock;
    }

    #Resources for the Shibboleth error pages. This can be customised.
    location /shibboleth-sp {
        alias /usr/share/shibboleth/;
    }

    underscores_in_headers on;

    #A secured location, but only a specific sub-path causes Shibboleth
    #authentication.
    location / {
        proxy_pass https://jhub:8000;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $connection_upgrade;
	proxy_set_header Host $host; # or $http_host if port is to be proxied?
	proxy_set_header Origin "";  # ???

        location = /hub/login { # edit shibboleth2.xml accordingly to this...
            include shib_clear_headers;
            #Add your attributes here. They get introduced as headers
            #by the FastCGI authorizer so we must prevent spoofing.
            more_clear_input_headers 'displayName' 'email' 'persistent-id';
            # check further attributes and their capitalization which
            # can be extracted from the header obtained from your IDp:
            #more_clear_input_headers
            #    'Mail' 'Givenname' 'Eppn' 'Displayname' 'Affiliation' 'Sn' 'Ou'
            #    'Persistent-Id' 'Shib-Session-Id' 'Auth_type' 'Remote_user';

             shib_request /shibauthorizer;
             shib_request_use_headers on;
             proxy_pass https://jhub:8000;
         }
     }
}

Advanced setup:

docker image pull jwilder/docker-gen
mkdir -p /var/lib/docker-nginx/{conf.d,vhost.d}
curl -s https://github.com/nginx-proxy/nginx-proxy/commits/main/nginx.tmpl >/var/lib/docker-nginx/nginx.tmpl

split nginx_shibboleth.conf:

• rows from location = /shibauthorizer to underscore_in_headers: vhost.d/FQDN.YOUR.SERV.ER
• rows of group location = /hub/login: vhost.d/FQDN.YOUR.SERV.ER_local

link certificates in /etc/letsencrypt folder (assuming that only one certificate pair is requested for every CNAME or virtual host):

ln -s live/FQDN.YOUR.SERV.ER/fullchain.pem default.crt
ln -s live/FQDN.YOUR.SERV.ER/privkey.pem default.key

continue with the modification of docker-compose.yaml below...

Populate nginx/shibboleth_conf (it is another story) and edit shibboleth2.xml, attribute-map.xml, etc.

Get and patch docker-compose.yaml

patch docker-compose.yaml <<EOF
--- example-docker-compose.yaml	2022-01-20 22:30:25.795316503 +0100
+++ docker-compose.yaml	2022-01-20 22:30:40.019305291 +0100
@@ -3,8 +3,8 @@
 services:
 
     jhub:
-        image: gesiscss/jupyterhub-jsa:v0.8.1
-        deploy:
+        image: jupyterhub/jupyterhub
+        deploy: # considered only with `docker stack deploy`
           replicas: 1
           restart_policy:
               condition: on-failure
@@ -12,19 +12,21 @@
               limits:
                   cpus: "0.2"
                   memory: 512M
-        volumes:
-            - path/to/jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py
         restart: always
+        volumes:
+            - /var/lib/docker-jhub/jupyterhub:/srv/jupyterhub
+            - /etc/letsencrypt:/etc/letsencrypt
+            - JHUB_HOME:/home
         expose:
             - "8000"
         networks:
-            - webnet
+            - jhub-net
 
-    nginx-shibboleth:
+    nginx-shib:
         depends_on:
             - jhub
         image: gesiscss/nginx-shibboleth:v0.2.2
-        deploy:
+        deploy: # considered only with `docker stack deploy`
           replicas: 1
           restart_policy:
               condition: on-failure
@@ -34,33 +36,18 @@
                   memory: 512M
         volumes:
             # add Shibboleth configuration
-            - path/to/shibboleth:/etc/shibboleth
+            - /var/lib/docker-nginx/shibboleth_conf:/etc/shibboleth
             # add nginx configuration
-            - path/to/shibboleth_nginx.conf:/etc/nginx/conf.d/shibboleth_nginx.conf
-            # below here depends on your project requirements
-            # if you want to use shibboleth eds
-            - path/to/embedded_discovery_service:/home/shibboleth/embedded_discovery_service
+            - /var/lib/docker-nginx/nginx_shibboleth.conf:/etc/nginx/conf.d/nginx_shibboleth.conf
             # if you want to use letsencrypt to enable https
             - /etc/letsencrypt:/etc/letsencrypt
             - /etc/ssl:/etc/ssl
         ports:
-            - "80:80"
             - "443:443"
         restart: always
         command: /usr/bin/supervisord --nodaemon --configuration /etc/supervisor/supervisord.conf
         networks:
-            - webnet
-    visualizer:
-        image: dockersamples/visualizer:stable
-        ports:
-          - "8080:8080"
-        volumes:
-          - "/var/run/docker.sock:/var/run/docker.sock"
-        deploy:
-          placement:
-            constraints: [node.role == manager]
-        networks:
-          - webnet
+            - jhub-net
 networks:
-  webnet:
+    jhub-net:
EOF

Advanced setup (contd.):

Add VIRTUAL_* parameters to jhub service definition:

environment:
  - VIRTUAL_HOST=FQDN.YOUR.SERV.ER
  - VIRTUAL_PORT=8000
  - VIRTUAL_PROTO=https
  - CERT_NAME=default
  - DEBUG=true # if something went wrong

"Patch" nginx-shib service definition:

          image: gesiscss/nginx-shibboleth:v0.2.2
+         container_name: nginx-shib
[...]
-            - /var/lib/docker-nginx/nginx_shibboleth.conf:/etc/nginx/conf.d/nginx_shibboleth.conf
+            - /var/lib/docker-nginx/conf.d:/etc/nginx/conf.d
             # if you want to use letsencrypt to enable https
-            - /etc/letsencrypt:/etc/letsencrypt
+            - /etc/letsencrypt:/etc/nginx/certs:ro

Add nginx-gen service:

    nginx-gen:
      image: jwilder/docker-gen
      command: -notify-sighup nginx-shib -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
      container_name: nginx-gen
      restart: unless-stopped
      volumes:
        - nginx-conf:/etc/nginx/conf.d # :rw !!!
        - nginx-vhost:/etc/nginx/vhost.d
#        - /srv/www/nginx-proxy/html:/usr/share/nginx/html
        - /etc/letsencrypt:/etc/nginx/certs:ro
        - /var/run/docker.sock:/tmp/docker.sock:ro
        - /var/lib/docker-nginx/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
      networks:
        - jhub-net

Start containers:

docker-compose up -d
docker-compose logs -f --tail 50
# create /var/log:
#docker-compose logs -f jhub > /var/log/jhub.log &
# and create /etc/logrotate.d/jhub.conf ...

Upgrade packages

Execute commands in jupyterhub container:

docker-compose exec jhub bash
apt update
apt upgrade
apt install mc nano less # etc.
chmod o-rx /home # it is worth denying access to list /home
exit

Basic JupyterHub configuration

Generate basic jupyterhub_config.py:

docker-compose exec jhub bash
jupyterhub --generate-config # though this contains only commented lines...
# (change or) add the following lines to it:
cat >>jupyterhub_config.py <<EOF
c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/FQDN.YOUR.SERV.ER/fullchain.pem'
c.JupyterHub.ssl_key = '/etc/letsencrypt/live/FQDN.YOUR.SERV.ER/privkey.pem'

c.Authenticator.admin_users = {'UID_FROM_SHIBBOLETH_TO_BE_ADMIN'}

c.JupyterHub.authenticator_class = 'jhub_shibboleth_user_authenticator.shibboleth_user_auth.ShibbolethUserAuthenticator
# use a different cookie entry as the user name:
c.Authenticator.header_name = 'ATTRIBUTE_FROM_SHIBBOLETH'

# put some extra values in the auth_state for the spawner
# don't forget to activate c.Authenticator.enable_auth_state = True (??? IS THIS WORKING ???)
c.Authenticator.auth_state_header_names = [
     'Sn', 'Persistent-Id', 'Ou', 'Mail', 'Givenname', 'Eppn', 'Displayname',
     'Affiliation', 'Shib-Session-Id', 'Auth_type', 'Remote_user'
]
EOF
adduser -q --gecos "" --disabled-password UID_FROM_SHIBBOLETH_TO_BE_ADMIN
pip install jhub-shibboleth-user-authenticator==0.1.6
exit

If you want to test your setup and login to your freshly configure JupyterHub server, you need to install notebook (which is otherwise installed automatically with nbgrader):

docker-compose exec jhub pip install notebook

Now, restart jhub container and you can try to login with Shibboleth authentication. (If not using Shibboleth, the last 10 line can be dropped, and you can use the default PAM authenticator, or an OAUTH2 method...)

docker-compose restart jhub

Nbgrader setup

Install nbgrader in editable mode (if the source needs to be patched):

docker-compose exec jhub bash
apt install git
cd /usr/local
pip install -e git+https://github.com/jupyter/nbgrader#egg=nbgrader
jupyter-nbextension install nbgrader --system --py --overwrite
jupyter-nbextension enable nbgrader --system --py
jupyter-serverextension enable nbgrader --system --py

Disable formgrader and course_list modules for regular users (students):

for m in formgrader course_list; do
    jupyter-nbextension disable $m/main --system --section=tree
    jupyter-serverextension disable nbgrader.server_extensions.$m --system
done
# to disable nbgrader cell-toolbar for regular users (aka. students):
jupyter-nbextension disable create_assignment/main --system
#
mkdir -p /srv/nbgrader/exchange
chmod a+rw /srv/nbgrader/exchange
ln -s /srv/jupyterhub /etc/jupyter

Create global nbgrader_config.py:

cat <<EOF >/etc/jupyter/nbgrader_config.py
from nbgrader.auth import JupyterHubAuthPlugin
c = get_config()
c.Exchange.path_includes_course = True
c.Authenticator.plugin_class = JupyterHubAuthPlugin
EOF
exit # from docker container

Create and setup courses

Add service with necessary role privileges to jupyterhub_config.py:

jhub_api_token = 'output of `openssl rand -hex 32`'
c.JupyterHub.services = [
    { 'name': 'formgrader-service',
      'api_token': jhub_api_token
    },
]
# spawner needs JHUB_API_TOKEN for querying students' groups
# "required to run the exchange features of nbgrader"
c.Spawner.environment = {
    'JHUB_API_TOKEN': jhub_api_token,
}
c.JupyterHub.load_roles = [
    { # this is for auth API
        'name': 'formgrade-role',
        'scopes': [ 'read:users:groups', 'list:services', 'groups', 'admin:users' ],
        'services': [ 'formgrader-service' ]
    }
]
c.JupyterHub.load_groups = {
# populate with formgrade-* and nbgrader-* groups (see make courses below)
}

Remark: 'admin:users' in scopes is needed only if command line API access (for administration of users and groups) uses the same API token. This can be a security issue, hence, creating a separate service / role / API_token is recommended for this purpose.

Add this to jupyterhub/Makefile:

grader-%:
	adduser -q --gecos "" --disabled-password $@
	su - $@ -c "chmod go-rwx ."
	@c_id=`echo $* |tr [a-z] [A-Z]`; \
	cdir="$$(grep $@ /etc/passwd |cut -f6 -d:)/OPTIONAL_PREFIX-$$c_id"; \
	nbconf="c=get_config\(\)\|\
c.CourseDirectory.root = \'$$cdir\'\|\
c.CourseDirectory.course_id = \'$$c_id\'";\
	su - $@ -c "if ! [ -e .jupyter ]; then mkdir .jupyter; fi; \
echo $$nbconf |tr '|' '\n' >.jupyter/nbgrader_config.py; \
nbgrader quickstart $$cdir && rm -rf $$cdir/source/ps1; \
sed -ri '/course_id/s/^/#/' $$cdir/nbgrader_config.py"
	su - $@ -c "jupyter-nbextension enable formgrader/main --user --section=tree"
	su - $@ -c "jupyter-serverextension enable nbgrader.server_extensions.formgrader --user"
	su - $@ -c "jupyter-nbextension enable create_assignment/main --user"
	su - $@ -c "jupyter-nbextension disable assignment_list/main --user --section=tree"
	su - $@ -c "jupyter-serverextension disable nbgrader.server_extensions.assignment_list --user"

gr-%: grader-%
	@c_id=`echo $* |tr [a-z] [A-Z]`; \
	echo -e "Add this snippet to the appropriate sections in jupyterhub_config.py (replace UNIQUE_PORT!):\\n\
--- 8< ---\\n\
#c.JupyterHub.service = [\\n\
    { 'name': '$$c_id',\\n\
      'url': 'http://127.0.0.1:UNIQUE_PORT',\\n\
      'command': [\\n\
        'jupyterhub-singleuser',\\n\
        '--group=formgrade-$$c_id',\\n\
        '--debug'\\n\
      ],\\n\
      'environment': { 'JHUB_API_TOKEN': jhub_api_token },\\n\
      'user': '$<',\\n\
      'cwd': '/home/$<',\\n\
      #'api_token': '$$(openssl rand -hex 16)'\\n\
    },\\n\
#]\\n\
#c.JupyterHub.load_groups = {\\n\
    'formgrade-$$c_id': [ '$<' ],\\n\
    'nbgrader-$$c_id': [],\\n\
#}\\n\
#c.JupyterHub.load_roles = [\\n\
    { 'name':   'role-$*',\\n\
      'scopes': [ 'access:services!service=$$c_id' ],\\n\
      'groups': [ 'formgrade-$$c_id' ]\\n\
    },\\n\
#]\\n\
--- >8 ---"

Now, execute the command

docker-compose exec jhub make gr-NEW_COURSE_ID

and patch jupyterhub_config.py as suggested.

Create and setup tutors (instructors)

Assuming a TAB separated tutor_ids.txt file with first column as id and the second one as the list of COURSE_IDs which the tutor of the given id should be member of, run this script or add similar lines to a Makefile:

docker-compose exec jhub bash
for id in `cut -f1 tutor_ids.txt |tr [A-Z] [a-z]`; do
    adduser -q --gecos "" --firstuid 2000 --gid 2000 --disabled-password $id
    su - $id -c "chmod go-rwx ."
    su - $id -c "jupyter-nbextension enable course_list/main --user --section=tree"
    su - $id -c "jupyter-serverextension enable nbgrader.server_extensions.course_list --user"
    # the following three lines are optional:
    su - $id -c "jupyter-nbextension enable create_assignment/main --user"
    su - $id -c "jupyter-nbextension disable assignment_list/main --user --section=tree"
    su - $id -c "jupyter-serverextension disable nbgrader.server_extensions.assignment_list --user"
done
python3 api_request.py users add `cut -f1 tutor_ids.txt`
# see api_request.py below...

After generating a reverse map from tutor_ids.txt as course_graders.txt with first column as COURSE_ID and second column as the list of graders of that course, the courses can be populated with their graders:

cat course_graders.txt |while read COURSE_ID GRADER_LIST; do
    python3 api_request.py groups add formgrade-${COURSE_ID} ${GRADER_LIST}`
done
exit

The jupyterhub/api_request.py for user / group management from command line:

import sys

def query( q, m="GET", d="" ):
  api_url = 'http://127.0.0.1:8081/hub/api'
  url = api_url + q
  h = { 'accept': 'application/json',
	'Authorization': 'token YOUR_jhub_api_token_FROM_jupyterhub_config.py'
  }
  print( url + " %s:" % m )
  if m == "GET":
     return requests.get( url, headers=h )
  elif m == "POST":
     print(d)
     return requests.post( url, headers=h, json=d )
  elif m == "DELETE":
     print(d)
     return requests.delete( url, headers=h, json=d )

input= sys.argv
argc = len(input)

if argc < 2:
   print( "Usage: " + input[0] + " {users|groups} [add|remove] \$ug_list" )

ug = input[1] # users or groups
if argc == 2: # just listing the users / groups existing in jupyterhub
   q = query( '/'+ug )
   gu= 'groups'
   if ug == gu: gu = 'users'
   print(q.json())
   for e in q.json():
       print( "%s: " % e.get('name'), end="" )
       print( e.get(gu) )
elif input[2] == "add" or input[2] == "remove":
   input.pop(0)
   ug = input.pop(0)
   gu= 'usernames'
   if ug == 'groups': gu = 'users'
   m = input.pop(0)
   if m == "add": m = "POST"
   else: m = "DELETE" # remove
   if ug == 'groups':
      name = input.pop(0)
      # e.g. /groups/formgrade-${COURSE_ID}/users POST { 'users': ${TUTOR_IDS} }
      q = query( '/'+ug+'/'+name+'/'+gu, m, { gu: input } )
   elif ug == 'users' and m == 'POST':
      # e.g. /users POST { 'usernames': ${TUTOR_IDS} }
      q = query( '/'+ug, m, { gu: input } )
   else:
      print("Unknown combination!")
   print( q.json() )
else:
   print("Unknown method!")

Create and setup students

Assuming a students.csv in the form and with appropriate header (only the first column is necessary) as follows

id	last_name	first_name	email	lms_user_id

Create students' jupyterhub accounts and register theirs IDs in jupyterhub database:

docker-compose exec jhub bash
tail -n +2 students.csv |cut -f1 -d, |tr -d '"' |tr [A-Z] [a-z] |while read id; do
    adduser -q --gecos "" --firstuid 3000 --gid 3000 --disabled-password $id
    su - $id -c "chmod go-rwx ."
    echo -n "."
done
python3 api_request.py users add `tail -n +2 students.csv |cut -f1 -d, |tr -d '"'`

Now, if students.csv contains students of a course of certain COURSE_ID only, they can be imported:

su - grader-COURSE_ID -c "nbgrader db student import students.csv"
exit

Setup jupyterhub-idle-culler service

Source: https://github.com/jupyterhub/jupyterhub-idle-culler

Install:

docker-compose exec jhub pip install jupyterhub-idle-culler

Modify jupyterhub_config.py:

#c.JupyterHub.services = [
#insert the following lines in this section (services):
    { 'name': 'culler-service',
      'command': [
            'jupyterhub-idle-culler', '--timeout=21600', '--url=http://127.0.0.1:8081/hub/api'
      ],
      'api_token': 'output of `openssl rand -hex 16`, e.g.' # if wanted to use its role through API requests
    },
#...
#]
#c.JupyterHub.load_roles = [
#insert the following lines in this section (load_roles):
    {   'name':   'idle-culler-role',
        'scopes': [
            'list:users', 'read:users:activity',
            'read:servers', 'delete:servers',
            'read:users:groups', 'admin:users' # needed for api_request.py, e.g.
        ],
        'services': [ 'culler-service' ]
    },
    { # now, scopes of formgrade-role can be narrowed:
        'name':     'formgrade-role',
        'scopes':   [ 'read:users:groups', 'list:services', 'groups' ],
        'services': [ 'formgrader-service' ]
    },
#]