This is an example project to show the setup of python and singularity-based experimental research. The aim is to provide a generalised research-oriented framework/workflow that supports reproducible science. The basic idea is to decouple:
- Development environment (Singularity)
- Experimental code (Python module)
- Path references (based on this)
- Data storage (storage folder)
- Generated outputs (output folder)
- HPC scripting (hpc folder)
Production build (for deploying to system through SHUB)
git clone https://github.com/davidhin/singularity-example.git
cd singularity-example
singularity pull --name main.simg shub://davidhin/singularity-example:latest
singularity run main.simg -p initialise
Then, if on Phoenix, run:
sbatch hpc/run_notebook.sh
-e means editable, which is what we want when installing the package locally for development purposes.
bash get_data.sh
pip install -r requirements.txt
pip install -e .
Local build for development
sudo singularity build --sandbox main.img Singularity;
singularity run main.img -p main;
Production build (for deploying to phoenix through SCP)
sudo singularity build main.simg Singularity;
scp ./main.simg [email protected]:/hpcfs/users/aXXXXXXX/main.simg
jupyter notebook --generate-config -y
echo "c.NotebookApp.allow_remote_access = True" >> /root/.jupyter/jupyter_notebook_config.py