-
Notifications
You must be signed in to change notification settings - Fork 21
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
10 changed files
with
202 additions
and
51 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
#!/bin/bash | ||
#! | ||
#! Example SLURM job script for Wilkes3 (AMD EPYC 7763, ConnectX-6, A100) | ||
#! Last updated: Fri 30 Jul 11:07:58 BST 2021 | ||
#! | ||
|
||
#!############################################################# | ||
#!#### Modify the options in this section as appropriate ###### | ||
#!############################################################# | ||
|
||
#! sbatch directives begin here ############################### | ||
#! Name of the job: | ||
#SBATCH -J gpujob | ||
#! Which project should be charged (NB Wilkes2 projects end in '-GPU'): | ||
#SBATCH -A taylor-sl3-gpu | ||
#! How many whole nodes should be allocated? | ||
#SBATCH --nodes=1 | ||
#! How many (MPI) tasks will there be in total? | ||
#! Note probably this should not exceed the total number of GPUs in use. | ||
#SBATCH --ntasks=1 | ||
#! Specify the number of GPUs per node (between 1 and 4; must be 4 if nodes>1). | ||
#! Note that the job submission script will enforce no more than 32 cpus per GPU. | ||
#SBATCH --gres=gpu:1 | ||
#! How much wallclock time will be required? | ||
#SBATCH --time=02:00:00 | ||
#! What types of email messages do you wish to receive? | ||
#SBATCH --mail-type=NONE | ||
#! Uncomment this to prevent the job from being requeued (e.g. if | ||
#! interrupted by node failure or system downtime): | ||
##SBATCH --no-requeue | ||
|
||
#! Do not change: | ||
#SBATCH -p ampere | ||
|
||
#! sbatch directives end here (put any additional directives above this line) | ||
|
||
#! Notes: | ||
#! Charging is determined by GPU number*walltime. | ||
|
||
#! Number of nodes and tasks per node allocated by SLURM (do not change): | ||
numnodes=$SLURM_JOB_NUM_NODES | ||
numtasks=$SLURM_NTASKS | ||
mpi_tasks_per_node=$(echo "$SLURM_TASKS_PER_NODE" | sed -e 's/^\([0-9][0-9]*\).*$/\1/') | ||
#! ############################################################ | ||
#! Modify the settings below to specify the application's environment, location | ||
#! and launch method: | ||
|
||
#! Optionally modify the environment seen by the application | ||
#! (note that SLURM reproduces the environment at submission irrespective of ~/.bashrc): | ||
. /etc/profile.d/modules.sh # Leave this line (enables the module command) | ||
module purge # Removes all modules still loaded | ||
module load rhel8/default-amp # REQUIRED - loads the basic environment | ||
|
||
#! Insert additional module load commands after this line if needed: | ||
|
||
#! Full path to application executable: | ||
#! TODO - workout how to make version get passed to this script | ||
application="/home/js2430/rds/hpc-work/OceanBioME-runner/_work/_temp/julia-1.9.3/bin/julia -O0 --color=yes --project test/gpu_runtests.jl" | ||
|
||
#! Run options for the application: | ||
options="" | ||
|
||
#! Work directory (i.e. where the job will run): | ||
workdir="$SLURM_SUBMIT_DIR" # The value of SLURM_SUBMIT_DIR sets workdir to the directory | ||
# in which sbatch is run. | ||
|
||
#! Are you using OpenMP (NB this is unrelated to OpenMPI)? If so increase this | ||
#! safe value to no more than 128: | ||
export OMP_NUM_THREADS=1 | ||
|
||
#! Number of MPI tasks to be started by the application per node and in total (do not change): | ||
np=$[${numnodes}*${mpi_tasks_per_node}] | ||
|
||
#! Choose this for a pure shared-memory OpenMP parallel program on a single node: | ||
#! (OMP_NUM_THREADS threads will be created): | ||
CMD="$application $options" | ||
|
||
|
||
############################################################### | ||
### You should not have to change anything below this line #### | ||
############################################################### | ||
|
||
cd $workdir | ||
echo -e "Changed directory to `pwd`.\n" | ||
|
||
JOBID=$SLURM_JOB_ID | ||
|
||
echo -e "JobID: $JOBID\n======" | ||
echo "Time: `date`" | ||
echo "Running on master node: `hostname`" | ||
echo "Current directory: `pwd`" | ||
|
||
if [ "$SLURM_JOB_NODELIST" ]; then | ||
#! Create a machine file: | ||
export NODEFILE=`generate_pbs_nodefile` | ||
cat $NODEFILE | uniq > machine.file.$JOBID | ||
echo -e "\nNodes allocated:\n================" | ||
echo `cat machine.file.$JOBID | sed -e 's/\..*$//g'` | ||
fi | ||
|
||
echo -e "\nnumtasks=$numtasks, numnodes=$numnodes, mpi_tasks_per_node=$mpi_tasks_per_node (OMP_NUM_THREADS=$OMP_NUM_THREADS)" | ||
|
||
echo -e "\nExecuting command:\n==================\n$CMD\n" | ||
|
||
eval $CMD |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
name: CSD3 tests | ||
|
||
on: | ||
push: | ||
branches: | ||
- main | ||
pull_request: | ||
branches: | ||
- main | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref }} | ||
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} | ||
|
||
env: | ||
DATADEPS_ALWAYS_ACCEPT: true | ||
JULIA_VERSION: "1.9.3" | ||
JULIA_MINOR_VERSION: "1.9" | ||
RDS_HOME: "/home/js2430/rds/hpc-work/OceanBioME-runner/_work" | ||
|
||
jobs: | ||
test: | ||
name: Deploy CSD3 tests | ||
runs-on: self-hosted | ||
steps: | ||
- label: "initialize gpu enviroment" | ||
commands: | ||
- "wget -N -P $CSD3_HOME/_temp https://julialang-s3.julialang.org/bin/linux/x64/$JULIA_MINOR_VERSION/julia-$JULIA_VERSION-linux-x86_64.tar.gz" | ||
- "tar xf $CSD3_HOME/_temp/julia-$JULIA_VERSION-linux-x86_64.tar.gz -C $CSD3_HOME/_temp" | ||
|
||
# Run the tests on slurm | ||
- "sbatch .github/workflows/gpu-test.sbatch" | ||
|
||
# Clean up | ||
- "rm -rf $CSD3_HOME/_temp/julia-$JULIA_VERSION" # presumably this happens anyway |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
using Pkg | ||
|
||
Pkg.instantiate() | ||
|
||
Pkg.add("Test", "CUDA", "DataDeps", "Documenter", "Statistics", "JLD2") | ||
|
||
Pkg.precompile() | ||
|
||
using Oceananigans | ||
|
||
arch = GPU() | ||
|
||
include("runtests.jl") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters