The j parameter

[s-canchi]

What is the -j flag ? Do we always need to specify the -j parameter?

[s-canchi]

The snakemake syntax --cores, --jobs, -j are used to specify the number of cores available to run the job. You can specify all as the argument to indicate the use of all available CPU cores. However, a word of caution on using this option when operating on a shared resource such as a HPC system. To find out how many cores are available to you, you can use lscpu to list the configuration. In the recent versions of Snakemake (v6.0+), it is necessary to specify the -j parameter when running snakemake. Find more about the different snakemake parameters in their documentation.

[s-canchi]

Will the number of cores used always be limited by the -j flag? What if a rule was programmed to use more (e.g. bwa mem -t 16)?

[ctb]

if the rule is setup properly (so that it's bwa mem -t {threads} and not bwa mem -t 16) then snakemake will adjust properly. But snakemake has no insight into what a shell command does unless it is invited to give parameters to that shell command :).

The consequences are minimal on dedicated (non-shared) systems, where using too many threads (e.g. more threads than CPUs) will at worst just slow down the command and maybe consume more memory than otherwise.

On shared systems like HPCs, if you use more threads than you allocated (e.g. on SLURM systems, if you ask for -c 4 in srun/sbatch and then run snakemake with -j 16), your job may be killed, just as will happen if you request too little memory.