Batch submission example templates

See Style Guide and the batch submission guide.

Introduce scripts as example batch submission scripts. Replace placeholder programs and example resources with verified application commands and requests.

Directive order

Use long --option=value options, one per line, before shell commands. Keep the application’s launch commands and resource settings when adapting a header. Follow this order, omitting options that do not apply:

Group	Directive order
Job name	`--job-name`
Placement	`--account`, `--partition`, `--qos`
Resources	`--nodes`, `--ntasks` or `--ntasks-per-node`, `--cpus-per-task`, `--gres`
Limits	`--mem-per-cpu` (CPU) or `--mem` (GPU), then `--time`
Workflow	`--array`, `--dependency`
Logs	`--output`, `--error`
Notifications	`--mail-user`, `--mail-type`

Complete scripts should specify a job name, resources, time and output. Keep teaching fragments brief; email and specialised options are optional.

Use numeric memory values without a suffix. For CPU jobs, use --mem-per-cpu, for example --mem-per-cpu=2000. Specify --nodes only when the workload needs a particular node count; otherwise let Slurm place the requested tasks. For GPU jobs, use --mem for host RAM per node, following the GPU memory guidance: request slightly more host RAM than the combined device memory of the requested GPUs.

Job names and log filenames

Use short names such as castep-graphite, preferably lower case with hyphens. Default to combined output and errors; split them when useful.

Job	Combined log	Optional separate errors
Ordinary	`--output=%x-%j.out`	`--error=%x-%j.err`
Array	`--output=%x-%A_%a.out`	`--error=%x-%A_%a.err`

%x is the job name, %j the job ID, %A the array job ID and %a the array index. For example: castep-graphite-123456.out. Use Slurm tokens in directives and shell variables such as $SLURM_JOB_ID in commands; shell variables do not expand inside directives.

Keep logs in the submission directory. If using a logs/ prefix, create it before submission:

mkdir -p logs
sbatch example.sh

Keep application results separate from Slurm logs. A nonempty .err file does not itself indicate failure. Requeued jobs may retain their job ID, so restart logging needs separate consideration.

Template layouts

CPU templates use Stanage’s default partition. Load the required modules before the application command. Resource values below are illustrative.

Serial CPU

One task uses one CPU.

#!/bin/bash
#SBATCH --job-name=serial-example
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

srun ./my_serial_program

SMP and OpenMP

One OpenMP process uses four CPUs on one node.

#!/bin/bash
#SBATCH --job-name=openmp-example
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=2000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun ./my_openmp_program

MPI tasks

Four MPI tasks use one CPU each; Slurm may place them on more than one node. Load the MPI library used to build the program.

#!/bin/bash
#SBATCH --job-name=mpi-example
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

export OMP_NUM_THREADS=1
srun --export=ALL ./my_mpi_program

MPI across nodes

Two nodes each run four MPI tasks, with one CPU per task: eight tasks in total.

#!/bin/bash
#SBATCH --job-name=mpi-multinode-example
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

export OMP_NUM_THREADS=1
srun --export=ALL ./my_mpi_program

Hybrid MPI and OpenMP

Two nodes, two MPI tasks per node and four CPUs per task: 16 CPUs in total.

#!/bin/bash
#SBATCH --job-name=hybrid-example
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=2000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
srun --export=ALL ./my_hybrid_program

One GPU

One task uses an A100 and eight CPUs; see Using GPUs on Stanage.

#!/bin/bash
#SBATCH --job-name=gpu-example
#SBATCH --partition=gpu
#SBATCH --qos=gpu
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:a100:1
#SBATCH --mem=82000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

srun ./my_gpu_program

Several GPUs on one node

Resource header for four tasks and four A100 GPUs. Add the application’s launcher and rank-to-GPU mapping. A single process using all GPUs needs a one-task layout. Stanage’s public GPU partition currently permits only single-node allocations.

#!/bin/bash
#SBATCH --job-name=gpu-multirank-example
#SBATCH --partition=gpu
#SBATCH --qos=gpu
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:a100:4
#SBATCH --mem=328000
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out

# Add the application's launcher and rank-to-GPU mapping.

Job arrays

Ten serial jobs, with at most four running concurrently. Resources are per element.

#!/bin/bash
#SBATCH --job-name=array-example
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4000
#SBATCH --time=00:10:00
#SBATCH --array=0-9%4
#SBATCH --output=%x-%A_%a.out

srun ./my_serial_program "inputs/input-${SLURM_ARRAY_TASK_ID}.dat"

Arrays can also use SMP, MPI or GPU layouts. Keep the relevant resource and launch settings, add --array, and use the array log pattern and input index.

See the upstream sbatch reference, job array guide and MPI guide for Slurm semantics.