Happy Holidays

Wishing everyone a peaceful break and fewer broken jobs.

Advanced Job Submission and Control

Tip

If you are not familiar with basic job submission and control you should first read our Job Submission and Control page.

Introduction

This page details more advanced job submission and control methods that can be used with the SLURM scheduler on Stanage.

---

Advanced Job Submission Methods

In this section, the concept of each advanced submission method will be described with subsequent explanations of how to implement these on each scheduler.

---

Job or task arrays

An “array job” is a set of tasks run from a single batch job submission script. Each of these tasks should consume a relatively large amount of compute time to maintain optimum efficiency. If each task is short (seconds or even a few minutes), array jobs will saturate the scheduler and more time is spent managing jobs than running your jobs. This will also negatively impacts other users!

Advantages of array jobs:

• You only need to submit one job to run a series of very similar tasks;

• These tasks are independent and do not all need to run at once so the job scheduler can efficiently run one or more queued tasks as the requested computational resources become available;

• They are particularly useful for embarrassingly parallel problems such as:

  • Monte Carlo simulations;

  • Parameter sensitivity analysis;

  • Batch file processing.

Disadvantages of array jobs:

• If a single task fails to run correctly it can be a pain to determine and re-submit failed tasks.

• If the tasks are small, the scheduler will spend more time managing and queueing your tasks than computing them.

See also

See our tutorial with worked examples using job arrays for embarrassingly parallel tasks.

Warning

Array jobs using SLURM can have a maximum of 1000 tasks.

SLURM job arrays are only supported for batch jobs and the array index values are specified using the --array or -a option of the sbatch command as follows:

# Submit a job array with index values between 0 and 31
$ sbatch --array=0-31

# Submit a job array with index values of 1, 3, 5 and 7
$ sbatch --array=1,3,5,7

# Submit a job array with index values between 1 and 7
# with a step size of 2 (i.e. 1, 3, 5 and 7)
$ sbatch --array=1-7:2

Job arrays will have two additional environment variable set. SLURM_ARRAY_JOB_ID will be set to the first job ID of the array. SLURM_ARRAY_TASK_ID will be set to the job array index value. SLURM_ARRAY_TASK_COUNT will be set to the number of tasks in the job array. SLURM_ARRAY_TASK_MAX will be set to the highest job array index value. SLURM_ARRAY_TASK_MIN will be set to the lowest job array index value.

For example a job submission of this sort:

$ sbatch --array=1-3
Submitted batch job 39319

Will generate a job array containing three jobs with the environment variables set as follows:

SLURM_JOB_ID=39319
SLURM_ARRAY_JOB_ID=39319
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=39320
SLURM_ARRAY_JOB_ID=39319
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=39321
SLURM_ARRAY_JOB_ID=39319
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

All SLURM commands and APIs recognize the SLURM_JOB_ID value. Most commands also recognize the SLURM_ARRAY_JOB_ID plus SLURM_ARRAY_TASK_ID values separated by an underscore as identifying an element of a job array. Using the example above, “39320” or “39319_1” would be equivalent ways to identify the second array element of job 39319 as shown in the following sacct command example.

 $ sacct -j 39319_1
 JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
 ------------ ---------- ---------- ---------- ---------- ---------- --------
 39319_1      array.sh  sheffield       free          1  COMPLETED      0:0
 39319_1.b+      array                  free          1  COMPLETED      0:0

 $ sacct -j 39320
 JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
 ------------ ---------- ---------- ---------- ---------- ---------- --------
 39319_1      array.sh  sheffield       free          1  COMPLETED      0:0
 39319_1.b+      array                  free          1  COMPLETED      0:0

Note that the parent job runs the final task. In the following sacct command example using SLURM_ARRAY_JOB_ID (39319 i.e spawning job) will retrieve details of the whole task array:

 $ sacct -j 39319
 JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
 ------------ ---------- ---------- ---------- ---------- ---------- --------
 39319_1      array.sh  sheffield       free          1  COMPLETED      0:0
 39319_1.b+      array                  free          1  COMPLETED      0:0
 39319_2      array.sh  sheffield       free          1  COMPLETED      0:0
 39319_2.b+      array                  free          1  COMPLETED      0:0
 39319_3      array.sh  sheffield       free          1  COMPLETED      0:0
 39319_3.b+      array                  free          1  COMPLETED      0:0

The same output could have been achieved with sacct -j 39319_3 .

Using email notifications

By default in SLURM, the emails for events BEGIN, END and FAIL apply to the job array as a whole rather than individual tasks. So:

#SBATCH --mail-type=BEGIN,END,FAIL

Would result in one email per job, not per task. If you want per task emails, specify:

#SBATCH --mail-type=BEGIN,END,FAIL,ARRAY_TASKS

Which will send emails for each task in the array.

Managing output and error files

SLURM uses the %A and %a replacement strings for the master job ID and task ID, respectively.

For example:

#SBATCH --output=Array_test.%A_%a.out
#SBATCH --error=Array_test.%A_%a.error

The error log is optional as both types of logs can be written to the ‘output’ log.

#SBATCH --output=Array_test.%A_%a.log

Warning

If you only use %A in the log all array tasks will try to write to a single file. The performance of the run will approach zero asymptotically. Make sure to use both %A and %a in the log file name specification.

Grouping tasks for efficiency

Limiting number of concurrent tasks

A maximum number of simultaneously running tasks from the job array may be specified using a % separator. For example --array=0-15%4 will limit the number of simultaneously running tasks from this job array to 4.

---

Dependent jobs

Dependent jobs, or jobs submitted with dependencies on other jobs will wait until the job they are dependent on has met with a certain conditon. This can allow you to build workflows with pre-processing or post-processing steps.

Job dependencies with the SLURM scheduler are specified with the --dependency option to sbatch using job IDs only in the format:

$ sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> ...

The different dependency types available are:

after:jobid[:jobid...]        #job can begin after the specified jobs have started
afterany:jobid[:jobid...]     #job can begin after the specified jobs have terminated
afternotok:jobid[:jobid...]   #job can begin after the specified jobs have failed
afterok:jobid[:jobid...]      #job can begin after the specified jobs have run to completion with an exit code of zero (see the user guide for caveats).
singleton                     #jobs can begin execution after all previously launched jobs with the same name and user have ended. This is useful to collate results of a swarm or to send a notification at the end of a swarm.

The most simple way to use dependencies are to use the afterany type for single consecutive jobs e.g. :

$ sbatch job1.sh
Submitted batch job 12345678
$ sbatch --dependency=afterany:12345678 job2.sh

In this case when job 1 finishes (terminates), job 2 will become eligible for scheduling. This means even if job 1 fails, job 2 will run.

A further example with more complicated conditions is shown below:

#! /bin/bash

# first job - no dependencies
cmd1=$(sbatch --parsable --mem=12g --cpus-per-task=4 job1.sh)
jid1=$( $cmd1 | cut -d ";" -f 1)

# multiple jobs can depend on a single job
cmd2=$(sbatch --parsable --dependency=afterany:$jid1 --mem=20g job2.sh)
jid2=$( $cmd2 | cut -d ";" -f 1)

cmd3=$(sbatch --parsable  --dependency=afterany:$jid1 --mem=20g job3.sh)
jid3=$( $cmd3 | cut -d ";" -f 1)


# a single job can depend on multiple jobs
cmd4=$(sbatch --parsable  --dependency=afterany:$jid2:$jid3 job4.sh)
jid4=$( $cmd4 | cut -d ";" -f 1)


# a single job can depend on all jobs by the same user with the same name
cmd5=$(sbatch --parsable --dependency=afterany:$jid4 --job-name=dtest job5.sh)
jid5=$( $cmd5 | cut -d ";" -f 1)

cmd6=$(sbatch --parsable --dependency=afterany:$jid5 --job-name=dtest job6.sh)
jid6=$( $cmd6 | cut -d ";" -f 1)


sbatch --dependency=singleton --job-name=dtest job9.sh

---

Timed start jobs

Jobs can be submitted to the scheduler to run at a specific time. This section explains how to achieve this with SLURM.

Timed start jobs using the Slurm scheduler are requested with the --begin argument in the following formats:

sbatch --begin=16:00 job.sh
sbatch --begin=now+60 job.sh #(seconds by default)
sbatch --begin=now+1hour job.sh
sbatch --begin=2023-06-30T12:34:00 job.sh

The scheduler will immediately submit these jobs but will wait until the elected date/time has passed before starting them.