DMTCP

Guidance last updated: 3 November 2026

DMTCP (Distributed MultiThreaded CheckPointing) saves a running program’s state so it can be restarted without adding checkpoint code to the program. For applications with their own restart feature, see Checkpointing and restarting jobs.

The example below is for a serial CPU program. MPI, GPU and applications that connect to external services need application-specific checkpoint support.

Usage

After connecting to Stanage (see Establishing a SSH connection), start an interactive session:

salloc

You can load a specific version with the following command:

module load DMTCP/4.2.0-GCCcore-15.2.0
module load DMTCP/4.1.0-GCCcore-14.2.0
module load DMTCP/4.0.0-GCCcore-14.3.0
module load DMTCP/3.0.0-GCCcore-13.2.0
module load DMTCP/3.0.0-GCCcore-12.3.0
module load DMTCP/3.0.0-GCCcore-12.2.0
module load DMTCP/3.0.0-GCCcore-11.3.0

Batch example

Save the following as counter.py. It adds the numbers 1 to 10, taking about 10 seconds:

import time

total = 0
for step in range(1, 11):
    total += step
    print(f"Step {step}: {total}", flush=True)
    time.sleep(1)
print(f"Total: {total}", flush=True)

Save this example batch submission script as checkpoint_job.sh.

Note

The two-second checkpoint interval is only for this short demonstration. For a real workload, 300 seconds (five minutes) may be more reasonable, depending on how long checkpoints take to write and how much work you can afford to repeat.

#!/bin/bash
#SBATCH --job-name=dmtcp_example
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2000
#SBATCH --time=00:01:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module load DMTCP/4.2.0-GCCcore-15.2.0
mkdir -p checkpoints

dmtcp_launch --new-coordinator --coord-port 0 \
    --ckptdir "$PWD/checkpoints" --interval 2 --no-gzip \
    /usr/bin/python3 counter.py > counter.log 2>&1

Submit with sbatch checkpoint_job.sh. The program finishes normally and counter.log ends with Total: 55, leaving checkpoints for the restart example.

Restarting from a checkpoint

Once the first job has finished, save this example batch submission script as restart_job.sh:

#!/bin/bash
#SBATCH --job-name=dmtcp_restart
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2000
#SBATCH --time=00:01:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -e
module load DMTCP/4.2.0-GCCcore-15.2.0

dmtcp_restart --new-coordinator --coord-port 0 \
    --ckptdir "$PWD/checkpoints" --interval 2 \
    checkpoints/*.dmtcp > counter-restarted.log 2>&1

Submit with sbatch restart_job.sh from the same directory. It resumes the saved counter and running total, repeating the final few steps and finishing with Total: 55 in counter-restarted.log.

For a real calculation, replace the Python command, resource requests and checkpoint interval with suitable values. Keep all files from the latest complete checkpoint together in a directory used only for that calculation.

Installation notes

DMTCP was installed using Easybuild, build details can be found in $EBROOTDMTCP/easybuild with a given module loaded.

Further information

See the DMTCP quick start guide.