Attention

SERVICE RISK: The Bessemer HPC cluster will be at risk from 9am on July 15th until 5pm on July 19th 2024

For further detail, please click this link.

Quick Reference (Cheat Sheets)

Note

These cheat sheets are a quick reference guide to be used once you are familiar with our documentation.

HPC Cluster Cheat Sheets

This cheat sheet covers basic functionality for our clusters.

CONNECTING TO THE CLUSTER AND TRANSFERRING FILES

ssh -X YOUR_USERNAME@stanage.shef.ac.uk

Connect - Link

srun –pty bash -i

Start an interactive session - Link

srun –partition=gpu –qos=gpu –gres=gpu:1 –pty bash

Start an interactive GPU session - Link

scp /path/to/file.txt YOUR_USERNAME@stanage.shef.ac.uk:/path/to/directory/

Upload - Link

scp YOUR_USERNAME@stanage.shef.ac.uk:/path/to/file.txt /path/to/directory/

Download file - Link

scp -r YOUR_USERNAME@stanage.shef.ac.uk:/path/to/my_results /path/to/directory/

Download directory - Link

rsync -avzP /path/to/directory/ YOUR_USERNAME@stanage.shef.ac.uk:/path/to/directory/

Sync/transfer directory - Link

wget https://software.github.io/program/files/myprogram.tar.gz

Download direct from website - Link

curl -O https://software.github.io/program/files/myprogram.tar.gz

Download direct from website - Link

BATCH JOB SUBMISSION, MONITORING AND CONTROL

sbatch MY_SCRIPT.sh

Submit a batch job - Link

squeue -u $USER

Investigate jobs in queue (Running R and Pending PD) - Link

sstat -j 1234567

Investigate running job - Link

sacct -j 1234567

Investigate historical job - Link

scancel 1234567

Cancel a job - Link

scontrol <action> 1234567

Control a job (hold/release) - Link

salloc

Allocate resources to an interactive job - Link

srun

Start a task inside a job - Link

PARTITION INFORMATION

sinfo

Node and partition information - Link

General CPU nodes

256GB Memory/node; 64 cores/node; 96 hrs

Large Mem CPU nodes

1TB Memory/node; 64 cores/node; 96 hrs

V Large Mem CPU nodes

2TB Memory/node; 64 cores/node; 96 hrs

GPU nodes

512GB Memory/node; 48 cores/node; 80GB Memory/GPU; 96 hrs

WHERE’S MY DATA AND BACKUPS? - Link

/home/$USER/

Home (not backed up)

/mnt/parscratch/users/$USER/

Fastdata (not backed up)

MODULES (ACTIVATING SOFTWARE) - Link

module avail

List available modules

module -t –redirect avail |& grep -i somename

Find a module

module spider <name>/<version>

Detailed module information

module load <name>/<version>

Load a module

module unload <name>/<version>

Unload a module

module list

List loaded modules

module purge

Unload all modules

ml --help

Shorthand options

Tip

You can find more information about each command with the man command. i.e man <command>


Linux Cheat Sheet

DIRECTORY OPERATIONS

pwd

Show current working directory

mkdir dir

Make a new working directory (dir)

cd dir

Change directory to dir

cd ..

Go up a directory

cd -

Go to previous directory

cd or cd ~

Navigate to home directory

du -sh dir

Size of directory dir

rmdir dir

Deletes empty directory dir

rm -r dir

Deletes directory dir and contents

mv dir dir2

Rename dir to dir2

ls

List contents of directory

ls -la

List all contents with permissions

LS OPTIONS

-lrath

A combination of options

-a

Show all (including hidden)

-R

Recursive list

-r

Reverse order

-t

Sort by last modified

-S

Sort by file size

-l

Long listing format

-h

Human-readable memory

BASH SHORTCUTS

Ctrl + c

Stop current execution

Ctrl + r

Search history

history

Show history

!1234

Repeat 1234 command in history

!!

Repeat last command in history

Ctrl + l

Clears the terminal

Ctrl + d

Exit shell or terminal

COMMAND LISTS

cmd1 ; cmd2

Run cmd1 then cmd2

cmd1 && cmd2

Run cmd2 if cmd1 succeeds

cmd1 || cmd2

Run cmd2 if cmd1 fails

cmd &

Run cmd in a subshell

FILE OPERATIONS

touch file_name

Create new file file_name

cat file_name

Display contents of file file_name

cat file1 file2 > file3

Concatenate two files in a new file (file3)

mv file /new/file/path/

Moves the file to the new location

cp file /new/file/path/

Copies the file to the new location

mv file file_new

Renames the file to file_new

rm file

Deletes the file

head file

Display first 10 lines of file

tail file

Display last 10 lines of file

less file

Display contents of file with a pager

more file1 file2

Display contents of multiple files with a pager

more begins*

Display contents of files starting with begins

SEARCH FILES

grep pattern files

Search for pattern in files

grep -i

Case insensitive search

grep -r

Recursive search

grep -v

Inverted search

grep -o

Show matched part of file only

find /dir/ -name dir_name

Find files starting with name in dir

find /dir/ -mmin n

Find files in dir modified in the last n minutes

whereis command

Find binary / source / manual for command

locate file

Find file (quick search of system index)

FILE PERMISSIONS

chmod 777 file

File read, write, execute permissions to everyone

chmod 755 file

Full permission to owner, read permissions for others

chmod 766 file

Full permission to owner, read and write for others

chown user file

Change file ownership

chown user:group file

Change file owner and group

ENVIRONMENT VARIABLES

Listing environment variables

env

List current environment variables

printenv

List specified environment variables

echo ${MYVARIABLE}

Print specified variable

Setting environment variables

set

Sets or unsets shell variables

unset

Deletes shell and environment variables

export NAME=value

Sets environment variables

Commonly used environment variables

HOME

Path of your user’s home directory

USER

Username of your current user.

PATH

Any executables in the listed directories will be available from the terminal via their name.

LD_LIBRARY_PATH

Any libraries in the listed directories will be available to programs.

Tip

You can find more information about most commands with the man command. i.e man <command>


SGE to SLURM conversions

User Commands

SGE

SLURM

Interactive login

qrshx (site specific)

srun --pty bash -i

Job submission

qsub [script_file]

sbatch [script_file]

Job deletion

qdel [job_id]

scancel [job_id]

Job status by job

qstat [-j job_id]

squeue [job_id]

Job status by user

qstat [-u user_name]

squeue -u [user_name]

Job hold

qhold [job_id]

scontrol hold [job_id]

Job release

qrls [job_id]

scontrol release [job_id]

Queue list

qconf -sql

squeue

List nodes

qhost

sinfo -N OR scontrol show nodes

Cluster status

qhost -q

sinfo

GUI

qmon

sview

Environmental variables

Job ID

$JOB_ID

$SLURM_JOBID

Submit directory

$SGE_O_WORKDIR

$SLURM_SUBMIT_DIR

Submit host

$SGE_O_HOST

$SLURM_SUBMIT_HOST

Node list

$PE_HOSTFILE

$SLURM_JOB_NODELIST

Job Array Index

$SGE_TASK_ID

$SLURM_ARRAY_TASK_ID

Parallel environmental variables set by schedulers

Job ID

$JOB_ID

$SLURM_JOB_ID

Number of cores

$NSLOTS

$SLURM_NPROCS

Job Specification

Script directive

#$

#SBATCH

queue/partition

-q [queue/partition]

-p [queue/partition]

count of nodes

N/A

-N [min[-max]]

CPU count

-pe [PE] [count]

-c [count(cpus-per-task)]

Wall clock limit

-l h_rt=[seconds]

-t [min] OR -t [days-hh:mm:ss]

Standard out file

-o [file_name]

-o [file_name]

Standard error file

-e [file_name]

-e [file_name]

Combine STDOUT & STDERR files

-j yes

(use -o without -e)

Copy environment

-V

--export=[ALL | NONE | variables]

Event notification

-m abe

--mail-type=[events]

Send notification email

-M [address]

--mail-user=[address]

Job name

-N [name]

--job-name=[name]

Restart job

-r [yes|no]

--requeue OR --no-requeue (NOTE: configurable default)

Set working directory

-wd [directory]

--workdir=[dir_name]

Resource sharing

-l exclusive

--exclusive OR --shared

Real memory per core

-l rmem=[memory per core][K|M|G]

--mem=[mem per node][M|G|T] OR --mem-percpu= [mem][M|G|T]

Charge to an account

-A [account]

--account=[account]

Tasks per node

(Fixed allocation_rule in PE)

--tasks-per-node=[count] --cpus-per-task=[count]

Job dependency

-hold_jid [job_id | job_name]

--depend=[state:job_id]

Job project/account

-P [name]

--account=[account]

Job host preference

-q [queue]@[node] OR -q [queue]@@[hostgroup]

--nodelist=[nodes] AND/OR --exclude= [nodes]

Quality of service

--qos=[name]

Job arrays

-t [array_spec]

--array=[array_spec] (Slurm version 2.6+)

Generic Resources

-l [resource]=[value]

--gres=[resource_spec]

Licenses

-l [license]=[count]

--licenses=[license_spec]

Begin Time

-a [YYMMDDhhmm]

--begin=YYYY-MM-DD[THH:MM[:SS]]

SGE

SLURM

qstat

squeue

qstat -u username

squeue -u username

qstat -f

squeue -al

qsub

sbatch

qsub -N jobname

sbatch -J jobname

qsub -m beas

sbatch --mail-type=ALL

qsub -M user@shef.ac.uk

sbatch --mail-user=user@shef.ac.uk

qsub -l h_rt=24:00:00

sbatch -t 24:00:00

qsub -pe smp 4

sbatch -N 1 -n 1 -c 4

qsub -l mem=4G

sbatch --mem=4000

qsub -P projectname

sbatch -A projectname

qsub -o filename

sbatch -o filename

qsub -e filename

sbatch -e filename

qdel

scancel