Attention

WARNING: From 9am on 19th August until 5pm on 2nd September there will be no access to the Stanage HPC cluster.

We will send an email to notify you when Stanage is back online and available for job submission.

Quick Reference (Cheat Sheets)

Note

These cheat sheets are a quick reference guide to be used once you are familiar with our documentation.

HPC Cluster Cheat Sheets

This cheat sheet covers basic functionality for our clusters.

StanageBessemer

CONNECTING TO THE CLUSTER AND TRANSFERRING FILES

ssh -X YOUR_USERNAME@stanage.shef.ac.uk	Connect - Link
srun –pty bash -i	Start an interactive session - Link
srun –partition=gpu –qos=gpu –gres=gpu:1 –pty bash	Start an interactive GPU session - Link
scp /path/to/file.txt YOUR_USERNAME@stanage.shef.ac.uk:/path/to/directory/	Upload - Link
scp YOUR_USERNAME@stanage.shef.ac.uk:/path/to/file.txt /path/to/directory/	Download file - Link
scp -r YOUR_USERNAME@stanage.shef.ac.uk:/path/to/my_results /path/to/directory/	Download directory - Link
rsync -avzP /path/to/directory/ YOUR_USERNAME@stanage.shef.ac.uk:/path/to/directory/	Sync/transfer directory - Link
wget https://software.github.io/program/files/myprogram.tar.gz	Download direct from website - Link
curl -O https://software.github.io/program/files/myprogram.tar.gz	Download direct from website - Link

BATCH JOB SUBMISSION, MONITORING AND CONTROL

sbatch MY_SCRIPT.sh	Submit a batch job - Link
squeue -u $USER	Investigate jobs in queue (Running R and Pending PD) - Link
sstat -j 1234567	Investigate running job - Link
sacct -j 1234567	Investigate historical job - Link
scancel 1234567	Cancel a job - Link
scontrol <action> 1234567	Control a job (hold/release) - Link
salloc	Allocate resources to an interactive job - Link
srun	Start a task inside a job - Link

PARTITION INFORMATION

sinfo	Node and partition information - Link
General CPU nodes	256GB Memory/node; 64 cores/node; 96 hrs
Large Mem CPU nodes	1TB Memory/node; 64 cores/node; 96 hrs
V Large Mem CPU nodes	2TB Memory/node; 64 cores/node; 96 hrs
GPU nodes	512GB Memory/node; 48 cores/node; 80GB Memory/GPU; 96 hrs

WHERE’S MY DATA AND BACKUPS? - Link

/home/$USER/	Home (not backed up)
/mnt/parscratch/users/$USER/	Fastdata (not backed up)

MODULES (ACTIVATING SOFTWARE) - Link

module avail	List available modules
module -t –redirect avail |& grep -i somename	Find a module
module spider <name>/<version>	Detailed module information
module load <name>/<version>	Load a module
module unload <name>/<version>	Unload a module
module list	List loaded modules
module purge	Unload all modules
ml --help	Shorthand options

Tip

You can find more information about each command with the man command. i.e man <command>

---

Linux Cheat Sheet

DIRECTORY OPERATIONS

pwd	Show current working directory
mkdir dir	Make a new working directory (dir)
cd dir	Change directory to dir
cd ..	Go up a directory
cd -	Go to previous directory
cd or cd ~	Navigate to home directory
du -sh dir	Size of directory dir
rmdir dir	Deletes empty directory dir
rm -r dir	Deletes directory dir and contents
mv dir dir2	Rename dir to dir2
ls	List contents of directory
ls -la	List all contents with permissions

LS OPTIONS

-lrath	A combination of options
-a	Show all (including hidden)
-R	Recursive list
-r	Reverse order
-t	Sort by last modified
-S	Sort by file size
-l	Long listing format
-h	Human-readable memory

BASH SHORTCUTS

Ctrl + c	Stop current execution
Ctrl + r	Search history
history	Show history
!1234	Repeat 1234 command in history
!!	Repeat last command in history
Ctrl + l	Clears the terminal
Ctrl + d	Exit shell or terminal

COMMAND LISTS

cmd1 ; cmd2	Run cmd1 then cmd2
cmd1 && cmd2	Run cmd2 if cmd1 succeeds
cmd1 || cmd2	Run cmd2 if cmd1 fails
cmd &	Run cmd in a subshell

FILE OPERATIONS

touch file_name	Create new file file_name
cat file_name	Display contents of file file_name
cat file1 file2 > file3	Concatenate two files in a new file (file3)
mv file /new/file/path/	Moves the file to the new location
cp file /new/file/path/	Copies the file to the new location
mv file file_new	Renames the file to file_new
rm file	Deletes the file
head file	Display first 10 lines of file
tail file	Display last 10 lines of file
less file	Display contents of file with a pager
more file1 file2	Display contents of multiple files with a pager
more begins*	Display contents of files starting with begins

SEARCH FILES

grep pattern files	Search for pattern in files
grep -i	Case insensitive search
grep -r	Recursive search
grep -v	Inverted search
grep -o	Show matched part of file only
find /dir/ -name dir_name	Find files starting with name in dir
find /dir/ -mmin n	Find files in dir modified in the last n minutes
whereis command	Find binary / source / manual for command
locate file	Find file (quick search of system index)

FILE PERMISSIONS

chmod 777 file	File read, write, execute permissions to everyone
chmod 755 file	Full permission to owner, read permissions for others
chmod 766 file	Full permission to owner, read and write for others
chown user file	Change file ownership
chown user:group file	Change file owner and group

ENVIRONMENT VARIABLES

Listing environment variables
env	List current environment variables
printenv	List specified environment variables
echo ${MYVARIABLE}	Print specified variable
Setting environment variables
set	Sets or unsets shell variables
unset	Deletes shell and environment variables
export NAME=value	Sets environment variables
Commonly used environment variables
HOME	Path of your user’s home directory
USER	Username of your current user.
PATH	Any executables in the listed directories will be available from the terminal via their name.
LD_LIBRARY_PATH	Any libraries in the listed directories will be available to programs.

Tip

You can find more information about most commands with the man command. i.e man <command>

---

SGE to SLURM conversions

User Commands	SGE	SLURM
Interactive login	qrshx (site specific)	srun --pty bash -i
Job submission	qsub [script_file]	sbatch [script_file]
Job deletion	qdel [job_id]	scancel [job_id]
Job status by job	qstat [-j job_id]	squeue [job_id]
Job status by user	qstat [-u user_name]	squeue -u [user_name]
Job hold	qhold [job_id]	scontrol hold [job_id]
Job release	qrls [job_id]	scontrol release [job_id]
Queue list	qconf -sql	squeue
List nodes	qhost	sinfo -N OR scontrol show nodes
Cluster status	qhost -q	sinfo
GUI	qmon	sview
Environmental variables
Job ID	$JOB_ID	$SLURM_JOBID
Submit directory	$SGE_O_WORKDIR	$SLURM_SUBMIT_DIR
Submit host	$SGE_O_HOST	$SLURM_SUBMIT_HOST
Node list	$PE_HOSTFILE	$SLURM_JOB_NODELIST
Job Array Index	$SGE_TASK_ID	$SLURM_ARRAY_TASK_ID
Parallel environmental variables set by schedulers
Job ID	$JOB_ID	$SLURM_JOB_ID
Number of cores	$NSLOTS	$SLURM_NPROCS
Job Specification
Script directive	#$	#SBATCH
queue/partition	-q [queue/partition]	-p [queue/partition]
count of nodes	N/A	-N [min[-max]]
CPU count	-pe [PE] [count]	-c [count(cpus-per-task)]
Wall clock limit	-l h_rt=[seconds]	-t [min] OR -t [days-hh:mm:ss]
Standard out file	-o [file_name]	-o [file_name]
Standard error file	-e [file_name]	-e [file_name]
Combine STDOUT & STDERR files	-j yes	(use -o without -e)
Copy environment	-V	--export=[ALL | NONE | variables]
Event notification	-m abe	--mail-type=[events]
Send notification email	-M [address]	--mail-user=[address]
Job name	-N [name]	--job-name=[name]
Restart job	-r [yes|no]	--requeue OR --no-requeue (NOTE: configurable default)
Set working directory	-wd [directory]	--workdir=[dir_name]
Resource sharing	-l exclusive	--exclusive OR --shared
Real memory per core	-l rmem=[memory per core][K|M|G]	--mem=[mem per node][M|G|T] OR --mem-percpu= [mem][M|G|T]
Charge to an account	-A [account]	--account=[account]
Tasks per node	(Fixed allocation_rule in PE)	--tasks-per-node=[count] --cpus-per-task=[count]
Job dependency	-hold_jid [job_id | job_name]	--depend=[state:job_id]
Job project/account	-P [name]	--account=[account]
Job host preference	-q [queue]@[node] OR -q [queue]@@[hostgroup]	--nodelist=[nodes] AND/OR --exclude= [nodes]
Quality of service		--qos=[name]
Job arrays	-t [array_spec]	--array=[array_spec] (Slurm version 2.6+)
Generic Resources	-l [resource]=[value]	--gres=[resource_spec]
Licenses	-l [license]=[count]	--licenses=[license_spec]
Begin Time	-a [YYMMDDhhmm]	--begin=YYYY-MM-DD[THH:MM[:SS]]

SGE	SLURM
qstat	squeue
qstat -u username	squeue -u username
qstat -f	squeue -al
qsub	sbatch
qsub -N jobname	sbatch -J jobname
qsub -m beas	sbatch --mail-type=ALL
qsub -M user@shef.ac.uk	sbatch --mail-user=user@shef.ac.uk
qsub -l h_rt=24:00:00	sbatch -t 24:00:00
qsub -pe smp 4	sbatch -N 1 -n 1 -c 4
qsub -l mem=4G	sbatch --mem=4000
qsub -P projectname	sbatch -A projectname
qsub -o filename	sbatch -o filename
qsub -e filename	sbatch -e filename
qdel	scancel

---