Attention

The ShARC HPC cluster was decommissioned on the 30th of November 2023 at 17:00. It is no longer possible for users to access that cluster.

Using the HPC Systems

If you have not used a High Performance Computing (HPC) cluster, the Linux operating system or even a command line before this is the place to start. This guide will get you set up using the University’s clusters (Stanage, Bessemer and ShARC) fairly quickly.

• What is High Performance Computing?
  • Introduction
  • Who can use or should use a HPC cluster?
  • What is a HPC cluster?
  • When should I use HPC?
  • When should I not use HPC?
  • How do I get started?
• Getting an Account
  • For Staff
  • For Students
  • Conditions for HPC access / usage
• Connecting to a cluster using myApps (web browser)
• Connecting to a cluster using SSH
  • Connecting with a password
  • Connecting with SSH keys
  • Suggested SSH clients
  • Establishing a SSH connection
  • What if I cannot use the VPN or I need a persistent long term connection
  • What Next?
• Tips for working with MFA
  • Reuse SSH connections
  • TMUX/screen
  • Transferring files
• Filestores
  • Choosing the correct filestore
  • Home directories
  • Data directories
  • Fastdata areas
  • Shared (project) directories
  • Scratch directories
  • Community areas for software
  • How to check your quota usage
  • If you exceed your filesystem quota
  • Recovering files from snapshots
• Transferring files
  • Transfers with SCP/SFTP
  • How to download files directly to the cluster
• Job Submission and Control
  • Introduction
  • Key Concepts
  • Job Submission / Control on ShARC
  • Job Submission / Control on Bessemer
  • Job Submission / Control on Stanage
  • Cluster job resource limits
  • Advanced / Automated job submission and management
  • Reference information and further resources
• Advanced Job Submission and Control
  • Introduction
  • Advanced Job Submission Methods
  • Advanced workflow management tools
• Activating software using Environment Modules
  • Overview and rationale
  • Basic guide
  • Searching for Modules
  • Behind the scenes
  • Convenient ways to set up your environment for different projects
  • Managing your own module files
  • Compiling software dependent on modules
  • Module Command Reference
• Choosing appropriate compute resources
  • Introduction
  • Cluster choice
  • Time Allocation Limits
  • CPU Allocation Limits
  • Memory Allocation Limits
  • Filestore Limits / file store performance characteristics
  • Special limits and alternative queues
• Installing software to the clusters
  • General background prequisites
  • Installing software from binaries
  • Installing software by compiling from source
  • Making installed software available to execute
  • Why should I install from source?
  • What alternative methods exist?
• Hardware-accelerated graphics rendering (qsh-vis)
  • Usage instructions
  • Resources available to qsh-vis sessions
  • Availability
  • Technical details
• Jupyter and JupyterHub
  • Introduction
  • Using Jupyter on ShARC
  • Maintenance of ShARC’s JupyterHub service
  • JupyterHub on a Grid Engine cluster: internal workings
  • Credits
• HPC Gateway Service
  • Service description
  • Access conditions
  • Overview of the connection process
  • Specific usage examples