Stanage Specifications
Operating System and software
OS: SUSE Liberty Linux 7 (previously CentOS 7.9; both binary compatible with RedHat Enterprise Linux 7.9) on all nodes.
Interactive and batch job scheduling software: Slurm.
Many applications, compilers, libraries and parallel processing tools. See Software on Stanage.
Networking
Intel OmniPath Architecture (OPA) (100 Gb/s) to all public nodes.
Gigabit Ethernet.
General CPU node specifications
152 nodes are publicly available (not exclusive to research groups).
Machine: Dell PowerEdge R650.
CPUs: 2 x 32 core Intel Xeon Platinum 8358:
Ice Lake processor microarchitecture;
Base clock 2.60 GHz; Boost clock 3.40 GHz;
Support for AVX-512 vectorisation instructions (simultaneously apply the same operation to multiple values in hardware);
Support for Fused Multiply-Add instructions (expedites operations involving the accumulation of products e.g. matrix multiplication);
Hyperthreading is disabled on all nodes.
RAM: 256 GB (i.e. 4.0 GiB / core):
3200 MT/s;
DDR4.
Large memory node specifications
To complement the standard nodes with 256GB of memory per node (4GB/core), there are 12 large memory nodes with 1TB (16GB/core), and a further 12 very large nodes with 2TB (32GB/core).
Large memory nodes
12 nodes are publicly available (not exclusive to research groups).
Machine: Dell PowerEdge R650.
CPUs: 2 x 32 core Intel Xeon Platinum 8358:
Ice Lake processor microarchitecture;
Base clock 2.60 GHz; Boost clock 3.40 GHz;
Support for AVX-512 vectorisation instructions (simultaneously apply the same operation to multiple values in hardware);
Support for Fused Multiply-Add instructions (expedites operations involving the accumulation of products e.g. matrix multiplication);
Hyperthreading is disabled on all nodes.
RAM: 1024 GB (i.e. 16.0 GiB / core):
3200 MT/s;
DDR4.
Very large memory nodes
12 nodes are publicly available (not exclusive to research groups).
Machine: Dell PowerEdge R650.
CPUs: 2 x 32 core Intel Xeon Platinum 8358:
Ice Lake processor microarchitecture;
Base clock 2.60 GHz; Boost clock 3.40 GHz;
Support for AVX-512 vectorisation instructions (simultaneously apply the same operation to multiple values in hardware);
Support for Fused Multiply-Add instructions (expedites operations involving the accumulation of products e.g. matrix multiplication);
Hyperthreading is disabled on all nodes.
RAM: 2048 GB (i.e. 32.0 GiB / core):
3200 MT/s;
DDR4.
GPU nodes
A100 GPU node specifications
15 nodes are publicly available (not exclusive to research groups). Prior to December 2022 these were temporarily available in the Bessemer cluster.
Machine: Dell PowerEdge XE8545.
CPUs: 2 x 24 core AMD EPYC 7413:
Zen 3 processor microarchitecture;
Base clock 2.65 GHz; Boost clock 3.60 GHz;
Hyperthreading is disabled on all nodes.
RAM: 512 GB (i.e. 32.0 GiB / core):
3200 MHz;
DDR4.
Local storage: 460 GB boot device (SSD) plus 2.88 TB ‘/scratch’ temporary storage (RAID 0 on SSDs).
GPUs: 4x NVIDIA A100, each with:
High-bandwidth, low-latency NVLink GPU interconnects (3-4x the bandwidth of PCIe for inter-GPU comms)
80GB memory (HBM2e).
H100 GPU node specifications
6 nodes are publicly available (not exclusive to research groups).
Machine: Dell PowerEdge R7525.
CPUs: 2 x 24 core AMD EPYC 7413:
Zen 3 processor microarchitecture;
Base clock 2.65 GHz; Boost clock 3.60 GHz;
Hyperthreading is disabled on all nodes.
RAM: 512 GB (i.e. 32.0 GiB / core):
3200 MHz;
DDR4.
Local storage: 460 GB boot device (SSD) plus 960 GB ‘/scratch’ temporary storage (SSD).
GPUs: 2x NVIDIA H100, each with:
PCIe connectivity (NB 3-4x less bandwidth than NVLink for inter-GPU comms)
80GB memory (HBM2e).
Non-worker nodes
Two login nodes (for resilience).
Other nodes to provide:
Lustre parallel filesystem.
Slurm scheduler ‘head’ nodes.
Total capacity
With all workers including GPU and large memory nodes:
Worker nodes: 204.
CPU cores: 12608.
Intel Cores: 11264.
AMD Cores: 1344.
Total memory: 90112 GiB.
GPUs: 76.
Fast network filesystem (Lustre): 2 PiB.