Attention

GPU users: GPU nodes run EL9, while this documentation still reflects the EL7 software stack.

Run module avail on an EL9 login node, or within an interactive session on the relevant GPU type, to see the software available.

The documentation will be updated once all nodes have migrated to EL9.

Stanage Specifications

Operating System and software

Some of the Stanage hardware.

  • OS: SUSE Liberty Linux 7 (previously CentOS 7.9; both binary compatible with RedHat Enterprise Linux 7.9) on all nodes.

  • Interactive and batch job scheduling software: Slurm.

  • Many applications, compilers, libraries and parallel processing tools. See Software on Stanage.

Networking

  • Intel OmniPath Architecture (OPA) (100 Gb/s) to all public nodes.

  • 25 Gigabit Ethernet.

General CPU node specifications

142 nodes are currently publicly available.

  • Machine: Dell PowerEdge R650.

  • CPUs: 2 x 32 core Intel Xeon Platinum 8358:

    • Ice Lake processor microarchitecture;

    • Base clock 2.60 GHz; Boost clock 3.40 GHz;

    • Support for AVX-512 vectorisation instructions (simultaneously apply the same operation to multiple values in hardware);

    • Support for Fused Multiply-Add instructions (expedites operations involving the accumulation of products e.g. matrix multiplication);

    • Hyperthreading is disabled on all nodes.

  • RAM: 256 GB (251 GB available, i.e., 4016 MB / core):

    • 3200 MT/s;

    • DDR4.

Large memory node specifications

To complement the standard nodes with 251 GB of memory per node (4016 MB/core), there are 12 large memory nodes with 1 TB (16112 MB / core), and a further 12 very large nodes with 2 TB (32224 MB / core).

Large memory nodes

12 nodes are currently publicly available.

  • Machine: Dell PowerEdge R650.

  • CPUs: 2 x 32 core Intel Xeon Platinum 8358:

    • Ice Lake processor microarchitecture;

    • Base clock 2.60 GHz; Boost clock 3.40 GHz;

    • Support for AVX-512 vectorisation instructions (simultaneously apply the same operation to multiple values in hardware);

    • Support for Fused Multiply-Add instructions (expedites operations involving the accumulation of products e.g. matrix multiplication);

    • Hyperthreading is disabled on all nodes.

  • RAM: 1024 GB (1007 GB available, i.e., 16112 MB / core):

    • 3200 MT/s;

    • DDR4.

Very large memory nodes

12 nodes are currently publicly available.

  • Machine: Dell PowerEdge R650.

  • CPUs: 2 x 32 core Intel Xeon Platinum 8358:

    • Ice Lake processor microarchitecture;

    • Base clock 2.60 GHz; Boost clock 3.40 GHz;

    • Support for AVX-512 vectorisation instructions (simultaneously apply the same operation to multiple values in hardware);

    • Support for Fused Multiply-Add instructions (expedites operations involving the accumulation of products e.g. matrix multiplication);

    • Hyperthreading is disabled on all nodes.

  • RAM: 2048 GB (2014 GB available, i.e., 32224 MB / core):

    • 3200 MT/s;

    • DDR4.

GPU nodes

A100 GPU node specifications

13 nodes are currently publicly available.

  • Machine: Dell PowerEdge XE8545.

  • CPUs: 2 x 24 core AMD EPYC 7413:

    • Zen 3 processor microarchitecture;

    • Base clock 2.65 GHz; Boost clock 3.60 GHz;

    • Hyperthreading is disabled on all nodes.

  • RAM: 512 GB (503 GB available, i.e., 10731 MB / core):

    • 3200 MHz;

    • DDR4.

  • Local storage: 460 GB boot device (SSD) plus 2.88 TB scratch temporary storage (RAID 0 on SSDs).

  • GPUs: 4x NVIDIA A100, each with:

    • High-bandwidth, low-latency NVLink GPU interconnects (3-4x the bandwidth of PCIe for inter-GPU comms)

    • 80GB memory (HBM2e).

H100 GPU node specifications

6 nodes are currently publicly available.

  • Machine: Dell PowerEdge R7525.

  • CPUs: 2 x 24 core AMD EPYC 7413:

    • Zen 3 processor microarchitecture;

    • Base clock 2.65 GHz; Boost clock 3.60 GHz;

    • Hyperthreading is disabled on all nodes.

  • RAM: 512 GB (503 GB available, i.e.,, 10731 MB / core):

    • 3200 MHz;

    • DDR4.

  • Local storage: 460 GB boot device (SSD) plus 960 GB scratch temporary storage (SSD).

  • GPUs: 2x NVIDIA H100, each with:

    • PCIe connectivity (NB 3-4x less bandwidth than NVLink for inter-GPU comms)

    • 80GB memory (HBM2e).

H100-NVL GPU node specifications

4 nodes are currently publicly available.

Note

While both the H100-NVL and H100 GPU nodes contain “H100” GPUs, the H100 NVL variant in the H100-NVL nodes is substantially different from the H100 GPUs in the H100 nodes. The H100 NVL is a newer GPU featuring 15% more CUDA and tensor cores alongside 95% more memory bandwidth than the older H100 GPU. In some work loads the H100 NVL may be up to 50% faster in practice.

Non-worker nodes

  • Two login nodes (for resilience).

  • Other nodes to provide:

    • Lustre parallel filesystem.

    • Slurm scheduler ‘head’ nodes.

Total capacity

With all workers including GPU and large memory nodes:

  • Worker nodes: 205.

  • CPU cores: 12896.

    • Intel Cores: 11744.

    • AMD Cores: 1152.

  • Total memory: 88991 GB available.

  • GPUs: 104.

  • Fast network filesystem (Lustre): 2.2 PB.