Filestores

On the Stanage cluster, every HPC user has access to up to 3 different storage areas:

Home directories: per-user storage with no backups.
Fastdata areas: per-user Lustre storage with no backups.
Scratch directories : per node, node-local storage with no backups.

Home directories

All users have a home directory on Stanage. This area is seperate from the other clusters

System	Path	Type	Quota per user	Shared between system login and worker nodes?	Shared between systems?
Stanage	`/users/$USER`	NFS	50 GB or 300000 files	Yes	No

Where $USER is the user’s username.

Snapshotting and mirrored backups

Snapshotting is not enabled for home areas and these areas are not backed up.

Fastdata areas

Fastdata areas are optimised for large file operations. These areas are Lustre filesystems.

They are are faster than Home directories and Shared (project) directories when dealing with larger files but are not performant when reading/writing lots of small files (Scratch directories are ideal for reading/writing lots of small temporary files within jobs). An example of how slow it can be for large numbers of small files is detailed here.

System	Path	Type	Quota per user	Filesystem capacity	Shared between systems?	Network bandwith per link
Stanage	`/mnt/parscratch/users/$USER`	Lustre	No limits	2 PiB	No	100Gb/s (Omni-Path)

Where $USER is the user’s username.

Tip

This folder doesn’t exist by default, you can create it with safe permissions by running the command: mkdir -m 0700 /mnt/parscratch/users/$USER

Snapshotting and mirrored backups

Snapshotting is not enabled for fastdata areas and these areas are not backed up.

Shared (project) directories

Shared project storage areas are not yet available on the Stanage cluster.

Scratch directories

For jobs that need to read/write lots of small files the most performant storage will be the temporary storage on each node (under the /tmp directory).

This is because with Home directories, Fastdata areas and Shared (project) directories, each time a file is accessed the filesystem needs to request ownership/permissions information from another server and for small files these overheads are proportionally high.

For the scratch storage area, such ownership/permissions metadata is available on the local machine, thus it is faster when dealing with small files.

Each user is encouraged to use /tmp/users/$USER (where $USER is the user’s username) for node-local storage.

Tip

This folder doesn’t exist by default, you can create it with safe permissions by running the command: mkdir -m 700 -p /tmp/users/$USER

You should run this command in each batch submission script prior to using this directory!

Further conditions also apply:

Anything in the /tmp/users/$USER area may be deleted periodically when the worker-node is idle or rebooted.
The /tmp/users/$USER area is not backed up.
There are no quotas for /tmp/users/$USER storage.
The /tmp/users/$USER area uses the ext4 filesystem.

Danger

/tmp/users/$USER area is temporary and has no backups. If you forget to copy your output data out of the /tmp/users/$USER area before your job finishes, your data cannot be recovered!

How to check your quota usage

To find out your storage quota usage for your home directory you can use the quota command with the -u (your user) and -s (human readable) :

[user@login1 [stanage] ~] quota -u -s
    Filesystem   space   quota   limit   grace   files   quota   limit   grace
storage:/export/users
                 3289M  51200M  76800M            154k    300k    350k

To determine usage in a particular Shared (project) directories you can use the df command like so:

[user@login1 [stanage] ~]  df -h /shared/myproject1
Filesystem                        Size  Used Avail Use% Mounted on
172.X.X.X:/myproject1/myproject1   10T  9.1T  985G  91% /shared/myproject1

To assess what is using up your quota within a given directory, you can make use of the ncdu module on Stanage or the

The ncdu utility will give you an interactive display of what files/folders are taking up storage in a given directory tree.

If you exceed your filesystem quota

If you reach your quota for your home directory then many common programs/commands may cease to work as expected or at all and you may not be able to log in.

In addition, jobs may fail if you exceed your quota for your Shared (project) directory.

In order to avoid this situation it is strongly recommended that you:

Check your quota usage regularly.
Copy files that do not need to be backed up to a Fastdata area or remove them from Stanage completely.