Filestores
Every HPC user has access to up to six different storage areas:
Home directories: per-user backed-up, snapshotted storage
Data directories: additional per-user snapshotted storage (not on Bessemer)
Fastdata areas: high-performance shared filesystem for temporary data - optimised for reading/writing large files from multiple nodes and threads simultaneously
Shared (project) directories: per-PI shared storage areas (snapshotted and backed-up) for project data - can be accessed from non-HPC machines too
Scratch directories: per-node temporary storage - useful for reading/writing lots of small files within one job
Community areas for software: cluster-wide storage areas to allow users to share software.
The storage areas differ in terms of:
the amount of space available;
whether they are available from multiple nodes;
whether they are shared between clusters;
whether the underlying storage system is performant if reading/writing large files;
whether the underlying storage system is performant if reading/writing small files;
frequency of storage snapshotting, whether storage is mirrored and the maximum duration data can be retained for;
whether they handle permissions like a typical Linux filesystem.
At present none provide encryption at rest.
Choosing the correct filestore
To make a quick assessment of what storage area is likely to best fulfil your needs, please take a look at the provided decision tree below:
Warning
This decision tree only provides a quick assessment, please check the full details of each filestore before committing to using them for your work.
Home directories
All users have a home directory on each system, some of which are shared between systems:
System |
Path |
Type |
Quota per user |
Shared between system login and worker nodes? |
Shared between systems? |
---|---|---|---|---|---|
Bessemer |
|
NFS |
100GB |
Yes |
No |
ShARC |
|
NFS |
10GB |
Yes |
No |
See also: How to check your quota usage and * If you exceed your filesystem quota.
Snapshotting and mirrored backups
Frequency of snapshotting |
Snapshots retained |
---|---|
Every 4 hours |
10 most recent |
Every night |
Last 7 days |
Frequency of mirrored backups |
Backups retained |
---|---|
Every 4 hours |
6 most recent |
Every night |
28 most recent |
See also: Recovering files from snapshots.
Data directories
Every user on ShARC (not Bessemer) has access to a larger data storage area:
System |
Path |
Type |
Quota per user |
Shared between system login and worker nodes? |
Shared between systems? |
---|---|---|---|---|---|
Bessemer |
N/A |
NFS |
N/A |
N/A |
N/A |
ShARC |
|
NFS |
100GB |
Yes |
No |
See also: How to check your quota usage and * If you exceed your filesystem quota.
Snapshotting and mirrored backups
Frequency of snapshotting |
Snapshots retained |
---|---|
Every 4 hours |
10 most recent |
Every night |
Last 7 days |
Frequency of mirrored backups |
Backups retained |
---|---|
Every 4 hours |
6 most recent |
Every night |
28 most recent |
See also: Recovering files from snapshots.
Automounting
Data directories are made available to you (mounted) on demand:
if you list the contents of just /data
after first logging on then your /data/te1st
subdirectory (where te1st
is your username) might not be shown.
However, if you list the contents of /data/te1st
itself or change into that directory
then its contents will appear.
Later on if you list the contents of /data
again
you may find that /data/te1st
has disappeared again, as
it is automatically unmounted following a period of inactivity.
Fastdata areas
Fastdata areas are optimised for large file operations. These areas are Lustre filesystems.
They are are faster than Home directories, Data directories and Shared (project) directories when dealing with larger files but are not performant when reading/writing lots of small files (Scratch directories are ideal for reading/writing lots of small temporary files within jobs). An example of how slow it can be for large numbers of small files is detailed here.
System |
Path |
Type |
Quota per user |
Filesystem capacity |
Shared between systems? |
Network bandwith per link |
---|---|---|---|---|---|---|
Bessemer |
|
Lustre |
No limits |
460 TB |
No |
25Gb/s Ethernet |
ShARC |
|
Lustre |
No limits |
669 TB |
No |
100Gb/s (Omni-Path) |
Snapshotting and mirrored backups
Snapshotting is not enabled for fastdata areas and these areas are not backed up.
Managing your files in fastdata areas
In order to avoid interference from other users’ files
it is important that you store your files in a directory created and named the same as your username
e.g. if your username is te1st
then you can a fastdata area for yourself using:
mkdir /fastdata/te1st
By default the directory you create will have world-read access. If you want to restrict read access to just your account then run
chmod 700 /fastdata/te1st
after creating the directory. A more sophisticated sharing scheme would have private and public directories
mkdir /fastdata/te1st
mkdir /fastdata/te1st/public
mkdir /fastdata/te1st/private
chmod 755 /fastdata/te1st
chmod 755 /fastdata/te1st/public
chmod 700 /fastdata/te1st/private
Automatic file deletion
Warning
There are no quota controls in fastdata areas but older files are automatically deleted: a report of files older than 60 days is regularly generated, the owners of these files are then notified by email then a week after the email(s) are sent the identified files are deleted.
We reserve the right to change this policy without warning in order to ensure efficient running of the service.
It is important to therefore not use fastdata areas for long-term storage and copy important data from these areas to areas suitable for longer-term storage (Home directories, Data directories or Shared (project) directories).
You can use the lfs
command to find out which files in a fastdata directory are older than a certain number of days and hence approaching the time of deletion.
For example, if your username is te1st
then you can find files 50 or more days old using:
lfs find -ctime +50 /fastdata/te1st
File locking
As of September 2020 POSIX file locking is enabled on all Lustre filesystems. Prior to this the lack of file locking support on the University’s Lustre filesystems caused problems for certain workflows/applications (e.g. for programs that create/use SQLite databases).
Scratch directories
For jobs that need to read/write lots of small files the most performant storage will be
the temporary storage on each node (under the /scratch
directory).
This is because with Home directories, Data directories, Fastdata areas and Shared (project) directories, each time a file is accessed the filesystem needs to request ownership/permissions information from another server and for small files these overheads are proportionally high.
For the /scratch
store, such ownership/permissions metadata is available on the local machine,
thus it is faster when dealing with small files.
As the /scratch
areas are node-local storage and files/folders are deleted when jobs end:
any data used by the job must be copied to
/scratch
when the jobs starts.any output data stored in
/scratch
must also be copied off to another area before the job finishes. (e.g. to Home directories or Data directories).
Further conditions also apply:
Anything in the
/scratch
area may be deleted periodically when the worker-node is idle.The
/scratch
area is not backed up.There are no quotas for
/scratch
storage.The
/scratch
area uses the ext4 filesystem.
Danger
/scratch
areas are temporary and have no backups. If you forget to copy your output data out of the
/scratch
area before your job finishes, your data cannot be recovered!
Where to store data inside /scratch
areas:
The schedulers automatically create a per-job directory for you under /scratch
.
The name of this directory is stored in the $TMPDIR
environment variable e.g.
On ShARC:
[te1st@sharc-login1 ~]$ qrshx
[te1st@sharc-node003 ~]$ cd $TMPDIR
[te1st@sharc-node003 667443.1.all.q]$ pwd
/scratch/667443.1.all.q
On Bessemer:
[te1st@bessemer-login1 ~]$ srun -c 1 --mem=4G --pty bash -i
[te1st@bessemer-node001 ~]$ cd $TMPDIR
[te1st@bessemer-node001 2660172]$ pwd
/scratch/2660172
The scheduler will then clean up (delete) $TMPDIR
at the end of your job,
ensuring that the space can be used by other users.
Warning
If using qrsh
on ShARC to start an interactive job then
the TMPDIR
environment variable will unfortunately be undefined
so you will need to manually create a directory under /scratch
(named using your username)
and this will not be cleaned up when the job ends.
Community areas for software
Most data that researchers want to share with their collaborators at the University should reside in Shared (project) directories.
However, as mentioned in Permissions behaviour, these areas may not be ideal for storing executable software/scripts
due to the way permissions are handled beneath /shared
.
Also, users may want to install software on the clusters that they want to be accessible by all cluster users.
To address these two needs users can request the creation of a new directory beneath of the three directories listed below and if their request is granted they will be given write access to this area:
System |
Path |
Type |
Software install guidelines |
Public index of areas |
Notes |
---|---|---|---|---|---|
Bessemer |
|
NFS |
|||
ShARC |
|
NFS |
Also available at |
Note that:
Software installation should follow our installation guidelines where provided.
Software installations must be maintained by a responsible owner.
Software which is not actively maintained may be removed.
How to check your quota usage
To find out your storage quota usage for your home directory and data directory (if not on Bessemer)
you can use the quota
command:
[foo11b@sharc-node004 binary]$ quota
Size Used Avail Use% Mounted on
10G 10G 0G 100% /home/foo11b
100G 0 100G 0% /data/foo11b
In the above, you can see that the quota was set to 10 gigabytes and all of this is in use which is likely to cause jobs to fail.
To determine usage in a particular Shared (project) directories you can use the df
command like so:
[foo11b@sharc-node004 binary]$ df -h /shared/myproject1
Filesystem Size Used Avail Use% Mounted on
172.X.X.X:/myproject1/myproject1 10T 9.1T 985G 91% /shared/myproject1
To assess what is using up your quota within a given directory, you can make use of the ncdu module on ShARC or the ncdu module on Bessemer. The ncdu utility will give you an interactive display of what files/folders are taking up storage in a given directory tree.
If you exceed your filesystem quota
If you reach your quota for your home directory then many common programs/commands may cease to work as expected or at all and you may not be able to log in.
In addition, jobs may fail if you exceed your quota for your data directory or a Shared (project) directory.
In order to avoid this situation it is strongly recommended that you:
Check your quota usage regularly.
Copy files that do not need to be backed up to a Fastdata area or remove them from Bessemer/ShARC completely.
Recovering files from snapshots
Home directories, Data directories and Shared (project) directories are regularly snapshotted. See above for details of the snapshot schedules per area. A subset of snapshots can be accessed by HPC users from the HPC systems themselves by explicitly browsing to hidden directories e.g.
Storage area |
Parent directory of snapshots |
---|---|
|
|
|
|
|
From within per-snapshot directories you can access (read-only) copies of files/directories. This allows you to attempt recover any files you might have accidentally modified or deleted recently.
Note that .snapshot
directories are not visible when listing all hidden items within their parent directories
(e.g. using ls -a /home/$USER
):
you need to explicitly cd
into .snapshot
directories to see/access them.