Read Me

https://github.com/rcgsheffield/sheffield_hpc/actions/workflows/static.yml/badge.svg

The following is the README from our GitHub repository.

Sheffield High Performance Computing Documentation

This is the source for the documentation of The University of Sheffield’s High Performance Computing cluster(s).

It is written in the reStructuredText (rst) format and the Sphinx tool is used to convert this to a set of HTML pages.

For a guide on the rst file format see this document.

Rendered Documentation

This website is currently automatically built from this repository: each push to the master branch causes a GitHub Actions workflow, which:

  • Builds the documentation using Sphinx

  • Adds a robots.txt file to the web root to guide indexing and reference the sitemap

  • Adds a Google Search Console verification file for ownership validation

  • Generates a sitemap using the sphinx-sitemap plugin to support better SEO

  • Deploys the resulting HTML to GitHub Pages

Test builds

For a review build, an administrator creates a remote test_* branch from current master. Open a pull request from the content branch into it, check the build result and merge for preview review. After approval, publish the original content branch through a separate PR into master. For a coordinated release, combine related PRs on a dedicated test branch, remove preview-only changes such as a test banner, then open a PR from that branch into master. The workflow publishes the preview branch at https://docs.hpc.shef.ac.uk/en/<branch>/ alongside /en/latest/ while the test branch exists. For example, test_newsletter appears at /en/test_newsletter/. See the test-build guide for branch naming, updates, removal and public access.

Redirects

The published site retains the /en/latest/ URL structure from its former Read the Docs hosting so existing links keep working. The deployment workflow moves the built pages into that directory and copies rtd-root-redirect.html to the web root, redirecting https://docs.hpc.shef.ac.uk/ to /en/latest/. Local Sphinx builds serve the pages directly from the build root.

The alternate domain, docs.hpc.sheffield.ac.uk, uses the separate sheffield_hpc_redirect repository. Its CNAME associates that domain with an organisation-controlled GitHub Pages site, whose HTML redirects point to https://docs.hpc.shef.ac.uk/en/latest/. This allows both custom domains to remain configured on our Pages sites. Keep that configuration in place while DNS points to GitHub Pages; see GitHub’s domain verification guidance for protection against domain takeovers.

For renamed pages, html_additional_pages in conf.py maps each old page path, without .html, to a redirect template in _templates. For example, nvcc-stanage-redirect.html redirects the old NVIDIA compiler page to the CUDA page. Add the old path to this mapping and set the template’s HTML refresh target to the new relative URL. This generates the redirect at the old URL without adding an obsolete page to the table of contents through a placeholder RST file.

How to Contribute

To contribute to this documentation, you need to clone both the main repository and its submodule hpc-examples to your machine, use:

git clone --recurse-submodules git@github.com:rcgsheffield/sheffield_hpc.git

Once you have the git repository locally on your computer, install uv version 0.13.5 or newer. uv manages the local Python environment.

Please see our Documentation Reference which is a valuable resource for admins of our documentation.

Follow the style guide for language, command examples, software pages and validation conventions. For copyable email-contact examples and automatic compose-link hover tips, see the email contact guidance.

Once you have made your changes and pushed them to GitHub you will need to Open a Pull Request. All changes to the repository should be made through Pull Requests, including those made by the people with direct push access.

If you make changes to hpc-examples:

  1. Open a PR to hpc-examples/main first.

  2. Do not update the submodule pointer in sheffield_hpc until the PR is merged.

  3. Once merged, update the submodule pointer and create a separate PR in sheffield_hpc.

Building the documentation

From the repository root, install the locked dependencies and build:

uv sync --locked
uv run --locked sphinx-build -b html -W -d _build/doctrees . _build/html

uv uses the Python version in .python-version and can download it if needed. The HTML is written to _build/html. On systems with Make, make html runs that same build. Warnings fail the build, as they do in CI.

For a local preview that rebuilds when files change, run:

uv run --locked sphinx-autobuild -b html -W -d _build/doctrees . _build/html

Or use make serve (make serve PORT=8080 to choose another port). Stop an existing preview before running a separate build in the same worktree.

Existing contributors can follow the Tox-to-uv transition guide, including an old-to-new command table and dependency-maintenance instructions.

Testing the building of the documentation

The validity of the reStructuredText in this repo and the ability to convert that to HTML with Sphinx can be tested in three ways:

  • Locally by contributors when they run e.g. make html

  • By a GitHub Actions Workflow each time a contributor creates or updates a Pull Request.

  • By a GitHub Actions) Workflow on each push to the master branch.

Updating documentation dependencies

Use uv add <package> (uv add --dev <tool> for development tools), uv remove <package>, or uv lock --upgrade-package <package>. To upgrade an exactly pinned direct dependency, use uv add 'package==new-version'. Run uv sync --locked and the HTML build, then review and commit the relevant pyproject.toml and uv.lock changes together. Do not edit uv.lock by hand. Dependency updates do not change the Python version in .python-version.

Important files / folders

  • conf.py - Sphinx configuration file.

  • pyproject.toml - Direct dependencies, development tools and supported Python versions.

  • uv.lock - Exact resolved dependency versions; commit this when dependencies change.

  • .python-version - Python version used for local builds and CI.

  • Makefile - Short commands for building, previewing and checking links.

  • global.rst - A globally included file (goes into all pages) which is excluded from direct building using exclude_patterns in conf.py.

  • referenceinfo/imports - sub-folder tree of files to be included by not directly built. This is excluded from direct building using exclude_patterns in conf.py.

  • _static/css/custom.css - custom CSS overrides for the theme.

  • themes/sheffieldhpc - Sheffield HPC custom theme components (Sphinx HTML templates, media files, CSS etc…). This functions as an overlay to the default Sphinx RTD theme.

  • .github/workflows - GitHub Actions workflows for pull requests, pushes to master and link checking.

Custom Google Search Engine

As the built in Sphinx search is naive / poor, a custom Google CSE has been added (currently implemented as per https://github.com/rcgsheffield/sheffield_hpc/pull/1971).

This is implemented with the following steps:

  1. Create Google custom search on console: https://cse.google.com/cse/all

  2. Copy HTML snippet for Google custom search

  3. Paste it into _templates/searchbox.html.

  4. Configure html_sidebars to use searchbox.html in your document.

  5. Ensure your templates_path is set to correctly source the templates directory.

  6. Customise the theming, search domain and other settings at https://cse.google.com/cse/all if not done already.

  7. Test the search is configured and functioning as desired.

Google Analytics 4 (GA4)

  • We use Google Analytics 4 (GA4) to track usage statistics for our documentation.

  • This helps us understand which pages are most visited, how users navigate the docs, and where improvements can be made.

  • GA4 is enabled via the sphinxcontrib-googleanalytics plugin.

  • The Privacy Policy is linked in the footer of every documentation page.

The tracking ID is set in conf.py:

googleanalytics_id = 'G-HTK1JMDS73'
googleanalytics_enabled = True

If tracking needs to be disabled, set:

googleanalytics_enabled = False

Making or using imported files from the referenceinfo/imports area

This area is intended to be used to contain .rst files which we wish to use in more than one location which can be imported. The general method for making use of imported files is as follows:

  • Make a new file to be imported within a sensible subdirectory within this area e.g. /referenceinfo/imports/software/mysoftware/import.rst

  • Import your new file into your main page with: .. include:: /referenceinfo/imports/software/mysoftware/import.rst

  • Build the documentation and ensure that hierarchical elements are correct e.g. titles within toctrees must be correct to fit in the parent document properly.

  • Add a comment within the import / parent document to explain why the import is necessary if it is not immediately obvious.

(Re)-generating PNG images from Mermaid.js diagram definitions

Some diagrams, such as images/hpcgateway-sequence-diag.png have been generated with mermaid-cli and Mermaid.js diagram definitions such as images/hpcgateway-sequence-diag.mmd. How to install mermaid-cli and regenerate one of these diagrams:

yarn add @mermaid-js/mermaid-cli
./node_modules/.bin/mmdc -i images/hpcgateway-sequence-diag.mmd -o images/hpcgateway-sequence-diag.png

Use of Tags in Documentation Repository

Tags allow us to easily access historical documentation states before key events, such as cluster decommissioning. To add a tag whilst on recently updated master branch:

git tag -a v.0.0.1 <commit hash> -m "Enter message here"
git push origin v.0.0.1

Where v.0.0.1 is the tag name.

Retrieving Documentation with Git Tags

To find relevant documentation just before decommissioning a cluster, we can now use these tags. For example, to view files from a tag version (v.0.0.1 “Pre Iceberg Decommissioning”), follow these steps:

Use the following to list all files where the file paths match some condition e.g. each path contains the substrings rst, apps and iceberg:

git ls-tree -r v.0.0.1 --name-only | grep iceberg | grep rst | grep apps

Where v.0.0.1 is the tag name. Here we could have specified a commit hash.

To view one of the listed files we can use for example:

git show v.0.0.1:iceberg/software/apps/povray.rst

Software guidance dates

Maintain software guidance dates as described in the contributing guide.