Attention

SERVICE RISK: The Bessemer HPC cluster will be at risk from 9am on July 15th until 5pm on July 19th 2024

For further detail, please click this link.

Attention

The ShARC HPC cluster was decommissioned on the 30th of November 2023 at 17:00. It is no longer possible for users to access that cluster.

OpenFOAM

OpenFOAM is leading software for computational fluid dynamics (CFD). It is licensed free and open source only under the GNU General Public Licence (GPL) by the OpenFOAM Foundation.

Usage

OpenFOAM can be activated using one of the module files below:

module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
module load apps/openfoam/v2012/gcc-8.2-openmpi-4.0.3

OpenFOAM can be used in an interactive or batch job. OpenFOAM modules for version 8 and v2012 can be activated using the module file and sourcing the OpenFOAM environment script e.g.

module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
source $FOAM_BASH

OpenFOAM version 4.1 uses the module file to match the shell environment and can be loaded by simplying running:

module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1

Hint

Users should investigate OpenFOAM documentation to determine which OpenFOAM executables are parallel compatible and which are serial only. Only the simpleFoam executable shown below is parallel compatible and is executed with srun in multiple core jobs.


Interactive Usage

The following is an example single core interactive session running the pitzDaily example model.

After connecting to ShARC (see Establishing a SSH connection), you can start an interactive graphical session. and assuming a single core request:

rm -r /fastdata/$USER/tests/openfoam/run/
mkdir -p /fastdata/$USER/tests/openfoam/run
cd /fastdata/$USER/tests/openfoam/run
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
source $FOAM_BASH
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
chmod 700 -R pitzDaily && cd pitzDaily
blockMesh
simpleFoam

Batch Usage

The following is an example batch job running the pitzDaily example model:

Important

You will need to supply a decomposeParDict in the system subdirectory of the case if using parallel processing - check the installation script for an example using the EOF method to add it :

#!/bin/bash
#$ -V
#$ -cwd
#$ -M a.person@sheffield.ac.uk
#$ -m abe
#$ -l h_rt=01:00:00
#$ -l rmem=2G
#$ -pe mpi 4
#$ -N test_OpenFOAM8_parallel-4

rm -r /fastdata/$USER/tests/openfoam/run/
mkdir -p /fastdata/$USER/tests/openfoam/run
cd /fastdata/$USER/tests/openfoam/run
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
source $FOAM_BASH
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
chmod 700 -R pitzDaily && cd pitzDaily
cp /fastdata/$USER/tests/openfoam/decomposeParDict system/decomposeParDict
blockMesh
decomposePar
mpirun -n $NSLOTS simpleFoam -parallel

Installation notes

OpenFOAM v2012

OpenFOAM v2012 was installed by Git cloning the requisite repositories from https://develop.openfoam.com/Development/openfoam/ and https://develop.openfoam.com/Development/ThirdParty-common/ followed by running the compileOFv2012.sge installation script. (Third party downloads may be necessary prior to compiling.)

Persistent configuration settings can be found in /usr/local/packages/apps/openfoam/v2012/OpenFOAM-v2012/etc/prefs.sh.

All available third party dependencies were also (manually in some cases) downloaded, compiled and installed.

The module file was built by observing shell environment changes / looking at the OpenFOAM documentation and can downloaded: gcc-8.2-openmpi-4.0.3.

The use of setenv  OMPI_MCA_btl_openib_allow_ib 1 in the module file is required in order to get the correct connectivity with the Omnipath interconnect.

Installation was tested as follows as above with the example batch script modified (Getting Started example from https://openfoam.org/download/8-source/) with the below decomposeParDict:

https://openfoamwiki.net/index.php/DecomposePar

Note

Note that OpenFOAM v2012 has been compiled with its own included OpenMPI 4.0.3.

OpenFOAM 8

OpenFOAM 8 was installed by Git cloning the requisite OpenFOAM-8 and ThirdParty-8 directories from https://github.com/OpenFOAM/OpenFOAM-8 followed by running the compileOF8pf.sge and compileOF8pf_Third_Party.sge installation scripts. (Third party downloads may be necessary prior to compiling.)

Persistent configuration settings can be found in /usr/local/packages/apps/openfoam/8/gcc-8.2-openmpi-4.0.1/site/8/prefs.sh.

All available third party dependencies were also (manually in some cases) downloaded, compiled and installed. FoamyHexMesh was enabled.

The module file was built by observing shell environment changes / looking at the OpenFOAM documentation and can downloaded: gcc-8.2-openmpi-4.0.1.

The use of setenv  OMPI_MCA_btl_openib_allow_ib 1 in the module file is required in order to get the correct connectivity with the Omnipath interconnect.

Installation was tested as follows as above with the example batch script modified (Getting Started example from https://openfoam.org/download/8-source/) with the below decomposeParDict:

https://openfoamwiki.net/index.php/DecomposePar

OpenFOAM 4.1

OpenFOAM 4.1 was installed using the install_openfoam.sh script, the module file is gcc-6.2-openmpi-2.0.1. The following optional dependencies were built as part of the installation process: Scotch/PT-Scotch v6.0.3 (located in /usr/local/packages/apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1/ThirdParty-4.1). The following optional dependencies were not installed: ParaView and CGAL.

Installation was tested as follows (Getting Started example from https://openfoam.org/download/4-1-source/):

$ mkdir /data/$USER/tests/openfoam/run
$ cd /data/$USER/tests/openfoam/run
$ module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1
$ cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
$ cd pitzDaily
$ blockMesh
$ simpleFoam
$ paraFoam