Attention
The ShARC HPC cluster was decommissioned on the 30th of November 2023 at 17:00. It is no longer possible for users to access that cluster.
OpenFOAM
OpenFOAM is leading software for computational fluid dynamics (CFD). It is licensed free and open source only under the GNU General Public Licence (GPL) by the OpenFOAM Foundation.
Usage
OpenFOAM can be activated using one of the module files below:
module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
module load apps/openfoam/v2012/gcc-8.2-openmpi-4.0.3
OpenFOAM can be used in an interactive or batch job. OpenFOAM modules for version 8 and v2012 can be activated using the module file and sourcing the OpenFOAM environment script e.g.
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
source $FOAM_BASH
OpenFOAM version 4.1 uses the module file to match the shell environment and can be loaded by simplying running:
module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1
Hint
Users should investigate OpenFOAM documentation to determine which OpenFOAM executables are parallel compatible and
which are serial only. Only the simpleFoam executable shown below is parallel compatible and is executed with srun
in multiple core jobs.
Interactive Usage
The following is an example single core interactive session running the pitzDaily example model.
After connecting to ShARC (see Establishing a SSH connection), you can start an interactive graphical session. and assuming a single core request:
rm -r /fastdata/$USER/tests/openfoam/run/
mkdir -p /fastdata/$USER/tests/openfoam/run
cd /fastdata/$USER/tests/openfoam/run
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
source $FOAM_BASH
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
chmod 700 -R pitzDaily && cd pitzDaily
blockMesh
simpleFoam
Batch Usage
The following is an example batch job running the pitzDaily example model:
Important
You will need to supply a decomposeParDict in the system subdirectory of the case if using parallel processing - check the installation script for an example using the EOF method to add it :
#!/bin/bash
#$ -V
#$ -cwd
#$ -M a.person@sheffield.ac.uk
#$ -m abe
#$ -l h_rt=01:00:00
#$ -l rmem=2G
#$ -pe mpi 4
#$ -N test_OpenFOAM8_parallel-4
rm -r /fastdata/$USER/tests/openfoam/run/
mkdir -p /fastdata/$USER/tests/openfoam/run
cd /fastdata/$USER/tests/openfoam/run
module load apps/openfoam/8/gcc-8.2-openmpi-4.0.1
source $FOAM_BASH
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
chmod 700 -R pitzDaily && cd pitzDaily
cp /fastdata/$USER/tests/openfoam/decomposeParDict system/decomposeParDict
blockMesh
decomposePar
mpirun -n $NSLOTS simpleFoam -parallel
Installation notes
OpenFOAM v2012
OpenFOAM v2012 was installed by Git cloning the requisite repositories from https://develop.openfoam.com/Development/openfoam/ and https://develop.openfoam.com/Development/ThirdParty-common/ followed by running the
compileOFv2012.sge installation script. (Third party downloads may be necessary prior to compiling.)
Persistent configuration settings can be found in /usr/local/packages/apps/openfoam/v2012/OpenFOAM-v2012/etc/prefs.sh.
All available third party dependencies were also (manually in some cases) downloaded, compiled and installed.
The module file was built by observing shell environment changes / looking at the OpenFOAM documentation and can downloaded: gcc-8.2-openmpi-4.0.3.
The use of setenv OMPI_MCA_btl_openib_allow_ib 1 in the module file is required in order to get the correct connectivity with the Omnipath interconnect.
Installation was tested as follows as above with the example batch script modified
(Getting Started example from https://openfoam.org/download/8-source/) with the below decomposeParDict:
https://openfoamwiki.net/index.php/DecomposePar
Note
Note that OpenFOAM v2012 has been compiled with its own included OpenMPI 4.0.3.
OpenFOAM 8
OpenFOAM 8 was installed by Git cloning the requisite OpenFOAM-8 and ThirdParty-8 directories from https://github.com/OpenFOAM/OpenFOAM-8 followed by running the
compileOF8pf.sge and
compileOF8pf_Third_Party.sge installation scripts. (Third party downloads may be necessary prior to compiling.)
Persistent configuration settings can be found in /usr/local/packages/apps/openfoam/8/gcc-8.2-openmpi-4.0.1/site/8/prefs.sh.
All available third party dependencies were also (manually in some cases) downloaded, compiled and installed. FoamyHexMesh was enabled.
The module file was built by observing shell environment changes / looking at the OpenFOAM documentation and can downloaded: gcc-8.2-openmpi-4.0.1.
The use of setenv OMPI_MCA_btl_openib_allow_ib 1 in the module file is required in order to get the correct connectivity with the Omnipath interconnect.
Installation was tested as follows as above with the example batch script modified
(Getting Started example from https://openfoam.org/download/8-source/) with the below decomposeParDict:
OpenFOAM 4.1
OpenFOAM 4.1 was installed using the
install_openfoam.sh script, the module
file is
gcc-6.2-openmpi-2.0.1. The following optional dependencies were built as part of the installation process: Scotch/PT-Scotch v6.0.3 (located in /usr/local/packages/apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1/ThirdParty-4.1). The following optional dependencies were not installed: ParaView and CGAL.
Installation was tested as follows (Getting Started example from https://openfoam.org/download/4-1-source/):
$ mkdir /data/$USER/tests/openfoam/run
$ cd /data/$USER/tests/openfoam/run
$ module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1
$ cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
$ cd pitzDaily
$ blockMesh
$ simpleFoam
$ paraFoam