Attention

The ShARC HPC cluster was decommissioned on the 30th of November 2023 at 17:00. It is no longer possible for users to access that cluster.

Nextflow

Nextflow is a free and open-source software distributed under the Apache 2.0 licence, developed by Seqera Labs. The software is used by scientists and engineers to write, deploy and share data-intensive, highly scalable, workflows on any infrastructure.

Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages.

Its fluent DSL simplifies the implementation and the deployment of complex parallel and reactive workflows on clouds and clusters

Note

nf-core is a community effort to collect a curated set of analysis pipelines built using Nextflow.

Interactive Usage

After connecting to ShARC (see Establishing a SSH connection), start an interactive session with the qrsh or qrshx command.

The latest version of Nextflow (currently version 22.4.0) is made available with the command:

$ module load apps/Nextflow/22.04.0/binary

You can now run the nextflow command:

  $ nextflow -version
  N E X T F L O W
  version 22.04.0 build 5697
  created 23-04-2022 18:00 UTC (19:00 BST)
  cite doi:10.1038/nbt.3820
  http://nextflow.io
  .

A Simple Script

Write a file named tutorial.nf (source) with the following content:

params.str = 'Hello world!'

process splitLetters {
  output:
    path 'chunk_*'

  """
  printf '${params.str}' | split -b 6 - chunk_
  """
}

process convertToUpper {
  input:
    path x
  output:
    stdout

  """
  cat $x | tr '[a-z]' '[A-Z]'
  """
}

workflow {
  splitLetters | flatten | convertToUpper | view { it.trim() }
}

Execute the script by entering the following command in your terminal:

$ nextflow run tutorial.nf

It will output something similar to the text shown below:

N E X T F L O W  ~  version 19.04.0
executor >  local (3)
[69/c8ea4a] process > splitLetters   [100%] 1 of 1 ✔
[84/c8b7f1] process > convertToUpper [100%] 2 of 2 ✔
HELLO
WORLD!

Batch usage

Ensure you have produced the above tutorial.nf script then write a file named batch.sh with the following content:

#!/bin/bash
# your email address
#$ -M a.person@sheffield.ac.uk
#$ -m eba
## set the number of cores
#$ -pe smp 2
## set max runtime to 1 minute (for this test)
#$ -l h_rt=00:01:00
## set max memory to 1Gb per core (default is 2G)
#$ -l rmem=1G
#$ -cwd
module load apps/Nextflow/22.04.0/binary
nextflow run tutorial.nf

You can now submit this job to the SLURM scheduler with

$ sbatch batch.sh

Your output file content will be similar to the following:

N E X T F L O W  ~  version 22.04.0
Launching `tutorial.nf` [thirsty_mahavira] DSL2 - revision: 7ed0e799f3
[-        ] process > splitLetters   -
[-        ] process > convertToUpper -

[-        ] process > splitLetters   [  0%] 0 of 1
[-        ] process > convertToUpper -

executor >  local (1)
[be/6c0c6b] process > splitLetters   [  0%] 0 of 1
[-        ] process > convertToUpper -

executor >  local (1)
[be/6c0c6b] process > splitLetters   [100%] 1 of 1 ✔
[-        ] process > convertToUpper -

executor >  local (2)
[be/6c0c6b] process > splitLetters       [100%] 1 of 1 ✔
[4b/4c2a74] process > convertToUpper (1) [  0%] 0 of 2

executor >  local (3)
[be/6c0c6b] process > splitLetters       [100%] 1 of 1 ✔
[1b/1ed03e] process > convertToUpper (2) [ 50%] 1 of 2
HELLO

executor >  local (3)
[be/6c0c6b] process > splitLetters       [100%] 1 of 1 ✔
[1b/1ed03e] process > convertToUpper (2) [ 50%] 1 of 2
HELLO

executor >  local (3)
[be/6c0c6b] process > splitLetters       [100%] 1 of 1 ✔
[1b/1ed03e] process > convertToUpper (2) [100%] 2 of 2 ✔
HELLO
WORLD!

Installation notes

This installation was tested with the above examples.

Version 22.04.0