2. Specific Use Cases

2.1. MATLAB

MATLAB is available as a module, however it is not recommended to run intensive MATLAB calculations on Hydra: its performance is not optimal and parallel execution is not fully supported.

First check which MATLAB versions are available:

module spider MATLAB

Next load a suitable version:

module load MATLAB/2019b

Tip

Use the most recent version for new projects

It is possible to run MATLAB in console mode for quick tests. For example, with a MATLAB script called ‘testmatlab.m’, type:

matlab -batch "run('testmatlab.m');"

MATLAB scripts should be submitted to the queue in a job script. Before submitting, however, we highly recommend to first compile your script using the MATLAB compiler mcc (this can be done on the login node):

mcc -m testmatlab.m

This will generate a testmatlab binary file, as well as a ‘run_testmatlab.sh’ shell script (and a few other files). You can ignore the ‘run_testmatlab.sh’ file.

Now you can submit your matlab calculation as a batch job. Your job script should look like this

1
2
3
4
5
6
7
8
 #!/bin/bash
 #PBS -l walltime=01:00:00
 #PBS -l nodes=1:ppn=1

 module load MATLAB/2019b

 cd $PBS_O_WORKDIR
 ./testmatlab 2>&1 >testmatlab.out

The advantage of running a compiled matlab binary is that it does not require a license. We have only a limited number of MATLAB licenses that can be used at the same time, so in this way you can run your simulation even if the all licenses are in use.

See also

The mcc documentation for more information on using the MATLAB compiler.

2.2. R

Depending on your needs there are different methods to use R in Hydra

  • Interactive sessions should be performed in the compute nodes

    1. Start an interactive job in a compute node adjusting the number of cores ppn as needed (by default R only uses 1 core):

      qsub -I -l nodes=1:ppn=1
      
    2. Load your R module of choice (preferably a recent version):

      module load R/4.0.0-foss-2020a
      
    3. Start the interactive R shell:

      R
      
  • Scripts written in R can be executed with the command Rscript. A minimal job script for R only requires loading the R module and executing your scripts with Rscript

    1
    2
    3
    4
    5
    6
    7
    8
     #!/bin/bash
     #PBS -l walltime=01:00:00
     #PBS -l nodes=1:ppn=1
    
     module load R/4.0.0-foss-2020a
    
     cd $PBS_O_WORKDIR
     Rscript <path-to-script.R>
    

Tip

The quality of the graphics generated by R can be improved by changing the graphical backend to Cairo. Add the following lines to the file ~/.Rprofile to make these changes permanent for your user (create the file ~/.Rprofile if it does not exist)

1
2
3
4
5
6
 # Use cairo backend for graphics device
 setHook(packageEvent("grDevices", "onLoad"),
     function(...) grDevices::X11.options(type='cairo'))

 # Use cairo backend for bitmaps
 options(bitmapType='cairo')

2.3. Gaussian

The available modules for Gaussian can be listed with the command:

module --show-hidden spider Gaussian

We recommend using the module Gaussian/G16.A.03-intel-2017b for general use because its performance has been thoroughly optimized for Hydra. More recent modules, such as Gaussian/G16.B.01, should be used if you need any of their new features.

Gaussian jobs can use significantly more memory than the value specified by %mem in the input file or with g16 -m in the execution command. Therefore, it is recommended to submit Gaussian jobs requesting a total memory that is at least 20% larger than the memory value defined in the calculation.

Gaussian G16 should automatically manage the available resources and parallelization. However, it is known to under-perform in some circumstances and not use all cores allocated to your job. In Hydra, the command myresources will report the actual use of resources of your jobs. If any of your Gaussian calculations is not using all available cores, it is possible to force the total number of cores used by Gaussian G16 with the option g16 -p or by adding the Gaussian directive %nprocshared to the top of the input file.

The following job script is an example to be used for Gaussian calculations running on 1 node with multiple cores. In this case we are running a g16 calculation with 80GB of memory (-m=80GB), but requesting a total of 96GB of memory (20% more). Additionally, we are requesting 20 cores for this job and automatically passing this setting to g16 with the option -p=${PBS_NP}, where ${PBS_NP} is an environment variable that contains the number of cores allocated to your job.

1
2
3
4
5
6
7
8
 #!/bin/bash
 #PBS -l nodes=1:ppn=20
 #PBS -l mem=96GB

 ml Gaussian/G16.A.03-intel-2017b

 cd $PBS_O_WORKDIR
 g16 -p=${PBS_NP} -m=80GB < input_file.com > output_file.log

2.4. GaussView

GaussView is a graphical interface used with the computational chemistry program Gaussian. GaussView is installed in Hydra and can be used alongside Gaussian to enable all property visualizations.

  1. Login to Hydra enabling X11 forwarding. Linux and macOS users can do so by adding the option -Y to the ssh command used for login. See below:

    ssh -Y <username>@login.hpc.vub.be
    
  2. Load the modules of GaussView

    • GaussView 6 with Gaussian/G16.A.03:

      module load GaussView/6.0.16
      
    • GaussView 6 with Gaussian/G16.B.01:

      module load Gaussian/G16.B.01
      module load GaussView/6.0.16
      
  3. Launch GaussView:

    gview.sh
    

Keep in mind that using a graphical interface in Hydra is currently rather slow. Thus, for regular visualization tasks, the HPC team recommends installing GaussView in your personal computer. Binary packages of GaussView are available for Linux, Mac, and Windows and are provided upon request to VUB-HPC Support.

2.5. matplotlib with a graphical interface

The HPC environment is optimized for the execution of non-ineractive applications in job scripts. Therefore, matplotlib is configured with a non-GUI backend (Agg) that can save the resulting plots in a variety of image file formats. The generated image files can be copied to your own computer for visualization or further editing.

If you need to work interactively with matplotlib and visualize its output from within Hydra, you can do so with the following steps

  1. Login to Hydra enabling X11 forwarding. Linux and macOS users can use the following command:

    ssh -Y username@login.hpc.vub.be
    
  2. Enable the TkAgg backend at the very beginning of your Python script:

    1
    2
     import matplotlib
     matplotlib.use('TkAgg')
    

    Note

    The function matplotlib.use() must be done before importing matplotlib.pyplot. Changing the backend parameter in your matplotlibrc file will not have any effect as the system-wide configuration takes precedence over it.

2.6. CESM/CIME

The dependencies required to run CESM in Hydra are provided by the module CESM-deps. This module also contains the XML configuration files for CESM with the specification of machines, compiler and batch system of Hydra. Once CESM-deps is loaded, the configuration files can be found in ${EBROOTCESMMINDEPS}/machines.

The following steps show an example setup of a CESM/CIME case

  1. Load the module CESM-deps:

    module load CESM-deps/2-intel-2019b
    
  2. Data files for CESM have to be placed in $VSC_SCRATCH/cesm (DIN_LOC_ROOT):

    mkdir $VSC_SCRATCH/cesm
    

    Users needing data located elsewhere (e.g. in a Virtual Organization) can create symlinks in their $VSC_SCRATCH to the corresponding location:

    ln -sf $VSC_DATA_VO/cesm $VSC_SCRATCH/cesm
    
  3. Create the following folder structure for your CESM/CIME cases in $VSC_SCRATCH:

    mkdir $VSC_SCRATCH/cime
    mkdir $VSC_SCRATCH/cime/cases
    mkdir $VSC_SCRATCH/cime/output
    
  4. Download the source code of CESM/CIME into $VSC_SCRATCH/cime:

    Clone a public release of CESM
     cd $VSC_SCRATCH/cime
     git clone -b release-cesm2.1.3 https://github.com/ESCOMP/cesm.git cesm-2.1.3
    
    Clone external sources of CIME
     cd $VSC_SCRATCH/cime/cesm-2.1.3
     ./manage_externals/checkout_externals
    
  5. Add the configuration settings for Hydra and Breniac to your CESM/CIME source code:

    cd $VSC_SCRATCH/cime/cesm-2.1.3
    update-cesm-machines cime/config/cesm/machines/ $EBROOTCESMMINDEPS/machines/ machines compilers batch
    
  6. Optional Add support for iRODS

    Determine your version of CIME
     $ cd $VSC_SCRATCH/cime/cesm-2.1.3
     $ git -C cime/ describe --tags
     cime5.6.32
    
    Apply the patches for the closest CIME version in $EBROOTCESMMINDEPS/irods
     $ cd $VSC_SCRATCH/cime/cesm-2.1.3
     $ git apply $EBROOTCESMMINDEPS/irods/cime-5.6.32/*.patch
    
  7. The creation of a case follows the usual procedure for CESM:

    cd $VSC_SCRATCH/cime/cesm-2.1.3/cime/scripts
    ./create_newcase --case $VSC_SCRATCH/cime/cases/name_of_case --res f19_g17 --compset I2000Clm50BgcCrop
    
  8. Your CESM case in $VSC_SCRATCH/cime/cases/name_of_case can now be setup, built and executed. We provide a job script called case.job to automatically perform all these steps in the compute nodes, minimizing wait times in the queue and ensuring that the nodes building and running the case are compatible. You can copy the template of case.job from $EBROOTCESMMINDEPS/scripts/case.job to your case and modify it as needed (adding xmlchange or any other commands). Once the script is adapted to your needs, submit it to the queue with qsub as usual:

    Copy job template to your case folder
     cd $VSC_SCRATCH/cime/cases/name_of_case
     cp $EBROOTCESMMINDEPS/scripts/case.job $VSC_SCRATCH/cime/cases/name_of_case/
    
    Edit case.job if needed
     $EDITOR case.job
    
    Submit your CESM job (adjust computational resources as needed)
     qsub -l nodes=2:ppn=20 -l walltime=24:00:00 case.job
    

The module CESM-tools provides a set of tools commonly used to analyse and visualize CESM data. Nonetheless, CESM-tools cannot be loaded at the same time as CESM-deps because their packages have incompatible dependencies. Once you obtain the results of your case, unload any modules with module purge and load CESM-tools/2-foss-2019a to post-process the data of your case.

2.7. GAP

The GAP shell has a strong focus on being used interactively, whereas on Hydra the preferred way to run calculations is by submitting job scripts. Nonetheless, it is possible to use the interactive shell of GAP in our compute nodes with the following steps

  1. Request an interactive job session and wait for it to be allocated:

    $ qsub -I -l nodes=1:ppn=4 -l walltime=3:00:00
    qsub: waiting for job 3036036.master01.hydra.brussel.vsc to start
    qsub: job 3036036.master01.hydra.brussel.vsc ready
    vsc10xxx@node361 ~ $
    
  2. Load the module of GAP and start its shell as usual:

    vsc10xxx@node361 ~ $ module load gap/4.11.0-foss-2019a
    vsc10xxx@node361 ~ $ gap
    ********* GAP 4.11.0 of 29-Feb-2020
    * GAP * https://www.gap-system.org
    ********* Architecture: x86_64-pc-linux-gnu-default64-kv7
    [...]
    gap>
    

Submitting a job script using GAP is also possible and requires preparing two scripts. One is the usual job script to be submitted to the queue and the second one is the script with the commands for GAP.

  • The job script is a standard job script requesting the resources needed by your calculation, loading the required modules and executing the script with the code for GAP.

    Example job script for GAP
    1
    2
    3
    4
    5
    6
    7
    8
     #!/bin/bash
     #PBS -l walltime=00:60:00
     #PBS -l nodes=1:ppn=4
    
     module load gap/4.11.0-foss-2019a-modisomTob
    
     cd $PBS_O_WORKDIR
     ./gap-script.sh
    
  • The script gap-script.sh is a shell script that executes GAP and passes your code to it. It is necessary to execute GAP with the -A option and only load the required GAP packages at the beginning of your script to avoid issues.

    Example script to execute GAP in batch mode
    1
    2
    3
    4
    5
     #!/bin/bash
     gap -A -r -b -q << EOI
     LoadPackage( "Example" );
     2+2;
     EOI
    

    Important

    Keep in mind to make gap-script.sh executable with the command chmod +x gap-script.sh

2.8. Mathematica

First check which Mathematica versions are available:

module spider Mathematica

Next load a suitable version:

module load Mathematica/12.0.0

Tip

Use the most recent version for new projects

Run Mathematica in console mode in the terminal for quick tests:

wolframscript

Mathematica scripts (Wolfram Language Scripts) should be submitted to the queue in a job script. In the following example, we run the Mathematica script testmath.wls

Example job script for Mathematica
1
2
3
4
5
6
7
8
 #!/bin/bash
 #PBS -l walltime=01:00:00
 #PBS -l nodes=1:ppn=1

 module load Mathematica/12.0.0

 cd $PBS_O_WORKDIR
 wolframscript -file testmath.wls

Tip

Mathematica code is not optimized for performance. However, it supports several levels of interfacing to C/C++. For example, you can speed up your compute intensive functions by compiling them with a C compiler from inside your Mathematica script.

2.9. Stata

First check which Stata versions are available:

module spider Stata

Next load a suitable version:

module load Stata/16-legacy

Tip

Use the most recent version for new projects

Running Stata in console mode in the terminal for quick tests:

stata

Stata do-files should be submitted to the queue in a job script. In the following example, we run the Stata program teststata.do

1
2
3
4
5
6
7
8
 #!/bin/bash
 #PBS -l walltime=01:00:00
 #PBS -l nodes=1:ppn=1

 module load Stata/16-legacy

 cd $PBS_O_WORKDIR
 stata-se -b do teststata

Upon execution, Stata will by default write its output to the log file teststata.log.

Note

The recommended version of Stata in batch mode is stata-se, because it can handle the larger datasets.

2.10. GROMACS

2.10.1. Threading Models

GROMACS supports two threading models, which can be used together:

  • OpenMP threads

  • thread-MPI threads: MPI-based threading model implemented as part of GROMACS, incompatible with process-based MPI models such as OpenMPI

There are two variants of the GROMACS executable:

  • gmx: recommended for all single-node jobs, supports both OpenMP threads and thread-MPI threads

  • gmx_mpi: for multi-node jobs: must be used with mpirun, only supports OpenMP threads

The number of threads must always be specified, as GROMACS sets it incorrectly on Hydra:

  • gmx: use option -nt to let GROMACS determine optimal numbers of OpenMP threads and thread-MPI threads

  • gmx_mpi: use option -ntomp (not -ntmpi or -nt), and set number of threads equal to 1.

Running on 1 or more GPUs, by default GROMACS will:

  • detect the number of available GPUs, create 1 thread-MPI thread for each GPU, and evenly divide the available CPU cores between the GPUs using OpenMP threads. Therefore, ppn should be a multiple of gpus. Always check in the log file that the correct number of GPUs is indeed detected.

  • optimally partition the force field terms between the GPU(s) and the CPU cores, depending on the number of GPUs and CPU cores and their respective performances.

2.10.2. Job Scripts

To get good parallel performance, GROMACS must be launched differently depending on the requested resources (#nodes, #cores, and #GPUs). In the example job scripts given below, a molecular dynamics simulation is launched with run input file example.tpr:

  • single-node, multi-core

    1
    2
    3
    4
    5
    6
    7
    8
     #!/bin/bash
     #PBS -l walltime=01:00:00
     #PBS -l nodes=1:ppn=4
    
     module load GROMACS/2020.1-foss-2020a-Python-3.8.2
    
     cd $PBS_O_WORKDIR
     gmx mdrun -nt $PBS_NP -s example.tpr
    
  • multi-node

    1
    2
    3
    4
    5
    6
    7
    8
     #!/bin/bash
     #PBS -l walltime=01:00:00
     #PBS -l nodes=2:ppn=4
    
     module load GROMACS/2020.1-foss-2020a-Python-3.8.2
    
     cd $PBS_O_WORKDIR
     mpirun -np $PBS_NP gmx_mpi mdrun -ntomp 1 -s example.tpr
    
  • single-GPU, single-node, multi-core

    1
    2
    3
    4
    5
    6
    7
    8
     #!/bin/bash
     #PBS -l walltime=01:00:00
     #PBS -l nodes=1:ppn=4:gpus=1
    
     module load GROMACS/2019.3-fosscuda-2019a
    
     cd $PBS_O_WORKDIR
     gmx mdrun -nt $PBS_NP -s example.tpr
    
  • multi-GPU, single-node, multi-core

    1
    2
    3
    4
    5
    6
    7
    8
     #!/bin/bash
     #PBS -l walltime=01:00:00
     #PBS -l nodes=1:ppn=8:gpus=2
    
     module load GROMACS/2019.3-fosscuda-2019a
    
     cd $PBS_O_WORKDIR
     gmx mdrun -nt $PBS_NP -s example.tpr
    

See also

The chapter Getting good performance from mdrun in the GROMACS manual for more information on running GROMACS efficiently.

2.11. CP2K

To get good parallel performance with CP2K in Hydra, it is important to disable multi-threading. Below is an example job script which runs the CP2K input file ‘example.inp’

1
2
3
4
5
6
7
8
9
 #!/bin/bash
 #PBS -l walltime=01:00:00
 #PBS -l nodes=1:ppn=4

 module load CP2K/6.1-intel-2018a
 export OMP_NUM_THREADS=1

 cd $PBS_O_WORKDIR
 mpirun cp2k.popt -i example.inp -o example.out

2.12. PyTorch with optimal parallelization

PyTorch internally uses threads to run in parallel, but it makes some assumptions to determine the number of threads that do not apply to Hydra. This usually results in jobs with too many threads, saturating the allocated cores and hindering its performance. For optimal performance, the number of threads should be equal to the number of cores available to your job. PyTorch can be configured to follow this rule by adding the following lines near the beginning of your Python script:

1
2
 torch.set_num_threads(int(os.environ['PBS_NP']))
 torch.set_num_interop_threads(1)

Note

If you are using Python multiprocessing on top of PyTorch, then your job will generate N threads per each PyTorch process and the total number of threads will be (#threads * #processes). In this case, you should adapt your #threads and #processes such that the total number of threads is still equal to the number of cores allocated to your job.