1. Module System#
1.1. Where can I find installed software?#
Most end-user software on Hydra is available via modules that can be dynamically loaded by the user. That’s why it might seems that the software in Hydra is very limited, but it is not the case, we provide more than 2500 modules. To check the full list of installed software, type:
module av
Use the spider option to list all available versions of a given software
package:
module spider Python
Use the help option to list all extensions included in a given module:
module help Python/3.11.3-GCCcore-12.3.0
Modules can be loaded as follows:
module load Python/3.11.3-GCCcore-12.3.0
You can check which modules are currently loaded with:
module list
Unloading all currently loaded modules can be done like this:
module purge
See also
Helpdesk If you need software that is not yet installed in our cluster, we can install it.
1.2. How can I find specific Python or R packages?#
We extensively use module extensions to handle Python and R packages. Software modules can contain multiple extensions, which are additional software packages necessary or useful by the main software of the module.
Since 2023, the base Python and R modules only contain those extensions needed to build and install extra packages on top of them. Other Python or R packages missing in those modules may be provided in bundle modules or in their own specific modules. For instance, Python has the following bundles available:
Python-bundle-PyPI: collection of modules from PyPISciPy-bundle: collection of modules for scientific Python, includingnumpy,pandasandscipy
R has its own bundles of modules as well:
R-bundle-CRAN: collection of modules from CRANR-bundle-Bioconductor: collection of modules from Bioconductor
You can check the complete list of extensions in any module with the
command module help
module help SciPy-bundle/2023.07-gfbf-2023a
The command module spider can be used to search in Hydra for
specific Python or R packages, either as modules or extensions in bundles.
$ module spider ggplot2
--------------------------------------------------------
ggplot2:
--------------------------------------------------------
Versions:
ggplot2/3.3.6 (E)
ggplot2/3.4.4 (E)
Names marked by a trailing (E) are extensions provided by another module.
--------------------------------------------------------
For detailed information about a specific "Seurat" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider ggplot2/3.4.4
--------------------------------------------------------
In the previous example for the R package ggplot2, the command module
spider tells the available versions installed in the system. They are all
labelled with (E) because they are extensions of a module. You can
find the specific module that has a specific version of ggplot2 by executing
module spider with the full name of the extension
$ module spider ggplot2/3.4.4
--------------------------------------------------------
ggplot2: ggplot2/3.4.4 (E)
--------------------------------------------------------
This extension is provided by the following modules. To access the extension you must load one of the following modules. Note that any module names in parentheses show the module location in the software hierarchy.
R-bundle-CRAN/2023.12-foss-2023a
Names marked by a trailing (E) are extensions provided by another module.
Hence, loading the module R-bundle-CRAN/2023.12-foss-2023a will
make ggplot2 v3.4.4 loadable in our R environment.
Helpdesk If you need any R packages that are currently unavailable in Hydra, we can install them.
1.3. How can I find old software?#
Searching for software with module av or module spider will only show
software added in the last 2 years, older modules are hidden by default. You
can still search, load, and use old/hidden modules, but a warning message will
be shown upon loading, suggesting to upgrade to a more recent version.
To search for hidden modules, type:
module --show-hidden spider Python
Using the above command, all installed Python modules are shown.
1.4. How can I find legacy software?#
Legacy modules contain all software built for the former operating system. To
load those, first load the legacy-software module:
module load legacy-software
Note
Please be aware that this software stack is unavailable on the zen4
nodes. If you require legacy software, ensure you use the --partition
option when submitting your job, directing it to the appropriate partition(s).
Warning
While many legacy-software installations will work, we cannot guarantee their performance or functionality, as the software was built for a different operating system than the one currently used in the cluster. If you must continue using older software that is no longer working, we can provide specific solutions based on containers or, as a last resort, carry out re-installation, please contact VUB-HPC Support for help.
1.5. The toolchain of software packages#
The name of a software package such as TensorFlow/2.13.0-foss-2023a not
only contains the name of the package TensorFlow and its version
2.13.0, but also has the term foss-2023a, which is known as the
toolchain.
The toolchain foss is the set of tools used to build that package and it
has its own version 2023a. Software modules are made available in specific
toolchains depending on their requirements. The most common ones are:
- GCCcore
Toolchain based on the GNU Compiler Collection, which includes front ends for C, C++, Objective-C, Fortran, Java, and Ada. The
GCCcoreversion corresponds to the version of the GNU Compiler Collection (GCC).- GCC
GCCcoretoolchain with additional libraries provided bybinutils(libstdc++, libgcj,…).- gfbf
GCC-based compiler toolchain, including FlexiBLAS (BLAS and LAPACK support) and (serial) FFTW.
- gompi
GCC-based compiler toolchain, including OpenMPI for MPI support.
- foss
The Free and open-source software (FOSS) toolchain is based on
GCCcoreand also includes support for OpenMPI, OpenBLAS, FFTW and ScaLAPACK.- intel
Intel-based toolchain including Intel compilers, Intel MPI and Intel Math Kernel Library (MKL).
- iomkl
Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
Some toolchains offer improved performance on specialised hardware, for instance
certain software may be faster with the intel toolchain if used in compute
nodes with Intel hardware; whereas other toolchains offer additional features,
such as gompi which adds the MPI stack to GCC for inter-node
communication.
The name of software packages may contain after their toolchain specification
additional information regarding other dependencies of the package, such as
needed interpreters. For instance, the package
R-bundle-Bioconductor/3.15-foss-2022a-R-4.2.1 corresponds to Bioconductor
version 3.15 built with foss toolchain version 2022a and uses R
version 4.2.1.
Multiple modules can only be loaded together as long as their toolchains and toolchain versions are compatible. Otherwise, conflicts may occur.
1.5.1. Compatible toolchains#
Modules with different toolchains are generally not compatible, unless one
is a sub-toolchain of the other. Hence, a module in foss is not
compatible with another one in intel. The table below shows the
relation between the base toolchain GCCcore, sub-toolchains, and
full toolchains foss and intel:
base toolchain |
sub-toolchain |
sub-toolchain |
full toolchain |
|---|---|---|---|
GCCcore |
GCC |
gompi |
foss |
GCCcore |
GCC |
gfbf |
foss |
GCCcore |
iccifort |
iimpi |
intel |
1.5.2. Compatible toolchain versions#
Different versions of the same toolchain are always incompatible. For
example, you cannot load a module in foss-2022a or GCCcore-11.3.0
with another one in foss-2023a or GCCcore-12.3.0. Please check the
foss component table
showing the GCC or GCCcore version corresponding to a given foss
toolchain.
Helpdesk If you cannot find a compatible set of modules that provides all software required for your work, we can install them.