1. Module System

1.1. Where can I find installed software?

Most end-user software on Hydra is available via modules that can be dynamically loaded by the user. That’s why it might seems that the software in Hydra is very limited, but it is not the case, we provide more than 2500 modules. To check the full list of installed software, type:

module av

Use the spider option to list all available versions of a given software package:

Example to list all modules of Python
 module spider Python

Use the show option to list all extensions included in a given module:

Example to list all extensions in the module of Python v3.8.6
 module show Python/3.8.6-GCCcore-10.2.0

Modules can be loaded as follows:

Example to load Python v3.8.6
 module load Python/3.8.6-GCCcore-10.2.0

You can check which modules are currently loaded with:

module list

Unloading all currently loaded modules can be done like this:

module purge

Helpdesk If you need software that is not yet installed in our cluster, we can install it.

1.2. How can I find old software?

Searching for software with module av or module spider will only show software added in the last 2 years, older modules are ‘hidden’ by default. You can still search, load, and use old/hidden modules, but a warning message will be shown upon loading, suggesting to upgrade to a more recent version.

To search for hidden modules, type:

Example to list all hidden and visisble Python modules
 module --show-hidden spider Python

1.3. The toolchain of software packages

The name of a software package such as Python/3.6.6-foss-2018b not only contains the name of the package Python and its version 3.6.6, but also has the term foss-2018b, which is known as the toolchain. The toolchain is the set of tools used to build that package and it has its own version 2018b. Any specific software can be made available with different toolchains. The most common ones are

GCCcore

Toolchain based on the GNU Compiler Collection, which includes front ends for C, C++, Objective-C, Fortran, Java, and Ada.

GCC

GCCcore toolchain with additional libraries provided by binutils (libstdc++, libgcj,…).

foss

The Free and open-source software (FOSS) toolchain is based on GCCcore and also includes support for OpenMPI, OpenBLAS, FFTW and ScaLAPACK.

intel

Intel compilers, Intel MPI and Intel Math Kernel Library (MKL).

iomkl

Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.

fosscuda

The foss toolchain with support for CUDA.

Some toolchains offer improved performance on specialised hardware, for instance certain software may be faster with the intel toolchain if used in compute nodes with Intel hardware; whereas other toolchains offer additional features, such as fosscuda which is needed to execute code on GPUs.

The name of software packages may contain after the toolchain specification additional information regarding other dependencies of the package, such as needed interpreters. For instance, the package R-bundle-Bioconductor/3.7-foss-2018b-R-3.5.1 corresponds to Bioconductor version 3.7 built with the foss toolchain 2018b and requires R version 3.5.1.

It is important to only load modules built with a common toolchain (including its version), otherwise conflicts may occur. Hence, R-bundle-Bioconductor/3.7-foss-2018b-R-3.5.1 can be loaded along with other packages build with the foss-2018b toolchain, but not any other. The only exception to this rule are packages built with the GCCcore toolchain, which is compatible with both the foss and intel toolchains. Note that GCCcore is not the same as GCC, which is incompatible with the intel toolchain. The compatibility between toolchains is version dependent, some recent ones being:

  • GCCcore-10.2.0 is compatible with foss-2020b and intel-2020b

  • GCCcore-9.3.0 is compatible with foss-2020a and intel-2020a

  • GCCcore-8.3.0 is compatible with foss-2019b and intel-2019b

  • GCCcore-8.2.0 is compatible with foss-2019a and intel-2019a

Helpdesk If you cannot find a compatible set of modules that provides all software required for your work, we can install them.

1.4. How can I build/install additional software/packages?

You should first check if the software is already available in Hydra, either with its own module or as part of another module. If the software package you need is not available, we strongly recommend to request its installation to VUB-HPC Support. Installations carried out by VUB-HPC have several advantages:

  1. The HPC team will optimize the compilation for each CPU architecture present in Hydra, guaranteeing that your software/package runs efficiently on all nodes (and usually much faster than installations made by the users).

  2. Free software will be available to all users of Hydra and licensed software can be made available to specific groups of users.

  3. The package will be built in a reproducible way with EasyBuild: important for scientific reproducibility.

  4. Different versions of the software can be installed alongside each other.

If you still want to install additional software/packages yourself, you can find guidance for specific development environments in the questions below.

1.4.1. Compiling and testing your software on the HPC

We strongly recommend to use a suitable buildenv module to compile software in Hydra. This ensures that you work in a control and reproducible environment.

Example loading a build environment for toolchain foss/2019b
 module load buildenv/default-foss-2019b

The buildenv module:

  • loads the compiler and any math and/or MPI libraries that may be included in the respective toolchain

  • defines compiler flags for optimal performance: CFLAGS, FFLAGS, CXXFLAGS, LIBBLAS, LIBLAPACK, LIBFFT, …

  • defines flags and paths to make sure the build system finds the right libraries: LIBRARY_PATH, LD_LIBRARY_PATH, LDFLAGS, …

It is possible to load additional development tools compatible with the buildenv module. For instance, the following modules can be loaded alongside buildenv/default-foss-2019b:

  • CMake/3.15.3-GCCcore-8.3.0

  • Autotools/20180311-GCCcore-8.3.0

  • pkg-config/0.29.2-GCCcore-8.3.0

  • Ninja/1.9.0-GCCcore-8.3.0

  • Meson/0.51.2-GCCcore-8.3.0-Python-3.7.4

Users compiling their own software should be aware that software compiled on the login nodes may fail in older compute nodes if full hardware optimization is used. The CPU microarchitecture of the login nodes (Skylake) has some instruction sets not available in Hydra’s older compute nodes (Ivy Bridge and Broadwell). Therefore, there are two options to compile your own software

Best performance

Compile on the login node (with -march=native). The resulting binaries can only run on Skylake nodes, but they offer the best performance on those nodes. Jobs can be restricted to run on Skylake nodes with the qsub option -l feature=skylake.

Best compatibility

Compile on any Ivy Bridge node. Login to an Ivy Bridge node with qsub -I -l feature=ivybridge and compile your code on it. The resulting binaries can run on any node on Hydra with good performance. Alternatively, users knowing how to setup the compilation can compile on the login node with -march=ivybridge -mtune=skylake.

Helpdesk The environment in the HPC might differ significantly from your development system. We can help you in case of problems or questions to transfer your development to the HPC.

See also

VSC Docs: Software development for more information.

1.4.2. Installing additional Python packages

Helpdesk There is a large number of Python packages already available in Hydra, see for example the packages included in Python and Scipy-bundle modules. If the package you need is not available, we can install it for you.

  • If you would like to test some new Python package before requesting its installation, you can do so by using your personal site-packages directory in your home:

    1. Load the appropriate Python module, i.e. the specific Python version to use with the package:

      module load Python/3.8.6-GCCcore-10.2.0
      
    2. Install the Python package with pip in your user account. The following command will also download and install all required dependencies. The new package and all missing dependencies will be installed by default in ~/.local/lib/pythonX.Y/site-packages:

      pip install --user <new_python_package>
      
  • Developers, who require using/testing software in Python that is in active development, can also use pip to install their own packages in a personal site-packages directory:

    1. Load the appropriate Python module and install the Python package from the local directory containing the source code. In this case pip will also install any missing dependencies:

      module load Python/3.8.6-GCCcore-10.2.0
      pip install --user /path/to/your/source/code
      
    2. Optional Updating the installation can be done at any time with the command:

      pip install --user --no-deps --ignore-installed /path/to/your/source/code
      

1.4.3. Installing additional R packages

There are already many packages included in the library of R in Hydra. The complete list can be looked up for each R version in Hydra with the command module show

Example to list all extensions in the module of R v4.0.0
 module show R/4.0.0-foss-2020a

R packages missing in the main R module may be provided with their own module. Use the module command to search in the repository of Hydra. Please see Where can I find installed software?

Helpdesk If you need any R packages that are currently unavailable in Hydra, we can install them.

Developers can compile and install R packages in the local R library of their home directory. The R function install.packages() will specifically ask to use your personal library. Keep in mind that if your software requires code compilation beyond R, you might need a build environment as described in Compiling and testing your software on the HPC.

The packages of your personal R library can potentially break the rest of R packages provided in Hydra. This is due to conflicts with the global R library or due to a version change of R after the installation of local R packages. If you experience errors running R scripts that are related to a failed load of a package, it is helpful to check your script in a clean R environment without your personal R library:

  1. Remove all modules and load the desired version of R:

    module purge
    module load R/4.0.0-foss-2020a
    
  2. Enter into a clean R environment (not loading previous workspace):

    R --no-restore
    
  3. Use .libPaths inside the shell of R or at the beginning of your R script to disable your personal R library:

    $ R
    > .libPaths('')
    > your_code...
    

1.4.4. Installing additional Perl packages

See VSC Docs: Perl package management

1.5. How can I run graphical applications?

There are 2 methods to run graphical applications in Hydra: X11 forwarding and Remote desktop sharing.

1.5.1. X11 forwarding

X11 forwarding is the simplest solution, but also the slowest. This option is suitable for applications with minimal graphical requirements.

  • Linux and MacOS

    Users can enable X11 forwarding upon login by adding the option -Y to the ssh command:

    ssh -Y vsc10xxx@login.hpc.vub.be
    
  • Windows

    Users can take advantage of the built-in X11 support in MobaXterm, see VSC Docs: Setup MobaXterm. By default X11 forwarding is enabled; you can check this setting under the Advanced SSH settings tab in the SSH Session settings window.

After login, you can test whether X11 is enabled by running the command xclock in the login node. If it works, a clock will be displayed in a new window. Please keep in mind that the login node is restricted to graphical applications that perform non-intensive compute tasks, such as visualization tools.

Compute intensive graphical apps should be executed in a compute node. This can be achieved by starting an interactive job and adding the option -X to the qsub command:

qsub -I -X

Tip

To avoid having to keep your SSH connection active for the duration of the interactive job, you can submit your interactive job in a tmux session.

1.5.2. Remote desktop sharing

Alert Setting up remote desktop sharing is a complex multi-step process, recommended only for advanced users.

For applications that are graphics-heavy we provide an alternative solution to X11 forwarding that is more efficient, called Virtual Network Computing (VNC). The steps are outlined below.

  1. Create a VNC password

    The VNC password should be created only once and it will be used in all future connections to your VNC sessions in Hydra. Type the following command to set the password:

    vncpasswd
    

    When asked about a view-only password, reply no. The entered password is obfuscated and stored by default in the file ~/.vnc/passwd.

  2. Start a VNC server

    The following command starts a VNC server inside a job in a compute node. Adapt the requested resources and the screen resolution as needed:

    (echo vncserver -geometry 1600x900 -fg) | qsub -l walltime=12:0:0 -N VNC -e VNC.err -
    
  3. Verify that the VNC server has successfully started

    Once the VNC job has started, look for a line that starts with ‘New’ in your job error file (VNC.err):

    grep '^New' VNC.err
    

    If your VNC session is running, then a line similar to the following should be shown:

    1
    2
    $ grep '^New' VNC.err
    New 'node379.hydra.os:1 (vsc10xxx)' desktop is node379.hydra.os:1
    
  4. Take note of the compute node and the display number of your VNC session, which will be needed in the next steps. In the example above with node379.hydra.os:1, we have:

    • compute node: node379

    • display number: 1

  5. Start a VNC client in your local computer

    • Linux and macOS

      Linux and macOS users can use TigerVNC from the command line, which allows connecting via the gateway (the login node) in a single command. Linux users can install it via their package manager; macOS users can install it via Homebrew.

      • Adapt the following command: replace node379:1 with the compute node and display number of your VNC session:

        vncviewer node379:1 -via vsc10xxx@login.hpc.vub.be -maximize
        
      • A new window asking your password should pop up. Enter your VNC server password. The VNC session should now appear in a new window.

    • Windows

      Windows users can use the VNC client that is included in recent MobaXterm versions.

      • Click icon Session in the top toolbar to create a new session. A new Session settings window should pop up:

      ../../../_images/session_settings.png
      • Click icon VNC in the top toolbar.

      • Under Basic VNC settings, fill out:

      • Click tab Advanced settings and check Use new embedded VNC engine.

      • Click tab Network settings and click the rectangular bar SSH gateway (jump host). A new window MobaXterm jump hosts configuration should pop up:

      ../../../_images/jump_hosts.png
      • Fill out:

        • Gateway host: login.hpc.vub.be

        • Username: vsc10xxx (replace with your VSC-ID)

        • Port: 22

        • Select Use SSH key and click the little file icon to browse to your private SSH key.

        • Click OK.

      • Back in the window Session settings, click OK. A new window asking your password should pop up.

      • Enter your VNC server password. The VNC session should now appear in a new tab inside the MobaXterm interface:

      ../../../_images/vnc_client_running.png