4. Installing additional software#
Before you start building/installing your own software/packages, you should first check if the software is already available in Hydra, either with its own module or as part of another module. If the software package you need is not available, we strongly recommend to request its installation to VUB-HPC Support. Installations carried out by VUB-HPC have several advantages:
The HPC team will optimize the compilation for each CPU architecture present in Hydra, guaranteeing that your software/package runs efficiently on all nodes (and usually much faster than installations made by the users).
Free software will be available to all users of Hydra and licensed software can be made available to specific groups of users.
The package will be built in a reproducible way with EasyBuild: important for scientific reproducibility.
Different versions of the software can be installed alongside each other.
If you still want to install additional software/packages yourself, you can find guidance for specific development environments in the sections below.
4.1. Compiling and testing your software on the HPC#
We strongly recommend to use a suitable
buildenv module to compile software
in Hydra. A
buildenv module loads pre-defined collections of build tools and
compilers, ensuring that you work in a controlled and reproducible environment.
module load buildenv/default-foss-2021a
loads the compiler and any math and/or MPI libraries that may be included in the respective toolchain
defines compiler flags for optimal performance: CFLAGS, FFLAGS, CXXFLAGS, LIBBLAS, LIBLAPACK, LIBFFT, …
defines flags and paths to make sure the build system finds the right libraries: LIBRARY_PATH, LD_LIBRARY_PATH, LDFLAGS, …
If needed you can load additional development tools compatible with the buildenv
module. For instance, the following modules can be loaded alongside
Users compiling their own software should be aware that software compiled on the login nodes may fail in older compute nodes if full hardware optimization is used. The CPU microarchitecture of the login nodes (Skylake) has some instruction sets not available in Hydra’s older compute nodes (Ivy Bridge and Broadwell). Therefore, there are two options to compile your own software
- Best performance
Compile on the login node (with
-march=native). The resulting binaries can only run on Skylake nodes, but they offer the best performance on those nodes. Jobs can be restricted to run on Skylake nodes with the Slurm option
- Best compatibility
Compile on any Ivy Bridge node. Login to an Ivy Bridge node with
srun --partition=ivybridge_mpi --pty bash -l
and compile your code on it. The resulting binaries can run on any node on Hydra with decent performance. Alternatively, users knowing how to setup the compilation can compile on the login node with
Helpdesk The environment in the HPC might differ significantly from your development system. We can help you in case of problems or questions to transfer your development to the HPC.
VSC Docs: Software development for more information.
4.2. Installing additional Python packages#
Helpdesk There is a large number of Python packages already available in Hydra, see the question How can I find specific Python or R packages? If the package you need is not available, we can install it for you.
If you would like to test some new Python package before requesting its installation, you can do so by using your personal
site-packagesdirectory in your home:
Load the appropriate Python module, i.e. the specific Python version to use with the package:
module load Python/3.9.5-GCCcore-10.3.0
Install the Python package with
pipin your user account. The following command will also download and install all required dependencies. The new package and all missing dependencies will be installed by default in
pip install --user <new_python_package>
Developers, who require using/testing software in Python that is in active development, can also use
pipto install their own packages in a personal
Load the appropriate Python module and install the Python package from the local directory containing the source code. In this case
pipwill also install any missing dependencies:
module load Python/3.9.5-GCCcore-10.3.0 pip install --user /path/to/your/source/code
Optional Updating the installation can be done at any time with the command:
pip install --user --no-deps --ignore-installed /path/to/your/source/code
4.3. Python virtual environments#
A virtual environment is an isolated Python environment in which you can safely install Python packages, independent from those installed in the system or in other virtual environments. For Python developers, using virtual environments is very convenient as it allows working on multiple software projects at the same time. On the other hand, as explained in section Additional Software, it is highly recommended to use the software modules as much as possible.
In this section, we show how you can combine modules with virtual environments in Hydra to get the best of two worlds.
Load a Python module that is suitable for the Python packages that will be installed:
module load Python/3.9.6-GCCcore-11.2.0
Optional Load modules with additional Python packages.
The Python module already includes a long list of Python packages, but many other modules are available. An often used module is
SciPy-bundle, a bundle of data science packages such as
module load SciPy-bundle/2021.10-foss-2021b
Create a virtual environment with virtualenv.
The following command will create a new directory
myenv, where your new
myenvenvironment will be stored:
virtualenv --system-site-packages myenv
Before we can use the virtual environment, we must activate it.
The currently active environment will be added in front of the prompt:
$ source myenv/bin/activate (myenv) $
It’s recommended to upgrade
pipto the latest version:
(myenv) $ python -m pip install --upgrade pip
Now we can install additional Python packages, or different versions of available packages, in the the virtual environment.
For example, to install a version of the
requestspackage that is different from the version included in the Python module, you can type:
(myenv) $ python -m pip install requests==2.27.1
To exit the virtual environment, use the command
The environment name will be removed from the prompt:
(myenv) $ deactivate $
To reuse the virtual environment:
Load the same software modules that you loaded when you created the virtual environment
In this example:
module load Python/3.9.6-GCCcore-11.2.0 SciPy-bundle/2021.10-foss-2021b
Reactivate the environment:
4.4. Installing additional R packages#
Developers can compile and install R packages in the local R library of their
home directory. The R function
install.packages() will specifically ask to
use your personal library. Keep in mind that if your software requires code
compilation beyond R, you might need a build environment as described in
Compiling and testing your software on the HPC.
Handling a personal R library in Hydra can be tricky though, it can easily break
the rest of R packages provided by software modules (i.e the
This can be due to conflicts with the global R library, issues with the multiple
CPU micro-architectures in the compute nodes or due to a version change of R
after the installation of local R packages.
If you experience errors running R scripts that are related to a failed load of a package, it is helpful to check your script in a clean R environment without your personal R library:
Remove all modules and load the desired version of R:
module purge module load R/4.1.0-foss-2021a
Disable the R library in your home directory:
Enter into a clean R environment (not loading previous workspace):
You can check the paths where R will look for the requested packages
(i.e. after a call to
library()) with the function
paths at the beginning of the list have precedence over the rest.