Workflow Managers for HPC on November 25, 2024

On November 25 2024, the ICTS of KU Leuven organize a lunchbox session about workflow managers for HPC. The session will be hybrid (physical and online participation).

The purpose of this lunchbox session is to introduce few basic tools available on our Tier-2 clusters, and specifically allow users to present the specific workflow managers they use. The idea is to have an interactive session where users can share their experiences, and learn from one another, in order to promote using workflow manager tools for different scientific domains on our Tier-2 system.

In an HPC ecosystem, it is common to have a complex workflow to complete a full work cycle. For instance, input data needs to be staged in in advance, the input needs to be pre-processed, a number of jobs have to be submitted to process the data, and the output needs to be staged out to a remote project data inventory. In this workflow, a failed task needs to be resubmitted, and sometimes there exists a logical order among tasks.

Exploiting ‘workflow manager’ tools specific to HPC environments facilitates handling such complex HPC jobs. These tools are widely practiced in various scientific domains such as machine learning and AI, genomics research, large data analysis, etc. Some research teams have matured exploiting such tools like Ray, Nextflow, Snakemake. However, some individuals or research teams might not have yet adopted such workflow managers in their work, despite the huge benefits that such tools offer.

More information and registration can be found on https://admin.kuleuven.be/icts/onderzoek/hpc/workflows_4_hpc.