4. FAQ: Data Transfer on the HPC

4.1. How can I copy files to/from Hydra?

There are multiple options to move data in and out of Hydra. The section Data transfer lists the available tools in the cluster and has detailed instructions on how to use the following solutions:

• Your OneDrive in VUB

• Globus

• WebDAV services as Nextcloud/ownCloud

4.2. How can I automate the transfer of data to/from Hydra?

Automatic data transfers from external servers in your jobs scripts is described in the section Automatic data transfer in jobs.

4.3. How can I share files with other users?

There are two approaches available to share files with other users in Hydra:

• A Virtual Organization (VO) provides extra disk space in Hydra that is shared with the other members of your research group, where you can easily share any data files with them. Check also the docs on how to share a VO subdir.

• Globus allows sharing directories with other users via VSCdocGlobus data sharing.

If you want to setup a shared directory with other users in Hydra, but the available methods do not suit your needs, please contact VUB-HPC Support. We will evaluate the best solution for your use case.

4.4. How can I share a VO subdirectory with the other VO members?

Suppose that we have created a directory shared in $VSC_SCRATCH_VO, and we want to give the other VO members read/write permissions to this subdirectory. This can be done by changing the group of the shared directory recursively to the VO with chown -R and setting the setgid bit with chmod g+s:

cd $VSC_SCRATCH_VO
# replace bvoxxxxx with your VO ID
chown -R :bvoxxxxx shared
chmod -R g+s shared
ls -hld shared

If everything went well, the ls -hld command should show your VO in the 4th field, as well as the setgid bit:

output

drwxrwsr-x 266 vsc1xxxx bvoxxxxx 16K Sep  1 15:16 shared

The chmod g+s command ensures that any new files or directories created in shared will automatically inherit its group ownership. Note however that group ownership will NOT be updated for files or directories copied from another location. For those, you still have to change the permissions manually.

4.5. How can I recover/transfer data of a disabled VSC account?

Important

The best solution is always to arrange and transfer your data on your own before your VSC account is disabled. Check Data transfer for options on how to move data out of the HPC cluster while your account is active.

Depending on the status of your account you will have different options at hand. Former users first lose access to services needing authentication with your VUB such as Globus. At this stage, if you are still in Belgium, the best option might be to connect to the HPC and move data to your Virtual Organization (if you are in one) or to move data to your your OneDrive in VUB and share it with your colleagues from there.

If your VSC account was fully closed not long ago, your data might still be safeguarded in our systems. In such a case please contact support to transfer or recover your data as described in the section Sharing data from closed accounts.