1.3. Running a job on the cluster
1.3.1. Installing python on the cluster
1.3.1.1. At MPI-DS
To have a current version of python I recommend installing miniconda.
Download the Linux 64-bit installer and use scp
to put it in your home directory on the cluster.
Then ssh into the cluster and use the “Quick command line install” for Linux from the link above.
1.3.1.2. At GWDG
They have decent python.
Run module load python
when setting up the virtual environment for the first time.
Thereafter simply activating the virtual environment is enough.
1.3.2. Submission Scripts
A cluster job can be started using SubmitToCluster.sh
.
This script expects a YAML configuration file and a specification of the type of cluster run you want (see help for options).
Under the hood ClusterRun_<type>.py
takes care of initial setup, i.e. ensuring that data output directories exist and that the submission script is rendered according to the YAML configuration.
Under the hood drrc.Config.generate_submission_script_from_YAML()
parses the YAML configuration and fills a template for a specified queueing system.