Using the cluster

The sim cluster consists of 4 compute nodes with 48 cores each (96 threads with hyperthreading) that are connected by a fast network. A resource manager, in our case SLURM, takes care of starting MPI applications on the nodes. So far, we have used interactive sessions on a single node, where we could only use up to 8 processes.

The normal process of using a cluster is to write a job script and submit the job to the queue. A scheduler decides the order in which all jobs will be executed. When its the turn for our job, it will allocate the requested number of nodes and run the job. On busy systems, it will give smaller or shorter jobs priority over longer and larger jobs as well as optimize for good utilization of the whole cluster.

A sample job script, job.sh is the following:

 1#!/bin/bash
 2#
 3#SBATCH --job-name=submission
 4#SBATCH --output=result.txt
 5#
 6#SBATCH --ntasks=16
 7#SBATCH --ntasks-per-node=4
 8#SBATCH --time=10:00
 9
10module use /usr/local.nfs/sgs/modulefiles
11module load gcc/10.2
12module load openmpi/3.1.6-gcc-10.2
13module load vtk/9.0.1
14module load cmake/3.18.2
15
16srun -n 16 ./numsim_parallel lid_driven_cavity.txt

The scripts starts with the Shebang line 1 and some parameters to the scheduler that start with #SBATCH. It requests the number of processes (line 6) and how many processes should be placed on every node (line 4), as well as the maximum runtime (line 8). Usually in High Performance Computing, it is too slow to use hyperthreading. The given example will allocate 4 nodes and request 4 cores per node. All console output of the job will be written to the file result.txt, as specified in line 4.

We again need to load the modules as previously and additionally set PATH and CPATH to a custom mpi version because the default OpenMPI does not work with SLURM.

Instead of using mpirun we need to use SLURM to launch the parallel program. The command srun takes care of this and is given the total number of processes. This has to be less than or equal to the requested number of processes in the header of the file.

Note that we should load the modules at the beginning of the job script.

The script can be submitted by the command

sbatch job.sh

You can list all submitted and running jobs with their job id with

$ squeue
           JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           12817       all submissi  maierbn  R       0:02      4 sgscl1n[1-4]

To cancel a job, run scancel with the job id:

$ scancel 12817