What is a batch job? 1/2

• On a laptop you might be used to start a program (job) by clicking an icon, which starts the job instantly
• If we start many jobs at the same time, we occasionally encounter problems like running out of memory etc.
• In an HPC environment, the computer is shared among hundreds to thousands of other users who all have different resource needs
• HPC batch jobs include a resource request, which corresponds to an estimate of how much resources the job is expected to use

What is a batch job? 2/2

• A batch job consists of two parts: A resource request and the actual computing step
• A job does not start directly, but is sent to a queue
• Depending on the requested resources and current load on the system, the job may need to wait for a while before starting
• At CSC (and HPC systems in general), all heavy computing must be done via batch jobs (see our usage policy)

What is a batch job system?

• A resource management system that keeps track of all jobs that use, or would like to use, the computing resources
• Aims to share the resources in an efficient and fair way among all users
• Optimizes resource usage by filling the compute nodes so that there will be as little idling resources as possible

Queueing and fair share of resources

• A job is queued and starts when the requested resources become available
• The order in which the queued jobs start depends on their priority and currently available resources
• At CSC, the priority is configured to use “fair share”
  • The initial priority of a job decreases if the user has recently run lots of jobs
  • Over time (while queueing) its priority increases and eventually it will run
  • Some queues have a lower priority (e.g. longrun – use shorter if you can!)
• See our documentation for more information on Getting started with running batch jobs on Roihu and LUMI.

Schema of how the batch job scheduler works

The batch job system in CSC’s HPC environment

• CSC uses a batch job system called Slurm to manage resources
• Slurm is used to control how the overall computing resources are shared among all jobs in an efficient and fair manner
• Slurm controls how a single job request is allocated resources, such as:
  • computing time
  • number of cores
  • amount of memory
  • other resources like GPUs, local disk, etc.
• Getting started with Slurm batch jobs on Roihu and LUMI

An example serial batch job script for Roihu

• A batch script is a shell script (bash) that consists of two sections:
  • Resource requests flagged with #SBATCH and the actual computing step(s)

#!/bin/bash
#SBATCH --time=00:01:00             # Defines the max time the job can run (1 min)
#SBATCH --partition=test            # Defines the queue in which to run the job
#SBATCH --ntasks=1                  # Defines the number of tasks (processes)
#SBATCH --cpus-per-task=1           # Total number of cores is ntasks * cpus-per-task
#SBATCH --account=<project>         # Defines the billing project, e.g. project_2001234 (mandatory field)

srun echo "Hello $USER! You are on node $HOSTNAME"

• The options are described in Docs CSC: Create Roihu batch jobs
  • The actual program is launched using the srun command
  • The content above could be copied into a file simple_serial.bash and submitted to the queue with sbatch simple_serial.bash

Using an application-specific batch script template

Submitting, cancelling and status of batch jobs

• A batch job script is submitted to the queue with the command:
  • sbatch example_job.sh
• List all your jobs that are queuing/running:
  • squeue -u $USER
• Detailed info of a queuing/running job:
  • scontrol show job <jobid>
• A job can be deleted using the command:
  • scancel <jobid>
• Display the resource usage and efficiency of a completed job:
  • seff <jobid>

Available batch job partitions

• The available batch job partitions are listed in Docs CSC
  • The Slurm partitions available on LUMI are detailed in the LUMI documentation.
• In order to use the resources efficiently, it is important to estimate the resource request as accurately as possible
• By avoiding excessive “just-in-case” requests, the job will start earlier

Different types of HPC jobs

• Typically, an HPC job can be classified as serial, parallel or GPU, depending on the main requested resources
• The following slides will present an overview of different job types
• A serial job is the simplest type of job, whereas parallel and GPU jobs require advanced software and programming methods to fully utilise their capacity
  • Note that GPU-jobs are in principle also parallel, but they run on different hardware (GPUs instead of CPUs) and are programmed differently
• If you use pre-installed applications, please ensure what kind of resources they need to run efficiently (serial, parallel or GPU)

HPC serial jobs

• A serial software can only use a single core, so don’t reserve more!
• Why could your serial job benefit from being run using CSC’s resources instead of on your own computer?
  • Part of a larger workflow (high-throughput computing)
  • Avoid data transfer between the supercomputers and your own computer
  • Data sharing among other project members
  • CSC’s software licensing
  • Pre-installed software
  • Memory and/or disk demands

Running multiple serial jobs 1/2

• You can utilize HPC resources for running multiple independent serial jobs at the same time (task farming)
  • Array jobs
  • Other high-throughput tools
• When running many jobs, make sure that you don’t overload the batch queue system or the parallel file system (mind your I/O and job steps)!

Running multiple serial jobs 2/2

• On Roihu, pure serial resources are only available in small and longrun partitions
  • Some tools, e.g. HyperQueue, can make a set of serial jobs suitable also for medium partition
  • But, the workflow needs to fill (at least) one Roihu node (384 cores) and keep the CPUs busy for the job duration
  • Due to the high core count, it is important that all the resources are used efficiently inside a reserved node!

HPC parallel jobs

• A parallel job distributes the calculation over several cores in order to achieve a shorter wall-time (and/or a larger allocatable memory)
  • The total computational problem is divided into subtasks, which are processed by each core in parallel
• There are two major parallelization standards: OpenMP and MPI
  • Note, depending on the parallelization scheme there is a slight difference between how the resource reservation is done
• Batch job scripts for Roihu (how to create and examples) and LUMI (quickstart, CPU and GPU examples)
• The best starting point: Software specific batch scripts in Docs CSC

HPC GPU jobs 1/2

• A graphics processing unit (GPU, a graphics card), is capable of doing a certain type of simultaneous calculations very efficiently
• In order to take advantage of this power, an application must be (re)programmed to adapt to how the GPUs process data

HPC GPU jobs 2/2

• CSC’s GPU resources on Roihu are relatively scarce and should be used only by applications that really benefit from GPUs
  • A GPU on Roihu uses 270 times more GPU BUs than a single CPU core uses CPU BUs - see above for performance requirements
  • On Roihu GPU nodes, each reserved GPU grants access to up to 72 CPU cores
  • The CPU cores on Roihu GPU nodes don’t affect the billing but must be requested explicitely
  • Note that LUMI-G has a massive GPU capacity available, which is also “cheaper” as measured in BUs compared to Roihu

Interactive jobs

• When you login to CSC’s supercomputers, you end up on one of the login nodes of the supercomputer
  • These login nodes are shared by all users and they are not intended for heavy computing
• If you have a heavier job that still requires interaction
  • Request resources from the interactive partition using the sinteractive command
  • This will open an interactive shell where you can perform your computations directly on a compute node instead of the login node
• Interactive jobs and GUIs can also be launched in the web interfaces
  • Jupyter notebooks, RStudio, TensorBoard, MATLAB, VSCode, …