The one-slide lecture

• Some software packages on the CSC supercomputers are installed as containers
  • May cause some changes to usage
  • See the instructions in Docs CSC for each software for details
• Containers provide an easy way to install software
  • A single-command installation if a suitable Docker or Apptainer/Singularity container exists

Containers

• Containers are a way of packaging software and their dependencies (libraries, etc.)
• Popular container engines include Docker, Apptainer (open source fork of Singularity), Shifter
• Apptainer is most popular in HPC environments

A note on Apptainer/Singularity

• The open source project formerly called Singularity is now called Apptainer (since March 2022)
• Sylabs continues to develop commercially oriented products such as Singularity (formerly SingularityCE) and SingularityPro
• Apptainer 1.0 was forked from Singularity 3.9.5 with some changes
• Singularity and Apptainer are currently very similar, but they are now developed independently and will probably continue to diverge

Apptainer/Singularity differences

• All SINGULARITY_ and SINGULARITYENV_ variables work for now, but you will get a warning
  • There are currently corresponding APPTAINER_ and APPTAINERENV_ variables, but the naming may change in the future
  • If both SINGULARITY_/SINGULARITYENV_ and APPTAINER_/APPTAINERENV_ are defined for a given variable, APPTAINER values will be used.
• On CSC machines commands singularity and singularity_wrapper are symlinked to apptainer and apptainer_wrapper respectively

TLDR: Most older singularity instructions work currently as-is, but this will probably change as time goes on.

Containers vs. virtual machines (1/2)

Containers vs. virtual machines (2/2)

• Virtual machines can run a totally different OS than the host computer (e.g. Windows on a Linux host or vice versa)
• Containers share the kernel with the host, but they can have their own libraries
  • They can run, e.g., a different Linux distribution than the host

Benefits of containers: Ease of installation

• Containers are becoming a popular way of distributing software
  • A single-command installation
  • More portable since all dependencies are included
  • Normal user rights are enough when using an existing container
• Root access on build system is enough
  • Root access, package managers (yum, apt, etc.) can be utilized even when not available on the target system.
  • Makes installing libraries easier

Benefits of containers: Environment isolation

• Containers use the host system kernel, but they can have their own bins/libs layer
  • Can be a different Linux distribution than the host
  • Can solve some incompatibilities
  • Less likely to be affected by changes in the host system

Benefits of containers: Enviroment reproducibility

• Environment can be saved as a whole
  • Useful with, e.g., Python, where updating underlying packages (NumPy, etc.) can lead to differences in the behavior
    
• Sharing with collaborators is easy (a single file)

Apptainer in a nutshell

• Containers can be run with user-level rights
  • But: building new containers requires root access
• Minimal performance overhead
• Supports MPI
  • Requires containers tailored to the host system
• Can use host driver stack (Nvidia/CUDA)
  • Add the option --nv
• Can import and run Docker containers
  • Running Docker directly would require root privileges

Using Docker containers with Apptainer

• You can pull and convert a Docker container image with normal user rights:
  • apptainer pull docker://<address>:<tag>
  • apptainer build <image> docker://<address>:<tag>
• For example:
  • apptainer pull docker://nvcr.io/nvidia/pytorch:19.10-py3
• More information in our documentation:
  • Running Apptainer containers
  • Creating Apptainer containers
  • Using Tykky to create Apptainer containers
  • Apptainer containers on LUMI

Apptainer containers as an installation method

• Apptainer is a good option in cases where the installation is otherwise problematic:
  • Complex installations with many dependencies/files
  • Obsolete dependencies incompatible with the native environment
    • Still needs to be kernel-compatible
  • Native use of Conda has been deprecated on CSC supercomputers
    • Conda environments can be installed inside containers
    • Container wrapper tool Tykky provided for this
• Should be considered even when other methods exist

Just a random example (FASTX-toolkit)

• Tested installation methods:
  • Native: 47 files, total size 1.9 MB
  • Conda: 27464 files, total size 1.1 GB
  • Apptainer: 1 file, total size 339 MB
• Containers are not the solution for everything, but they do have their uses