Outline

• Primer on Docker images for Apptainer (Recap)
• HPC disk systems and efficient image conversion
• Good practice tips in converting images

Primer on Docker images for Apptainer (Recap)

Docker Images

• The docker is a leading container platform and is widely adopted in industry/cloud applications
• Plenty of images are archived in docker registries
• Provides even GPU-accelerated solutions
• Handy when directly building Singularity image seems more difficult
• Good news: Singularity/Apptainer nicely integrates with docker images
• Challenge: Some docker images may not work smoothly with Apptainer

Docker vs. Apptainer Images

• Docker image
  • Image is a layered structure
  • Stored in the local image cache when using docker client
  • Saves some space by sharing some layers across different local images when storing
  • Caching the build layers can speed up the build process
• Apptainer image
  • Single image file (.sif) / images are flat
  • Stored as a normal file
  • Easy portability : transferring and sharing images across a cluster is very easy
  • Image can be used as binary (./image.sif)

Running Apptainers from Prebuilt Images from Docker Registry

• Public images:
  • apptainer pull/build docker://privaterepo/imagename:tag
• Private images:
  • apptainer pull –docker-login docker://privaterepo/imagename:tag
• Local docker archive:
  • apptainer build local_apptainer_image.sif docker-archive://local_docker.tar

Sharing Images via Registries

• Sharing images without registries is usually discouraged
• Share your images via registries and maintain proper codes in GitHub/Gitlab
• Sharing via registries requires pushing your image to registry
• Production images needs to be properly named and tagged

Reminder: Take care of Apptainer Cache in Puhti

• Default location: $HOME/.apptainer
  • Disk space in $HOME in HPC systems is limited for a user
  • Your $HOME directory can be quickly filled up, causing disk space issues
• One can configure Singularity directories using two environment variables:
  • APPTAINER_CACHEDIR: Cache folder for images from a container registry.
  • APPTAINER_TMPDIR: Temporary directory to build container file-systems.
• Useful tips:
  • apptainer cache list # show storage capacity used by the cache
  • apptainer cache clean # clean up everything

HPC Disk Systems and Efficient Image Conversion

Disk and Storage Overview

NVMe Disks for Faster Image Conversion

• Use only when working with bigger images
• Not all compute nodes have these NVMe disks
• You can request these resources in batch script/interactive node
• Use the environment variable $LOCAL_SCRATCH to access local storage on compute node
• In batch jobs, remember to copy the files back to your scratch folders

Tiny Batch Script That Uses NVMe Disks

• Image conversion example

Good Practice Tips in Image Conversion

Good Practice Tips in Image Conversion (1/2)

• Maintain tags of the image (reproducibility!)
  • generic URI for docker: docker://<user >/<repo-name >[: <tag >]
  • developers release several different versions of the same container with different tags
• Hashes are even more unique
  • Always pull the same image: library://debian:sha256.b92c7fdfcc615
• Pay attention to special tag: latest
  • pulling an image with latest: apptainer pull library://debian:latest

Good Practice Tips in Image Conversion (2/2)

• Don’t use any image that would be running under root
  • Works fine in docker world and results in bugs in apptainer
• If you modify an image, don’t install to $HOME or $TMP
• Docker images with ENTRYPOINT scripts are usually broken
  • ENTRYPOINT [ “entrypoint_script.sh” ]