Modules and pre-installed software

The module system and how to use it on CSC supercomputers. The same information can be found in the module section of Docs CSC.

All materials (c) 2020-2023 by CSC – IT Center for Science Ltd. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 Unported License, http://creativecommons.org/licenses/by-sa/4.0/

Module systems in supercomputers

  • A variety of software with different (possibly conflicting) requirements are needed on a supercomputer
  • The solution for managing this situation: separating the applications into modules
  • Environment modules set up everything required by a particular application:
    • Load libraries, adjust paths, set environment variables
  • CSC uses Lmod environment modules which are based on the Lua programming language

How to access pre-installed software on CSC supercomputers

  • You can check the available applications and their respective modules in the application list
    • Each application page in Docs CSC contains information on basic usage
  • You can use module spider to search for an application
    • On LUMI you need to first run module use /appl/local/csc/modulefiles to see modules installed by CSC
  • Use the software by loading the module: module load modulename
    • For example: module load gromacs-env

How to use modules

  • The general syntax is simple: module command modulename
  • These modules are used both in interactive sessions and batch jobs
    • Include module purge and module load modulename in your batch scripts to always get the same modules in consecutive jobs
  • You can’t just load all the modules because of possibly conflicting dependencies

Useful commands for the module environment

  • These commands will help you navigate the module environment:
    • module list: See a list of the currently loaded modules
    • module avail: Modules currently available for loading (hides modules that can’t be loaded at the moment due to dependencies)
    • module spider modulename: Search for an application in the list of all existing modules
    • module spider modulename/version: Gives information on how to load a specific version of a module (prerequisites etc.)
  • If you try to load a module that is unavailable, you will get an error message

Various kinds of modules

  • Some modules contain software that have a graphical user interface
  • Some modules contain software that only provide a command-line interface
  • A module might contain only one program (e.g. gromacs-env), while some provide a collection of applications (e.g. Python or R packages)

Conda, Python and R packages

  • Some pre-installed software are distributed through the Conda package management tool
    • Do not install Conda environments directly on the Lustre parallel file system, containerize them instead using e.g. Tykky
    • See the usage policy for further details
  • Pre-installed Python packages are available in Python-related modules
    • Check Docs CSC for the available packages and usage instructions
    • It is not possible to load multiple modules with Python packages at once
    • pip list shows a list of packages installed in the current environment
  • Instructions on how to check installed R libraries
    • RStudio shows available packages in the sidebar

Customizing your own environment

  • If you “always” use certain modules, it is possible to load them in your .bashrc, but we do not recommend this
    • This causes the modules to be always loaded, also in batch jobs which may cause hard-to-spot issues
    • If you already did this, see the csc-env command
  • If it feels cumbersome to run multiple module load modulenames at the start of each session, you can define an alias for these in your .bashrc
    • Add this line to your .bashrc: alias mods="module load modulenames"
    • Now you can load all those modules easily with mods (after sourcing your .bashrc or logging out and in again)

Advanced module use

  • You can save your current set of modules with module save filename and load it later with module restore filename
  • You can also write your own module files:
    1. Add them to your home directory ($HOME/modulefiles)
    2. Add this path to the module search path (module use $HOME/modulefiles)
  • To see the parameters of a module file, use the command module show modulename