Outline

• Efficient file I/O in HPC systems
• Using Allas in batch scripts
• Moving data to/from Allas, IDA and LUMI-O
• Transferring data in sensitive data computing
• Cleaning and backing up data

Parallel file systems

• A parallel file system (PFS) provides a common file system area that can be accessed from all nodes in a cluster
• Without PFS users would have to always copy all needed data to compute nodes before runs (cf. local disk)
  • Also the results would not be visible outside the compute node
• CSC uses Lustre parallel file system on Roihu

Lustre

What happens when you access a file?

Managing file I/O (1/3)

• Parallel file system (Lustre):
  • Shared across all nodes in the cluster (e.g. /scratch)
  • Optimized for parallel I/O of large files, slow if accessing lots of small files!
• Temporary local storage (NVMe):
  • $TMPDIR: automatic on login + all compute nodes, no reservation needed
  • $LOCAL_SCRATCH: extra, billed storage reservable (--gres=nvme:<GB>) on XL nodes (Viz not yet implemented)
  • Disaggregated storage: separate, larger shared fast-storage pool for full-node jobs
  • Purged after job finishes; availability varies by partition

Managing file I/O (2/3)

• To avoid on Lustre:
  • Accessing lots of small files, opening/closing a single file in a rapid pace
  • Having many files in a single directory
• Use file striping to distribute large files across many OSTs
• Use more efficient file formats when possible
  • Simply using tar and compression is a good start
  • High-level I/O libraries and portable file formats like HDF5 or NetCDF
    • Enable fast I/O through a single file format and parallel operations
    • AI/ML example: TensorFlow’s TFRecords – a simple record-oriented binary format
• Docs CSC: How to achieve better I/O performance on Lustre

Managing file I/O (3/3)

• Use fast local disk to handle file I/O with lots of small files
  • Requires staging and unstaging of data
  • tar xf /scratch/<project>/big_dataset.tar.gz -C $TMPDIR
• Do not use databases on /scratch
  • Instead, consider hosting DBs on cloud resources (e.g. Pukki DBaaS)

Using Allas in batch jobs (1/2)

• Command-line interface: use either Swift or S3 protocol
  • Swift (multiple projects, 8-hour) vs. S3 protocol (fixed for a project, persistent)
  • Roihu defaults to S3, unlike Puhti and Mahti
• allas-conf needs setting up CSC password interactively
  • Jobs may start late and actual job may take longer than 8 hrs

Using Allas in batch jobs (2/2)

• For Swift connections, use allas-conf --swift -k (not needed for S3, which is already persistent)
  • stores password in variable $OS_PASSWORD to generate a new token automatically
    • a-tools regenerate a token using $OS_PASSWORD automatically
    • rclone requires explicitly setting environment variable in batch jobs:

source /appl/soft/manual/general/common/allas/allas-cli-utils/allas_conf --swift -f -k $OS_PROJECT_NAME

Configuring Allas for S3 protocol

• Opening Allas connection in S3 mode
  • source allas_conf -m S3
• Connection is persistent
• Usage:
  • s3cmd with endpoint s3:
  • rclone with endpoint s3allas:
  • a-put --s3 to force S3 (a-get already defaults to S3; use --swift to switch)

How to use LUMI-O from Roihu?

• LUMI-O is very similar to Allas, but it uses only S3 protocol
• In Roihu, connection to LUMI-O can be opened with command:
  • allas-conf --lumi
• Usage:
  • Using LUMI-O with rclone (endpoint is lumi-o:)
    • e.g. rclone lsd lumi-o:
  • One can use a-tools with option --lumi
    • e.g. a-list --lumi
• Docs CSC: Using Allas and LUMI-O from LUMI

Moving data between LUMI-O and Allas

• Requires activating connections to both LUMI-O and Allas at the same time:
  • allas-conf
  • allas-conf --lumi
• Use rclone with s3allas: as endpoint for Allas and lumi-o: for LUMI-O
  • rclone copy -P lumi-o:lumi-bucket/object s3allas:allas-bucket/

Moving data between Fairdata IDA and Allas

• Needs transfer of data via supercomputer (e.g. Roihu)
• Requires configuring Fairdata IDA in CSC supercomputers
  • Load IDA module: module load ida
  • Configure IDA database: ida_configure
  • Upload data to IDA: ida upload <target_in_ida> <local_file>
  • Download data from IDA: ida download <target_in_ida> <local_file>

Transferring data for sensitive data computing

• CSC sensitive data services: SD Connect and SD Desktop, use service-specific encryption
• SD Desktop is able to read encrypted data from Allas
  • If you want to make your data available in SD Desktop, you need to use SD Connect to upload data to Allas
  • Open SD Connect compatible connection to Allas with allas-conf --sdc
  • Data can then be uploaded by using command a-put with option --sdc
  • More information in Docs CSC

Cleaning and backing up data (1/3)

• Disk cleaning
  • In force for project disk areas under /scratch on Roihu
  • Files not accessed for 90 days (5+ TiB quota) or 180 days (smaller quotas) are removed periodically
    • You get at least 1 month’s advance notice, plus a list of the files about to be removed
    • Check your project’s cleaning cycle with csc-workspaces
• Best practice tips
  • Don’t save everything automatically
  • Use LUE tool to analyze your disk usage
    • Avoid du and find -size, these commands are heavy on the file system
  • Move important data not in current use to Allas

Cleaning and backing up data (2/3)

• allas-backup command provides an easy-to-use command-line interface for the restic backup tool
• Backing up differs from normal storing:
  • Incremental (efficient) and version control (no overriding)
  • Based on hashes and requires more computing
  • Efficient way to store different versions of a dataset
• New restic-based “data mover” tool coming soon!

Cleaning and backing up data (3/3)

• Please note that Allas is intended for storing active data
• Project lifetime is usually 1-5 years
• Commands for backing up data:
  • allas-backup --help
  • allas-backup [add] file-or-directory
  • allas-backup list
  • allas-backup restore snapshot-id