Container Platform Guide for HPC: Apptainer Architecture, Installation, and Best Practices

Docker revolutionized software deployment, but its architecture is fundamentally incompatible with shared HPC environments. The Docker daemon runs as root, and any user who can start a Docker container can escalate to root on the host. In a multi-user HPC cluster, this is unacceptable.

Apptainer (formerly Singularity) was designed from the ground up for HPC: it runs containers as the calling user (no privilege escalation), integrates transparently with the host filesystem and network, and works natively with SLURM and MPI. It is now the de facto container runtime for scientific computing.

Why Not Docker on HPC?

The single-file SIF format is particularly valuable for HPC: images can be stored on a parallel filesystem and shared among users without a running daemon.

Three-Component Architecture

Apptainer’s architecture has three concepts:

SIF image (.sif file): The immutable container image stored as a single file. Can be created from Docker Hub images, from definition files, or from interactive sandbox sessions. Stored on the shared filesystem and shared across users.

Definition file (.def file): A declarative recipe for building a SIF image. Specifies the base image, software to install, environment variables, and entry points. Checked into version control like any other code artifact.

Central registry: A container registry (Apptainer’s library, Docker Hub, GHCR, or a private Harbor instance) from which SIF images are pulled by name and version. Enables reproducible, versioned software environments.

Installation

# RHEL/Rocky Linux 8/9 — install from EPEL
dnf install epel-release
dnf install apptainer

# Ubuntu 22.04 — install from official PPA
add-apt-repository -y ppa:apptainer/ppa
apt-get update && apt-get install -y apptainer

# Verify installation
apptainer --version

For production HPC clusters, configure these system-wide settings in /etc/apptainer/apptainer.conf:

# /etc/apptainer/apptainer.conf

# Allow users to set SIF image paths for security
allow setuid = yes

# Bind the parallel filesystem automatically
bind path = /mnt/scratch, /mnt/project

# Bind GPU libraries (alternative to --nv flag)
mount dev = yes

# Restrict network namespace (security)
allow net networks = bridge, host

# Location for session files
sessiondir max size = 512

# Enable overlayfs for writable layers
enable overlay = try

Building SIF Images

Pulling from Docker Hub

# Pull Python image from Docker Hub and convert to SIF
apptainer pull python-3.11.sif docker://python:3.11-slim

# Pull from GHCR (GitHub Container Registry)
apptainer pull myapp.sif oras://ghcr.io/myorg/myapp:v2.1

# Pull from Apptainer Library
apptainer pull ubuntu-22.04.sif library://library/default/ubuntu:22.04

Building from Definition Files

Definition files are the recommended approach for reproducible scientific software environments:

# gromacs-2024.def — GROMACS molecular dynamics container
Bootstrap: docker
From: nvidia/cuda:12.2-devel-ubuntu22.04

%labels
    Author HPC Team
    Version 2024.1
    Application GROMACS

%environment
    export GMXLIB=/usr/local/gromacs/share/gromacs/top
    export PATH=/usr/local/gromacs/bin:$PATH

%post
    # System dependencies
    apt-get update && apt-get install -y \
        cmake g++ libfftw3-dev libhdf5-dev \
        libopenmpi-dev openmpi-bin wget

    # Build GROMACS with GPU and MPI support
    wget https://ftp.gromacs.org/gromacs/gromacs-2024.1.tar.gz
    tar xzf gromacs-2024.1.tar.gz
    cd gromacs-2024.1
    mkdir build && cd build
    cmake .. \
        -DGMX_BUILD_OWN_FFTW=OFF \
        -DGMX_MPI=ON \
        -DGMX_GPU=CUDA \
        -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs
    make -j8 && make install

    # Clean up build artifacts to reduce image size
    cd / && rm -rf /gromacs-2024.1*

%runscript
    exec gmx_mpi "$@"

# Build SIF from definition file (requires root or fakeroot)
apptainer build gromacs-2024.sif gromacs-2024.def

# Build with fakeroot (no root required)
apptainer build --fakeroot gromacs-2024.sif gromacs-2024.def

Running Containers in SLURM

#!/bin/bash
#SBATCH --job-name=gromacs-md
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --gres=gpu:4
#SBATCH --time=24:00:00

# Run GROMACS in Apptainer across 4 nodes with GPU support
mpirun -np 256 \
  apptainer exec --nv \
  --bind /mnt/scratch/project:/data \
  /mnt/software/containers/gromacs-2024.sif \
  gmx_mpi mdrun -v -deffnm md -ntmpi 1 -ntomp 16 -gpu_id 0123

The --nv flag binds NVIDIA GPU drivers and libraries from the host into the container, eliminating the need to install GPU drivers inside the SIF image. The --bind flag makes the job’s data directory accessible at /data inside the container.

Common Problems and Solutions

“No space left on device” during build: Apptainer uses /tmp for build scratch space. Set APPTAINER_TMPDIR=/mnt/scratch/apptainer-tmp before building large images.

MPI applications hang at startup: Ensure the MPI inside the container matches the version on the host, or use the host MPI by binding host MPI libraries: --bind /usr/lib/x86_64-linux-gnu/openmpi:/opt/openmpi. For PMI-based launchers, use srun instead of mpirun.

GPU not detected inside container: Run apptainer exec --nv container.sif nvidia-smi. If this fails, the NVIDIA driver version inside the container image may be newer than the host driver. Always build images with CUDA version matching or older than the host driver.

Permission errors on shared images: SIF images on shared filesystems must be readable by all users. Set chmod 644 *.sif or use a group-readable directory with chmod 2775.

Best Practices

• Version all definition files in a central Git repository. Treat container recipes as code.
• Store SIF images on the shared filesystem in a structured directory (e.g., /mnt/software/containers/appname-version.sif) accessible to all compute nodes.
• Use a private registry (Harbor, GitLab Container Registry) for internal images to avoid Docker Hub pull rate limits.
• Pin base image versions in definition files (From: ubuntu:22.04 not From: ubuntu:latest) to ensure reproducibility.
• Minimize image size by removing build tools and caches in the %post section.

For HPC container platform deployment, image management infrastructure, and SLURM integration, visit Mevasis HPC Solutions or contact us.