Slurm HPC Cluster Management: A Comprehensive Guide

Slurm (Simple Linux Utility for Resource Management) is the open-source job scheduler used by over 60% of HPC clusters worldwide. From TOP500 supercomputers to university research labs, from enterprise R&D centers to AI training infrastructure — Slurm ensures fair and efficient resource sharing at every scale.

This comprehensive guide covers every critical aspect of Slurm HPC cluster management: architecture fundamentals, installation steps, partition design, GPU scheduling, monitoring infrastructure, and common troubleshooting scenarios. Whether you’re setting up a new cluster or optimizing an existing Slurm deployment, this guide is your roadmap.

Slurm Architecture: Core Components

Slurm is built on a modular architecture where each component has a defined responsibility, with all inter-component communication secured by MUNGE authentication.

• slurmctld (Controller Daemon): The brain of Slurm — tracks all resource states, manages the job queue, and makes scheduling decisions. Only one active controller runs per cluster.
• slurmd (Node Daemon): Runs on every compute node, executes job launch instructions from the controller, and reports resource status back.
• slurmdbd (Database Daemon): Stores job accounting, user accounts, quota definitions, and fair-share data in MySQL/MariaDB.
• MUNGE: Encrypts and authenticates all communication between Slurm daemons. The MUNGE key must be identical across all nodes.

Slurm Installation: Step by Step

Prerequisites

Before installation, clarify: node count, cores per node, memory, GPU types and counts, number of users/groups, software stack, and whether you’re migrating from another scheduler.

MUNGE Setup

# On controller node
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chmod 400 /etc/munge/munge.key
chown munge:munge /etc/munge/munge.key
systemctl enable --now munge

# Distribute key to all compute nodes
for node in cn{01..32}; do
  scp /etc/munge/munge.key ${node}:/etc/munge/
  ssh ${node} "chmod 400 /etc/munge/munge.key && systemctl restart munge"
done

Main Configuration (slurm.conf)

ClusterName=research
SlurmctldHost=slurm-controller(10.0.0.10)
SchedulerType=sched/backfill
SelectType=select/cons_tres

NodeName=cn[01-24] CPUs=64 RealMemory=256000
NodeName=gpu[01-04] CPUs=64 RealMemory=512000 Gres=gpu:h100:8

PartitionName=debug   Nodes=cn01         MaxTime=00:30:00
PartitionName=short   Nodes=cn[02-24]    MaxTime=04:00:00 Default=YES
PartitionName=long    Nodes=cn[02-24]    MaxTime=7-00:00:00
PartitionName=gpu     Nodes=gpu[01-04]   MaxTime=24:00:00

Partition Design

A well-designed partition architecture dramatically improves cluster utilization:

Job Scheduling Examples

Batch Job Submission

#!/bin/bash
#SBATCH --job-name=cfd_sim
#SBATCH --partition=long
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --time=48:00:00

module load openfoam/11.0
mpirun -np 256 simpleFoam -parallel

GPU Job Submission

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --gres=gpu:h100:4
#SBATCH --cpus-per-task=32
#SBATCH --mem=256G
#SBATCH --time=24:00:00

torchrun --nproc_per_node=4 train.py

Cgroup Resource Isolation

# /etc/slurm/cgroup.conf
CgroupAutomount=yes
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
AllowedSwapSpace=0
MaxRAMPercent=98

GPU Management with GRES

# gres.conf
NodeName=gpu01 Name=gpu Type=h100 File=/dev/nvidia[0-7] Count=8
NodeName=gpu02 Name=gpu Type=a100 File=/dev/nvidia[0-3] Count=4

Monitoring with Prometheus + Grafana

Critical metrics: slurm_queue_pending, slurm_nodes_drain, slurm_cpu_utilization, slurm_gpu_utilization, node_memory_MemAvailable.

Common Issues and Solutions

Nodes draining: Check reason with scontrol show node <name> | grep Reason. Common causes: OOM, hardware error, slurmd communication loss. Resume with scontrol update NodeName=<name> State=resume.

Jobs stuck pending: Run squeue -j <id> --start to see why. Common: insufficient resources, partition time limit exceeded, QoS caps.

MUNGE failures: Verify clock sync (chronyc tracking), check key consistency across nodes (md5sum /etc/munge/munge.key).

Mevasis Slurm Services

Mevasis delivers enterprise Slurm deployment, optimization, and managed operations across Turkey. From greenfield cluster setup to existing Slurm optimization, 24/7 managed services to custom training programs — we cover the full HPC lifecycle.

• Slurm Services — Installation, consulting, and managed support
• HPC Maintenance — Continuous monitoring and proactive intervention
• HPC Training — Admin and end-user training programs

Schedule a free discovery call to discuss your project.