Slurm Performance Optimization Guide

Most HPC clusters run at only 60-70% of their potential with default Slurm configuration. Here are 5 critical settings that dramatically improve Slurm performance, based on years of enterprise HPC experience.

1. Maximize Utilization with Backfill Scheduling

SchedulerType=sched/backfill
bf_interval=30
bf_max_job_test=200
bf_window=1440

Backfill lets small jobs run in gaps while resources are reserved for large high-priority jobs. bf_window=1440 (24 hours) is optimal for most clusters — small enough to avoid controller CPU overload, large enough to maximize utilization.

2. Balance Job Mix with Partition Optimization

PartitionName=short   Nodes=cn[01-20] MaxTime=04:00:00 Priority=200 MaxNodes=8
PartitionName=long    Nodes=cn[01-20] MaxTime=7-00:00:00 Priority=100
PartitionName=gpu     Nodes=gpu[01-04] MaxTime=48:00:00 Priority=300
PartitionName=debug   Nodes=cn01       MaxTime=00:30:00 Priority=500

Key: MaxNodes=8 on short partition prevents large jobs from blocking quick-turnaround work. GPU partition gets highest priority after debug.

3. Prevent OOM with Aggressive Cgroup Memory Management

ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
AllowedSwapSpace=0
MaxRAMPercent=98
MinRAMSpace=2048

No swap for HPC workloads — swap kills performance. Instead, OOM-kill the offending job while protecting other jobs on the same node. Leave 2GB for OS and slurmd overhead.

4. Maximize GPU Throughput

# gres.conf — heterogeneous GPU types
NodeName=gpu01 Name=gpu Type=h100 File=/dev/nvidia[0-7] Count=8
NodeName=gpu02 Name=gpu Type=a100 File=/dev/nvidia[0-3] Count=4
NodeName=gpu03 Name=gpu Type=l40s File=/dev/nvidia[0-7] Count=8

Never lump different GPU models under the same type — users will accidentally get the wrong hardware. Separate types let users be specific: --gres=gpu:h100:2.

5. Optimize slurmdbd Database Performance

ALTER TABLE cluster_event_table ADD INDEX idx_time (time_end);
ALTER TABLE cluster_job_table ADD INDEX idx_user (id_user, time_end);

Archive events older than 6 months and jobs older than 12 months. Keep the live database lean — sacct queries should return in under a second.

Real Results

With these 5 optimizations, typical enterprise clusters see:

Queue wait time: 40-60% reduction
GPU utilization: 40% → 85%+
OOM-caused job failures: 90% reduction
Night/weekend utilization: 30% increase via backfill

For professional Slurm optimization, see our Slurm Services or our comprehensive Slurm guide.

Slurm Performance Optimization: 5 Critical Settings