HPC Optimisation — Get More From Your Cluster

You may not be using the full theoretical capacity of your existing HPC cluster. Misconfigured scheduler policies, untuned network parameters or I/O bottlenecks all cut significantly into the return on your hardware investment.

Areas of Optimisation

SLURM Scheduler Optimisation

SLURM’s default configuration is rarely specific to your workload. The scope of optimisation:

# Backfill scheduler - lets small jobs slot in
SchedulerType=sched/backfill
SchedulerParameters=bf_window=10080,bf_resolution=600

# Priority weights
PriorityType=priority/multifactor
PriorityWeightJobSize=1000
PriorityWeightAge=500
PriorityWeightPartition=250

# GPU resource tracking
GresTypes=gpu

Network Tuning

InfiniBand and Ethernet parameters do not run optimally at their defaults:

# RDMA buffer sizes
echo 1000 > /proc/sys/net/core/netdev_max_backlog

# TCP buffers (for Ethernet)
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728

# InfiniBand QoS services
/etc/rdma/rdma.conf: IPOIB_LOAD=yes

Storage Performance

BeeGFS or Lustre configuration improvements:

• Stripe count and chunk size optimisation
• Client cache tuning
• Metadata server load balancing
• Journal and cache sizing

Application Profiling

Finding the bottlenecks in the computation itself:

# MPI communication profiling
mpirun -np 32 -mca pml ob1 --map-by socket \
  valgrind --tool=callgrind ./my_simulation

# GPU profiling
nsys profile --trace=cuda,nvtx \
  python train_model.py

Initial Assessment Report

As a first step we offer a free cluster health assessment:

• Analysis of SLURM queue statistics
• Resource efficiency (wait time versus run time ratio)
• Storage I/O benchmark (short test)
• Network latency measurement

The report shows which optimisation steps would return the most. Get in touch for the free assessment.