Multi-Cluster HPC Management with SLURM Federation: Technical Guide

As enterprise HPC infrastructure grows, a single cluster can no longer serve all workloads cost-effectively or reliably. Different hardware generations, geographically distributed data centers, and high-availability requirements drive the need to operate multiple clusters in a coordinated fashion. SLURM Federation enables exactly this — multiple independent clusters managed through a unified control plane.

Core Architecture and Components

The foundation of multi-cluster management is the federation layer. Each cluster continues to run its own local slurmctld (SLURM controller), but all clusters connect through a shared slurmdbd (database daemon). This architecture allows each cluster to operate independently while a central orchestration mechanism coordinates cross-cluster decisions.

Core components:

• slurmdbd: Stores all accounting data, user accounts, and job history centrally. This is the single source of truth for the entire federation.
• slurmctld (per cluster): Manages local queues and nodes; implements federation decisions.
• Unified authentication (LDAP/AD): Ensures users can work with the same credentials across all clusters.
• Unified storage (optional): Shared filesystem (Lustre, GPFS) for cross-cluster data access.

SLURM Federation Setup

Setting up a federation requires all clusters to share the same slurmdbd instance. Use these commands to add clusters to the federation:

# Check current federation status
sacctmgr show federation

# Add a new cluster to the federation
sacctmgr add cluster compute2 \
  controlhost=compute2-mgmt \
  controlport=6817

# Create a federation containing two clusters
sacctmgr add federation hpc-fed \
  clusters=compute1,compute2

# Submit a job to a specific cluster
sbatch --cluster=compute2 job.sh

# List jobs across all clusters
squeue --federation

# Query job priority federation-wide
sprio --federation

When a user submits a job, SLURM first estimates the wait time on the local cluster, then evaluates capacity on other clusters, and routes the job to whichever cluster can start it soonest. The user experiences this process as completely transparent.

Workload Balancing Policies

Effective multi-cluster strategy goes beyond simply asking “which cluster has free capacity?” Different organizational needs call for different policy types:

• Priority-based: Critical production workloads always route to the designated primary cluster.
• Shortest wait: Jobs automatically go to the cluster with the lowest estimated wait time — ideal for general-purpose use.
• Capacity threshold: When a cluster’s utilization exceeds a defined threshold, jobs spill to neighboring clusters. Used to absorb sudden demand spikes.
• Data locality: Jobs are routed to the cluster closest to the storage system holding the data. Minimizes latency from large data transfers.
• Cost optimization: In hybrid cloud environments, the lowest-cost resource is automatically selected. Critical in cloud bursting scenarios.

Centralized Monitoring: Prometheus and Grafana

Monitoring multiple clusters separately multiplies operational complexity and creates blind spots. Metrics from each cluster (node utilization, queue depth, job completion time, network traffic) are aggregated to a central Prometheus instance. Unified dashboards in Grafana present the real-time state of all infrastructure on a single screen.

Any event across any cluster — node failure, disk filling, queue wait time exceeding a threshold — immediately notifies the relevant team. Automated remediation runbooks resolve a significant fraction of common issues without human intervention.

Common Problems and Solutions

Clock synchronization: If NTP synchronization is not maintained on all cluster nodes, federation communication inconsistencies occur. Regularly check synchronization status with chronyc tracking.

Account and project synchronization: User and project definitions on slurmdbd must be consistent across all clusters. Manage account changes centrally with sacctmgr; avoid manual per-cluster modifications.

Network latency: Inter-cluster control traffic requires low-latency network connectivity. Separate production workload traffic from management traffic onto dedicated network interfaces.

Missing rollback plan: Every deployment phase should have a rollback procedure ready. To temporarily disable a cluster added to the federation, use sacctmgr modify cluster.

Best Practices

These practices significantly improve operational maturity for enterprise multi-cluster infrastructure:

1. Phased migration: Deploy the federation in a test environment first. Perform production transition in incremental, reversible steps.
2. Policy documentation: Record in writing which workloads go to which cluster and why.
3. Capacity planning: Define maximum utilization thresholds for each cluster in advance; test spillover scenarios with drills.
4. Security: Run inter-cluster communication over encrypted channels (TLS); restrict slurmdbd access with firewalls.
5. Regular auditing: Review federation status, account consistency, and monitoring alerts on a weekly basis.

Conclusion

SLURM Federation-based multi-cluster management unites distributed HPC infrastructure under a central control plane, meaningfully improving both operational efficiency and resource utilization. When correct architectural design, consistent policy enforcement, and centralized monitoring are applied together, organizations can maintain business continuity through both planned maintenance and unexpected failures.

For detailed information about multi-cluster architecture and to request an organization-specific assessment, visit Mevasis Multi-Cluster Solutions or contact us.