Slurm on Kubernetes: Slinky, Kueue and Volcano Options Explained

Slurm on Kubernetes: Slinky, Kueue and Volcano Options Explained

Slurm on Kubernetes: Slinky, Kueue and Volcano Options Explained

Published
20 November, 2026
Category
Technical Guide

Short answer: there are three ways to get Slurm-style job scheduling on a Kubernetes GPU cluster. You can run Slurm and Kubernetes side by side on separate nodes, run Slurm inside Kubernetes with SchedMD’s Slinky project, or skip Slurm and add a Kubernetes-native batch layer with Kueue or Volcano. If your users depend on sbatch, fairshare and job accounting, Slinky or the side-by-side model keeps those intact. If everything is already containerised and your team is Kubernetes-first, Kueue or Volcano is shorter.

This post is about combining the two. For a head-to-head view of each scheduler on its own, see our Slurm vs Kubernetes comparison.

Why teams want both

SchedMD’s own slurm-operator documentation sums up the split well. Kubernetes is good at workloads that run for an indefinite time, often on a single node, with loose policy and a resource pool that can grow. Slurm is good at workloads that run for a finite time, with well-defined resources and topology, across multiple nodes, under strict policy, on a known pool of hardware.

A shared GPU cluster can end up hosting both kinds of work: training runs and simulations are finite multi-node batch jobs, while inference endpoints, notebooks and pipelines are long-running services. With two separate clusters, GPUs in one pool can sit idle while the other has a queue. Combining the schedulers lets both groups share the hardware without changing tools.

Three patterns: side by side, Slurm inside Kubernetes, Kubernetes-native batch

PatternWhat runs whereWho schedules batch jobs
Side by sideSlurm on bare-metal nodes, Kubernetes on a separate node poolSlurm, unchanged
Slurm inside KubernetesSlurm daemons run as pods (slurm-operator), or Slurm schedules Kubernetes pods (slurm-bridge)Slurm
Kubernetes-native batchOnly Kubernetes; Kueue or Volcano adds queues and quotasKueue with kube-scheduler, or Volcano

Side by side is the simplest to operate, but moving capacity between the two is a manual step. Slurm inside Kubernetes keeps the Slurm user experience while Kubernetes manages the Slurm components. Kubernetes-native batch has the fewest moving parts if everything is containerised, but users give up the Slurm command line and policy model.

Slinky (SchedMD’s Slurm operator): what it deploys

Slinky is SchedMD’s set of projects for running Slurm with Kubernetes. NVIDIA announced on 15 December 2025 that it had acquired SchedMD and said it will keep developing and distributing Slurm as open-source, vendor-neutral software. The active Slinky repositories are:

  • slurm-operator: a Kubernetes operator that deploys and manages Slurm clusters.
  • slurm-bridge: a Kubernetes scheduler backed by Slurm, so Kubernetes and Slurm workloads can share the same nodes.
  • slurm-client and containers: a Go client for the Slurm REST API and the OCI images for the Slurm daemons.

slurm-operator defines custom resources (CRD version v1beta1) for the Controller (slurmctld), NodeSet (a group of identical slurmd workers), LoginSet (login nodes with SSSD for user identity), RestApi (slurmrestd), Accounting (slurmdbd) and Token. Accounting is optional and needs a MySQL or MariaDB compatible database, which the docs suggest providing through the mariadb-operator, the mysql-operator or a managed database.

Operational details worth knowing:

  • The operator requires Slurm’s configless mode, auth/slurm, auth/jwt and dynamic nodes. Workers fetch configuration from slurmctld, so no shared config filesystem is needed.
  • Before scaling in or upgrading a NodeSet, the operator drains the affected Slurm nodes. NodeSets and LoginSets can scale to zero, which pairs with a Horizontal Pod Autoscaler.
  • A hybrid cluster is supported: slurmd can stay on bare metal and still join a Slurm control plane running in Kubernetes, and services such as LDAP, NFS or MariaDB can live outside the cluster.
  • Version 1.3.0, released on 19 November 2026, added slurmctld high availability, which needs a ReadWriteMany volume for shared state.

Status: the 1.x line started with the v1.0.0 tag in November 2025. The 1.3.0 release supports Kubernetes 1.35 to 1.37 and Slurm 25.11 or later, and its Helm chart defaults to Slurm 26.05. The code is Apache 2.0 licensed. GitHub issues are handled on a best-effort basis; SchedMD’s official issue tracker is support.schedmd.com.

slurm-bridge works the other way around. A Pod, PodGroup, Job, JobSet or LeaderWorkerSet submitted to Kubernetes becomes an external Slurm job; when slurmctld allocates nodes, slurm-bridge binds the pods to them. Native sbatch and salloc jobs keep running through slurmd. Two documented limits: bridge jobs take whole nodes exclusively by default, and on a hybrid node native Slurm jobs and bridge-managed pods must not run at the same time. Check the slurm-bridge compatibility table separately, as it differs from the operator’s for older release series.

Kueue and Volcano: which Slurm features they do and do not replace

Kueue is a kubernetes-sigs project that manages quotas and decides when a job is admitted, when it waits and when it is preempted. It does not replace kube-scheduler; pod-to-node placement stays with the default scheduler. Its building blocks are the ClusterQueue (a cluster-scoped pool with quotas and fair-sharing rules), the namespaced LocalQueue that users submit to, ResourceFlavors that map quota to node types, and cohorts in which ClusterQueues lend and borrow unused quota. Supported job types include Kubernetes Jobs, JobSet, MPIJob, RayJob and others.

Volcano is a CNCF incubating project with its own batch scheduler; a Volcano Job uses schedulerName: volcano by default. It provides gang scheduling through minAvailable, hierarchical Queues with weight, capability and guaranteed resources, and scheduling actions named enqueue, allocate, preempt, reclaim and backfill.

Slurm featureKueueVolcano
Partitions and limitsClusterQueue quotas, ResourceFlavorsQueues with capability and guarantee
Priority and preemptionPriority-based preemption inside a ClusterQueue, reclaim across a cohortpreempt and reclaim actions
Fairshare with usage decayAdmission Fair Sharing: decayed historical usage among LocalQueues of one ClusterQueue (beta since v0.15)DRF and queue weights
Gang start (all nodes at once)All-or-nothing via waitForPodsReady: a timeout, then eviction and requeueGang plugin with minAvailable
Topology-aware placementTopology-Aware Scheduling (beta since v0.14)Network topology aware scheduling with HyperNodes
Backfill using job time limitsNo equivalent in the documented queueing strategies (StrictFIFO, BestEffortFIFO)A backfill action, documented as filling nodes to improve utilisation
User interfaceKubernetes manifestsKubernetes manifests

Slurm’s backfill scheduler starts lower-priority jobs only when that does not delay the expected start of higher-priority jobs, and it relies on job time limits to work this out. If your utilisation depends on that behaviour, or on per-job accounting through slurmdbd and sacct, test it explicitly before moving to a Kubernetes-native queue.

Kubernetes itself is also adding gang scheduling: it is beta in Kubernetes 1.37 and disabled by default behind the GenericWorkload feature gate.

GPU scheduling in each pattern

GPU-aware scheduling looks different in each pattern.

Side by side. Slurm handles GPUs with GRES as on any bare-metal cluster; our Slurm GPU scheduling guide covers GRES and MIG. The Kubernetes pool uses the NVIDIA GPU Operator.

slurm-operator. GPU scheduling happens in two layers. Kubernetes gives GPUs to the slurmd pod through the device plugin (nvidia.com/gpu) or Dynamic Resource Allocation (DRA). Slurm then schedules jobs on those GPUs through GRES: the docs set AutoDetect=nvidia in gres.conf, add gpu to GresTypes and declare a matching Gres on the NodeSet. For GB200 and GB300 NVL72 systems, IMEX channels are handled through the NVIDIA DRA driver rather than Slurm prolog and epilog scripts. The chart can also map GPUs to Slurm jobs in DCGM metrics.

slurm-bridge. GPU requests from pods can come through the NVIDIA DRA driver, which slurm-bridge maps to Slurm GRES, or through the classic nvidia.com/gpu device-plugin path.

Kueue and Volcano. GPUs are Kubernetes resources provided by the NVIDIA GPU Operator, which installs the driver, container toolkit, device plugin, GPU Feature Discovery labels, DCGM exporter and MIG Manager. With MIG’s mixed strategy, each profile appears as its own resource, such as nvidia.com/mig-1g.10gb. Time-slicing is also available, but NVIDIA notes it gives no memory or fault isolation between replicas, unlike MIG. In Kueue, separate ResourceFlavors with node labels let you put quotas on different GPU models. Our Kubernetes GPU cluster guide walks through the GPU Operator and Volcano setup.

Storage and MPI/InfiniBand considerations

Storage. By default slurmctld keeps its state on a persistent volume, and the docs advise against disabling this in production. Controller HA needs a ReadWriteMany volume, and accounting needs its database. Home directories and datasets are mounted into NodeSet and login pods as ordinary Kubernetes volumes, for example NFS. Parallel filesystems reach pods through a CSI driver where one exists, such as the BeeGFS CSI driver, or through host mounts.

InfiniBand and RDMA. To use InfiniBand from pods, the RDMA devices and drivers have to be exposed to them. The NVIDIA Network Operator deploys DOCA-OFED drivers, supports RDMA over InfiniBand and RoCE, SR-IOV and IP over InfiniBand secondary networks, and works alongside the GPU Operator. Slinky documents SR-IOV setup, and notes that virtual functions must be created on the hosts beforehand and re-created after each reboot.

Topology. slurm-operator copies each Kubernetes node’s topology annotation to its slurmd pod so Slurm’s topology-aware scheduling works without listing every node in advance. Kueue’s Topology-Aware Scheduling uses node labels and a Topology object instead. Volcano can discover HyperNodes for an InfiniBand fabric from NVIDIA UFM.

MPI launch. Inside Slurm, srun launches MPI ranks with PMIx as usual (see the Slurm MPI guide). On Kubernetes-native batch, MPI jobs can run through the Kubeflow MPI Operator’s MPIJob, which Kueue supports, or through a Volcano Job.

Decision table

Your situationPattern to start with
Existing Slurm users, bare-metal MPI jobs, Kubernetes only for servicesSide by side
You want Slurm policy and commands, but Kubernetes to manage nodes, upgrades and scalingSlinky slurm-operator
Kubernetes workloads should queue under Slurm’s priority and fairshareSlinky slurm-bridge
Everything is containerised, the team runs Kubernetes, you want to keep kube-schedulerKueue
Containerised batch with gang scheduling and a dedicated batch schedulerVolcano
You need Slurm accounting, QOS and time-limit backfill exactly as todayKeep Slurm (side by side or Slinky)

Whichever pattern you choose, test it with your own jobs first: a multi-node MPI or NCCL run, a GPU-sharing case and a preemption case. Our Slurm solution page and Kubernetes solution page describe the architectures we build on each side.

Mevasis installs Slurm clusters and provides consulting and managed support. If you are weighing these options for a GPU cluster, our Slurm services page explains how we work.

Frequently Asked Questions

Yes. SchedMD's Slinky project provides slurm-operator, which runs slurmctld, slurmd, slurmrestd, login nodes and optionally slurmdbd as pods managed through Kubernetes custom resources. The 1.3 release supports Kubernetes 1.35 to 1.37 and Slurm 25.11 or later.

slurm-operator runs a Slurm cluster inside Kubernetes, and users keep submitting with sbatch, srun and salloc. slurm-bridge makes Slurm the scheduler for selected Kubernetes workloads such as Pods, Jobs and JobSets by translating them into Slurm jobs.

Not one to one. Kueue adds queues, quotas, fair sharing, preemption and topology-aware admission to Kubernetes, but it works alongside kube-scheduler and users submit Kubernetes objects instead of batch scripts. Teams that rely on Slurm accounting, QOS and time-limit backfill should test their policies before switching.

Kueue keeps the default kube-scheduler and controls when jobs are admitted against quota. Volcano is a separate batch scheduler, selected with schedulerName volcano, with its own queues, gang scheduling and scheduling actions. The deciding question is whether you want to keep kube-scheduler in charge of pod placement.

In two layers. Kubernetes assigns GPUs to the slurmd pods through the NVIDIA device plugin or Dynamic Resource Allocation, and Slurm then schedules jobs on those GPUs through GRES defined in gres.conf.

  • Share:

Let's Build Future Together.