Slurm vs Kubernetes for AI Training: The 2026 GPU Scheduling Decision Guide

A 512-GPU training run does not fail because the model is wrong. It fails because 63 of 64 nodes were allocated, the last pod sat Pending, and the other 63 spent six hours holding expensive GPUs while waiting for a peer that never arrived. That single failure mode, partial allocation of an all-or-nothing job, is the reason the Slurm vs Kubernetes question exists at all. Slurm was built around it. Kubernetes was built around services that tolerate it.

In 2026 the gap has narrowed from both sides. Kubernetes gained queueing and topology-aware admission through Kueue and Volcano, and the core project itself now ships early workload-aware scheduling primitives. Slurm gained an official Kubernetes operator, Slinky, from its maintainer SchedMD. The honest answer is no longer “pick one,” but “decide where the scheduling authority lives.”

What this covers: how each scheduler actually makes decisions, where Kueue and Volcano differ, what Slinky and SUNK-style bridges change, how checkpointing and fairness interact with each choice, a decision matrix by team size and workload, an sbatch script, a Kueue ClusterQueue manifest, and the migration pitfalls that bite in production.

Context and Background

Slurm (originally the Simple Linux Utility for Resource Management) has run most of the world’s large HPC systems for two decades. Its design assumes a fixed pool of nodes, long-lived queues, users who submit scripts with explicit time limits, and a scheduler whose job is to pack a calendar. A job is a request for N nodes, G GPUs and a wall-clock limit, and the scheduler’s output is a start time.

Kubernetes was designed in the opposite direction. Its scheduler places one pod at a time, immediately, against current free capacity, and assumes that a pending pod is a transient condition that an autoscaler or a retry will resolve. That works for stateless services and is exactly wrong for a distributed training job whose ranks must all start together. Deep-learning teams adopted Kubernetes anyway because they already ran serving, data pipelines and CI there, and because containers, namespaces and RBAC are the lingua franca of platform teams.

The result was a decade of bolt-ons. The Kubernetes Scheduling Framework allowed out-of-tree schedulers; Volcano, a CNCF project, introduced batch-oriented queues and gang scheduling. The Kubernetes SIG Scheduling and Batch communities built Kueue, a job queueing controller that decides when a workload should be admitted and leaves pod placement to the default scheduler. For a deeper look at how the default scheduler itself works, see our walkthrough of the Kubernetes scheduling framework, topology spread constraints and descheduler.

Meanwhile, GPU clusters grew faster than the tooling. When a rack of eight-GPU nodes costs more per hour than a small team’s salary, idle time is a budget line. The scheduling layer is where idle time is created or destroyed, so it deserves the same rigor as the network fabric. If you are sizing that fabric as well, our comparison of RoCE v2, InfiniBand and Ultra Ethernet for AI cluster networking covers the other half of the problem.

One caution on framing. “Kubernetes” is not a scheduler for batch jobs; it is a platform, and the real comparison is Slurm against a stack: kube-scheduler plus a queueing layer (Kueue or Volcano) plus an operator for the framework (JobSet, Kubeflow Trainer, KubeRay). Comparing Slurm to bare kube-scheduler is a strawman, and so is comparing a mature Slurm site to an unconfigured Kubernetes cluster.

Two Scheduling Philosophies: A Reference Architecture View

Slurm schedules a calendar of jobs against a static inventory, using priority, fairshare and backfill. Kubernetes schedules individual pods against live capacity; batch semantics such as queues, quotas, preemption and gang admission come from add-on controllers like Kueue and Volcano, which decide when a whole workload may start.

That one-paragraph answer hides the structural difference, so it is worth drawing both stacks before comparing features.

How Slurm is put together

A Slurm cluster has a small number of daemons. slurmctld is the central controller holding the job queue, the node table and the scheduler. slurmd runs on every compute node, launches slurmstepd helpers that supervise job steps, and enforces resource limits through cgroups. slurmdbd fronts a SQL database that stores accounting records, which in turn feed fairshare and quality-of-service (QOS) policy.

Figure 1: Slurm control and data plane. The scheduler lives in slurmctld; slurmdbd supplies the usage history that drives fairshare; slurmd and slurmstepd enforce allocations on each node.

The figure shows the key architectural property: there is one authority. A job is either allocated by slurmctld or it is not, and an allocation is a set of whole nodes, specific GPUs and an expiry time. The scheduler knows the future because every job carries a time limit.

GPUs are modelled as generic resources (GRES). You declare GresTypes=gpu in slurm.conf, give each node a GPU count, and describe devices in gres.conf. According to the Slurm GRES documentation, AutoDetect=nvml can fill in device details from the NVIDIA Management Library, though the count in slurm.conf is still required and a node with fewer devices than configured is drained. File entries map to device paths such as /dev/nvidia1, and the docs warn that NVML indexes GPUs by PCI bus ID while device files follow Linux minor numbers, so the mapping should be verified after boot.

Isolation comes from cgroup.conf. The cgroup.conf reference documents ConstrainCores, ConstrainRAMSpace and ConstrainDevices, all defaulting to no. ConstrainDevices limits a job to the device files tied to its allocated GRES, which is what keeps a job that asked for two GPUs from seeing the other six. The same page notes that the cgroup/v1 plugin is deprecated and recommends moving to cgroup/v2.

How the Kubernetes batch stack is put together

On the Kubernetes side, responsibility is split across layers, and that split is the source of both its flexibility and its footguns.

Figure 2: Kueue-centred admission. Kueue wraps a job in a Workload, reserves quota in a ClusterQueue, optionally computes topology placement, and only then lets kube-scheduler bind pods.

Per the Kueue concepts page, a ClusterQueue is a cluster-scoped pool of resources with usage limits and fair-sharing rules; a LocalQueue is a namespaced queue that groups one tenant’s workloads; a ResourceFlavor describes a class of nodes (GPU model, architecture, price tier); a Cohort is a group of ClusterQueues that can borrow unused quota from each other; and a Workload is the unit of admission. A workload is admitted when it has a quota reservation, fits physical node capacity if topology-aware scheduling is used, and all admission checks are ready.

Note what Kueue is not. It does not place pods. It suspends a job, decides whether quota exists, and unsuspends it, after which the ordinary kube-scheduler binds pods. This division is deliberate and is the main architectural contrast with Volcano, discussed below.

Where gang semantics come from

Gang scheduling means all pods of a group start together or none do. In Kueue the guarantee comes from admission: a Workload with a multi-pod PodSet reserves quota for the whole set before any pod is created, so quota is never partially consumed. Be careful about what that does and does not promise. Quota reservation prevents the accounting deadlock; it does not by itself guarantee that kube-scheduler can bind every pod if node-level constraints such as fragmentation, taints or topology interfere. This is precisely why Kueue added topology-aware scheduling, covered later.

The core project is also moving. The Kubernetes project’s documentation lists a Workload API and, from v1.36, a separate PodGroup API for runtime state, with a PodGroup scheduling cycle in kube-scheduler that processes a group atomically, plus first iterations of topology-aware scheduling and workload-aware preemption, according to the v1.36 workload-aware scheduling announcement. I could not verify the maturity level or feature-gate names of these features from the source I retrieved, so treat them as early-stage and check the release notes for your cluster version before depending on them.

Slurm Internals That Matter for AI Training

Most Slurm-versus-Kubernetes debates compare feature lists. The more useful comparison is how each system turns a pile of pending jobs into a decision, because that determines utilization, queue wait and fairness.

Priority, fairshare and the multifactor plugin

Slurm computes a priority for every pending job as a weighted sum of factors. The multifactor priority documentation lists age, fairshare, job size, partition, QOS and per-TRES factors (TRES, trackable resources, covers CPUs, memory, GPUs and licenses), each normalized between 0.0 and 1.0 and multiplied by an admin-set weight. All weights default to zero, so nothing influences priority until you configure it. The docs recommend large weights, on the order of 1000 or more, so small fairshare differences still change the ordering, and their example uses PriorityWeightFairshare=10000 with PriorityWeightAge=1000.

Fairshare compares what an account is entitled to with what it has consumed, and under-served accounts go first. Usage decays with a half-life, PriorityDecayHalfLife, which defaults to seven days. A longer half-life means a team that burned a huge allocation last month is still penalized today; a shorter one forgives quickly. Fair Tree has been the default fairshare algorithm since Slurm 19.05. I have deliberately not reproduced the exact fairshare formula, since the page I fetched did not state it.

This matters for AI work because GPU-hours are the scarce unit. Setting a TRES weight on GPUs, and charging usage by GPU rather than by node, makes a team that holds eight GPUs for ten hours look heavier than one that holds a single CPU node for ten hours. Many sites forget this and inherit CPU-centric fairness on GPU machines.

Backfill: the feature Kubernetes does not have by default

Backfill is Slurm’s answer to the fragmentation that gang jobs cause. The scheduler configuration guide explains that the backfill scheduler can start lower-priority jobs early, but only if doing so will not delay the expected start of any higher-priority job. It reserves whole nodes for pending jobs, accounts for GRES and memory, and exposes expected start times through squeue --start.

The mechanism depends entirely on honest time limits: the guide states that reasonably accurate time limits are important for backfill to work well. Key parameters are bf_window (how far ahead it plans, default 1440 minutes, which should be at least the longest allowed time limit to avoid starvation), bf_interval (default 30 seconds) and bf_continue. Set DefaultTime and MaxTime per partition so users who omit a limit do not get an unbounded one.

Consider the practical effect. A 64-node job is at the head of the queue and will start in four hours when enough nodes free up. Meanwhile, sixteen idle nodes sit waiting. Backfill sees that a two-hour, eight-node job can run on those nodes and finish before the big job’s reservation begins, so it starts it. No one is delayed and utilization rises. That behavior is a direct consequence of knowing the future, which in turn comes from time limits.

Kubernetes batch queueing has no universal equivalent, because pods and many Kueue-managed workloads (serving, interactive notebooks, open-ended training) have no declared duration. Kueue’s behavior is closer to “admit the highest-priority workload that fits quota now, preempt if policy allows” than to “build a reservation calendar.” For AI training that is often acceptable, since long runs use checkpointing and are preemptible by design, but it is a real difference in how idle capacity gets used.

A concrete sbatch script

The following script requests two eight-GPU nodes, one task per GPU, with an explicit time limit so backfill can reason about it. The srun line launches under Slurm’s own process manager.

#!/bin/bash
#SBATCH --job-name=llm-pretrain
#SBATCH --partition=gpu
#SBATCH --account=team-research
#SBATCH --qos=normal
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8
#SBATCH --cpus-per-task=12
#SBATCH --time=12:00:00
#SBATCH --signal=B:USR1@600
#SBATCH --requeue
#SBATCH --output=logs/%x-%j.out

export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
export MASTER_PORT=29500
export NCCL_DEBUG=WARN

# Ask the training loop to checkpoint ten minutes before the time limit
trap 'touch "$CKPT_DIR/.save_now"' USR1

srun --kill-on-bad-exit=1 \
  python train.py --resume-from-latest --ckpt-dir "$CKPT_DIR"

Three details deserve comment. --time is not decoration; it is the input to backfill. --signal=B:USR1@600 delivers a signal to the batch shell ten minutes before the limit so the job can checkpoint, and --requeue lets Slurm restart the job after preemption or node failure. --kill-on-bad-exit=1 ensures one dead rank tears down the step instead of leaving the others hanging in a collective.

Everything above is declarative about resources and silent about containers. Slurm sites typically add Pyxis and Enroot or Apptainer for containerized execution, which is the part Kubernetes gives you for free. This is the first of several places where the two ecosystems have complementary gaps.

Kubernetes Batch Scheduling: Kueue, Volcano and Native Primitives

Kueue in practice

Kueue’s strength is that it composes with the Kubernetes ecosystem rather than replacing parts of it. Its integration guides cover Jobs, JobSets, Ray, Kubeflow training operators, plain pods and more, and the controller stays out of pod placement. The manifest below is a ClusterQueue for a team-owned slice of GPU capacity that can borrow idle quota from a shared cohort.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: h100-80gb
spec:
  nodeLabels:
    nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: team-research-cq
spec:
  namespaceSelector:
    matchLabels:
      kueue-team: research
  cohortName: gpu-shared
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: h100-80gb
      resources:
      - name: "cpu"
        nominalQuota: 512
      - name: "memory"
        nominalQuota: 4Ti
      - name: "nvidia.com/gpu"
        nominalQuota: 32
        borrowingLimit: 16
        lendingLimit: 8
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: research
  namespace: team-research
spec:
  clusterQueue: team-research-cq

The field names follow the structure on the ClusterQueue concept page, which showed kueue.x-k8s.io/v1beta2 in its examples; the page’s prose and examples disagree on whether the cohort field is spelled cohort or cohortName, so confirm against the API reference for your installed Kueue version. The quota numbers, node label and GPU model above are illustrative, not recommendations.

Read the policy as follows. The team owns 32 GPUs nominally. It may borrow up to 16 more from idle cohort peers, and may lend at most 8 of its own. If a peer’s workload needs its quota back, reclaimWithinCohort: Any lets the owner preempt borrowers. withinClusterQueue: LowerPriority lets higher-priority workloads in the same queue preempt lower-priority ones. The docs note that borrowWithinCohort requires reclaim to be enabled and cannot be combined with Fair Sharing, so choose between strict quota semantics and fair-sharing semantics deliberately.

A job attaches to the queue with a single label, kueue.x-k8s.io/queue-name, which is why adopting Kueue for existing manifests is cheaper than adopting a new job API.

Topology-aware scheduling: where Kueue closes the gap

Quota is not placement. Eight pods that each fit somewhere in the cluster may still land across four racks and cripple all-reduce bandwidth. Kueue’s topology-aware scheduling (TAS) addresses this. An administrator creates a Topology object whose levels map to node label keys, references it from a ResourceFlavor via spec.topologyName, and then annotates pod templates.

Per the Kueue TAS guide, kueue.x-k8s.io/podset-required-topology forces all pods into one domain at the named level (the workload waits if none has capacity), while kueue.x-k8s.io/podset-preferred-topology tries a domain and widens upward if necessary. These annotations belong on the pod template, and exactly one is used per PodSet. The same page reports that the TopologyAwareScheduling gate has been beta and on by default since v0.14, with some related profile and layering gates in alpha or beta.

The practical lesson: for communication-bound training, use “required” at the rack or block level when the job is small enough to fit, and “preferred” when you would rather run slowly than not at all. If an annotation seems ignored, the first thing to check is that the flavor has topologyName set.

Kueue vs Volcano

The Kueue vs Volcano question is mostly an architectural one. Volcano replaces the scheduling path: it provides its own scheduler, a Queue resource with multi-dimensional quotas and hierarchy, a VolcanoJob (vcjob) API, and plugin-based actions. Its documentation lists gang scheduling, queue borrowing, reclaim and preemption, network topology-aware scheduling, online and offline workload colocation, and multi-cluster scheduling. CNCF describes it as a batch system that moved to the incubator in 2022.

Kueue instead layers admission on top of the default scheduler. That means you inherit the default scheduler’s features (DRA integration, scheduling gates, the same plugins) and avoid running a second scheduler, but you also inherit its limits on pod-level placement.

Dimension	Kueue	Volcano
Where it sits	Admission controller; kube-scheduler still binds pods	Separate scheduler (`volcano-scheduler`) plus its own CRDs
Gang semantics	All-or-nothing quota reservation per Workload	All-or-nothing PodGroup scheduling in its own scheduler
Tenancy model	ClusterQueue, LocalQueue, Cohort	Queue, with hierarchy
Job API	Annotates existing Jobs, JobSets, Ray, Kubeflow	Native VolcanoJob plus framework integrations
Topology	Topology CRD and pod annotations	Network topology-aware plugins
Second scheduler to operate	No	Yes
Typical fit	Teams that want to stay close to upstream	Sites needing rich queue policy and plugin control

Neither is universally better. If your priority is upstream alignment and minimal moving parts, Kueue is the lower-friction path. If you need Volcano’s queue hierarchy or plugin actions today, or you already run it, replacing it for fashion is unwarranted risk. Avoid running both over the same pods; two controllers each believing they own admission produces confusing deadlocks.

Slurm on Kubernetes: Slinky and SUNK

The third option is to stop choosing. Two projects put a real Slurm scheduler inside Kubernetes.

Slinky, SchedMD’s operator

Slinky is SchedMD’s Kubernetes project for running Slurm. Its slurm-operator documentation describes custom controllers and CRDs that manage the lifecycle of Slurm clusters, with a Helm chart for a reference deployment.

Figure 3: Slurm on Kubernetes with Slinky. The operator reconciles a Slurm controller pod, NodeSets whose pods each represent a slurmd worker, and LoginSets for user access.

According to that documentation, the main components are a Controller (the Slurm control plane, restarted by Kubernetes if it crashes, so Slurm’s backup-controller HA is not used and no shared filesystem is required), NodeSet (a group of identical Slurm workers, scaled either as a StatefulSet with stable ordinal names or as a DaemonSet with one pod per matching node), and LoginSet (login nodes managing identity through SSSD). The CRDs listed on the uninstall page include accountings, loginsets, nodesets, restapis and tokens, with the earlier clusters CRD marked defunct.

Each NodeSet pod represents a Slurm worker, and its Slurm state (Idle, Allocated, Drain, Down) is reflected in pod conditions. Before scale-in or upgrades, the operator drains affected Slurm nodes so running work is considered. That is the essential trick: Kubernetes lifecycle events are translated into Slurm-aware operations instead of killing a slurmd under a running job.

The documented requirements are Kubernetes v1.29 or later, Slurm 25.11 or later, cgroup v2, and cert-manager with Helm. The docs show a v1.1.0 page with later tags up to v1.2.3 in the version list, use Semantic Versioning, and keep older CRD versions working through conversion within a v1 line, whereas v0 upgrades could be breaking. The page I retrieved did not give a production-readiness label, and community issues are handled on a best-effort basis with official support through SchedMD. Check the current release and support terms before committing.

CoreWeave SUNK

SUNK (Slurm on Kubernetes) is CoreWeave’s implementation, introduced in their October 2023 announcement. Per that post, compute nodes are bare-metal Kubernetes nodes each running a slurmd pod mapped one-to-one, managed by a custom Nodeset CRD. A Syncer uses Slurm’s REST API to keep Kubernetes and Slurm state consistent in both directions, so a Slurm drain can annotate a pod and a Kubernetes cordon can drain the Slurm node.

The notable idea is the reverse bridge. The post describes a Slurm-based Kubernetes scheduler: a pod that selects it causes a Slurm job to be created, which can preempt low-priority work, after which the pod lands on the allocated node. The same post says this integration was forthcoming at the time of writing, so verify its current availability. The stated goal is one pool of compute shifting between batch training and inference. SUNK is a CoreWeave product offering, while Slinky is the SchedMD project, so treat them as related in idea, not interchangeable.

What a bridge actually buys you

A Slurm-on-Kubernetes bridge preserves what researchers already know (sbatch, squeue, accounts, fairshare, backfill) while giving the platform team Kubernetes lifecycle, Helm-based GitOps, container images and one control plane for the rest of the fleet. It does not erase the fundamental duality. Two schedulers still exist: Slurm decides what runs on the GPU nodes it owns, and Kubernetes decides everything else. Sharing the same physical pool between them still requires an explicit mechanism, and “which one is the source of truth for GPU ownership” remains a design decision, not a default.

Deeper Analysis: Checkpointing, Fairness, Observability and the Decision

Preemption is only as good as your checkpoints

Both worlds preempt. Slurm preempts through partition priority tiers or QOS, then signals, requeues or cancels. Kueue preempts by evicting an admitted Workload, which deletes or suspends its pods, and the Workload returns to the queue. In both cases the cost of preemption is the work since the last checkpoint plus the restart overhead of re-forming the process group and reloading state.

That makes checkpoint cadence the real scheduling policy. If a job checkpoints every 30 minutes and preemption is frequent, a typical preempted job loses about half an interval, so roughly 15 minutes on average, before counting restore time. Multiply by the number of GPUs and the number of preemptions per day, and a “free” utilization gain from aggressive borrowing can turn negative. This arithmetic is simple on purpose: it is a first-principles estimate, not a benchmark, and your own checkpoint size and storage bandwidth set the true numbers. Our guide on spot GPU training, checkpointing and preemption-tolerant LLM jobs goes deeper on saving state quickly enough for this to be viable.

The mechanics differ. In Slurm, you request advance warning with --signal and handle it in the batch script, as in the script earlier. In Kubernetes, the equivalent is a preStop hook plus a sufficient terminationGracePeriodSeconds, and the grace period must exceed your slowest checkpoint write. A common production bug is a default 30-second grace period against a multi-gigabyte shard flush, which silently produces a truncated checkpoint that fails to load on resume. Write checkpoints atomically (write to a temporary path, then rename) so a half-written file is never mistaken for the latest.

Elastic training changes the scheduling problem

Classic gang scheduling assumes a fixed world size. Elastic training relaxes that. PyTorch’s torchrun accepts a range such as --nnodes=2:8 with a rendezvous backend such as c10d, and restarts workers when membership changes, bounded by --max-restarts. An elastic job can therefore start on the minimum and grow if capacity appears, which makes it a good citizen on a shared cluster.

The scheduler consequences are significant. Under Slurm, elasticity is awkward: an allocation is a fixed set of nodes for a fixed time, and growth means submitting additional work or using heterogeneous and dependency patterns. Under Kubernetes, scaling a replica count is natural, and a queueing layer can admit a job at its minimum size. Whether the admission layer supports partial admission of a workload depends on the integration and Kueue version, so verify that against the documentation for the job type you use rather than assuming it.

The trade-off is that elastic jobs change effective batch size or require learning-rate adjustment if world size changes, so the training code, not the scheduler, must be elastic-correct. Most large pretraining runs are therefore still fixed-size with fast resume, not elastic. Elasticity pays off most for fine-tuning, hyperparameter sweeps and spot capacity.

Multi-tenant fairness: two different mental models

Slurm fairshare is historical. It compares an account’s consumption over a decaying window to its configured share, which rewards teams that have been quiet and delays teams that just finished a big run. It is also hierarchical through the account tree, so a department’s share is split among its users. The decay half-life is the tuning knob, and its seven-day default means a month-old burst is mostly forgotten.

Kueue and Volcano quotas are instantaneous. A ClusterQueue has a nominal quota, may borrow idle capacity, and reclaims it through preemption when the owner returns. There is no memory of last month, so a team that always uses its full quota is never penalized for it. Kueue also provides Fair Sharing, but the docs note that borrowWithinCohort cannot be combined with it, so you must pick the model.

Neither is better; they answer different questions. Quota systems give predictable guarantees (“team A always gets 32 GPUs on demand”), which finance and contract-driven organizations like. Fairshare gives equitable long-run access and handles uneven demand gracefully, which research institutions prefer. A good hybrid is hard quotas for production and customer-facing workloads, and fairshare or fair-sharing for the research pool.

GPU observability: scheduler state is not GPU state

Neither scheduler tells you whether an allocated GPU is doing useful work. A job can hold eight GPUs and keep them at 5 percent utilization because the data loader is starved. That wastes the scarcest resource in the building while every scheduler dashboard looks healthy.

NVIDIA’s DCGM (Data Center GPU Manager) exporter publishes GPU metrics in Prometheus format. Commonly used fields include DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED for framebuffer memory, DCGM_FI_DEV_XID_ERRORS for driver-reported errors, and the profiling field DCGM_FI_PROF_GR_ENGINE_ACTIVE; confirm the exact set enabled in your exporter’s metrics configuration, because it is configurable. Join those series to job identity, using the Kubernetes pod labels the exporter attaches or, on Slurm, a prolog that records job-to-GPU mappings, and you can answer the question that matters: which jobs hold GPUs but do little with them.

Operationally, three alerts pay for themselves. First, allocated-but-idle GPUs (allocated by the scheduler, low engine activity for more than a threshold window). Second, XID errors on a node, which should trigger a drain: scontrol update nodename=... state=drain reason="xid" in Slurm, or a cordon and taint in Kubernetes. Third, queue wait time per team, because rising wait times are the early signal that quotas or priorities are miscalibrated. Our look at FinOps and GreenOps for carbon-aware cloud scheduling shows how to turn the same utilization data into cost and carbon accounting.

Dynamic Resource Allocation and the road ahead

Kubernetes Dynamic Resource Allocation (DRA) replaces the opaque nvidia.com/gpu: 1 counter with structured device claims, so a pod can request a GPU with specific attributes, or a MIG slice. That gives Kubernetes something Slurm GRES has had for years: typed, attribute-aware devices. The v1.36 announcement above also mentions ResourceClaim support for workloads, so DRA is being connected to PodGroups. Our DRA GPU scheduling guide covers the claim model in detail.

Slurm’s GRES documentation supports Multi-Instance GPU (MIG) from version 21.08 with AutoDetect=nvml, treating each MIG instance as an individual GPU with cgroup isolation, though it states that Slurm does not dynamically partition MIGs. It also supports “shards”, which let multiple jobs share one GPU without process fencing and so suit homogeneous workloads only. Whatever scheduler you choose, GPU-sharing policy needs an explicit owner, because nothing prevents two noisy jobs from sharing a card badly.

The Decision: Matrix by Team Size and Workload

Figure 4: A decision flow for choosing a scheduling authority. The answers depend on who your users are, whether Kubernetes is already the platform of record, and whether serving and batch must share GPUs.

The flow reduces to a few questions. Are your users researchers who think in sbatch and queues, or ML engineers who think in containers and manifests? Does the platform team already run Kubernetes for everything else? Must inference and training draw from one pool? The matrix below applies those questions to common situations. Treat it as opinion informed by the mechanisms above, not as measured data.

Situation	Suggested primary scheduler	Why
Small team, under roughly 64 GPUs, mixed training and serving	Kubernetes with Kueue	One platform, simple quotas, no second system to learn
Research lab, hundreds of GPUs, many users, long fixed-size jobs	Slurm	Backfill and fairshare fit the workload; users already know it
Platform team standardized on Kubernetes, many tenants	Kubernetes with Kueue, add Volcano only for specific policy needs	Stays close to upstream; avoids a second scheduler
Existing Slurm site that needs containers, GitOps and shared fleet	Slurm on Kubernetes (Slinky or similar)	Keeps user workflow, modernizes operations
Heavy HPC plus AI on the same machines (MPI, simulation, training)	Slurm	MPI launch, binding and accounting are first-class
Spiky inference plus batch training sharing GPUs	Kubernetes, possibly with a bridge	Autoscaling and serving are native to Kubernetes
Multi-cluster or multi-cloud burst	Kubernetes-based stack	Slurm federation exists, but Kubernetes tooling for cloud elasticity is more common
Strict contractual GPU guarantees per customer	Kubernetes quotas (Kueue or Volcano queues)	Instantaneous nominal quota matches the contract

Three rules of thumb sit under the table. First, follow your users: a scheduler researchers refuse to use becomes shelfware regardless of its elegance. Second, follow your operations: a team that cannot patch Slurm’s database or debug Slurm’s cgroups will struggle, as will a team with no Kubernetes expertise running Volcano. Third, count schedulers: every additional authority over the same GPUs multiplies the places a deadlock or silent starvation can hide.

For model serving after training, the scheduling question changes shape, since serving wants autoscaling and latency guarantees rather than queues. Our comparison of KServe, Ray Serve and BentoML for model serving on Kubernetes is a natural companion for that half of the pipeline.

Trade-offs, Gotchas, and What Goes Wrong

Time limits are policy, not paperwork. Backfill rewards accurate limits and punishes padded ones. If users routinely request 24 hours for 4-hour jobs, the scheduler plans around phantom occupancy and utilization drops. Enforce partition MaxTime, publish typical durations, and show users their wall-time-requested versus used ratio.

Quota reservation is not placement. On Kubernetes, a Workload can be admitted by Kueue, yet pods remain Pending because a node taint, an unschedulable DaemonSet, or fragmentation blocks binding. The quota is then held while nothing runs. Topology-aware scheduling reduces this for placement, but monitor “admitted but pending” as its own metric and set an admission timeout where your tooling supports one.

Two schedulers, one GPU. The most expensive mistake in a hybrid design is letting both Slurm and kube-scheduler believe they own the same nodes. If a Slurm job is running on a node and Kubernetes also schedules a GPU pod there, one of them will fail in a confusing way. Partition the fleet explicitly, with taints and labels, or use a bridge that makes one system authoritative.

Network and storage hide in the scheduler. A job admitted with perfect gang semantics can still crawl if its ranks are split across oversubscribed uplinks or if all of them hit the same shared filesystem metadata server at checkpoint time. Scheduler decisions about topology are only worthwhile if the fabric underneath was designed for the collective pattern.

Slurm-on-Kubernetes inherits both failure surfaces. You now debug cgroup v2 behavior, Slurm configuration, pod lifecycle and operator reconciliation at once. Operator versioning matters too: the Slinky docs warn that v0 upgrades could be breaking, and that requirements include a minimum Slurm and Kubernetes version. Pin versions and rehearse upgrades on a staging NodeSet.

Checkpoint truncation and grace periods. As noted, a short termination grace period produces unusable checkpoints. Test resume from a checkpoint taken at forced preemption, not only from a clean one.

Migration drift. Moving users from sbatch to YAML sounds trivial and is not. Environment modules, shared home directories, srun semantics, job arrays, dependencies (--dependency=afterok) and accounting reports all have to be re-created or consciously dropped. Plan for a dual-run period and a translation guide, and expect the long tail of workflows (job arrays especially) to take longer than the headline training jobs.

Practical Recommendations

Start by writing down your workload truth: GPU count, job size distribution, typical duration, fraction that is preemptible, and whether serving shares hardware. Most bad scheduler choices come from skipping this step and adopting whatever the loudest engineer prefers.

If you are on Slurm and it works, do not migrate for fashion. Add containers through Pyxis and Enroot or Apptainer, instrument with DCGM, and tune fairshare weights for GPU TRES. If the pain is operational (patching, provisioning, a fleet that is mostly Kubernetes), evaluate Slinky in a staging environment and read its current support and maturity statements first.

If you are on Kubernetes, adopt Kueue before reaching for a second scheduler. Define ResourceFlavors per GPU model, one ClusterQueue per team or tier, a cohort for borrowing, and turn on topology-aware scheduling for any job whose speed depends on the interconnect. Add Volcano only when you can name the feature Kueue cannot give you.

Whichever you choose, make checkpointing and resume a platform contract, not a user favor.

Record GPU utilization per job with DCGM and alert on allocated-but-idle GPUs.
Set explicit time limits (Slurm) or priority classes and preemption policy (Kubernetes) per tier.
Test preemption end to end, including resume from a forced checkpoint.
Pick one authority per GPU node; never share a node between two schedulers.
Use topology-required placement only for jobs that actually need it.
Pin and rehearse upgrades of Kueue, Volcano or the Slinky operator.
Review queue wait time per team monthly and adjust quota or fairshare weights.

Frequently Asked Questions

Is Slurm better than Kubernetes for AI training?

It depends on the workload. Slurm is better when jobs are large, fixed-size, long-running and submitted by researchers, because backfill, fairshare and explicit time limits make it excellent at packing a queue. Kubernetes, with Kueue or Volcano added, is better when training must coexist with serving and data pipelines on one platform. Neither wins universally; the deciding factors are user familiarity, existing infrastructure and whether inference shares the GPU pool.

What is the difference between Kueue and Volcano?

Kueue is an admission controller: it queues workloads, enforces quotas through ClusterQueues, and then lets the default kube-scheduler place pods. Volcano is a separate batch scheduler with its own Queue and VolcanoJob resources, gang scheduling and plugin actions. Kueue stays closer to upstream Kubernetes and avoids a second scheduler; Volcano offers richer scheduler-level policy but adds another component to operate.

Does Kubernetes support gang scheduling natively?

Not fully in the traditional sense. Kubernetes has added early workload-aware scheduling primitives, with a Workload API and a PodGroup API described in the v1.35 and v1.36 release material, but this is still evolving and you should confirm maturity for your version. Today most production clusters get gang behavior from Kueue’s all-or-nothing admission or Volcano’s gang scheduling rather than from the core scheduler alone.

Can you run Slurm on Kubernetes?

Yes. SchedMD’s Slinky project provides a slurm-operator with Controller, NodeSet and LoginSet components, and CoreWeave’s SUNK is a separate implementation. Each Slurm worker runs as a pod, and the operator drains Slurm nodes before scale-in or upgrades. Slinky documents requirements of Kubernetes 1.29 or later, Slurm 25.11 or later, cgroup v2 and cert-manager. Check support terms and maturity before production use.

How does GPU scheduling differ from CPU scheduling?

GPUs are scarce, expensive, indivisible by default and topology-sensitive. Jobs usually need several at once and need them physically close, connected by NVLink or a fast fabric. Schedulers therefore need typed devices (Slurm GRES, Kubernetes device plugins or DRA), gang admission, topology awareness, and checkpoint-aware preemption. Plain CPU bin-packing optimizes the wrong thing for distributed training.

What does backfill do and does Kubernetes have it?

Backfill lets lower-priority jobs start early on idle resources provided they do not delay the expected start of a higher-priority job, which depends on accurate time limits. Kubernetes has no universal equivalent because pods often lack declared durations. Queueing layers admit what fits quota now and use preemption instead of a reservation calendar, so utilization behavior differs even when the workloads look the same.