Kubernetes 1.37 Gang Scheduling vs Volcano, Kueue and YuniKorn

For six years the answer to “how do I run a distributed training job on Kubernetes without deadlocking my GPU fleet?” was the same: install a second scheduler. Kubernetes gang scheduling now ships in-tree, and as of v1.37 the Workload and PodGroup APIs are Beta, sitting in scheduling.k8s.io/v1beta1 alongside a brand-new hierarchical CompositePodGroup type. That changes the calculus for every platform team currently running Volcano, Kueue or Apache YuniKorn — but not in the direction the release notes imply. The all-or-nothing placement problem is genuinely solved upstream. The problem most people actually bought a batch scheduler for — quota, fair share and queueing across tenants — is not, and will not be in v1.38 either.

This post separates those two things precisely, so you can decide what to keep, what to delete, and what to migrate.

What this covers: what v1.37 actually puts in-tree, the CompositePodGroup tree model and its alpha caveat, a concrete v1alpha2-to-v1alpha3 migration with the disruptionMode rename, and a decision matrix for Volcano, Kueue and YuniKorn.

Context and Background

Kubernetes has scheduled Pods one at a time since the beginning. That is the right default for a stateless web service and exactly the wrong default for anything that only makes progress when every replica is running. A data-parallel training job with eight workers does not run at 5/8 speed when five Pods land and three stay pending; it runs at zero, while holding five accelerators hostage. Two such jobs arriving together can each grab a partial allocation and block the other indefinitely — the classic batch deadlock, and the reason that every serious AI platform on Kubernetes has carried an out-of-tree scheduler.

The three incumbents solved it in three different places. Volcano replaces kube-scheduler outright with an action-and-plugin pipeline — enqueue, allocate, preempt, reclaim, backfill — where a gang plugin enforces minAvailable inside the allocation loop. Kueue sits above the scheduler as an admission controller: it holds Workload objects in a ClusterQueue until quota is available, then lets the stock scheduler place the Pods. Apache YuniKorn replaces the scheduler too, and implements gangs by first binding placeholder Pods that reserve the exact shape of each task group, then swapping real Pods into the reservations.

The upstream effort, Workload-Aware Scheduling, arrived in v1.35 as a concept, gained flat PodGroup objects and single-level topology awareness in v1.36, and reached its first Beta milestone in v1.37. The direction of travel is explicit in the SIG Scheduling roadmap: Workload and PodGroup are targeted at GA in v1.38, and the working group states an intent for Kueue to eventually use Workload-Aware Scheduling as its underlying gang and topology engine rather than duplicating it. That is the single most important strategic signal in the release, and it is worth understanding before you rip anything out. If you are already running accelerators under Dynamic Resource Allocation in Kubernetes 1.37, the two features now interlock directly.

What Kubernetes 1.37 Actually Puts In-Tree

Kubernetes v1.37 “Garhwal”, released on 26 August 2026 with 67 enhancements (16 Stable, 23 Beta, 27 Alpha, 1 deprecation), promotes the core Workload and PodGroup APIs to scheduling.k8s.io/v1beta1. Gang scheduling, workload-aware preemption and shared DRA ResourceClaims for PodGroups all graduate to Beta together. The hierarchical CompositePodGroup API enters Alpha at scheduling.k8s.io/v1alpha3.

Figure 1: Why per-Pod queueing deadlocks a distributed job, and what all-or-nothing gang admission does instead.

The left branch is the failure mode every batch platform engineer has watched in kubectl get pods: five of eight Pods bound, three pending, accelerators consumed and no forward progress. The right branch is what a PodGroup with a gang scheduling policy buys — the scheduler evaluates placement for the whole group, and binds either all of it or none of it. Nothing is bound speculatively, so nothing is held hostage.

Beta means available, not enabled

This is the first thing to internalise, because it inverts the usual Kubernetes Beta expectation. The GenericWorkload feature gate, which now covers the Workload API, gang scheduling and workload-aware preemption, is Beta but disabled by default on kube-apiserver, kube-controller-manager and kube-scheduler. The same is true of DRAWorkloadResourceClaims. Alpha features — TopologyAwareWorkloadScheduling and CompositePodGroup — are naturally off too.

Practically, this means no managed Kubernetes service gives you native gang scheduling by simply upgrading to 1.37. On EKS, GKE and AKS you cannot set arbitrary API server feature gates, so unless your provider ships the gate enabled you are looking at a self-managed control plane or a custom scheduler deployment to use any of this. That constraint alone keeps Volcano and YuniKorn in business for a large slice of the market through at least the v1.38 cycle, independent of any capability argument.

The work landed as KEP #4671 (gang scheduling), KEP #5710 (workload-aware preemption) and KEP #5729 (DRA ResourceClaims for workloads), all led by SIG Scheduling.

Native PodGroup queueing changes scheduler throughput, not just semantics

In v1.36, every member Pod of a PodGroup was still queued individually in the scheduling queue. The group semantics were enforced during placement, but the queue itself still saw N independent items, each cycling through backoff and retry on its own schedule. In v1.37 only the top-level PodGroup object is queued.

The consequence is not cosmetic. A 512-Pod training job previously contributed 512 entries to the active and backoff queues, each triggering its own scheduling cycle, each failing for the same reason, each pushing unrelated Pods further down the queue. Now it contributes one. All member Pods share identical queueing behaviour by construction, which eliminates a whole class of ordering artefact where some members of a gang were repeatedly retried ahead of others. The release notes also credit this work with addressing livelock scenarios where several workloads scheduled concurrently interfere with one another without any of them making progress — a failure that is notoriously hard to reproduce and even harder to diagnose from scheduler metrics.

If you run a cluster where GPU jobs and ordinary service Pods share a scheduler, this is the change most likely to show up as a measurable improvement in scheduler_pending_pods and end-to-end scheduling latency for the non-batch workloads.

minCount is now mutable, which makes elastic gangs possible

In earlier iterations, minCount — the minimum number of Pods that must be placeable for a PodGroup to be admitted — was strictly immutable. If you wanted a gang of eight and the cluster could only offer six, your options were to wait or to delete and recreate the group with a smaller floor, losing any Pods already assumed.

As of v1.37, minCount is mutable. A controller can shrink the required size of a gang on a job that is already running, or grow it, without evicting the Pods that are already scheduled. That is the API primitive elastic training needs: a framework that supports dynamic world size can now negotiate downward under contention and back up when capacity returns, and the scheduler will respect the new floor on the next cycle.

The same mutability is wired into the batch/v1 Job integration. Job.spec.scheduling is immutable once the Job is created, with exactly one exception — schedulingPolicy.gang.minCount can be updated, which is how you resize a running gang through the standard Job API rather than by touching scheduling objects directly. Note that this is a placement floor, not an autoscaling signal. Nothing upstream reconciles minCount against actual cluster capacity for you; that decision stays with whatever controller owns the job.

Workload-aware preemption got both correct and faster

Two distinct improvements landed here, and they are worth separating because only one of them is a behaviour change you can observe in a test.

The correctness fix concerns default, single-Pod preemption. In v1.36, when the scheduler preempted an individual Pod to make room for another individual Pod, it did not consult the PodGroup that Pod belonged to. A group that had explicitly declared all-or-nothing disruption could still lose a single member to an unrelated preemption, which silently broke the gang the moment the eviction landed. Kubernetes v1.37 removes this limitation: default preemption now respects the disruptionMode of the victim’s group. If you have been running WAS in v1.36 and seeing gangs mysteriously lose a worker, this is the explanation.

The performance fix concerns how the scheduler evaluates preemption candidates. To decide whether a preemptor can fit, the scheduler simulates removing all potential victims and re-runs the scheduling algorithm, then tries to reprieve as many victims as it can — putting back the ones whose removal turned out to be unnecessary. In v1.36 the full scheduling algorithm was re-run for each reprieval attempt. In v1.37 the algorithm runs once, the preemptor’s placement is assumed from that single output, and each reprieval is checked against that assumed placement. On a large cluster with dozens of candidate victims, this collapses an O(victims) sequence of full scheduling passes into one pass plus O(victims) cheap checks.

PodGroup finally has an authoritative preemptionPolicy

In v1.36 a PodGroup had no preemptionPolicy of its own. Whether the group could preempt was inferred from its members: the group could preempt as long as no member Pod carried preemptionPolicy: Never. That is a fragile inference — it depends on every Pod template in the group agreeing, and it produces surprising results when a controller injects a Pod with different settings.

With the new PodGroupPreemptionPolicy feature gate enabled, a PodGroup carries its own preemptionPolicy field, and that field is authoritative. The group-level declaration wins over whatever the member Pods say. For anyone writing admission policy — “inference workloads in this namespace may never preempt training” — this is the difference between a rule you can enforce on one object and a rule you have to validate across every Pod template in the workload.

CompositePodGroup and the LeaderWorkerSet Shape

Everything above describes a flat group: one PodGroup, N homogeneous Pods, one minCount. Real AI workloads are not flat. A disaggregated inference deployment built on LeaderWorkerSet has one leader and N workers per replica, and the leader has a different resource profile, a different failure semantic and a different co-location requirement from the workers. A JobSet has several ReplicatedJob entries that must start together but are not interchangeable.

Put a direct answer up front: CompositePodGroup is a scheduling.k8s.io/v1alpha3 object that groups other groups, letting you express a tree of scheduling requirements — gang policies and topology constraints at every level — which the scheduler evaluates as one unit and binds atomically.

Figure 2: A two-level CompositePodGroup tree. The root picks a zone; each child PodGroup then picks a rack confined inside that zone; all Pods bind atomically.

The diagram shows the two mechanisms that matter. First, the hierarchy: a root CompositePodGroup with gang.minGroupCount: 2 requires two of its child groups to be placeable, and each child PodGroup carries its own gang.minCount over its own Pods. Second, top-down topology resolution: the root’s zone constraint is resolved first, and the rack domains considered for each child are confined inside the zone the parent already assumed.

How the tree is declared

The Workload object gains spec.compositePodGroupTemplates. Each CompositePodGroupTemplate directly nests the templates its children derive from — podGroupTemplates for leaves, compositePodGroupTemplates for further composites. This is a template hierarchy, not the runtime objects; a controller stamps out the actual CompositePodGroup and PodGroup objects from it.

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: lws-inference
  namespace: serving
spec:
  compositePodGroupTemplates:
  - name: root
    schedulingPolicy:
      gang:
        minGroupCount: 2
    schedulingConstraints:
      topology:
      - key: topology.kubernetes.io/zone
    podGroupTemplates:
    - name: leader
      schedulingPolicy:
        gang:
          minCount: 1
      schedulingConstraints:
        topology:
        - key: topology.example.com/rack
    - name: workers
      schedulingPolicy:
        gang:
          minCount: 8
      schedulingConstraints:
        topology:
        - key: topology.example.com/rack

Note the version skew in the manifests themselves: the Workload that contains the composite templates is v1beta1, while the CompositePodGroup objects stamped out from it are v1alpha3. That is not a typo in the upstream examples — it is the actual shape of the release, and it has real consequences discussed below.

At runtime the controller creates one root CompositePodGroup referencing the root template, plus two PodGroup objects that link upward through spec.parentCompositePodGroupName and reference their leaf templates through spec.workloadRef:

apiVersion: scheduling.k8s.io/v1alpha3
kind: CompositePodGroup
metadata:
  name: lws-inference-root
  namespace: serving
spec:
  workloadRef:
    workloadName: lws-inference
    templateName: root
  schedulingPolicy:
    gang:
      minGroupCount: 2
  schedulingConstraints:
    topology:
    - key: topology.kubernetes.io/zone
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: lws-inference-workers
  namespace: serving
spec:
  parentCompositePodGroupName: lws-inference-root
  workloadRef:
    workloadName: lws-inference
    templateName: workers
  schedulingPolicy:
    gang:
      minCount: 8
  schedulingConstraints:
    topology:
    - key: topology.example.com/rack

Individual Pods join a group through spec.schedulingGroup.podGroupName, which the owning controller sets when it creates them.

Recursive evaluation and atomic binding

The scheduler treats the whole tree as a single scheduling unit. It traverses from the root down to the leaves. A parent CompositePodGroup is schedulable only when enough of its children satisfy its policy — at least minGroupCount children under a gang policy — and each leaf PodGroup is schedulable only when at least minCount of its member Pods can be placed. Once a valid combination is found for the root, every Pod across the entire hierarchy is bound atomically. If the root cannot satisfy its constraints, nothing binds at all.

That last sentence is the whole value proposition. A two-level workload with one leader and eight workers per replica previously required either a controller that manually coordinated two flat gangs (and got the interleaving wrong under contention), or an out-of-tree scheduler that understood the shape. Now the shape is expressible in the API.

Multi-level topology makes it concrete. The scheduler evaluates candidate zones for the root; for each candidate zone it subdivides the nodes by rack and explores feasible rack placements for the leader and worker groups strictly within that zone; and it does this across multiple zone-and-rack combinations before committing. That search is the expensive part, which is why single-level topology-aware scheduling also received placement-evaluation performance work in this release.

Preemption across a hierarchy

A CompositePodGroup can be a preemption victim as well as a preemptor. The disruptionMode on the composite decides what happens when the scheduler wants some of its capacity:

single — child groups inside the composite can be preempted and disrupted independently. This is the default when disruptionMode is unset.
all — all-or-nothing disruption across the entire hierarchy. If any Pod in the descendant subtree must go, the scheduler evicts every Pod in the hierarchy together.

The default matters. If you build a hierarchy and do not set disruptionMode: {all: {}} on the root, your carefully gang-scheduled tree can be dismembered one child group at a time — the exact failure the gang was supposed to prevent, arriving through the preemption path instead of the placement path.

The controller integration layer

Alongside the user-facing APIs, v1.37 adds reusable primitives (KEP #6089) so that every workload controller exposes the same scheduling concepts without reimplementing the translation logic. Types prefixed WorkloadPodGroup describe a leaf group; types prefixed WorkloadCompositePodGroup describe a group of groups. A controller embeds them verbatim under whatever field name fits its own domain — WorkloadPodGroupSchedulingPolicy (either basic or gang with a minCount), WorkloadPodGroupSchedulingConstraints (topology keys), WorkloadPodGroupDisruptionMode (single or all) and WorkloadPodGroupResourceClaim.

The workloadbuilder Go library turns that intent into objects. A controller describes its workload as a tree of WorkloadItem nodes — a node with children compiles to a CompositePodGroupTemplate, a node without to a PodGroupTemplate — then Validate() reports problems at the exact field path in the controller’s own API, BuildWorkload() compiles the tree, and NewPodGroup() / NewCompositePodGroup() stamp out runtime objects. Validation is deny-by-default: a controller declares what it supports through AllowedPolicies and AllowedDisruptionModes, and anything outside those lists is rejected, so primitives added in future releases stay unavailable until a controller explicitly opts in.

The in-tree Job controller is the first consumer (KEP #5547). batch/v1 Jobs now carry an explicit .spec.scheduling field instead of having the controller infer scheduling behaviour from the Job’s shape:

apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training-job
spec:
  parallelism: 8
  completions: 8
  scheduling:
    schedulingPolicy:
      gang: {}          # minCount omitted, defaults to parallelism
    schedulingConstraints:
      topology:
      - key: topology.kubernetes.io/zone
    disruptionMode:
      all: {}
  template:
    spec:
      containers:
      - name: trainer
        image: registry.example.com/trainer:v1

Omitting .spec.scheduling selects basic, which behaves exactly as Job scheduling does today — a Workload and PodGroup are still created, but without a minCount gate.

Migrating from v1alpha2 to v1alpha3

If you were an early adopter running Workload-Aware Scheduling on v1.36, this section is the work item. v1alpha2 has been entirely replaced by v1alpha3. There is no conversion path and no deprecation window — the alpha version is gone, and the reason it is gone is a naming cleanup around disruptionMode.

Figure 3: The preemption decision path. The policy check is now authoritative at group level, the algorithm runs once instead of per reprieval, and the victim’s disruptionMode is honoured.

The rename, and why it was necessary

In v1alpha2 the disruptionMode values were named after the object they applied to: PodGroup meant “disrupt the whole group together”, Pod meant “individual Pods may be disrupted”. That worked while PodGroup was the only grouping object. It breaks the moment CompositePodGroup exists, because a value literally named PodGroup on a CompositePodGroup reads as nonsense — does it mean the composite, or the child PodGroups?

So the values were decoupled from the object name. PodGroup became all, and Pod became single. The semantics are unchanged; only the spelling moved.

Before, on v1.36:

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: training-workers
  namespace: ml
spec:
  workloadRef:
    workloadName: training-run
    templateName: workers
  schedulingPolicy:
    gang:
      minCount: 16
  disruptionMode:
    PodGroup: {}

After, on v1.37:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-workers
  namespace: ml
spec:
  workloadRef:
    workloadName: training-run
    templateName: workers
  schedulingPolicy:
    gang:
      minCount: 16
  disruptionMode:
    all: {}

Two things changed on the same object: the API group version moved from v1alpha2 straight to v1beta1 for PodGroup, and disruptionMode.PodGroup became disruptionMode.all. The equivalent single-Pod mode moves from disruptionMode.Pod: {} to disruptionMode.single: {}.

A migration order that does not lose gangs

The rename is mechanical, but the sequencing is not, because you are changing a scheduling contract on objects that may be governing running jobs. A workable order:

First, inventory. kubectl get podgroups.v1alpha2.scheduling.k8s.io -A -o yaml before the upgrade, and store the output. Once the API version is gone, so is your ability to read the old spec from the cluster. Grep the same manifests out of Git while you are at it — the disruptionMode: {PodGroup: {}} and disruptionMode: {Pod: {}} shapes are trivially greppable and there is no ambiguity in the mapping.

Second, drain the batch queue if you can. WAS objects are created by controllers on behalf of workloads; the cleanest migration is one where no gang is mid-flight. For a training cluster this usually means a maintenance window at a checkpoint boundary, which you want anyway for a minor-version control-plane upgrade.

Third, upgrade and enable. Turn on GenericWorkload across kube-apiserver, kube-controller-manager and kube-scheduler — all three, or the objects will exist without being acted upon. Add PodGroupPreemptionPolicy if you intend to use group-level preemption policy, DRAWorkloadResourceClaims (apiserver, controller-manager, scheduler and kubelet) if you share claims across a group, and TopologyAwareWorkloadScheduling (apiserver and scheduler) for topology constraints.

Fourth, reapply the rewritten manifests and verify that each PodGroup reaches a scheduled state before resubmitting real work.

If you also intend to enable CompositePodGroup, note the coupling: the gate goes on kube-apiserver, kube-controller-manager and kube-scheduler, you must enable the scheduling.k8s.io/v1alpha3 API version explicitly on the apiserver, and enabling CompositePodGroup on kube-controller-manager also requires TopologyAwareWorkloadScheduling to be enabled there. That dependency is easy to miss and produces a controller that silently declines to stamp out composite objects.

The alpha-inside-beta problem

Here is the caveat that deserves more attention than the release notes give it. Workload and PodGroup are Beta at v1beta1. CompositePodGroup is Alpha at v1alpha3. They are used together — the Workload that declares compositePodGroupTemplates is a beta object whose most interesting field only does anything if an alpha API is enabled.

The practical exposure is version skew risk on a schedule you do not control. Alpha APIs in Kubernetes carry no compatibility guarantee: v1alpha3 can be replaced by v1alpha4 in v1.38 with breaking changes and no conversion, exactly as v1alpha2 was replaced this cycle. The roadmap targets CompositePodGroup for Beta in v1.38, but “targeted” is not “guaranteed”, and you have just watched an alpha version get deleted outright one release after it shipped.

So: adopt flat PodGroup gang scheduling now if your control plane lets you enable the gate. Treat CompositePodGroup as something to prototype against, not something to build a production control loop on, until it reaches Beta. If you need hierarchical gangs in production today, that requirement alone keeps an out-of-tree scheduler in your stack.

Do You Still Need Volcano, Kueue or YuniKorn?

The honest answer is that v1.37 takes one job away from these projects and leaves their main job untouched.

Figure 4: What moved in-tree in v1.37, what is in-tree but alpha, and what remains exclusively out of tree.

Gang placement, workload-aware preemption and group-shared DRA claims are now upstream primitives. Quota, hierarchical borrowing, fair share, job ordering policy, admission checks and multi-cluster dispatch are not upstream at all, are not in the v1.38 roadmap as upstream features, and are precisely what most organisations installed a batch scheduler to get.

What each comparator actually brings, at current versions

Kueue v0.19.5 (17 September 2026; v0.20.0-rc.0 is in pre-release) is a quota and admission system, not a scheduler. Its model is a cluster-scoped ClusterQueue holding a pool of resources, namespaced LocalQueue objects per tenant, ResourceFlavor objects describing heterogeneous node pools, and Cohort grouping that lets ClusterQueues borrow each other’s unused nominal quota — bounded by borrowingLimit on the borrower and lendingLimit on the lender. On top of that sit Fair Sharing, WorkloadPriorityClass (priority for queueing and preemption independent of Pod priority), flavor fungibility, AdmissionCheck hooks including cluster-autoscaler ProvisioningRequest integration, partial admission, dynamic reclaim, and MultiKueue for dispatching to worker clusters (Beta since v0.9, enabled by default).

Crucially, Kueue’s own documentation describes its all-or-nothing support as “All-or-nothing with ready Pods: a timeout-based implementation”. Kueue admits the job, then waits for Pods to become ready within a timeout and requeues if they do not. That is an admission-time approximation of gang scheduling, not an atomic bind. Native PodGroup gang scheduling is strictly stronger on that specific axis — which is exactly why SIG Scheduling’s roadmap envisions Kueue eventually leveraging WAS as its gang and topology engine rather than maintaining its own.

Kueue’s Topology-Aware Scheduling is currently more expressive than the in-tree equivalent. It drives placement through PodSet annotations — kueue.x-k8s.io/podset-required-topology, podset-preferred-topology, podset-unconstrained-topology, podset-group-name — and supports multi-layer slice constraints with up to three nested topology layers, each inner slice size dividing the outer. The in-tree model expresses topology as a list of keys per group level; the hierarchy carries the nesting. Comparable in power for two-level cases, not yet comparable for slice-based packing.

Volcano v1.15.2 (29 August 2026) is a full scheduler replacement with an action pipeline — enqueue, allocate, preempt, reclaim, backfill — and a plugin set that includes gang, drf (dominant resource fairness), priority, proportion, predicates, nodeorder, binpack and conformance. Its Queue CRD (scheduling.volcano.sh/v1beta1) carries capability as a hard upper bound, guarantee as reserved resources other queues cannot touch, deserved as the expected share under the capacity plugin, weight for proportional division under the proportion plugin, and reclaimable to control whether other queues may reclaim its excess. The reclaim action exists specifically to claw resources back across queue boundaries when a new job arrives — there is no in-tree equivalent, because there is no in-tree notion of a queue that owns resources.

Apache YuniKorn v1.9.0 (29 July 2026) is also a scheduler replacement, built around fully hierarchical queues (root.namespaces.level1) with guaranteed and max resource vectors at every level, maxapplications limits that must decrease monotonically down the tree, and per-user and per-group limits within a queue. Gang scheduling is driven by pod annotations: yunikorn.apache.org/task-group-name and yunikorn.apache.org/task-groups, where each task group declares minMember, minResource, and the nodeSelector, tolerations, affinity and topologySpreadConstraints that placeholders must mirror. schedulingPolicyParameters control placeholderTimeoutInSeconds (15 minutes by default) and gangSchedulingStyle (Soft or Hard). Queues running gang-scheduled apps must use FIFO sorting, otherwise the scheduler reserves partial resources per app and fragments the cluster.

That placeholder mechanism is worth contrasting with the in-tree design. YuniKorn reserves capacity by actually binding placeholder Pods, which is robust across scheduler restarts but makes reserved capacity visible to everything else as running Pods, and requires a timeout to release a gang that never completes. In-tree WAS reserves nothing: it simulates placement and binds only on success. Cleaner, but it means an unschedulable gang competes fresh on every cycle rather than holding ground.

The decision matrix

Capability	In-tree v1.37	Kueue v0.19.5	Volcano v1.15.2	YuniKorn v1.9.0
All-or-nothing gang bind	Yes, Beta, atomic	Timeout-based on ready Pods	Yes, `gang` plugin, `minAvailable`	Yes, via placeholder Pods
Hierarchical gang (leader plus workers)	Alpha only (`CompositePodGroup`)	PodSet groups	Job-level task roles	Multiple task groups per app
Multi-level topology placement	Alpha (`TopologyAwareWorkloadScheduling`)	Yes, up to 3 slice layers	Via plugins and node ordering	Node sorting, spread constraints
Workload-aware preemption	Yes, Beta	Yes, at admission level	Yes, `preempt` and `reclaim` actions	Partition-level, off by default
Tenant quota pools	No	ClusterQueue and LocalQueue	Queue CRD	Hierarchical queues
Borrowing and lending across tenants	No	Cohorts, borrowing and lending limits	`deserved`, `reclaimable`	`guaranteed` and `max` per level
Fair share	No	Fair Sharing	DRF and proportion plugins	Queue-level fairness
Job ordering policy	FIFO-ish via queue	StrictFIFO, BestEffortFIFO, priority	Configurable plugin order	Per-queue sorting policies
Multi-cluster dispatch	No	MultiKueue, Beta	No	No
Replaces kube-scheduler	No	No	Yes	Yes
Works on managed control planes	Only if the provider enables the gate	Yes	Yes	Yes

The three bolded rows are the answer to the title question. If you use Kueue, Volcano or YuniKorn primarily for quota, borrowing and fair share between teams, v1.37 does not replace it and nothing on the v1.38 roadmap will. If you adopted one of them only to stop distributed jobs from deadlocking, and you control your control plane, v1.37 can plausibly replace it — with the caveat that hierarchical shapes stay alpha.

The interesting middle case is Kueue. Because it does not replace kube-scheduler, Kueue and in-tree WAS compose rather than compete: Kueue decides whether and when a job is admitted against quota, and the scheduler decides where its Pods land, atomically. That is the pairing SIG Scheduling has signalled it wants, and it is the configuration I would build toward on a multi-tenant GPU cluster. Volcano and YuniKorn, by contrast, own the placement decision, so running either alongside in-tree gang scheduling means picking one — the in-tree path only applies to Pods scheduled by kube-scheduler.

Trade-offs, Gotchas, and What Goes Wrong

The feature gate is the whole story on managed Kubernetes. Everything in this post is unavailable to you unless someone enables GenericWorkload on your API server, controller manager and scheduler. Enabling it on two of the three yields objects that are accepted and then ignored. Check all three before filing a bug.

Alpha versions get deleted, not deprecated. v1alpha2 was replaced outright this cycle. Anything you build against v1alpha3 — CompositePodGroup in particular — carries the same risk in v1.38. If your platform’s contract with internal users includes API stability, do not expose composite groups as a supported surface yet.

Unset disruptionMode defaults to single. A hierarchy without an explicit disruptionMode: {all: {}} on the root can be taken apart child group by child group under preemption pressure. You get the gang guarantee on the placement path and lose it on the preemption path, which produces a failure that looks like a scheduler bug and is actually a missing field.

Gang scheduling without quota is a denial-of-service primitive. A single tenant submitting a PodGroup with minCount equal to the whole cluster will sit at the front of the queue consuming preemption evaluation and, with a high enough priority, evicting other tenants to make room. Nothing in-tree stops this, because nothing in-tree models tenants. This is the strongest operational argument for keeping Kueue in front of the scheduler regardless of what graduates upstream.

Atomic binding does not mean atomic running. The scheduler binds all Pods together; it does not guarantee they all start. An image pull failure, an init container crash or a node problem on one member still leaves you with a partially running gang. Gang scheduling addresses placement deadlock, not startup failure. Job-level restart policy and a readiness deadline are still your responsibility.

Multi-level topology search is not free. Evaluating candidate zones and then subdividing each by rack for every child group is combinatorially larger than single-level placement. The release includes performance work on single-level evaluation, but if you enable multi-level constraints on large heterogeneous clusters, watch scheduler cycle latency before and after rather than assuming parity.

Elastic minCount mutation has no upstream controller. Mutability is an API affordance. Deciding when to shrink a gang under contention and when to grow it back is logic you or your framework must write.

Practical Recommendations

Start by deciding which problem you actually have. Write down whether your incumbent batch scheduler is deployed for placement correctness, for tenant quota, or for both — and check against real usage, not the original justification. On most clusters the answer is “both, but quota is what we tune weekly”, and that answer means the in-tree work changes your architecture rather than your bill of materials.

If you run a self-managed control plane and your only requirement is flat gang scheduling, plan to enable GenericWorkload on 1.37 and evaluate removing Volcano or YuniKorn. Run both paths in parallel on a non-production queue first: in-tree gangs do not hold reservations the way YuniKorn placeholders do, so contention behaviour under a loaded cluster will not be identical.

If you run Kueue, do not remove anything. Upgrade to 1.37, enable the gate, and start feeding Kueue-admitted Jobs into native gang scheduling through Job.spec.scheduling. That is the composition upstream is building toward.

If you are on EKS, GKE or AKS without feature-gate control, this release changes nothing operational for you this quarter. Track it, migrate your manifests off v1alpha2 if you were experimenting, and revisit when Workload and PodGroup reach GA — targeted for v1.38, when gates are typically on by default.

A short checklist before you change anything:

Inventory every v1alpha2 PodGroup and Workload object and export the YAML before upgrading.
Rewrite disruptionMode: {PodGroup: {}} to {all: {}} and {Pod: {}} to {single: {}}.
Enable GenericWorkload on all three control-plane components, not two.
Set disruptionMode: {all: {}} explicitly on every root group you care about.
Keep a quota layer in front of the scheduler — in-tree WAS has none.
Do not put CompositePodGroup on a production support contract until it is Beta.
Benchmark scheduler cycle latency before and after enabling multi-level topology.

Frequently Asked Questions

Does Kubernetes 1.37 have built-in gang scheduling?

Yes. Gang scheduling graduated to Beta in Kubernetes v1.37 through the Workload and PodGroup APIs at scheduling.k8s.io/v1beta1, delivered as KEP #4671 by SIG Scheduling. It implements true all-or-nothing placement: a group is bound only when the cluster can accommodate at least minCount members, and nothing is bound otherwise. The important qualifier is that the GenericWorkload feature gate is disabled by default, so a 1.37 upgrade alone does not give you the feature.

Can I still use Volcano or Kueue on Kubernetes 1.37?

Yes, and for most clusters you should. Kueue composes cleanly with native gang scheduling because it manages quota and admission rather than placement, and SIG Scheduling has stated an intent for Kueue to eventually use Workload-Aware Scheduling as its underlying engine. Volcano and YuniKorn replace kube-scheduler entirely, so their gang implementations and the in-tree one are mutually exclusive — Pods scheduled by Volcano never reach the native PodGroup path.

What replaced disruptionMode PodGroup in v1alpha3?

The value PodGroup was renamed to all, and Pod was renamed to single. The semantics are identical — all means the whole group is disrupted together, single means individual members can be disrupted independently — but the names were decoupled from the PodGroup object so they read correctly on CompositePodGroup too. v1alpha2 was replaced entirely by v1alpha3 with no conversion path, so you must rewrite the manifests yourself.

Is CompositePodGroup production ready?

No. CompositePodGroup is Alpha at scheduling.k8s.io/v1alpha3 in v1.37, even though the Workload and PodGroup APIs it works with are Beta at v1beta1. Alpha APIs carry no compatibility guarantee, and the immediately preceding alpha version was deleted rather than deprecated. Beta is targeted for v1.38. Prototype against it, but do not expose it as a supported interface to your users yet.

Does native gang scheduling handle tenant quotas?

No, and this is the key limitation. Kubernetes v1.37 provides placement primitives — gang binding, workload-aware preemption, shared DRA claims and topology constraints — but has no concept of a tenant, a queue that owns resources, borrowing between teams, or fair share. Those remain exclusively out of tree in Kueue’s ClusterQueues and Cohorts, Volcano’s Queue CRD, and YuniKorn’s hierarchical queues. Nothing on the announced v1.38 roadmap changes that.

What is the difference between gang scheduling and workload-aware preemption?

Gang scheduling governs placement: it decides whether to bind a group of Pods all at once or not at all. Workload-aware preemption governs eviction: it lets the scheduler consider an entire PodGroup when choosing victims, so it does not disrupt individual Pods without freeing enough capacity for the preemptor’s whole group to run. In v1.37 both graduated to Beta under the same GenericWorkload gate, after the separate WorkloadAwarePreemption gate was merged into it.

Comments

Leave a Reply Cancel reply

Tag Cloud

Categories