Kubernetes Dynamic Resource Allocation (DRA) for GPU Scheduling Explained (2026)

For almost a decade, asking Kubernetes for a GPU meant writing nvidia.com/gpu: 1 and hoping. The scheduler saw an opaque integer, the node saw a device plugin handing out whatever was free, and everything interesting about the hardware, such as memory size, architecture, MIG geometry or NVLink topology, was invisible to placement. Kubernetes dynamic resource allocation (DRA) replaces that integer with a claim model: workloads describe the device they need, drivers publish what they have, and the scheduler matches the two.

This matters now because DRA is no longer an experiment. Its core APIs went generally available in Kubernetes 1.34, the features that make it useful for real GPU fleets (prioritized alternatives, device taints, extended-resource mapping, partitionable devices) have advanced through beta and stable in later releases, and NVIDIA ships a production DRA driver. This guide explains the object model, walks through working manifests, compares DRA with device plugins, covers MIG and sharing, lays out a migration plan, and spends serious time on what breaks.

What this covers: the DRA API objects, the scheduling and kubelet flow, CEL selectors, runnable YAML, NVIDIA driver setup, GPU sharing options, migration from device plugins, observability, and failure modes.

Context and Background

The device plugin framework, introduced in the Kubernetes 1.8 era and stable for years, solved a narrow problem: let a node advertise countable resources such as nvidia.com/gpu and let the kubelet wire the right device files into a container. It worked, and it still runs most GPU clusters today. But the Kubernetes documentation itself lists the structural limits: device plugins require per-container device requests, do not support device sharing between Pods, and do not support expression-based device filtering (Kubernetes DRA concept docs).

In practice those limits show up as ugly workarounds. Teams label nodes by GPU model and use nodeSelector to pick hardware. They run a separate MIG manager to pre-partition GPUs before any Pod arrives, which locks the geometry in advance. They encode “two GPUs on the same NVLink domain” in affinity rules that the scheduler cannot actually verify against the hardware. The scheduler treats nvidia.com/gpu: 2 as two interchangeable units, even when one is a 16 GB part and one is an 80 GB part on the same node.

DRA is the answer from the Kubernetes Device Management working group. The design is deliberately modelled on dynamic storage: just as a PersistentVolumeClaim draws from a StorageClass, a ResourceClaim draws from a DeviceClass. The Kubernetes docs make the same analogy: allocating resources with DRA “offers a similar experience to dynamic volume provisioning.” The original 2022-era prototype let a vendor controller participate in scheduling decisions, which did not scale and made cluster autoscaling impossible. The shipped design uses structured parameters: drivers publish device inventory as data, and the scheduler itself evaluates claims against it. That is why a cluster autoscaler can reason about DRA, a topic we cover in our comparison of Karpenter and Cluster Autoscaler for GPU node scaling.

The maturity timeline is worth stating precisely, because it is frequently misquoted. The DynamicResourceAllocation feature gate was alpha in 1.30, beta in 1.32 and went stable in 1.34, and the Kubernetes 1.34 release notes describe “the core of DRA” as GA, with the resource.k8s.io/v1 API as the default version (Kubernetes v1.34: DRA has graduated to GA). The concept documentation currently labels the feature “stable since v1.35”, which reflects the gate being locked on. In both framings, the practical rule is the same: on 1.34 or later, the base API is stable and cannot break under you, while individual add-on features carry their own maturity, which we tabulate later.

For GPU operators, the cost angle is real. Idle accelerators are the most expensive line item in most AI platforms, and coarse allocation is a major cause of idleness. We look at that economics in our guide to Kubernetes cost optimization and GPU right-sizing; DRA gives you the knobs, but you still have to turn them.

Kubernetes Dynamic Resource Allocation: Object Model and Scheduling Flow

In short: Kubernetes dynamic resource allocation works through four API kinds in the resource.k8s.io/v1 group. Drivers publish ResourceSlices describing devices, admins define DeviceClasses as categories, workloads submit ResourceClaims (usually via ResourceClaimTemplates), and the scheduler allocates matching devices before placing the Pod.

Figure 1: Kubernetes dynamic resource allocation. The driver publishes inventory, the scheduler matches claims to inventory, and the kubelet asks the driver to prepare the device.

Figure 1 shows the whole loop. Read it left to right as three roles. The device owner (usually the vendor’s driver) publishes what exists. The cluster admin groups devices into classes. The workload operator asks for devices by property. The scheduler is the only component that decides, and it decides using data that lives in the API server, which is what makes DRA auditable and autoscaler-friendly.

The four API kinds

The Kubernetes API documentation defines each kind precisely, and it pays to keep them straight because error messages refer to them by name.

A ResourceSlice represents one or more devices attached to a node (or, for network-attached devices, available to a set of nodes). Drivers create and update them; humans rarely edit them. Each device in a slice has a name, a set of typed attributes (strings, ints, bools, versions) and capacities (quantities such as memory). A node with eight GPUs typically has one slice listing eight devices, each carrying attributes like product name and architecture and a memory capacity. Slices also carry pool metadata, including a generation number, so the scheduler can tell a complete, current pool from a partially updated one.

A DeviceClass defines a category of devices that can be claimed and how to select them. It is cluster-scoped, created by admins or drivers, and usually contains a CEL (Common Expression Language) selector such as “the driver is gpu.nvidia.com“. Think of it as the StorageClass analogue: a stable name that workload authors reference, hiding driver details.

A ResourceClaim is a request for access to devices. It names one or more requests, each pointing at a DeviceClass and optionally adding selectors, a count, or constraints. Claims are namespaced and can outlive Pods. If you want several Pods, or several containers, to share the same device, you create a ResourceClaim by hand and reference it by name.

A ResourceClaimTemplate is a template from which Kubernetes stamps out a separate ResourceClaim for each Pod. The generated claim is bound to that Pod and deleted when the Pod terminates. This is the right default for batch workers and inference replicas, because each gets its own device with the same shape.

What the scheduler actually does

The scheduler includes a DRA plugin that participates in the normal filter and score cycle. For a Pod that references claims, it first checks the claims exist (a Pod referencing a missing claim stays Pending until the claim appears, as the task documentation notes). It then, per candidate node, reads the ResourceSlices visible to that node, evaluates each request’s CEL expressions against each device’s attributes and capacity, applies constraints, and tries to assemble an allocation that satisfies every request without double-booking a device.

If a node works, the scheduler records the result in the claim’s status.allocation and reserves the claim for the Pod in status.reservedFor, then binds the Pod. There is no separate “provisioning” round trip to a vendor controller, which is the central difference from the older classic design. Allocation is a pure function of API objects, so it can be simulated, and an autoscaler can ask “if I add a node of this type, would this Pending Pod’s claim be satisfiable?”

One limitation deserves an early mention because it surprises people. The Kubernetes documentation states that the scheduler does not support preemption for DRA resources: a running Pod holding DRA devices cannot be preempted by a higher-priority Pod that needs those devices. The high-priority Pod stays Pending until the device frees up or the holder is deleted. If your platform relies on priority-based eviction to reclaim GPUs, plan for that explicitly.

The node side: kubelet plugin and CDI

Allocation is only half the story. After binding, the kubelet calls the driver’s kubelet plugin over a local gRPC interface, asking it to prepare the allocated devices. The driver configures the device (for example setting up a MIG instance, an MPS daemon or a vendor-specific mode), then returns identifiers for CDI, the Container Device Interface. The kubelet passes those identifiers to the container runtime, which applies the CDI spec to inject device nodes, libraries and environment variables into the container.

This is why DRA requires a CDI-capable runtime. NVIDIA’s documentation lists a container runtime that supports CDI, configured to use it for DRA-allocated devices, as a prerequisite. If your cluster still depends on a legacy runtime hook for GPU injection, that is the first thing to modernize.

Hands-On: From DeviceClass to a Running GPU Pod

This section builds a working setup step by step. The manifests use the resource.k8s.io/v1 API, which is the default from Kubernetes 1.34. Device names, attributes and class names depend on your driver, so treat the CEL expressions as patterns and check your own ResourceSlices first.

Step 1: install the driver and inspect what it publishes

On NVIDIA hardware, the DRA Driver for NVIDIA GPUs manages two kinds of resources: GPUs and ComputeDomains (the abstraction for Multi-Node NVLink). Per NVIDIA’s GPU Operator documentation, the DRA path currently requires Kubernetes v1.34.2 or later, NVIDIA GPU driver 580 or later, and a CDI-enabled runtime. The same page shows the Helm install flow below; at the time of writing it pins GPU Operator v26.7.1 and DRA driver v0.5.0, so confirm current versions before copying (NVIDIA GPU Operator DRA docs).

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.7.1 \
  --namespace gpu-operator \
  --create-namespace \
  --set clusterPolicy.deployCR=false \
  --set gpuCluster.deployCR=true \
  --set driver.nvidiaDriverCRD.enabled=true
# Use --set driver.enabled=false instead if the GPU driver is pre-installed.
# Add --set draDriver.computeDomains.enabled=false to skip ComputeDomain support.

Note the constraint hiding in those flags: the operator’s DRA mode uses a GPUCluster resource, and the docs state a cluster can have either a GPUCluster (DRA) or a ClusterPolicy (device plugin), not both. That is a cluster-level choice, which shapes your migration plan later.

Once the driver is running, look at the inventory it publishes. These commands work on any DRA driver:

kubectl get deviceclasses
kubectl get resourceslices -o wide

# Show device names, driver, and node for every slice
kubectl get resourceslices \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,DRIVER:.spec.driver,POOL:.spec.pool.name

# Dump one slice to see the attribute names you can use in CEL
kubectl get resourceslice <name> -o yaml | less

According to the NVIDIA documentation, the NVIDIA driver registers the DeviceClasses gpu.nvidia.com, mig.nvidia.com and vfio.gpu.nvidia.com, plus two ComputeDomain classes when that feature is enabled. The YAML dump is where you learn the real attribute names. NVIDIA’s own quickstart specs select MIG devices with device.attributes['gpu.nvidia.com'].profile == '1g.5gb', so attributes are namespaced by driver domain, then accessed by name.

Step 2: define a DeviceClass for your platform

You can use the driver’s classes directly, but platform teams usually add their own so application authors never see raw CEL. A DeviceClass is a cluster-scoped object with a list of selectors. This illustrative example creates a class for large-memory inference GPUs. The attribute names are placeholders: replace them with names from your ResourceSlices.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu-large-memory
spec:
  selectors:
  - cel:
      expression: |-
        device.driver == "gpu.nvidia.com" &&
        device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0

Two CEL details matter here. The device variable exposes driver, attributes and capacity, and capacities are Kubernetes quantities, so you compare them with compareTo(quantity("40Gi")) rather than with raw numbers. The Kubernetes tutorial instead shows equality, device.capacity["driver.example.com"].memory == quantity("64Gi"), which also works but is rarely what you want for a memory floor. The API server validates CEL expressions and bounds their cost, but a wrong attribute name still compiles; it fails only when evaluated against real devices, so a typo tends to surface as an unschedulable claim rather than a rejected manifest. Guard optional attributes with has() and test expressions against a real ResourceSlice before rollout.

Step 3: write a ResourceClaimTemplate and a Pod

The Kubernetes tutorial shows the canonical shape. A claim holds one or more requests; each request uses exactly (a single option) or firstAvailable (an ordered list of alternatives). Here is a template that requests exactly one device from our class, and a Deployment that uses it:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: one-large-gpu
  namespace: inference
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu-large-memory
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server
  namespace: inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-server
  template:
    metadata:
      labels:
        app: llm-server
    spec:
      resourceClaims:
      - name: gpu
        resourceClaimTemplateName: one-large-gpu
      containers:
      - name: server
        image: ubuntu:24.04
        command: ["bash", "-c", "nvidia-smi -L && sleep infinity"]
        resources:
          claims:
          - name: gpu

The wiring has two halves, and mixing them up is the most common first-day error. The Pod-level spec.resourceClaims list declares which claims or templates the Pod can use and gives each a local name. The container-level resources.claims list says which of those local names this container receives. A claim declared at Pod level but not referenced by any container is allocated but unused. The reverse, a container referencing a name that is not declared, is rejected.

Because we used a template, each of the two replicas gets its own generated ResourceClaim and its own device. If you instead create a standalone ResourceClaim and reference it with resourceClaimName, every Pod referencing it shares the same allocation. The Kubernetes tutorial demonstrates both in a single Job: one container uses a per-Pod template claim, while two other containers share a named claim.

Figure 2: The lifecycle of a Pod that uses a ResourceClaimTemplate, from claim generation through allocation to CDI injection.

Figure 2 traces the runtime sequence. The control plane generates the claim, the scheduler allocates and binds, and only then does the kubelet involve the driver. Two consequences follow. First, scheduling latency includes CEL evaluation across candidate nodes, which is cheap for dozens of nodes and worth measuring at thousands. Second, device preparation failures occur after binding, so a Pod can be scheduled yet stuck in ContainerCreating with a driver error, an important triage distinction we return to in the failure-modes section.

Step 4: ask for alternatives with a prioritized list

Real fleets are heterogeneous. A model might run best on one 80 GB GPU but fit acceptably on two 40 GB GPUs. A prioritized list lets the claim express that preference. The Kubernetes 1.34 post describes it as letting users specify a list of acceptable devices, with the scheduler satisfying alternatives in order. Per the feature-gates reference, DRAPrioritizedList was beta in 1.34 and is stable from 1.36. This sketch uses the same placeholder class pattern:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: big-or-two-small
  namespace: inference
spec:
  spec:
    devices:
      requests:
      - name: accel
        firstAvailable:
        - name: one-big
          deviceClassName: gpu-large-memory
        - name: two-small
          deviceClassName: gpu-standard
          allocationMode: ExactCount
          count: 2

Mind two behaviours. The scheduler tries the first alternative on each node before falling back, so the ordering expresses preference per node, not a global ranking across the cluster. And your container must cope with whichever shape it receives, since a container requesting this claim sees one device or two depending on the outcome. Read the allocation back from the claim’s status, or from device metadata exposed by your driver, rather than assuming.

Step 5: pin a multi-device claim to one physical GPU

Constraints let a claim relate its requests to each other. NVIDIA’s quickstart for several MIG slices in one Pod uses matchAttribute: "gpu.nvidia.com/parentUUID" so that all requested MIG devices come from the same parent GPU. In that spec, one claim asks for two 1g.5gb slices, a 2g.10gb slice and a 3g.20gb slice, and the constraint with an empty requests list applies to all of them. This is exactly the sort of topology rule that node labels could never express reliably.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-devices
spec:
  spec:
    devices:
      requests:
      - name: mig-small
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: "device.attributes['gpu.nvidia.com'].profile == '1g.5gb'"
      - name: mig-medium
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: "device.attributes['gpu.nvidia.com'].profile == '2g.10gb'"
      constraints:
      - requests: []
        matchAttribute: "gpu.nvidia.com/parentUUID"

In a multi-container Pod, each container then names the specific request it wants with resources.claims[].request, so container A gets mig-small while container B gets mig-medium from the same claim. Without the request field, a container gets every device in the claim.

Step 6: verify and debug

Verification is a three-object drill-down, which the Kubernetes tutorial recommends: go from workload to Pods to ResourceClaims and run kubectl describe at each level.

kubectl get pods -n inference
kubectl get resourceclaims -n inference
kubectl describe resourceclaim -n inference <generated-claim-name>
# Look at status.allocation.devices.results for the chosen driver, pool, device
kubectl get resourceclaim -n inference <name> -o jsonpath='{.status.allocation.devices.results}'
kubectl logs -n inference deploy/llm-server

If you prefer scripting over kubectl, the same data is available through the Python client. This runnable snippet lists every device in every slice with its driver and node, which is handy for building inventory reports:

from kubernetes import client, config

config.load_kube_config()
api = client.CustomObjectsApi()
slices = api.list_cluster_custom_object("resource.k8s.io", "v1", "resourceslices")

for item in slices["items"]:
    spec = item["spec"]
    node = spec.get("nodeName", "<non-node>")
    for dev in spec.get("devices", []):
        print(f'{node:20} {spec["driver"]:20} {dev["name"]}')

Kubernetes Dynamic Resource Allocation vs Device Plugins: Sharing and Maturity

Once the basics work, the interesting questions are comparative: what does DRA give you over the plugin model, how do you share a GPU, and which features can you rely on? This section answers them with a decision matrix and a maturity table.

Device plugin versus DRA, side by side

Dimension	Device plugin (nvidia.com/gpu: N)	DRA (ResourceClaim)
What the scheduler sees	An integer count per node	Typed attributes and capacities per device
Selecting hardware	Node labels plus nodeSelector	CEL expressions on device properties
Sharing a device between Pods	Not supported by the framework; vendors fake it with replicas	Native: reference one ResourceClaim from many Pods
Per-workload device config	Per-node config only	Config attached to the claim
Where requests live	Per-container resource limits	Pod-level claim list, referenced per container
Multi-device topology rules	Affinity heuristics	Constraints such as matchAttribute
Dynamic partitioning	Pre-partition before Pods arrive	Possible in drivers that support it (alpha for NVIDIA MIG)
Autoscaler visibility	Counts only	Claims evaluated against slices
Maturity	Years in production	Core GA since 1.34; add-ons vary

The first row is the root of everything else. Because the plugin interface reports a count, any property the scheduler needs must be smuggled in through labels that someone maintains. With DRA, the property lives on the device object itself and cannot drift from reality.

A practical illustration, with illustrative numbers rather than a benchmark: imagine a pool of 8-GPU nodes where half the GPUs are 80 GB parts and half are 40 GB parts. Under the plugin model, nvidia.com/gpu: 1 lands on either, and a memory-hungry job can OOM on the smaller part after it has already been scheduled. The usual fix is two node pools with two resource names or labels, which fragments capacity: spare 80 GB GPUs cannot serve a Pod that would happily run on a 40 GB one unless you add fallback logic. With a prioritized list, one claim can say “prefer a large-memory device, otherwise two standard ones”, and the pools remain a single schedulable fleet.

Four ways to share a GPU

Sharing is where DRA changes day-to-day operations, because the choice of sharing model is now a property of the claim rather than of the node. Figure 3 summarises the decision.

Figure 3: Choosing a GPU sharing model under Kubernetes dynamic resource allocation, based on isolation needs and how many Pods share a device.

Whole GPU. The simplest case: a claim against gpu.nvidia.com allocates a full device exclusively. Use it for training, large-model serving and anything where memory bandwidth is the bottleneck.

MIG. Multi-Instance GPU partitions supported NVIDIA GPUs into hardware-isolated instances with dedicated compute and memory slices. On 80 GB parts, for example, the smallest profile, 1g.10gb, allows up to seven instances per GPU according to NVIDIA’s MIG documentation. Under DRA, each MIG instance appears as a device in the mig.nvidia.com class with a profile attribute and a parent GPU UUID. NVIDIA’s operator documentation states that allocating full GPUs and preconfigured MIG devices is generally available, while the alpha DynamicMIG feature, which creates MIG devices on demand for a claim, conflicts with several other alpha gates (PassthroughSupport, NVMLDeviceHealthCheck and MPSSupport). In plugin mode, a MIG manager has to set the geometry before allocation. The dynamic version is the real prize: no pre-partitioning, geometry follows demand.

Time-slicing through a shared claim. Several containers or Pods reference one ResourceClaim, so they share a single GPU through the driver’s time-sliced mode. There is no memory isolation and no fault isolation, and throughput for each tenant depends on the others. NVIDIA’s documentation says time-slicing works through a shared DRA claim and that custom intervals need the alpha TimeSlicingSettings gate. Use it for dev notebooks, CI and light inference, not for latency-sensitive production traffic.

Consumable capacity. This is the new Kubernetes-native model: independent claims from unrelated Pods can each receive a share of the same device, with the scheduler tracking how much capacity each consumes. The Kubernetes 1.34 post describes it as sharing for scenarios where predefined partitions are not viable, governed by admin-defined sharing policies. The feature-gates reference lists DRAConsumableCapacity as beta from 1.36 and enabled by default. The NVIDIA driver’s counterpart, ConsumableShares, is documented as alpha, and it cannot be combined with MPS. So the upstream machinery is further along than the vendor driver support, a theme that repeats across DRA.

A fifth concept sits beside these: partitionable devices. Instead of treating each MIG slice as an unrelated device, the driver publishes counter sets in a ResourceSlice (for example, a GPU with 8 Gi of memory counters) and lets devices declare how many counters they consume. The scheduler then guarantees you never allocate overlapping devices. The Kubernetes docs give a minimal example in which two devices each consume 6 Gi from a shared 8 Gi counter, so only one can be allocated at a time. DRAPartitionableDevices is beta and on by default from 1.36, and an alpha follow-on in 1.37, device compatibility groups, lets the scheduler reject mutually exclusive modes such as MIG versus vGPU on the same GPU before node-side preparation fails.

Which DRA features are safe to depend on

Maturity is per feature. The table below summarises the Kubernetes feature-gates reference as of this writing, whose pages carry documentation for v1.37. Always recheck against the exact minor version you run.

Feature	Gate	Alpha	Beta	Stable
Core DRA	DynamicResourceAllocation	1.30	1.32	1.34
Prioritized list	DRAPrioritizedList	1.33	1.34	1.36
Admin access	DRAAdminAccess	1.32	1.34	1.36
Device taints	DRADeviceTaints	1.33	1.36	1.37
Extended resource mapping	DRAExtendedResource	1.34	1.36	1.37
Claim device status	DRAResourceClaimDeviceStatus	1.32	1.33	1.37
Partitionable devices	DRAPartitionableDevices	1.33	1.36	not yet
Consumable capacity	DRAConsumableCapacity	1.34	1.36	not yet
Binding conditions	DRADeviceBindingConditions	1.34	1.36	not yet

Read this table as a risk map. The GA core plus prioritized lists and admin access are safe bets on 1.36 and later. Partitionable devices and consumable capacity are beta, on by default, and appropriate for staging with a plan to watch release notes. Anything alpha in the NVIDIA driver, such as dynamic MIG, MPS and consumable shares, should stay in test clusters.

Two features deserve a closer look because they change operations. Admin access, when requested on a claim, lets privileged tooling (a monitoring agent, for instance) reach a device that is already allocated to someone else; Kubernetes restricts it by requiring the resource.k8s.io/admin-access: "true" label on the namespace, so ordinary users cannot grant themselves device-level access. Extended resource mapping, for its part, lets a DeviceClass advertise DRA-managed devices under a legacy resource name, which is the bridge for existing manifests and the subject of the next section.

Migrating to Kubernetes Dynamic Resource Allocation Without an Outage

Figure 4 sketches a phased migration. The key idea is that DRA adoption should be incremental by node pool, not a cluster-wide flip, because the driver, the runtime configuration and the workload specs all change.

Figure 4: A phased migration from the device plugin to Kubernetes dynamic resource allocation, with a canary pool and an explicit rollback path.

Phase 0: audit. List every workload that requests nvidia.com/gpu (or other plugin resources), every node label used for hardware selection, and every MIG or time-slicing configuration. This inventory tells you which workloads can move as-is and which depend on plugin-specific replicas.

Phase 1: platform prerequisites. You need Kubernetes 1.34 or later (NVIDIA’s operator docs ask for 1.34.2 or later), a CDI-capable runtime, and a recent GPU driver (580 or later per NVIDIA). Upgrade first, separately from any DRA change, so a regression is attributable.

Phase 2: a canary node pool. Here the operator constraint bites: the GPU Operator’s DRA mode and plugin mode are mutually exclusive per cluster, according to NVIDIA’s documentation. If you run the operator in plugin mode today, you cannot simply add DRA nodes beside it through the same operator. Plan for either a separate canary cluster, or a transition in which the operator install is switched over while workloads are drained pool by pool. Whichever you choose, taint the canary pool so only opted-in workloads land there.

Phase 3: classes and templates. Create platform DeviceClasses and one ResourceClaimTemplate per workload shape (for example inference-small, inference-large, training-8gpu). Keep CEL in the classes where possible and keep manifests boring.

Phase 4: move one workload. Pick a stateless inference service, convert resources.limits to a claim, and compare placement, startup time and GPU utilization before and after. Use a canary rollout rather than editing production Deployments in place.

Phase 5: expand and retire. Move pools one at a time. Delete the plugin DaemonSet for a pool only after all its workloads have moved and a soak period has passed.

Using extended resource mapping as a bridge

Where your drivers and clusters are on a release that supports it, extended resource mapping removes much of the manifest churn. The Kubernetes 1.34 post describes it as letting admins advertise DRA-managed resources as extended resources so developers can keep using the familiar request syntax while the scheduler still allocates dynamically; the feature-gates reference lists DRAExtendedResource as stable in 1.37. NVIDIA’s documentation likewise notes that extended-resource-name support for backward compatibility arrives with Kubernetes 1.36. The trade-off is that a legacy-style request cannot express the richer selection that justified DRA in the first place, so use the bridge to migrate and claims to optimize.

A sensible end state looks like this: legacy nvidia.com/gpu requests still work for simple services through the mapping, while GPU-sensitive workloads such as multi-slice inference, topology-aware training and shared notebooks use explicit claims. Our GPU cost guide on right-sizing GPU workloads in Kubernetes explains how to measure which workloads belong in which bucket.

Scale-out context: why topology matters

DRA matters most at the top of the scale range. NVIDIA’s ComputeDomain abstraction, documented as generally available in the operator’s DRA mode, guarantees Multi-Node NVLink reachability between Pods in the domain and isolates them from others; it needs Grace Blackwell systems with Multi-Node NVLink such as GB200 or GB300 NVL72 racks. A ComputeDomain’s lifetime is bound to its consuming workload. That is a scheduling problem the plugin model cannot express at all, and it is the clearest example of DRA enabling hardware that Kubernetes previously handled badly. For the network side of these clusters, see our analysis of the Spectrum-X AI Ethernet fabric for 100k-GPU clusters.

Trade-offs, Gotchas, and What Goes Wrong

DRA is the right long-term model, but it moves complexity rather than deleting it. These are the failure modes that cost real time.

No preemption for DRA devices. As the Kubernetes documentation states, a running Pod that holds DRA devices cannot be preempted by a higher-priority Pod needing them. Platforms that depend on priority classes to evict low-priority GPU jobs will see the urgent Pod stay Pending. Mitigations are operational: reserve a node pool for urgent work, use quotas to cap low-priority claims, or delete lower-priority workloads explicitly through a controller.

Allocation succeeds, preparation fails. Scheduling happens against API data, but device preparation happens later on the node. A Pod can bind and then sit in ContainerCreating because the driver failed to configure a MIG instance, the CDI spec was not applied, or the runtime is not configured for CDI. Triage by looking at the claim (allocated?), then the kubelet and driver pod logs on that node. Device compatibility groups, still alpha in 1.37, exist precisely to move one class of these failures from node time to scheduling time.

Silent mismatch from CEL and attribute drift. An expression that references the wrong attribute name does not match any device, and the symptom is an unschedulable claim. Driver upgrades can rename or add attributes. Keep selectors in DeviceClasses where you can test them centrally, and add a smoke-test claim to your upgrade pipeline that allocates one device per class.

Driver is now a control-plane dependency. If the driver crashes or is uninstalled, ResourceSlices go stale or disappear, and new Pods cannot be allocated. Running Pods typically keep their devices, but nothing new schedules. Run the driver with sensible priority classes and alert on missing slices per node, for example a node with GPUs but an empty pool.

Webhook and tooling compatibility. The Kubernetes tutorial documents a specific symptom: creating a Pod fails with must specify one of: resourceClaimName, resourceClaimTemplateName, and one cause is a mutating Pod webhook built against APIs older than Kubernetes 1.32 that drops fields it does not know. Admission webhooks, GitOps diff tools and policy engines that pre-date DRA are a common blocker.

Claim lifecycle leaks. Standalone ResourceClaims outlive Pods. A forgotten claim holds a device even with no consumer, which looks like capacity vanishing from the cluster. Template-generated claims are cleaned up with their Pod, so prefer templates unless you truly need sharing. Audit with kubectl get resourceclaims -A and look for allocated claims with no reservation.

Fragmentation, now with more dimensions. MIG and partitionable devices are powerful, but they create bin-packing problems. If every node’s GPUs are carved into odd slice combinations, a request for one whole GPU may fail while plenty of slices sit idle. Decide per node pool whether it serves whole GPUs or slices, and avoid mixing without a plan.

Security and multi-tenancy. Sharing devices through one claim shares more than you may realise: time-sliced tenants have no memory isolation. Admin access exists to prevent privilege escalation, which is why it is gated by a namespace label. Kubernetes also publishes a task page on hardening DRA, and beta granular authorization for claim status updates from 1.36 narrows who can write allocation state. Treat claim creation like volume creation: restrict it by RBAC and cap it with quotas.

Observability gaps. Utilization is not visible from the claim. You still need device-level telemetry such as DCGM metrics; NVIDIA’s operator exposes DCGM and exporter settings for this. Kubernetes 1.34 added alpha resource health reporting into Pod status, and the kubelet’s PodResources API now reports DRA allocations so node agents can correlate Pods and devices. Both are useful, but build dashboards on utilization first and allocation second.

Version skew. Beta features are enabled by default from 1.36 but depend on both the API server and the scheduler (and sometimes the kubelet). During rolling upgrades, a claim using a newer field may be handled differently by components on different versions. Upgrade control plane first, validate with canary claims, then roll node pools.

Practical Recommendations

If you operate GPU clusters today, the pragmatic path is to adopt DRA where it pays back immediately and defer where vendor support is still alpha.

Start with a platform-level view. Run Kubernetes 1.36 or later if you can, because prioritized lists and admin access are stable there and several sharing features are beta and on by default. Treat DeviceClasses as your public API: name them for use cases (inference-small, training-whole-gpu), not for hardware SKUs, so you can swap hardware underneath. Keep application teams on ResourceClaimTemplates, and reserve hand-written ResourceClaims for deliberate sharing.

Then choose sharing by isolation need: whole GPUs for training and large serving, preconfigured MIG for multi-tenant inference with hard boundaries, shared claims for low-stakes dev work. Hold alpha vendor features such as dynamic MIG, MPS and consumable shares in a test cluster until the driver matures. Pair all of this with cost visibility, because sharing models only save money if utilization is measured.

A short checklist you can paste into a runbook:

• Confirm Kubernetes is 1.34 or later (1.36 or later recommended) and the runtime supports CDI.
• Decide DRA mode or plugin mode per cluster for the NVIDIA GPU Operator; they are mutually exclusive.
• Inventory ResourceSlices after install and verify every GPU node publishes a complete pool.
• Encode hardware selection in DeviceClasses, with CEL tested against real slices.
• Default to ResourceClaimTemplates; audit standalone claims weekly.
• Plan for no preemption of DRA devices: reserve capacity or use quotas.
• Add a per-class smoke-test claim to upgrade pipelines.
• Alert on empty pools, Pending claims older than a threshold, and allocated claims with no consumers.
• Check admission webhooks and policy tools for pre-DRA assumptions.
• Keep alpha driver features out of production.

For autoscaling, test your autoscaler’s DRA awareness with a Pending Pod and a claim before trusting it at scale; our Karpenter versus Cluster Autoscaler comparison for GPU nodes covers what to verify.

Frequently Asked Questions

What is Kubernetes dynamic resource allocation (DRA)?

Kubernetes dynamic resource allocation is an API for requesting and sharing devices such as GPUs. Drivers publish ResourceSlices describing available devices, admins define DeviceClasses, and workloads submit ResourceClaims with CEL selectors. The scheduler allocates matching devices and places Pods on nodes that can reach them. It is modelled on dynamic volume provisioning and replaces the integer-count model of device plugins, adding device attributes, sharing and richer topology rules.

Is DRA generally available, and in which Kubernetes version?

The core of DRA graduated to GA in Kubernetes 1.34, with resource.k8s.io/v1 as the default API version, and the Kubernetes concept docs label the feature stable since 1.35. Individual capabilities have their own maturity: prioritized lists and admin access are stable in 1.36, device taints and extended resource mapping in 1.37, while partitionable devices and consumable capacity are beta from 1.36. Always check the feature-gates reference for your exact version.

How is a DRA ResourceClaim different from nvidia.com/gpu: 1?

The plugin request is an anonymous integer that the scheduler matches against node capacity. A ResourceClaim describes the device itself: class, CEL selectors on attributes like memory or architecture, counts, alternatives and constraints. Claims can be shared by several Pods, carry per-workload configuration and relate multiple devices through rules like same parent GPU. The result is placement based on real hardware properties rather than labels someone must keep accurate.

Can DRA and the NVIDIA device plugin run together?

Not through the NVIDIA GPU Operator in a single cluster. NVIDIA’s documentation says a cluster can have a GPUCluster resource for DRA or a ClusterPolicy for the device plugin, but not both. Upstream Kubernetes itself does not forbid mixing drivers and plugins on different nodes, but you must avoid exposing the same physical GPUs through both paths. Migrate by node pool and verify with canary workloads.

Does DRA support MIG and GPU sharing?

Yes, with caveats on maturity. NVIDIA documents full GPU and preconfigured MIG device allocation as generally available through DRA. Time-slicing works through a shared claim. Dynamic MIG, which creates instances on demand, MPS and scheduler-accounted consumable shares are documented as alpha. Upstream, partitionable devices and consumable capacity are beta from Kubernetes 1.36. Use whole GPUs or preconfigured MIG for production, and test the alpha options before relying on them.

Does the scheduler preempt Pods that hold DRA devices?

No. The Kubernetes documentation lists this as a current limitation: a running Pod using DRA resources cannot be preempted by a higher-priority Pod that needs the same resources. The urgent Pod remains Pending until the device frees up. Plan for it with reserved pools, quotas on low-priority workloads, or an external controller that evicts workloads deliberately when urgent jobs arrive.

Further Reading

• Karpenter vs Cluster Autoscaler for GPU node scaling: how node autoscaling interacts with claims and GPU pools.
• Kubernetes cost optimization and GPU right-sizing: measure utilization before and after you adopt sharing.
• NVIDIA Spectrum-X AI Ethernet fabric for 100k-GPU clusters: the network layer beneath large GPU clusters.
• Kubernetes documentation: Dynamic Resource Allocation: the canonical concept and task pages.
• Kubernetes v1.34: DRA has graduated to GA: the release announcement that defines what went stable.
• NVIDIA GPU Operator: DRA Driver for NVIDIA GPUs: prerequisites, install flags and feature status.

By Riju — about