Kubernetes Cost Optimization and GPU Rightsizing (2026)

Most Kubernetes bills are not high because the workloads are heavy. They are high because the cluster is full of air: reservations nobody uses, nodes nobody drains, and accelerators idling at single-digit utilization while the meter runs at full price. Kubernetes cost optimization and GPU rightsizing in 2026 is the discipline of finding that air and removing it without turning your savings into an outage.

It matters more now than it did two years ago. A single eight-GPU training node can cost more per hour than an entire web tier, and inference fleets are routinely deployed one model per card because that is what the default scheduler hands out. The tooling has also changed under us: Karpenter is a stable v1 API with new consolidation behavior, in-place pod resizing has matured, and Dynamic Resource Allocation (DRA) has reached general availability and is replacing the integer-only GPU model.

You will leave with a control loop you can run every week, a cost model you can adapt with your own prices, and a decision framework for MIG, time-slicing, MPS and DRA-based sharing.

What this covers: the request-to-usage gap, node bin-packing with Karpenter, workload autoscaling, GPU sharing mechanisms, spot GPU handling, measurement, and the failure modes that make savings dangerous.

Context and Background

Kubernetes overspend has three structural causes, and nearly every cluster suffers from all three at once.

The first is inflated requests. Developers copy a manifest, double the numbers to be safe, and never revisit them. The scheduler treats a request as a reservation, so a pod requesting two cores and using 0.2 holds 1.8 cores hostage. Published industry surveys have repeatedly reported that a large share of requested CPU and memory goes unused; the exact percentage varies by report and methodology, so treat any single headline number with suspicion and measure your own clusters. In practice, a request-to-usage ratio of three to five on CPU is common in clusters that have never been audited.

The second is idle and fragmented nodes. Autoscalers add nodes quickly under pressure and remove them reluctantly. A node that drops to 15 percent utilization after a spike can linger for hours because one pod with a restrictive PodDisruptionBudget pins it in place. Over a day of scale-ups and scale-downs, pods scatter across nodes the way files fragment on a disk, leaving each node 40 to 60 percent used and none empty enough to remove.

The third, and now the largest, is idle GPUs. Kubernetes exposes GPUs through device plugins as extended resources such as nvidia.com/gpu, which are integers and cannot be overcommitted. One pod asks for one GPU and receives the whole device, whether it needs 3 GB of memory or 70 GB. For a small embedding model or a low-traffic chat endpoint, that means paying for an entire accelerator to use a sliver of it.

These three causes share a root. Kubernetes optimizes for availability and isolation by default, and cost is a property you opt into. The scheduler honors requests as hard reservations because anything else risks noisy-neighbor failures. Autoscalers scale down cautiously because eviction risks availability. GPUs are handed out whole because partitioning needs explicit hardware support and plugin configuration. None of this is a bug. Cost optimization is therefore the act of relaxing safe defaults deliberately, with enough measurement to know you have not relaxed them past stability. It also explains why one-time cleanups decay: every new deployment arrives with the same defensive defaults.

The organizing framework is FinOps. The FinOps Foundation frames the practice as inform, optimize, operate: make spend visible by team and workload, remove waste, then bake the discipline into platform defaults. Our companion piece on FinOps and GreenOps with carbon-aware scheduling covers the organizational and carbon side. Here we stay on the Kubernetes mechanics.

A note on versions, verified at the time of writing (November 2026). The Kubernetes release page lists 1.37, 1.36 and 1.35 as the supported minor versions, with 1.34 reaching end of life on 27 October 2026. If you are still on 1.34, you are now outside the patch window, which matters here because several cost-relevant features (in-place resize, DRA improvements) mature across 1.33 through 1.36. Check your own provider’s version calendar, since managed services trail upstream by weeks to months.

The Cost Optimization Control Loop

To reduce Kubernetes cost, close three gaps in this order: the request-to-usage gap at the pod level, the node-packing gap at the cluster level, and the replica-count gap at the workload level. Right-sizing requests first makes every downstream packing and scaling decision cheaper and more accurate, because later stages inherit the error of earlier ones.

Figure 1: The cost optimization control loop. Visibility feeds recommendations, which drive changes to requests and replicas, which Karpenter packs onto nodes, with quota and admission policy closing the loop.

The loop starts with observability (OpenCost or a commercial equivalent mapping spend to workloads, plus GPU telemetry from NVIDIA’s DCGM exporter). Recommendations come from the Vertical Pod Autoscaler and idle-node reports. You act by tuning requests and replicas, Karpenter bin-packs the resulting demand onto the fewest and cheapest nodes, and guardrails (quotas, admission policy, budgets) stop waste from creeping back. The spot and on-demand mix sits beside the packing step because it changes the unit price, not the quantity.

Why order matters more than any single technique

Consider the three levers as multipliers on the bill. Right-sizing reduces the quantity of CPU and memory reserved. Bin-packing reduces the number of nodes needed to hold that quantity. Capacity-type selection (spot, savings plans, reserved) reduces the price per node. Because they multiply, doing them out of order wastes effort. If you buy a one-year commitment before right-sizing, you lock in a discount on capacity you were about to stop needing. If you bin-pack before right-sizing, you densely pack inflated reservations and still pay for the air.

The correct sequence is therefore: measure, right-size requests, fix replica counts, let the node autoscaler repack, and only then commit to baseline capacity sized to the new steady state. Commitments go last because they are the only lever that is hard to undo.

Allocation: who pays for idle?

Cost allocation tools divide node cost among pods in proportion to their requests or usage, whichever is higher. The leftover, capacity nobody requested, is “idle” cost. Decide up front who owns idle. If you charge it to the platform team, they have a reason to improve bin-packing. If you spread it across tenants proportionally, tenants have a reason to request less. Both are defensible. What is not defensible is leaving idle unowned, because then nobody moves it. OpenCost, the vendor-neutral open source project for measuring and allocating container costs, is the common reference implementation, and most commercial tools build on the same allocation model.

Requests, Limits and Rightsizing at the Pod Level

Requests are the foundation, because the scheduler makes every placement decision from them. Inflated requests make the bin-packer provision nodes for capacity nobody uses. Fix requests first or everything below inherits the error.

The VPA in recommendation mode

The Vertical Pod Autoscaler watches historical consumption and emits recommendations: a target plus lower and upper bounds, derived from usage percentiles. Run it in Off (recommendation-only) mode first. You get the numbers without having the controller restart pods on its own schedule.

Two changes in recent Kubernetes releases are worth knowing. Historically, changing a pod’s resources meant recreating the pod, which is why auto-apply on production services was unpopular. In-place pod resize (KEP-1287) lets the kubelet change a running container’s CPU and memory without a restart; it entered beta in 1.33 and is reported to have graduated to stable in 1.35. VPA gained an InPlaceOrRecreate update mode that uses it. I could not retrieve the 1.35 release notes during this run, so confirm the stable status and the exact VPA mode name in the current upstream documentation before relying on them in production.

Even with in-place resize, memory decreases are the dangerous direction. Shrinking a memory limit below current working set can trigger an OOM kill, and not every runtime returns freed memory to the OS promptly (the JVM and Go runtimes are classic examples). Treat CPU resizes as cheap and memory shrinks as something to do with headroom and a restart policy.

A sane default manifest

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    memory: "512Mi"   # hard ceiling protects the node
    # no CPU limit, see throttling discussion below

Memory carries both request and limit because memory is incompressible. Exceed the limit and the kernel OOM-kills the container, so the limit is a safety ceiling for the node. CPU carries a request but no limit because CPU is compressible. A CPU limit is enforced by the Completely Fair Scheduler quota mechanism, which throttles the container within each scheduling period even when the node has idle cycles. The result is latency spikes at the tail with no cost benefit. This is a widely held practice rather than a universal law: multi-tenant clusters with untrusted workloads sometimes keep CPU limits for fairness, and the Linux kernel has improved quota behavior over the years. Measure throttling (container_cpu_cfs_throttled_periods_total in cAdvisor metrics) before and after you decide.

QoS classes decide who dies first

Requests and limits together set a pod’s Quality of Service class. If requests equal limits for every resource in every container, the pod is Guaranteed. If at least one request or limit is set but the pod does not meet the Guaranteed criteria, it is Burstable. If nothing is set, it is BestEffort. Under node memory pressure, the kubelet evicts BestEffort pods first, then Burstable pods that exceed their requests, and Guaranteed pods last.

Notice the trap. A pod with a memory limit but no CPU limit is Burstable, not Guaranteed. That is fine for most services, but if you strip limits from everything to save money you quietly demote critical services into eviction-prone classes. Keep latency-critical and stateful pods Guaranteed or tightly Burstable on memory (request close to limit). Let sacrificial batch work be BestEffort or low priority so it absorbs pressure instead of your API tier.

Targets and the waste metric

Aim for a request-to-usage ratio around 1.3 to 1.5 at the 95th percentile of usage, not the three to five most clusters start with. Rank workloads by absolute waste: (requested minus used) times replica count times unit price per core or gigabyte. The top ten deployments usually carry most of the recoverable spend, so fix those by hand and automate the long tail.

One caution on VPA recommendations: they are percentile-driven over a window, so a service with rare real spikes (a nightly report, a cache warm-up, a JIT compile at startup) can be sized to a number that starves it during the spike. Compare recommendations against peak behavior, not only the median. Startup is a special case: JVM services often need two to three times their steady-state CPU for the first minute. In-place resize combined with the startup CPU boost pattern addresses this, and Kubernetes has been adding first-class support for it, but verify the feature gate status on your version.

Admission-time defaults

Right-sizing once is easy. Keeping it right is the hard part. A LimitRange per namespace sets default requests and limits and caps the maximum a single container can ask for, so a developer who omits requests gets a sane default instead of an unbounded reservation. A ResourceQuota caps total requested CPU, memory and GPU per namespace, turning the namespace into a budget a runaway deployment cannot exceed. Beyond the built-ins, a policy engine such as Kyverno or ValidatingAdmissionPolicy (CEL-based and built into the API server) can reject manifests that violate cost rules, for example a pod requesting more than four cores without an exception label, or a GPU pod without a sharing profile. This is the “operate” leg of FinOps made concrete: optimization that lives only on a dashboard erodes, while optimization encoded as admission policy holds.

Node Bin-Packing and Autoscaling

Once requests are honest, the question is how few nodes can hold the demand. That is a bin-packing problem, and it is where Karpenter changed the economics relative to the older Cluster Autoscaler.

Cluster Autoscaler works from predefined node groups. You choose instance types in advance and it scales those groups. It is reliable and well understood, but coarse: it cannot pick a cheaper shape for the actual pending pods, and scale-down is conservative. Karpenter is provisioner-driven. It reads the aggregate resource shape of pending pods and launches the cheapest instance across families, sizes and capacity types that fits them. Its NodePool API reached v1 (karpenter.sh/v1) in 2024, and the project now lives under Kubernetes SIGs with provider implementations for AWS and Azure, among others. Other managed offerings (GKE Autopilot, EKS Auto Mode) wrap similar logic behind a managed control plane; they trade configurability for operational simplicity.

Figure 2: Pending pods are bound to existing nodes when they fit, otherwise a right-shaped node is launched. A separate consolidation scan drains empty or underused nodes, subject to disruption budgets.

A pending pod is tried against existing nodes first. If nothing fits, Karpenter launches the cheapest viable instance. The step that actually removes spend is the periodic consolidation scan, which finds nodes that are empty or whose pods would fit elsewhere, drains them, and terminates them.

Consolidation fights fragmentation

The scheduler places only new pods and never relocates running ones, so a cluster fragments over time. Consolidation is the defragmenter. It deliberately disturbs running placement to recover stranded capacity, which is why it must be allowed to evict and reschedule. A cluster that forbids disruption through blanket PodDisruptionBudgets or karpenter.sh/do-not-disrupt annotations freezes its own fragmentation and pays for it indefinitely. The lever and its cost are the same mechanism: controlled disruption now in exchange for fewer nodes.

A NodePool that consolidates, with guardrails

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "0 8 * * mon-fri"   # UTC cron
        duration: 10h
        reasons: ["Underutilized"]

Several details here are easy to get wrong. WhenEmptyOrUnderutilized consolidates underused nodes, not just empty ones; WhenEmpty is the conservative option. consolidateAfter is a cooldown after the last pod change before a node becomes a candidate, so a very small value increases churn. Budgets control how fast voluntary disruption proceeds, not whether it happens. With no budget defined, the default limit is 10 percent of nodes, budgets can be scoped by reason (Empty, Drifted, Underutilized) and by a UTC cron schedule, and when several budgets apply the most restrictive wins. A budget of "0" freezes voluntary disruption only.

That last phrase matters. Expiration, spot interruption and node auto-repair are forceful disruption methods and are not blocked by budgets. If you rely on a zero budget to protect a business-hours window, a node reaching expireAfter can still be removed. Pair expireAfter with a sensible terminationGracePeriod so you control the worst case.

Recent Karpenter releases (the project’s GitHub release notes show v1.14 and v1.15 in mid-2026) continue to add consolidation controls, including a newer consolidation policy variant and extra disruption metrics. I could not independently confirm the precise names and semantics of every new option, so check the release notes for the version you run before copying configuration from a blog post, including this one.

Scheduling constraints cap density

The bin-packer is only as dense as your constraints allow. Strict pod anti-affinity or topologySpreadConstraints with DoNotSchedule forbid co-location, so consolidation cannot repack as tightly as the arithmetic suggests. A “high availability” rule pinning one replica per node can double node count for a workload that would be safe at two per node. Prefer ScheduleAnyway or zone-level spread with a maxSkew greater than one for stateless services, and reserve hard anti-affinity for things that truly need it. Audit affinity with the same rigor you apply to requests. Our guide to Gateway API, Ingress and service meshes is a useful companion here, since sidecars and proxies add per-pod overhead that belongs in the same cost model.

Workload Autoscaling: Replicas, Events and Scale-to-Zero

Vertical sizing tunes a pod. Horizontal scaling tunes the number of pods, which is where you stop paying for idle replicas in quiet periods.

The Horizontal Pod Autoscaler scales replicas on CPU, memory, or custom and external metrics. For request-driven services, scaling on requests per second, in-flight concurrency or queue depth tracks load far better than CPU, which lags and overshoots. KEDA extends HPA with dozens of scalers (Kafka lag, SQS depth, cron, Prometheus queries) and supports scale-to-zero, which core HPA does not do by default. A consumer for a nightly batch can sit at zero replicas all day. Combined with Karpenter consolidation, the nodes backing that workload also disappear, so you pay nothing for idle consumers. See the KEDA documentation for the current scaler list.

A worked example (illustrative numbers)

Take a service of 40 replicas, each requesting 2 CPU cores and using 0.4 on average, a five-times over-provision. These numbers are illustrative, not measured.

Right-size. Setting requests to 0.6 core (a 1.5x ratio) cuts reserved CPU from 80 cores to 24, a 70 percent reduction before touching a node.
Repack. Suppose the original 80 cores were spread over eight 16-core nodes at roughly 62 percent requested (128 cores total, 80 requested). With 24 cores of demand plus daemon overhead and a 25 percent buffer, about 40 cores of capacity is enough: three 16-core nodes at most. Node count falls from 8 to 3, a 62 percent reduction.
Reprice. Place the stateless replicas on spot at an assumed 60 to 70 percent discount to on-demand (discounts vary by region, instance family and time; check your provider’s advisor). On the three nodes, if two are spot, the blended price falls again.

If an on-demand 16-core node cost $0.70 per hour (an assumed round number), the original eight nodes cost $5.60 per hour, about $4,090 per month. After steps 1 and 2 the three nodes cost $2.10 per hour. If two of the three run spot at an assumed 65 percent discount, the hourly cost is $0.70 + 2 x $0.245 = $1.19, about $870 per month. That is a reduction of roughly 79 percent. The point is not the exact figure but the structure: the levers multiply, and none works alone. The sequence matters more than any individual technique.

Avoid thrashing

Scaling reactions must be slower than the workload’s natural noise, or the system oscillates. HPA exposes stabilization windows and scaling policies so a brief dip does not trigger a scale-down that a brief recovery immediately reverses. Each oscillation costs: pods cold-start, caches re-warm, and Karpenter churns nodes underneath. Keep scale-up responsive, set scale-down stabilization to minutes for most services, and let the node layer follow. HPA and KEDA change pod count, which changes pending demand, which Karpenter then packs. Get the pod-level signal right first.

GPUs invert the cost model. On CPU, a few percent of waste is irritating. On GPU, one underused accelerator can outweigh the rest of the namespace. GPU rightsizing in Kubernetes is therefore less about trimming requests and more about sharing a physical device, because the default one-pod-one-GPU model leaves most of the silicon idle for inference, notebooks and fine-tuning experiments.

Direct answer: use MIG when tenants need hardware isolation and predictable latency, time-slicing when you want many light users on one card and can tolerate contention, and MPS when trusted cooperating processes should run kernels concurrently. In 2026, also evaluate DRA-based allocation, which expresses partitions as first-class, schedulable devices.

Figure 3: Three ways to share one physical GPU. Isolation strength falls from MIG to MPS to time-slicing, while flexibility rises in the opposite direction.

MIG: hardware partitions

Multi-Instance GPU partitions supported NVIDIA data-center GPUs (Ampere A100 and A30, Hopper H100 and H200, and newer generations) into as many as seven instances. Each instance gets dedicated compute slices, dedicated memory and dedicated memory bandwidth, and its own fault domain. A noisy neighbor cannot starve or crash a pod in a different slice. Profile names encode the split: on an 80 GB H100, a 1g.10gb instance is one-seventh of the compute slices with about 10 GB of memory. The NVIDIA MIG user guide holds the authoritative per-GPU profile tables, and I did not retrieve them in this run, so consult them for exact profile names on your SKU.

The costs of MIG are rigidity and operations. Profiles are fixed sizes, a workload needing slightly more than a slice cannot borrow from an idle neighbor, and reconfiguring a GPU typically requires that no workloads be running on it. Not every card supports MIG: consumer and many workstation or inference-oriented parts do not. Fragmentation applies here too. A GPU laid out as 3g + 3g + 1g can be unable to host a 4g request even with free slices elsewhere, so the mix of profiles you offer should match your actual model sizes.

Time-slicing: oversubscription by turns

The NVIDIA device plugin and GPU Operator can advertise each physical GPU as N schedulable replicas. Pods then share the device by context switching. There is no memory isolation and no performance isolation: one process can exhaust memory and OOM-kill neighbors, and latency becomes a function of how many tenants are active. It is trivial to enable and ideal for notebooks, CI and development clusters, where utilization matters more than tail latency. Be careful with the resource semantics: requesting nvidia.com/gpu: 1 on a time-sliced node means one replica, not one GPU, so quotas that count GPUs now count slices.

MPS: concurrent kernels, cooperative trust

CUDA Multi-Process Service lets kernels from multiple processes execute concurrently on the GPU through a shared context, which improves throughput over time-slicing for small kernels that individually underfill the device. Isolation is soft: processes cooperate, memory limits and active-thread percentages can be configured, but a fault in one client can affect others. It fits trusted internal pipelines, such as many small inference replicas from one team, better than multi-tenant platforms. MIG and MPS can be combined on supported setups (MPS inside a MIG instance), but check the current NVIDIA documentation for the supported matrix before building on it.

Mode	Isolation	Granularity	Best use	Main risk
MIG	Hardware, strong	Up to 7 fixed profiles	SLA-bound multi-tenant inference	Rigid profiles, fragmentation
Time-slicing	None	N replicas per GPU	Dev, CI, notebooks	OOM and latency interference
MPS	Soft, process level	Concurrent kernels	Trusted batch and small inference	Shared fault domain
Whole GPU	Full	1 pod per GPU	Large training, big models	Idle silicon

Dynamic Resource Allocation changes the model

Device plugins give the scheduler an opaque integer. Dynamic Resource Allocation replaces that with structured claims: a pod references a ResourceClaim that selects devices by attributes (model, memory, MIG profile) using CEL expressions, and the scheduler allocates against ResourceSlice objects published by a vendor driver. The upstream Kubernetes blog states that DRA graduated to general availability in Kubernetes 1.34, with many resource.k8s.io APIs reaching GA while related features such as partitioned devices and consumable capacity moved through beta and alpha stages. I could not retrieve the detailed per-feature maturity table in this run, so verify each feature gate against your cluster version before depending on it.

Why this matters for cost: with DRA, a MIG slice or a share of a GPU becomes something the scheduler can reason about, claim and release dynamically, rather than a static node label created by an operator reconfiguring the card. Dynamic partitioning means a single GPU can serve a 10 GB model in the morning and a 40 GB model in the afternoon without draining the node. NVIDIA ships a DRA driver for GPUs, and cloud providers are integrating DRA into their managed offerings at different speeds. Expect a migration period in which device plugins and DRA coexist, and avoid enabling both for the same devices on the same node.

Picking a profile by model size

Rightsizing a GPU means matching memory first, then compute. A rough working set for serving a model is weights plus KV cache plus activations and runtime overhead. Weight memory is parameter count times bytes per parameter: a 7-billion-parameter model in 16-bit precision needs about 14 GB for weights, roughly 7 GB at 8-bit and about 3.5 to 4 GB at 4-bit (these are arithmetic facts, not benchmarks). The KV cache then grows with context length and concurrent sequences, and can exceed the weights for long contexts at high concurrency. A model whose weights fit a 10 GB slice may still need a 20 GB slice once you account for cache. Measure peak memory from DCGM_FI_DEV_FB_USED under realistic load before choosing a profile, and leave 10 to 20 percent headroom. For the economics of cache reuse on the serving side, our post on LLM prompt caching architecture and economics shows how to reduce the compute you need to begin with.

Spot GPUs, Checkpointing and Interruption Handling

Spot or preemptible GPUs are the other major price lever, with the largest discounts and the highest operational risk. Discount depth varies widely across providers, regions and GPU generations, and availability for the newest accelerators is thin. Treat any quoted percentage as a snapshot and read your provider’s spot advisor and pricing pages the day you decide.

The interruption notice is short. AWS documents a two-minute warning for EC2 spot; other clouds give shorter notice (commonly around 30 seconds on Google Cloud and Azure; verify current terms). That is enough to flush a checkpoint for a small job and not enough to save a large sharded training state unless you have been checkpointing continuously.

Figure 4: On an interruption notice, the node is cordoned and drained, the job receives SIGTERM and writes a checkpoint, and a replacement node resumes from it.

Design rules for spot GPU workloads

First, make work resumable. Checkpoint at a cadence set by the cost of lost work: if a node is reclaimed on average every few hours, checkpointing every 10 to 15 minutes bounds lost work to a small fraction. Write checkpoints to object storage or a shared filesystem, not node-local disk. Second, handle SIGTERM and set terminationGracePeriodSeconds to fit the notice window. Third, spread across multiple instance types, zones and, where possible, regions, because interruption correlation within a single pool is high. Fourth, separate latency-sensitive inference from interruptible batch. A customer-facing endpoint on spot needs redundancy across pools and a fallback to on-demand, which Karpenter can express by allowing both capacity types and weighting NodePools.

Batch queueing systems help. Kueue provides quota-aware, preemption-aware admission for batch jobs and integrates with cluster autoscaling, so GPU jobs wait in a queue rather than each triggering a new expensive node. Gang scheduling support matters for distributed training: a job needing eight GPUs across two nodes should start only when all of them can be placed, otherwise you pay for idle partial allocations while waiting.

Measuring It: Observability and Signals

You cannot rightsize what you cannot see, and GPU utilization is easy to misread.

For CPU and memory, compare container working set and CPU usage (cAdvisor and kube-state-metrics, scraped by Prometheus) to requests. Report the ratio per workload and per namespace. For nodes, track allocatable versus requested versus used. The gap between requested and allocatable is a bin-packing loss; the gap between used and requested is a rightsizing loss. Keeping those two apart tells you which lever to pull.

For GPUs, the NVIDIA DCGM exporter exposes metrics. DCGM_FI_DEV_GPU_UTIL reports the fraction of time any kernel was running, which can show 100 percent while the card does tiny amounts of work with most streaming multiprocessors idle. Metrics such as DCGM_FI_PROF_GR_ENGINE_ACTIVE, SM activity and tensor-core activity give a truer picture when profiling metrics are enabled. Memory used (DCGM_FI_DEV_FB_USED) is the right signal for choosing a MIG profile. Use both: a GPU with 15 percent memory used and low SM activity is a candidate for slicing; a GPU with high SM activity is genuinely busy even if memory looks low.

Attribute cost per GPU hour to the namespace or team holding the claim, whether a whole GPU or a slice. With time-slicing, attribute by replica count; with MIG, by profile fraction of the card. A cost tool that bills a time-sliced replica as a whole GPU will overstate cost by the replica factor and mislead teams into thinking sharing does not help. Validate your tool’s GPU accounting against a known configuration before you publish a chargeback report.

Trade-offs, Gotchas, and What Goes Wrong

Every saving here is purchased with a risk. Knowing which risk lets you spend it deliberately.

Right-sizing too tightly. Setting requests to the median starves a service at its peak. Symptoms are throttling, OOM kills during spikes and a rising p99. Mitigation: size on the 95th to 99th percentile for latency-critical services, keep a documented headroom factor, and alert on throttled-period ratios and OOM events after every change.

Consolidation churn. Aggressive consolidateAfter values evict pods frequently. Stateful or slow-starting services suffer; batch jobs restart from scratch. Use karpenter.sh/do-not-disrupt on long-running jobs, schedule budgets around business hours, and set PodDisruptionBudgets that allow movement rather than forbid it. A PDB with maxUnavailable: 0 is a hidden node pin.

Disruption that budgets do not cover. Expiration, interruption and repair are forceful. Do not promise a change freeze based on a zero budget alone.

Time-slicing surprises. Out-of-memory in one tenant kills another’s process, and latency tails widen as tenants are added. Never time-slice for workloads with strict latency SLOs. Keep a memory ceiling per tenant in application config, since the GPU itself will not enforce one.

MIG fragmentation and reconfiguration pain. A mix of profiles that does not match demand strands slices. Reconfiguration needs a drain, so treat the profile layout as a capacity-planning decision reviewed monthly, not a per-deployment toggle, until DRA-based dynamic partitioning matures in your environment.

GPU memory is the real limit. Slicing fails when the model does not fit. Quantization, smaller models, or paged KV-cache serving can shrink the footprint, but each carries accuracy or complexity trade-offs that must be evaluated, not assumed.

Anti-patterns to avoid. Buying commitments before right-sizing. Removing every CPU limit on a multi-tenant cluster with untrusted code. Running VPA in auto mode on stateful workloads without testing restarts. Putting spot on single-replica services. Measuring GPU health by GPU_UTIL alone. Treating the cost dashboard as the deliverable rather than the change in the manifest.

Practical Recommendations

Start with measurement, because you cannot defend a saving you did not baseline. Deploy a cost allocation tool and GPU telemetry, then record last month’s cost per namespace, the request-to-usage ratios and GPU memory and SM activity. That baseline is what you will compare against at the end of the quarter.

Next, fix requests on the top ten wasteful workloads using VPA recommendations, with peak checks. Enable admission defaults and quotas so the improvement persists. Then let Karpenter consolidate, starting with WhenEmptyOrUnderutilized, a conservative consolidateAfter, and a scheduled budget that protects business-critical windows. Broaden instance-type flexibility so consolidation and spot have room to work.

For GPUs, classify workloads into three buckets. Large training and big-model serving keep whole GPUs or multi-GPU nodes. Production inference with SLAs moves to MIG profiles matched to measured memory, or to DRA claims if your cluster and driver support it. Dev, notebooks and CI move to time-slicing with per-user memory ceilings. Put interruptible batch on spot with checkpointing and a queue.

Finally, commit only the measured baseline, review monthly, and keep the cost owner a named team.

Checklist:

[ ] Baseline cost per namespace and per workload, with a unit-cost metric
[ ] VPA in recommendation mode on all long-running deployments
[ ] Memory limits set, CPU limits removed or justified, QoS class reviewed
[ ] LimitRange, ResourceQuota and an admission policy for GPU requests
[ ] Karpenter NodePools with broad instance flexibility, budgets and expireAfter
[ ] Anti-affinity and topology spread audited for density impact
[ ] GPU workloads bucketed: whole, MIG or DRA, time-sliced
[ ] DCGM metrics collected, using SM activity and memory, not utilization alone
[ ] Spot jobs checkpoint and handle SIGTERM within the notice window
[ ] Commitments sized to post-optimization baseline only
[ ] Monthly reconciliation of tool reports against the invoice

Frequently Asked Questions

What is the fastest way to cut Kubernetes costs?

Fix inflated requests on your largest deployments first, then let a node autoscaler such as Karpenter consolidate the freed capacity. Right-sizing reduces how much you reserve, and consolidation turns that reduction into fewer billable nodes. Skipping the first step means consolidation packs inflated reservations. Most clusters recover a substantial share of spend from the top ten wasteful workloads alone, though the exact proportion depends on how long the cluster has been running unaudited and how defensive teams have been.

Choose by isolation need. MIG gives hardware-isolated partitions, up to seven per supported GPU, and suits multi-tenant production inference with latency targets. Time-slicing has no isolation and suits notebooks, CI and development. MPS lets trusted processes run kernels concurrently and suits cooperative batch. If you need dynamic partition changes without draining nodes, evaluate Dynamic Resource Allocation, which reached general availability in Kubernetes 1.34, against your driver’s support.

Do I need to remove CPU limits to save money?

Removing CPU limits does not reduce your bill directly. It removes throttling, which lowers tail latency and can let you run with smaller requests because bursts use idle headroom. The saving is indirect. Keep memory limits, since memory cannot be reclaimed without killing processes. On multi-tenant clusters running untrusted workloads, CPU limits can still be justified for fairness. Measure throttled periods before and after the change rather than treating it as a universal rule.

Is Karpenter better than Cluster Autoscaler for cost?

For most dynamic workloads on supported clouds, yes, because it matches instance shape to pending pods and continuously consolidates. Cluster Autoscaler remains a sound choice when you need predefined node groups, work on a provider without a Karpenter implementation, or want fewer moving parts. Either way the savings depend on honest requests and relaxed scheduling constraints. A tool cannot pack tighter than your anti-affinity rules allow.

Is spot safe for GPU workloads?

Spot is safe for work that can be interrupted and resumed, such as training with frequent checkpoints, batch inference and experimentation. It is risky for single-replica, latency-sensitive services. AWS gives a two-minute interruption warning, and other clouds give less, so checkpoints must be small enough or frequent enough to survive. Spread across instance types and zones, and keep an on-demand fallback for anything customer-facing.

How do I know a GPU is underutilized?

Do not rely on a single utilization percentage, which reports whether any kernel was running, not how much of the chip was busy. Combine memory used, SM activity and, where available, tensor-core activity from the DCGM exporter over a representative week. Low memory plus low SM activity indicates a slicing candidate. High memory with low compute suggests a memory-bound or poorly batched serving configuration, where better batching may beat adding hardware.