Last Updated: October 2026

Karpenter Node Autoscaling for Kubernetes: Production Deep Dive

Most Kubernetes clusters waste money in a boring way: nodes sized for a workload that no longer exists, half-empty machines nobody dares to drain, and a node group whose instance type was chosen two years ago. Karpenter attacks that waste at its root. It does not scale pre-defined pools. It reads the pending pods, computes the cheapest set of machines that can host them, launches those machines directly through the cloud API, and later repacks the cluster when a cheaper arrangement exists.

That model matters more in 2026 than when Karpenter first appeared. The v1 API has been stable for two years, the AWS provider tracks Kubernetes 1.36, Azure ships a Karpenter-based managed add-on, and Amazon EKS Auto Mode runs Karpenter underneath. Choosing between node autoscalers is no longer a research question. Operating Karpenter well is.

You will leave with a precise mental model of how Karpenter decides, a production-grade NodePool and EC2NodeClass you can adapt, a disruption-budget strategy that survives peak hours, and a list of the failure modes that cost real teams real incidents.

What this covers: the v1 resource model, the provisioning algorithm, consolidation and drift, disruption budgets, spot interruption handling, a comparison with Cluster Autoscaler, other providers, cost modelling, and what goes wrong.

Context and Background

Kubernetes separates two jobs that are easy to conflate. The scheduler (kube-scheduler) places pods onto existing nodes. A node autoscaler decides how many nodes exist and what shape they have. The scheduler has no authority over node supply, so something else must watch for pods that cannot be placed and add capacity.

For most of Kubernetes’ history that something was the Cluster Autoscaler. It works on node groups, which on AWS are Auto Scaling Groups (ASGs). Each group has a fixed instance shape. When pods are pending, Cluster Autoscaler simulates which group, if grown, would let them schedule, then raises that group’s desired size and waits for the cloud to deliver a node. Scale-down is the reverse: nodes below a utilization threshold for a period are drained and removed.

This design is simple and portable, which is why it supports dozens of providers. Its cost is structural. Capacity is quantized into whatever shapes you pre-declared, so a 3-vCPU pod request lands on a 16-vCPU machine. Diversifying shapes means creating more node groups, and the number of groups it can reason about grows into an operational burden.

Karpenter began at AWS in 2021 as an answer to that structure. Its core now lives in the Kubernetes SIG Autoscaling repository, kubernetes-sigs/karpenter, while each cloud supplies a provider that implements a small interface for creating, listing, and deleting instances. The AWS provider is the reference implementation, and as of this writing the documentation site lists v1.14 as the latest release. The first stable API, karpenter.sh/v1, arrived with v1.0, and the maintainers state that incompatible changes will require a v2.

Cost is the usual motivation, and it deserves scepticism. Public case studies report meaningful compute savings after migration, but those results mix several effects: moving to Spot, adopting Graviton, right-sizing, and better packing. Karpenter enables all of those cheaply; it does not by itself produce any particular percentage. Later sections give you a way to model your own savings instead of borrowing someone else’s. For the broader cost discipline around it, see our guide to FinOps and carbon-aware scheduling.

The Karpenter v1 Resource Model

Karpenter’s behaviour is driven by three kinds of objects. Understanding which one you own, which one Karpenter owns, and how they relate removes most of the confusion newcomers report.

The direct answer: a NodePool declares what kinds of nodes Karpenter may create and how they may be disrupted. A NodeClass (EC2NodeClass on AWS) holds cloud-specific launch settings. A NodeClaim is Karpenter’s own request for one machine, created and deleted by the controller, never by you.

NodePool: the policy object

A NodePool is cluster-scoped and has two jobs. Under spec.template it describes the nodes: labels, taints, startup taints, a reference to the NodeClass, expireAfter, terminationGracePeriod, and the requirements list. Under spec.disruption it describes when Karpenter may remove those nodes. It can also carry limits (a ceiling on the sum of resources it will provision) and a weight for ordering between pools.

Requirements are the heart of it. Each is a label key, an operator, and values. The operators are In, NotIn, Exists, DoesNotExist, Gt, Lt, Gte, and Lte. Well-known keys include kubernetes.io/arch, kubernetes.io/os, topology.kubernetes.io/zone, karpenter.sh/capacity-type (spot, on-demand, reserved), and on AWS the karpenter.k8s.aws/instance-* family such as category, family, generation, and CPU. A pod’s own selectors and affinities are intersected with the NodePool’s requirements to produce the set of machines that could host it.

The design principle is to constrain by properties, not by names. Allowing “any current-generation compute, general, or memory instance with 4 to 32 vCPUs” gives Karpenter a large candidate set; naming three instance types gives it three. Candidate-set size is the single biggest lever on both price and Spot availability.

NodeClass: where the cloud lives

On AWS the EC2NodeClass selects subnets, security groups, and AMIs, and names the IAM role Karpenter attaches to nodes through an instance profile it manages. In v1 amiSelectorTerms is required. The convenient form is an alias such as al2023@v20260101 or bottlerocket@v1.40.0, written as family@version, where AL2 and AL2023 pin by date, Bottlerocket pins by semantic version, and the alias must be the only term. The documentation warns against @latest in production because every new AMI release makes existing nodes drift and be replaced.

Kubelet configuration moved from the NodePool to the EC2NodeClass in v1, which is a common upgrade surprise. Block device mappings default to a 20 GiB encrypted gp3 root volume on AL2 and AL2023, and metadata options default to IMDSv2 required with a hop limit of 1. That hop limit of 1 breaks pods that call the instance metadata service through the pod network without host networking, which is intentional hardening but occasionally a debugging detour.

NodeClaim: the unit of capacity

When pods are pending, Karpenter creates a NodeClaim: a request for a machine that satisfies a particular set of resource and requirement constraints. The provider turns it into an instance, the instance boots and registers as a Node, and Karpenter ties the two together by the NodeClaim’s status. You normally never write one, but reading them is the best debugging tool you have. kubectl get nodeclaims shows what Karpenter decided and why a launch is stuck; kubectl describe nodeclaim shows conditions such as Launched, Registered, and Initialized.

Two timing facts shape expectations. A node counts as initialized only after its resources are reported, which for GPUs means the device plugin DaemonSet has run. And expireAfter defaults to 720 hours, so every node has a built-in maximum lifetime of 30 days unless you set it to Never.

How Provisioning Works: From Pending Pod to Running Node

Figure 1: The Karpenter provisioning path. Pending pods are batched, simulated against NodePool requirements, turned into NodeClaims, and launched through the provider.

The flow in Figure 1 starts when kube-scheduler marks a pod unschedulable. Karpenter’s provisioner watches for such pods, but it does not react to each one instantly. It opens a batching window: the window closes after one second of idleness (BATCH_IDLE_DURATION, default 1s) or ten seconds total (BATCH_MAX_DURATION, default 10s), whichever comes first. Batching exists so a Deployment scaling from 5 to 200 replicas becomes one scheduling decision rather than 200 racing ones.

The scheduling simulation

Inside the window, Karpenter runs its own scheduling simulation. It first tries to fit pods onto existing nodes and on NodeClaims it has already decided to create, then opens new virtual nodes for what remains. Each virtual node starts as the intersection of a NodePool’s requirements and the first pod’s constraints. Every additional pod that fits narrows the set of instance types further, because the node must still hold the combined requests.

This is a first-fit-decreasing style bin-packing: larger pods are placed first, since small pods fill gaps cheaply afterwards. The algorithm is a heuristic, not an optimal solver. Bin-packing with arbitrary shapes is NP-hard, and Karpenter trades optimality for decisions made in seconds. The practical consequence is that results are good but not guaranteed minimal, which is one reason consolidation runs continuously afterward.

Several inputs feed the simulation:

• Pod requests, not limits. Instance type selection uses only requests. A pod with a 200 MiB request and a 4 GiB limit is sized as 200 MiB. Honest requests are therefore the foundation of every Karpenter cost outcome.
• DaemonSet overhead. Karpenter adds the requests of DaemonSets that would land on a node when sizing it, so a node is chosen large enough for the pods plus its own agents. It does not reserve that capacity, it only accounts for it.
• Topology and affinity. Required node affinity, pod affinity and anti-affinity, and topology spread constraints (zone, hostname, capacity type) are honoured. Preferred affinities and ScheduleAnyway spreads are treated as required first and relaxed one at a time if unsatisfiable. The PREFERENCE_POLICY setting (Respect by default, or Ignore) controls this; Respect can create more nodes than expected.
• Volumes. Karpenter follows pod to PersistentVolumeClaim to StorageClass and folds the zone constraint of an existing EBS volume into the launch decision.
• Memory overhead. Because the VM reserves some memory, Karpenter subtracts a percentage (VM_MEMORY_OVERHEAD_PERCENT, default 0.075) when exact data is not cached. If your measured capacity differs, tune it rather than over-requesting.

Choosing instance types

Once the simulation yields a NodeClaim with a set of compatible instance types, the AWS provider sorts them by price and sends the cheapest, together with larger alternatives, to EC2 Fleet. The FAQ states that Karpenter passes up to 60 instance type options: the most efficient fit plus 59 larger ones. For on-demand it uses the lowest-price allocation strategy; for Spot it uses price-capacity-optimized, which weighs both price and the depth of the capacity pool so launches avoid the cheapest-but-about-to-be-reclaimed pools.

This explains a counterintuitive tuning rule: more flexibility yields lower price and lower interruption rate. Teams that pin Spot pools to two instance families are paying twice, once in higher hourly cost and once in reclaimed capacity.

Controlling the candidate set with minValues

The minValues field on a requirement forces the candidate set to contain at least that many distinct values for a key. In the reference example, minValues: 2 on instance category and minValues: 5 on instance family means a NodeClaim is not launched unless five families can serve it. The documentation marks this feature alpha, so verify it for your version. It is governed by MIN_VALUES_POLICY: Strict (default) fails scheduling on that NodePool when the minimum cannot be met, while BestEffort relaxes it. Use minValues on Spot pools to prevent a narrow pod mix from silently collapsing the pool into a single, fragile instance type.

After launch

The provider returns an instance ID, Karpenter records it on the NodeClaim, and the kubelet registers a Node. Karpenter then removes any startup taints once the node is initialized. kube-scheduler, not Karpenter, binds the pending pods. That separation matters: Karpenter predicts where pods will land, but if kube-scheduler scores a different node highest, pods can drift away from the machine Karpenter built for them. The scheduling documentation recommends the MostAllocated scoring strategy in kube-scheduler so placement agrees with Karpenter’s packing, which is only configurable where you control the scheduler profile. On managed control planes you may not.

Typical time to a ready node is a function of the instance boot, the AMI, and the CNI and agent startup, so measure your own. Teams that bake images with large container images pre-pulled, or use Bottlerocket’s small footprint, consistently shave tens of seconds. We treat any fixed number you read elsewhere as environment-specific.

Production Configuration: A NodePool and NodeClass That Hold Up

The following manifests are a defensible starting point for a general-purpose pool on AWS. The alias version and role name are placeholders; replace them with your own, and validate against the Karpenter version you run.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: general
spec:
  role: KarpenterNodeRole-my-cluster          # placeholder IAM role
  amiSelectorTerms:
    - alias: al2023@v20260101                 # illustrative pin, never latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  metadataOptions:
    httpTokens: required
    httpPutResponseHopLimit: 1
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
  kubelet:
    maxPods: 110
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    metadata:
      labels:
        workload-class: general
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: general
      expireAfter: 336h                       # 14 days
      terminationGracePeriod: 30m
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gte
          values: ["5"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["4", "8", "16", "32"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "30 3 * * mon-fri"          # cron is UTC
        duration: 10h
        reasons: ["Underutilized", "Drifted"]
  limits:
    cpu: "2000"
    memory: 8000Gi

A few choices deserve explanation. The Gte 5 generation floor excludes old, slower, and usually pricier generations without naming any. Allowing both architectures lets Karpenter choose Graviton when your images are multi-arch; if they are not, pods simply constrain themselves to amd64 with a node selector. The terminationGracePeriod is a safety valve: once draining starts, after thirty minutes remaining pods are force-deleted, so a stuck PodDisruptionBudget cannot pin a node forever. Combined with expireAfter, the maximum node lifetime is the sum of the two.

The limits block is your blast-radius guard against runaway scale-out from a bad deployment or a loop. It is eventually consistent, so a rapid scale-out can overshoot slightly. Treat it as a circuit breaker, not an exact quota, and keep the cloud account’s own service quotas as a second line.

Mutually exclusive pools are easier to reason about than overlapping ones. If a pod matches several NodePools, the highest weight wins, but a design where each pod matches exactly one pool, using labels and taints, makes behaviour predictable. A typical production layout has a general pool, a Spot-only batch pool with a taint, a GPU pool with a taint and a tight limits.nvidia.com/gpu, and a small on-demand pool for system and stateful components. Tainted pools mean only workloads that opt in with a toleration land there, which keeps cheap interruptible capacity away from databases.

Disruption: Consolidation, Drift, Expiry and Budgets

Figure 2: The graceful disruption loop. Drift runs before consolidation, every candidate passes through a budget check, and replacements are launched before the old node is drained.

Provisioning is half of Karpenter. The other half is removing nodes, and it is where most production incidents originate. Figure 2 shows the graceful path. The documentation defines three classes of method: automated graceful (Drift, then Consolidation), automated forceful (Expiration, Interruption, and the alpha Node Auto Repair), and manual (deleting nodes, NodeClaims, or NodePools).

Consolidation

Consolidation is Karpenter’s answer to the idle-node problem Cluster Autoscaler leaves to utilization thresholds. It tries three actions in order: delete empty nodes, delete multiple nodes by repacking onto others or a single cheaper replacement, and replace a single node with a cheaper one. Each candidate is evaluated by simulation. Karpenter asks whether all reschedulable pods on the candidate could fit on remaining capacity plus at most one new node that costs less than the candidates it replaces. If yes, the action is valid.

The policy is set per NodePool through consolidationPolicy. WhenEmpty considers only empty nodes. WhenEmptyOrUnderutilized (the default) also considers nodes whose pods could be repacked. Recent releases document a Balanced policy that consolidates only when savings outweigh the disruption; verify it exists in the version you run before depending on it. The companion field consolidateAfter defaults to 0s and is a stability timer: it resets whenever a pod is added to or removed from the node. Setting it to Never disables consolidation for the pool.

An important detail is that the default of zero makes Karpenter eager. A burst-and-settle workload will be consolidated the moment the burst ends, and re-provisioned at the next burst, paying boot latency again. For spiky services a consolidateAfter of a few minutes smooths this churn at the price of paying for idle capacity a little longer.

Spot adds a rule. Deletion consolidation of Spot nodes is on by default, but replacing a Spot node with a cheaper Spot node requires the SpotToSpotConsolidation feature gate, documented as alpha and off by default. The documentation also requires at least 15 instance types of flexibility for single-node Spot-to-Spot replacement, because the idea is to avoid trading into a thinner, more interruptible pool just to save pennies.

Drift

A NodeClaim is drifted when it no longer matches the NodePool or NodeClass that created it. Karpenter stores a hash of the template on the owners. Changing NodePool requirements, or an EC2NodeClass amiSelectorTerms, subnetSelectorTerms, or securityGroupSelectorTerms, flags matching nodes as drifted so they are replaced. Behavioural fields such as weight, limits, and disruption.* do not trigger drift, and widening requirements so a node’s current type remains valid does not either.

Drift is how AMI rollouts happen: edit the alias, and the fleet rolls node by node under your budgets. That is powerful and dangerous in equal measure. A one-line change to an AMI selector is a fleet-wide replacement event, which is why pinning versions and gating by budget are non-negotiable.

Expiry and the forceful methods

expireAfter is a maximum lifetime, not a minimum. It lives at spec.template.spec.expireAfter, defaults to 720h, and is forceful: expired nodes begin draining as soon as they expire, and budgets do not rate-limit them. Changing the NodePool value does not modify existing NodeClaims; it induces drift so replacements pick it up. Interruption handling, covered next, is likewise forceful. Because forceful methods ignore budgets, a large synchronized fleet can expire simultaneously if every node was launched in the same minute, such as after a full cluster build. Stagger by creating nodes over time, or accept it with generous PodDisruptionBudgets.

Disruption budgets

Budgets are the guardrail. Under spec.disruption.budgets each entry sets nodes as a count or percentage, optionally scoped by reasons (Drifted, Underutilized, Empty) and optionally active only during a cron schedule for a duration. With no budgets defined, the default is a single budget of nodes: 10%. When multiple budgets apply to a reason, the most restrictive wins. For percentages, allowed disruptions equal the rounded-up percentage of total nodes minus nodes already deleting or NotReady.

Two traps recur. The schedule is a UTC cron expression; time zones are unsupported, so a “business hours” freeze for a team in Asia must be written in UTC. And percentages round up, so with three nodes in a pool, 10% still permits one disruption at a time. A budget of "0" during a window is the supported way to freeze voluntary disruption entirely, and the earlier manifest uses one to protect a long overnight batch window.

Protecting workloads

Three mechanisms interact. PodDisruptionBudgets limit simultaneous evictions of a workload and are respected during draining. The annotation karpenter.sh/do-not-disrupt on a pod excludes its node from consolidation and conditionally from drift. Its value can be "true" for permanent protection or a Go duration such as "30m" for protection that begins when the pod starts running. On a node, the same annotation blocks voluntary disruption. It does not stop Expiration, Interruption, Node Repair, or manual deletion. And terminationGracePeriod on the NodePool overrides blocking PDBs and annotations for drift once it elapses.

The pattern that works: use PDBs for services, do-not-disrupt with a duration for batch jobs that should finish, and never use permanent "true" on anything that can be restarted, because one forgotten annotation holds an entire node, and its cost, indefinitely.

Spot Instances and Interruption Handling

Figure 3: Spot interruption handling. Karpenter launches the replacement the moment the warning arrives, and drains the doomed node in parallel.

Spot capacity is where Karpenter’s design pays most clearly, because its per-pod instance selection naturally spreads across pools. Interruption handling is optional but essential. It is enabled by setting the --interruption-queue argument (environment variable INTERRUPTION_QUEUE) to the name of an SQS queue; if unspecified, interruption handling is disabled. You create the queue and EventBridge rules that forward AWS events, and the Getting Started CloudFormation template provisions both.

Karpenter acts on Spot interruption warnings, scheduled-change health events, and instance terminating and stopping events. Instance status check failures are detected through DescribeInstanceStatus, which needs IAM permission but no queue. When it sees one, it taints, drains, and terminates the node ahead of time. For Spot, EC2 gives a two-minute notice. Karpenter starts a replacement immediately and drains in parallel, so if boot time is under two minutes the replacement is ready before the old node dies.

Notably, Karpenter does not act on Spot rebalance recommendations; it only publishes Kubernetes events. The documentation points out that AWS Node Termination Handler does cordon and drain on rebalance, which can cause extra churn. Running both is explicitly discouraged. Pick Karpenter’s native handling and remove Node Termination Handler.

Designing for two minutes

Two minutes is short. Pods must handle SIGTERM and finish within the grace period, readiness gates must route traffic away fast, and the replacement must boot quickly. Three practices help. Set terminationGracePeriodSeconds below 120 for Spot-resident services, or they get killed mid-drain regardless. Keep at least two replicas across zones with topology spread so one reclaim cannot remove a whole service. And ensure the Spot pool has wide instance diversity so a reclaim of one type does not mean the replacement cannot be found.

Interruption is a forceful method, not rate-limited by budgets, and not blocked by do-not-disrupt. A pod annotated against disruption on a Spot node will still lose its node when AWS reclaims it. Stateful or long-running work belongs on on-demand or reserved capacity, enforced with a taint and node selector rather than by hoping.

A mixed strategy is common: a baseline of on-demand or reserved capacity sized to steady load, with Spot for the elastic remainder. Karpenter supports a reserved capacity type for capacity reservations (the ReservedCapacity gate is beta and on by default per the settings reference), and a single NodePool listing both spot and on-demand will prefer Spot when it is available and fall back to on-demand when Spot capacity is exhausted.

Karpenter vs Cluster Autoscaler

Figure 4: Cluster Autoscaler scales pre-declared groups; Karpenter selects instance shapes directly and consolidates by cost simulation.

The two tools solve the same problem with different abstractions, as Figure 4 sketches. Cluster Autoscaler talks to a group abstraction (ASG, VMSS, managed instance group). Karpenter talks to the instance API directly and has no groups at all. The comparison below is qualitative; any latency or savings figure depends on your environment, and we label none as universal.

Dimension	Cluster Autoscaler	Karpenter
Unit of scaling	Node group, fixed shape	NodeClaim, shape chosen per pod set
Instance diversity	One group per shape set	Constraint-driven, wide candidate set
Scale-up path	Group desired size, then cloud	Direct fleet launch
Scale-down	Utilization threshold and timers	Simulation-based consolidation
Spot handling	Via group mixed policies	Native selection plus interruption queue
Image rollout	Group rotation or instance refresh	Drift under budgets
Provider breadth	Many providers	AWS, Azure, and a few others
Version coupling	Tracks Kubernetes minor	Independent, but with compatibility matrix
Failure surface	Group config sprawl	Single controller, broad cloud permissions

The compatibility point deserves nuance. The FAQ says Karpenter is not tied to a specific Kubernetes version the way Cluster Autoscaler is, but it does publish a matrix: Kubernetes 1.34 requires Karpenter 1.6 or later, 1.35 requires 1.9, and 1.36 requires 1.13. Upgrade Karpenter before bumping the control plane, not after.

Karpenter can coexist with Cluster Autoscaler during migration. A common path: keep a small managed node group running Karpenter itself and critical add-ons, scale Cluster Autoscaler’s groups down gradually, and let Karpenter take new load. Do not run Karpenter on nodes it manages, because a consolidation or drift event could evict the controller and stall recovery.

Choose Cluster Autoscaler when you need a provider Karpenter does not support, when node groups are tightly coupled to licensing, hardware reservations, or placement groups you manage elsewhere, or when your platform team wants the least dynamic behaviour. Choose Karpenter when workload shape is heterogeneous, Spot is meaningful, or image rollouts are painful.

Beyond AWS: Azure, EKS Auto Mode and Provider Reality

Karpenter’s core is cloud-agnostic, but the ecosystem is uneven. On Azure, the AKS Karpenter provider is open source under Apache-2.0. Microsoft packages it as Node Auto Provisioning (NAP), a managed add-on where AKS runs Karpenter for you and handles token rotation, Helm charts, and version updates. A self-hosted mode exists, with best-effort community support. The Azure analogue of EC2NodeClass is AKSNodeClass in the karpenter.azure.com group. The README lists limitations worth reading before committing: no Kubenet or Calico, no IPv6, no service principal authentication, and restricted Windows support. Verify the current state against Microsoft’s documentation, as these limits change.

On AWS, Amazon EKS Auto Mode builds on Karpenter. AWS operates compute autoscaling, networking, load balancing, DNS, block storage, and GPU support for you. Nodes launched by Auto Mode have a maximum lifetime of 21 days, after which they are replaced. The default NodePools and NodeClasses are managed and should not be edited, but you add your own. In Auto Mode the NodeClass uses the eks.amazonaws.com API group rather than karpenter.k8s.aws, so manifests are not interchangeable.

The decision between self-managed Karpenter and a managed variant is operational, not technical. Self-managed gives you control over versions, feature gates (Spot-to-Spot consolidation, static capacity, node overlays), and AMI strategy, and burdens you with upgrades and IAM. Managed variants reduce toil and constrain customization. A reasonable rule: if your team lacks the capacity to run a controller with broad cloud permissions through every Kubernetes upgrade, take the managed option and spend the saved effort on workload requests and PDBs, where the savings actually are.

Other providers exist in the ecosystem, and the core exposes a CloudProvider interface for them. Treat provider maturity as a project-by-project question: check release cadence, supported Kubernetes versions, and whether the features you need, such as consolidation and drift, are implemented.

Cost Modelling: Know Your Savings Before You Claim Them

Vendor case studies make Karpenter sound like a fixed discount. It is not. Savings come from four independent levers, and you should estimate each separately.

Packing efficiency. Define it as requested vCPU divided by allocatable vCPU across nodes. Illustrative numbers only: if a Cluster Autoscaler fleet runs at 45% requested-to-allocatable and consolidation raises it to 65%, the same pods need roughly 31% fewer node-hours (1 − 45/65). Your baseline and outcome will differ; measure with kube-state-metrics before and after.

Spot share. If 60% of node-hours move to Spot at a hypothetical 65% discount to on-demand, the blended price falls by 0.60 × 0.65 = 39%. Real Spot discounts vary by instance type, zone, and time, so use the Spot price history for your actual candidate set rather than a headline figure.

Architecture. Moving eligible workloads to Graviton changes price-performance. This is independent of Karpenter, but Karpenter makes mixed-architecture pools trivial. Benchmark your own workload; claims of fixed percentage gains do not transfer.

Right-sizing. Requests inflated by developers set the floor on everything above. A cluster packed perfectly around padded requests is still wasteful. Karpenter cannot fix this; tools like the Vertical Pod Autoscaler in recommendation mode, or observability-driven reviews, can.

Combine them multiplicatively, not additively, and subtract the costs: extra cross-zone traffic from spreading, interruption-induced restarts, and idle headroom you keep for burst latency. A defensible estimate is a range with stated assumptions, produced from a replay of a week of pod requests through Karpenter’s scheduler or a shadow NodePool, not a single number.

Observability is part of the cost story. Karpenter exports Prometheus metrics about NodeClaims, pods pending, disruption decisions, and cloud-provider call errors. Alert on pods pending beyond a threshold, NodeClaims stuck in Launched without registering, and sustained consolidation activity. Tag nodes through the NodeClass so cost allocation tools can attribute spend; the NodePool name is a natural cost-center label. For network-level observability that complements node-level cost data, see our post on Cilium eBPF networking, observability and security.

Trade-offs, Gotchas, and What Goes Wrong

Consolidation churn. Eager consolidation with consolidateAfter: 0s on a bursty workload creates a loop of launch, settle, delete, launch. Symptoms: nodes younger than ten minutes in steady state, elevated pod restarts, and EC2 API throttling. Mitigation: raise consolidateAfter, use budgets, and add do-not-disrupt durations for warm-up-heavy pods.

Blocking PodDisruptionBudgets. A PDB with maxUnavailable: 0, or a single-replica deployment with minAvailable: 1, makes its node undrainable. Drift and consolidation will retry forever while the budget slot is consumed. Audit PDBs, and set terminationGracePeriod as a backstop for drift.

Forceful methods ignore budgets. Expiration and Interruption proceed regardless. Synchronous node birth leads to synchronous node death at expireAfter. Randomize launch times or accept staggered rollouts via smaller pools.

Fragile single instance types. A NodePool constrained to one family loses Spot availability, loses cost optimization, and fails to launch when that family is exhausted in a zone. Widen requirements, use minValues, and monitor InsufficientCapacityError events.

IP exhaustion. Packing many small pods per node on a VPC CNI can exhaust subnet IPs well before CPU. The v1.13 release notes mention treating insufficient-IP conditions per subnet rather than per zone. Size subnets generously and set maxPods consciously.

The controller as single point of failure. Karpenter runs as a Deployment. If it is down, nothing scales and nothing consolidates, but existing nodes keep serving. Run it on capacity it does not manage, with two replicas and leader election, and protect its IAM role: it can launch instances and pass roles, which makes it a high-value credential.

Over-trust in latency claims. Pod startup time is dominated by image pull and application readiness as much as node boot. Karpenter shortens the provisioning stage; it cannot fix a 4 GB image.

Preferred affinities inflate node count. With PREFERENCE_POLICY=Respect, soft constraints behave as hard first, which can create extra nodes. Use required rules only when truly required.

Feature gate drift. Alpha gates (SpotToSpotConsolidation, NodeRepair, NodeOverlay, StaticCapacity, CapacityBuffer) can change between releases. Pin versions and read release notes before enabling any in production.

Practical Recommendations

Start by treating Karpenter as a policy system with three dials: how wide the candidate set is, how fast nodes may churn, and how much interruption the workload tolerates. Everything else follows.

Begin with two NodePools, a general pool and a tainted Spot pool, plus a small managed node group for the controller. Use constraint-based requirements across at least three instance categories and a generation floor. Pin AMI aliases to explicit versions and roll them out with a budget schedule that excludes your peak traffic window. Add interruption handling before the first Spot node, and remove any Node Termination Handler. Then work on the dial that Karpenter cannot touch: honest pod requests.

Checklist:

• Upgrade path: confirm the Kubernetes-to-Karpenter compatibility matrix before each control-plane bump.
• NodePools mutually exclusive, with taints for Spot, GPU, and stateful tiers.
• Requirements by property, not by instance name; minValues on Spot pools if your version supports it.
• consolidateAfter set deliberately for bursty services; budgets with a UTC freeze window.
• expireAfter and terminationGracePeriod set; PDBs audited for undrainable cases.
• limits set as a circuit breaker; cloud quotas as the backstop.
• Interruption queue, EventBridge rules, and two-minute-safe termination behaviour tested with a fault drill.
• Alerts on pending pods, stuck NodeClaims, and throttled provider calls.
• A shadow or replay test before claiming savings.

Frequently Asked Questions

What is Karpenter and how does it differ from Cluster Autoscaler?

Karpenter is a Kubernetes node autoscaler that watches unschedulable pods, computes which instance types can host them, and launches nodes directly through the cloud API. Cluster Autoscaler instead adjusts the size of pre-defined node groups. Because Karpenter has no group abstraction, it can mix instance shapes freely and consolidate nodes by simulating cheaper placements, rather than relying on utilization thresholds and timers.

What are NodePool, EC2NodeClass and NodeClaim?

A NodePool defines which nodes Karpenter may create, their taints, labels, requirements, limits, and disruption policy. An EC2NodeClass holds AWS launch settings such as AMI selectors, subnets, security groups, the IAM role, and kubelet options. A NodeClaim is the controller-created request for one specific machine, representing a single capacity decision. Operators write the first two and inspect the third for debugging.

Does Karpenter work with Spot instances safely?

Yes, provided interruption handling is configured. Karpenter reads EC2 events from an SQS queue, launches a replacement as soon as a Spot warning arrives, and drains the old node within the two-minute notice. Safety also depends on your workloads: graceful shutdown under 120 seconds, multiple replicas across zones, and wide instance diversity. Interruption is forceful, so budgets and do-not-disrupt annotations do not prevent it.

How do disruption budgets work in Karpenter?

Budgets under spec.disruption.budgets cap how many nodes can be voluntarily disrupted at once, as a count or percentage. They can be scoped to reasons (Drifted, Underutilized, Empty) and activated by a UTC cron schedule with a duration. The default is 10% of nodes. The most restrictive applicable budget wins. Expiration and interruption are forceful and are not limited by them.

Is Karpenter only for AWS?

No, but AWS is the reference provider. Azure offers a Karpenter-based provider, delivered as the managed Node Auto Provisioning add-on on AKS, with a self-hosted option. Amazon EKS Auto Mode also runs on Karpenter. Other providers exist in the ecosystem with varying maturity. Check each provider’s supported Kubernetes versions, feature coverage for consolidation and drift, and its support model before adopting.

How do I safely upgrade Karpenter?

Check the compatibility matrix first: for example, Kubernetes 1.36 requires Karpenter 1.13 or later. Upgrade Karpenter before the control plane, one minor version at a time, reading release notes for feature-gate and CRD changes. Test in a non-production cluster, keep budgets conservative during the rollout, and pin AMI aliases so the upgrade does not trigger an unrelated fleet-wide drift event.

Further Reading

• Cilium eBPF networking, observability and security for the networking layer that decides how many pods fit per node.
• FinOps and carbon-aware scheduling for the cost governance practices around autoscaling.
• Karpenter documentation for the NodePool, disruption, and settings references used throughout this post.
• Karpenter core repository for releases and the API source.
• Amazon EKS Auto Mode documentation for the managed Karpenter variant.

By Riju — about