Kubernetes Scheduler Internals: Topology Spread, Affinity and the Descheduler

A Deployment with six replicas rolls out, and four of them land in one availability zone. Nothing is broken: every Pod passed every Filter, the scheduler scored the nodes honestly, and the cluster still ended up with a failure domain you did not want. The kubernetes scheduler makes one decision per Pod, at one moment in time, with the information it has then. It never revisits that decision.

That single property explains most scheduling surprises: skewed zones after a node pool scale-up, Pods stuck Pending for reasons that look contradictory, preemption evicting the wrong workload, and affinity rules that quietly throttle a large cluster. Understanding the machinery turns these from mysteries into arithmetic.

This post walks through the scheduler from the inside, using the upstream Kubernetes documentation as the source of truth (v1.37 is the current docs version at the time of writing). You will leave with a mental model of the queues and plugin phases, exact skew arithmetic for topology spread, a cost model for affinity, and a practical policy for the descheduler.

What this covers: the scheduling and binding cycles, the three queues, Filter and Score plugins, affinity versus topology spread, taints and preemption, scheduler profiles, the descheduler, interaction with autoscalers and PDBs, scale tuning, and a debugging playbook.

Context and Background

The default scheduler, kube-scheduler, runs in the control plane, watches for Pods that have no spec.nodeName, and assigns each one to a Node. Upstream describes the job as a two-step operation: filtering finds the set of feasible nodes, and scoring ranks them. The highest score wins, and ties are broken at random. The decision is written back to the API server as a binding.

Everything else in this post is a refinement of that loop. Node selectors, node affinity, taints, resource requests, volume topology and Pod affinity are all Filter inputs. Spread preferences, image locality and resource-fit strategies are Score inputs. Since Kubernetes v1.19 the scheduling framework (a stable plugin architecture) has been the way both are implemented, so the in-tree features and your custom code run through the same extension points.

Why does this matter more in 2026 than it did five years ago? Three trends. First, clusters are heterogeneous: GPU pools, spot pools, and multi-zone node groups mean placement rules carry real cost and reliability weight. Second, autoscalers such as Karpenter provision nodes in response to what the scheduler cannot place, so scheduling constraints now directly shape your infrastructure bill (see our comparison of Karpenter versus Cluster Autoscaler for GPU node scaling). Third, batch and AI workloads increasingly need all-or-nothing placement, which is a different problem from the one-Pod-at-a-time model (covered in Kubernetes gang scheduling versus Volcano versus Kueue).

The thesis of this post: the scheduler is a greedy, stateless-between-decisions placer, so good placement is the product of three layers working together. Hard constraints (Filter) define what is legal. Soft preferences and spread (Score) bias what is chosen. A separate corrective loop, the descheduler, repairs drift that greedy placement cannot prevent. Teams that treat these as one tool end up over-constraining Pods and then wondering why the cluster is both full of Pending Pods and half empty.

For the canonical reference, keep the Kubernetes documentation on the scheduling framework and topology spread constraints open alongside this text.

Inside the Kubernetes Scheduler: Queues, Cycles and Plugins

The scheduler takes each unscheduled Pod through a scheduling cycle that picks a node, followed by a binding cycle that commits the choice. Scheduling cycles run serially, one Pod at a time, while binding cycles run concurrently, so slow operations like volume attachment do not stall the next placement.

Figure 1: The kubernetes scheduler pipeline. Pods move through the active, backoff and unschedulable queues; the scheduling cycle (PreFilter, Filter, Score, Reserve, Permit) runs serially and the binding cycle (PreBind, Bind, PostBind) runs concurrently.

Figure 1 shows the loop. A Pod enters through PreEnqueue, waits in a queue ordered by the QueueSort plugin, then runs the scheduling cycle. If no node is feasible, the Pod goes to the unschedulable pool (and PostFilter, usually preemption, gets a chance first). If a node is found, the Pod is reserved on that node and handed to a binding goroutine.

The three queues

The scheduler keeps three structures, which the upstream scheduling-framework page and the SIG Scheduling queue notes describe:

Active queue. Pods ready to be attempted, ordered by the QueueSort plugin. Only one QueueSort plugin may be enabled at a time, and the default orders by Pod priority and then by queue arrival time.
Backoff queue. Pods whose last attempt failed wait here before retrying. In the scheduler configuration API, podInitialBackoffSeconds defaults to 1 second and podMaxBackoffSeconds to 10 seconds, so a repeatedly failing Pod backs off exponentially up to that cap.
Unschedulable pool. Pods that failed Filter sit parked until something changes in the cluster that could plausibly help them. Without this, a large pending backlog would be retried against an unchanged cluster in a hot loop.

How does the scheduler know when to move a parked Pod? Through EnqueueExtension and QueueingHint. A plugin that rejected a Pod declares which cluster events (a node added, a Pod deleted, a label changed) could make it schedulable. The QueueingHint callback, executed on each event, answers “does this specific event help this specific Pod?” and routes the Pod to the active or backoff queue only if so. The upstream docs mark QueueingHint as a feature-gated capability (SchedulerQueueingHints); check the feature-gate table for your version before relying on its default state.

PreEnqueue deserves a mention because it explains a common source of confusion. A Pod with spec.schedulingGates is held back by a PreEnqueue plugin. Per the docs, such a Pod is placed in the unschedulable list without receiving an Unschedulable condition. If you see a Pod that is Pending but has no scheduling failure event, look for scheduling gates first. External controllers (quota systems, batch admission controllers) use gates to hold Pods until they are ready to compete for resources.

The scheduling cycle, phase by phase

Within one scheduling cycle the framework invokes plugins at defined extension points:

PreFilter pre-processes the Pod and computes state the Filter phase will reuse. An error aborts the cycle. For example, the inter-Pod affinity plugin computes which topology domains already contain matching Pods here, so Filter does not re-scan the cluster for every node.
Filter runs per node and removes nodes that cannot host the Pod. If any Filter plugin rejects a node, the remaining plugins are skipped for it. The docs note that nodes may be evaluated concurrently, which is why Filter plugins must be side-effect free.
PostFilter runs only when no node passed Filter. Preemption is the canonical PostFilter implementation: it tries to make room by evicting lower-priority Pods.
PreScore and Score rank the feasible nodes. Each Score plugin returns a value in a defined range, NormalizeScore rescales plugin outputs where needed, and the scheduler combines them using per-plugin weights.
Reserve tentatively assumes the Pod on the chosen node inside the scheduler’s cache, so the next serial cycle sees those resources as taken even though the binding has not reached the API server. If a later phase fails, Unreserve runs on all Reserve plugins in reverse order and must be idempotent.
Permit can approve, deny, or hold the Pod with a timeout. A “wait” result starts the binding goroutine but blocks it, which is the hook that coscheduling-style plugins use to hold a group until enough members are placed.

The binding cycle

PreBind does work that must precede binding, for instance finishing dynamic volume provisioning. Bind performs the write to the API server; the first Bind plugin that handles the Pod short-circuits the rest. PostBind is informational. Because this cycle is concurrent, the scheduler is already evaluating the next Pod against its assumed cache while the previous Pod is still being bound. That is a throughput optimisation with a consequence: a bind that fails after Reserve triggers Unreserve and sends the Pod back to the queue, and for a short window the cache and the API server disagree.

Why placement is greedy

Put the pieces together and the greediness is structural. The cycle evaluates exactly one Pod against a snapshot, picks the best node by current scores, and moves on. It cannot say “place this Pod badly now so that the next five fit better.” Nor can it later undo a placement because the world changed. Autoscaling, node drains, and rolling updates all change the world after the decision. That is the design gap the descheduler fills, and why we treat it as a first-class component in a later section.

Hard Constraints: Selectors, Affinity, Taints and Their Costs

Hard constraints are the Filter-phase rules that decide which nodes are legal for a Pod. The cheapest are label lookups (nodeSelector); the most expensive are cross-Pod rules (inter-Pod affinity), whose cost grows with the number of existing Pods the scheduler must examine.

nodeSelector and node affinity

nodeSelector is the simplest form: the Pod only fits nodes that carry every listed label. Node affinity expresses the same idea more richly. requiredDuringSchedulingIgnoredDuringExecution is a hard rule, equivalent to a selector with set operators (In, NotIn, Exists, DoesNotExist, Gt, Lt). preferredDuringSchedulingIgnoredDuringExecution is a Score input with a weight from 1 to 100.

Two semantics trip people up. If you specify both nodeSelector and node affinity, both must be satisfied. Within nodeSelectorTerms, multiple terms are ORed, while multiple matchExpressions inside a single term are ANDed. And the phrase “IgnoredDuringExecution” is literal: if node labels change after placement, the Pod is not evicted. That gap is again a descheduler job (the RemovePodsViolatingNodeAffinity strategy).

Node-label rules are cheap because they depend only on the node and the Pod. The scheduler evaluates them per node in Filter without looking at any other Pod.

Inter-Pod affinity and anti-affinity

Pod affinity attracts a Pod to nodes (or, more precisely, topology domains) that already run Pods matching a label selector. Pod anti-affinity repels it. The topologyKey names the node label that defines a domain: kubernetes.io/hostname for per-node, topology.kubernetes.io/zone for per-zone.

This is where the cost model changes. To decide whether a node is legal, the scheduler must know which Pods match the selector in which domains, including the existing Pods’ own anti-affinity terms against the incoming Pod (the rule is symmetric). The upstream docs carry an explicit warning that inter-Pod affinity and anti-affinity require substantial processing that can slow scheduling significantly in large clusters, and they recommend against using them in clusters larger than several hundred nodes. Treat that as the official guidance rather than a hard limit: the cost depends on the number of matching Pods, the number of terms, and whether you use preferred or required rules.

Some practical details:

Required anti-affinity at hostname level (one replica per node) is a legitimate and common use, but it is brittle. If you have five nodes and request six replicas, the sixth stays Pending forever, and a node drain during a rollout can block progress.
namespaceSelector widens matching beyond the Pod’s own namespace; an empty selector ({}) matches all namespaces, which also widens the PreFilter scan.
matchLabelKeys and mismatchLabelKeys for Pod affinity let a rule target “the same revision” without hand-editing selectors per rollout. The docs list the Pod-affinity matchLabelKeys field as beta and enabled by default in the current docs version; confirm the stage against your cluster version.
Preferred rules cost the same PreFilter work but never leave a Pod unschedulable, which is usually what you actually want.

Taints and tolerations

Taints work in the opposite direction: nodes repel Pods unless the Pod tolerates the taint. The three effects matter differently. NoSchedule blocks new placements, PreferNoSchedule is a soft Score penalty, and NoExecute also evicts running Pods that do not tolerate it (optionally after tolerationSeconds). Taints are the right tool for dedicated pools (GPU nodes, system nodes) because the default is exclusion; affinity alone cannot keep unrelated Pods off a pool.

A frequent GPU-pool pattern combines both: a taint such as nvidia.com/gpu=present:NoSchedule keeps general workloads out, a toleration plus a node-affinity rule or nodeSelector lets the GPU workloads in and pins them. Forgetting the toleration yields the classic 0/N nodes are available: N node(s) had untolerated taint event.

Which constraint for which job

Need	Best tool	Cost profile	Failure mode
Run only on a node class	nodeSelector or required node affinity	Cheap, per node	Pending if no node has the label
Keep others off a pool	Taint plus toleration	Cheap, per node	Missing toleration blocks the owner too
Even spread over zones or nodes	Topology spread constraints	Moderate, per domain	Skew not repaired after the fact
Co-locate with a specific Pod	Required or preferred Pod affinity	Expensive, per matching Pod	Hot spots, large-cluster slowdown
One replica per node, strictly	Required Pod anti-affinity (or spread with maxSkew 1)	Expensive	Pending when replicas exceed nodes

The right habit is to reach for topology spread before Pod anti-affinity whenever the goal is “distribute”, and reserve affinity for “this Pod needs that Pod nearby.”

Topology Spread Constraints: Skew Arithmetic Worked Through

Topology spread constraints let a Pod declare how evenly its siblings must be distributed across failure domains. The scheduler computes a skew per candidate domain and refuses, or penalises, placements that would exceed maxSkew. Unlike anti-affinity, spread constraints express a ratio rather than a prohibition, which makes them far more forgiving.

Figure 2: Topology spread constraints skew calculation. Placing a new Pod in a zone is legal only if that zone’s matching count plus one, minus the global minimum, is at most maxSkew.

The fields and what they actually do

The Pod-level topologySpreadConstraints list takes these fields, per the upstream documentation:

maxSkew (required, greater than zero). With DoNotSchedule, it is the maximum permitted difference between the matching-Pod count in the target domain and the global minimum. With ScheduleAnyway, it is a Score signal: the scheduler prefers domains that reduce skew but will still place the Pod.
topologyKey. The node label whose value defines a domain. Prefer the well-known topology.kubernetes.io/zone and kubernetes.io/hostname keys over private labels.
whenUnsatisfiable. DoNotSchedule (the default) makes it a Filter rule; ScheduleAnyway makes it a Score rule.
labelSelector. Selects which Pods are counted. Only Pods in the incoming Pod’s namespace are candidates.
minDomains (optional). Only valid with DoNotSchedule. If fewer eligible domains exist than minDomains, the global minimum is treated as 0.
matchLabelKeys. Lists Pod label keys whose values (taken from the incoming Pod) are merged into the selector. The standard use is pod-template-hash, so each Deployment revision is balanced independently. The docs describe it as beta and on by default since v1.27, and note that since v1.34 the key-value pairs are explicitly merged into labelSelector in the API server.
nodeAffinityPolicy (Honor by default when unset) and nodeTaintsPolicy (Ignore by default when unset). They decide which nodes count when computing the minimum. The docs state both became beta in v1.26 and graduated to GA in v1.33.

One structural rule: you can have only one constraint per unique combination of topologyKey and whenUnsatisfiable. Multiple constraints are combined with a logical AND.

Worked example one: the basic rule

Take three zones with matching Pod counts A=3, B=2, C=2, and maxSkew: 1, DoNotSchedule. The global minimum is 2.

Candidate zone	Count after placement	Skew (count minus min)	Legal?
A	4	4 – 2 = 2	No, exceeds 1
B	3	3 – 2 = 1	Yes
C	3	3 – 2 = 1	Yes

Zone A is filtered out; B and C survive and the Score phase breaks the tie. After the placement the counts are, for example, A=3, B=3, C=2 and the minimum is still 2, so zone A (3-2=1) is legal again next time. This is the steady state you want during a scale-up: counts move in lockstep.

Worked example two: minDomains and the empty-domain trap

Suppose you ask for minDomains: 3 with maxSkew: 1, but your cluster has only two eligible zones with nodes. Because eligible domains (2) are fewer than minDomains (3), the global minimum is treated as 0. The first Pod in zone A gives skew 1 – 0 = 1, which is legal. A second Pod in zone B also gives 1 – 0 = 1. A third Pod would make some zone hold 2, giving skew 2 – 0 = 2, which is illegal. The third replica stays Pending.

That is the intended use of minDomains: it turns “we must run in at least N zones” into a scheduling-level guarantee, and the Pending Pod is the signal that triggers an autoscaler to create nodes in a third zone. Karpenter and Cluster Autoscaler both react to unschedulable Pods, so this is a clean contract between the scheduler and the provisioning layer. Without minDomains, the scheduler would happily stack replicas into the two zones that exist and call it balanced.

Worked example three: why the defaults leave skew behind

If you define nothing, the scheduler applies built-in default constraints (stable since v1.24): maxSkew: 3 on kubernetes.io/hostname and maxSkew: 5 on topology.kubernetes.io/zone, both ScheduleAnyway. These are deliberately loose Score hints, not guarantees. A web tier with six replicas can legitimately land four-and-two across zones under defaults, because a skew of 5 is permitted and nothing here is a hard filter.

The docs also warn that the PodTopologySpread plugin does not score nodes that lack the configured topology keys. If your nodes do not all carry both kubernetes.io/hostname and topology.kubernetes.io/zone, define your own defaults instead of relying on the built-ins. You set cluster-level defaults through the PodTopologySpread plugin arguments in a scheduler profile; these apply only to Pods that define no constraints of their own and belong to a Service, ReplicaSet, StatefulSet or ReplicationController.

Interactions that bite

Node inclusion policies. With nodeAffinityPolicy: Honor, a Pod restricted by node affinity to a GPU pool counts skew only across GPU-pool nodes. With Ignore, domains that the Pod could never use still contribute a zero to the minimum, which can make every placement look skewed. Check which policy you have when constraints behave unexpectedly after you add a nodeSelector.

Taints. nodeTaintsPolicy defaults to Ignore, meaning tainted nodes the Pod cannot use are still counted when computing the minimum. A zone consisting only of tainted nodes can pin the global minimum at zero and block placement everywhere else. Setting Honor excludes domains the Pod cannot actually land in.

Missing labels. The scheduler considers only nodes that carry all referenced topology keys. A node with a mistyped label such as zone-typo is silently bypassed: no Pods land there, and Pods already on it are ignored in skew counts.

Conflicting constraints. Two constraints can have an empty intersection. The docs give an example where a hostname constraint and a zone constraint, together, allow no node; the Pod stays Pending until you raise maxSkew or switch one constraint to ScheduleAnyway.

Scale-down. Spread constraints apply only at placement time. When a Deployment scales down, the ReplicaSet controller chooses which Pods to delete without considering zone balance, so a 6-replica 2/2/2 layout can become 3/1/0 after scaling to four. Nothing in the scheduler corrects this. The descheduler’s RemovePodsViolatingTopologySpreadConstraint plugin exists for exactly this repair.

Selector mismatch. If the constraint’s labelSelector does not match the incoming Pod’s own labels, the Pod can be placed legally but never contributes to the count it is supposed to balance. Always select on labels that appear in the Pod template.

Spread versus anti-affinity in numbers

Consider 9 replicas across 3 zones. Required zone anti-affinity allows at most 3 Pods (one per zone) and leaves 6 Pending. A spread constraint with maxSkew: 1 places 3/3/3 and tolerates a zone outage with a predictable 33 percent capacity loss. When you lose a zone entirely, the surviving two zones can only absorb replicas up to their skew limit among eligible domains, so size your headroom for N-1 zones with that in mind. Spread gives you the distribution guarantee with no cap on replica count.

Scoring versus filtering as a design choice

Choose DoNotSchedule when a skewed layout is a correctness or availability violation (a quorum-based database, a zonal-redundancy SLO). Choose ScheduleAnyway when uneven placement is acceptable but undesirable and you would rather run than stay Pending (most stateless services). Many production teams combine both: a zone-level DoNotSchedule with a generous maxSkew, plus a hostname-level ScheduleAnyway with maxSkew: 1. Remember the one-constraint-per-key-and-effect rule: these two coexist because their topologyKey differs.

Priority, Preemption and Scheduler Profiles

When Filter yields no feasible node, PostFilter runs. In the default configuration that means preemption: the scheduler looks for a node where evicting lower-priority Pods would make room for the pending Pod.

How preemption works

Priority comes from a PriorityClass object. Per the upstream documentation, user-defined values range up to 1,000,000,000; larger numbers are reserved for system classes, where system-cluster-critical is 2,000,000,000 and system-node-critical is 2,000,001,000. The pending Pod’s priority also determines queue order, so high-priority Pods are tried first.

Preemption mechanics, in order:

The scheduler simulates removing lower-priority Pods from candidate nodes and checks whether the pending Pod would then fit.
It picks a node and a victim set, preferring victims that do not violate a PodDisruptionBudget. The docs are explicit: PDBs are supported but not guaranteed. If no victim set avoids a violation, the scheduler preempts anyway.
It sets status.nominatedNodeName on the preemptor. This is a hint, not a binding: victims get their full graceful-termination period, and if another node frees up first (or a higher-priority Pod arrives) the preemptor may land elsewhere and the nomination is cleared.
Once victims terminate, a normal scheduling cycle places the preemptor, trying the nominated node first.

Two consequences matter operationally. Preemption is not instantaneous: a victim with a 60-second terminationGracePeriodSeconds delays the preemptor by up to that long, which interacts with the lifecycle behaviour described in our guide to graceful shutdown, PodDisruptionBudgets and preStop hooks. And preemption considers the preemptor’s own hard constraints: the docs note that if a pending Pod has inter-Pod affinity to a lower-priority Pod on a node, removing that Pod would break the affinity, so that node is generally not a viable preemption target.

A PriorityClass with preemptionPolicy: Never lets a Pod jump ahead in the queue without evicting anyone, a good fit for batch jobs that should be scheduled first when capacity appears but never disrupt running services. The docs also warn that in clusters with untrusted users, high-priority Pods can be abused to evict others, and recommend using ResourceQuota to restrict who can create them.

The v1.37 docs additionally describe an alpha capability: preemption for in-place Pod resize, restricted to the Pod’s current node, where the scheduler frees capacity so the kubelet can actuate the resize. It is alpha, so do not build production policy on it.

Scheduler profiles and custom plugins

One kube-scheduler process can host several profiles, each with a schedulerName and its own plugin configuration. Pods opt in through spec.schedulerName; the default profile is default-scheduler. Profiles share the same queue and the same QueueSort plugin, so they differ in Filter, Score and later phases only.

A typical use is a second profile tuned for bin-packing. The default scoring favours spreading load, which is right for latency-sensitive services but wasteful for batch pools you want to consolidate so a node autoscaler can remove empty nodes. A NodeResourcesFit plugin configured with MostAllocated (or RequestedToCapacityRatio) scoring strategy produces that behaviour.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
percentageOfNodesToScore: 0
profiles:
  - schedulerName: default-scheduler
  - schedulerName: bin-pack-scheduler
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated
            resources:
              - name: cpu
                weight: 1
              - name: memory
                weight: 1
    plugins:
      score:
        disabled:
          - name: NodeResourcesBalancedAllocation

Custom plugins are compiled into a scheduler binary (the out-of-tree scheduler-plugins project is the common starting point) and registered by name in a profile. Because they run in the serial scheduling cycle, a slow Filter or Score plugin throttles cluster-wide throughput. Keep plugin work CPU-local, precompute shared state in PreFilter or PreScore, and avoid network calls inside the scheduling cycle. If a plugin implements Reserve or Permit it needs to be EnqueueExtension-aware too, or its rejected Pods will be requeued blindly.

Running a second scheduler binary is also possible, but two schedulers racing for the same nodes can double-book resources because neither sees the other’s assumed Pods until binding reaches the API server. Prefer profiles inside a single process.

The Descheduler and the Autoscaler Loop

The scheduler never moves a running Pod. The descheduler (a Kubernetes SIG Scheduling project) is the corrective loop: it finds Pods whose placement is now suboptimal and evicts them, trusting that the normal scheduler will place the replacements better. It is not a scheduler, and it never binds anything.

Figure 3: The descheduler loop. It evicts through the eviction API, so PodDisruptionBudgets apply; the controller recreates the Pod; the kubernetes scheduler places it again; an autoscaler reacts only if the new Pod is unschedulable.

What the descheduler does

According to the project README, you can run it as a Job, CronJob or Deployment, installed via Helm or Kustomize. The policy is a DeschedulerPolicy object, and the README shows apiVersion: descheduler/v1alpha2. At the time of writing the README references the v0.37.x release line and states that each descheduler release targets the matching Kubernetes minor version (v0.37 with v1.37) and is tested against the three latest minors. Always match your descheduler minor version to your cluster, and read the compatibility table before upgrading.

The plugins listed in the README are:

Plugin	What it repairs
RemoveDuplicates	Multiple Pods of one owner on the same node
LowNodeUtilization	Pods on overloaded nodes while others sit idle
HighNodeUtilization	Consolidates Pods off lightly used nodes so they can be removed
RemovePodsViolatingInterPodAntiAffinity	Anti-affinity broken after the fact
RemovePodsViolatingNodeAffinity	Node affinity no longer satisfied (the IgnoredDuringExecution gap)
RemovePodsViolatingNodeTaints	Pods that no longer tolerate a node’s taints
RemovePodsViolatingTopologySpreadConstraint	Skew above maxSkew after scale-down or node churn
RemovePodsHavingTooManyRestarts	Crash-looping Pods
PodLifeTime	Pods older than a configured age
RemoveFailedPods	Pods in failed states

A minimal policy that repairs zone skew while leaving everything else alone looks like this:

apiVersion: "descheduler/v1alpha2"
kind: "DeschedulerPolicy"
profiles:
  - name: spread-repair
    pluginConfig:
      - name: "DefaultEvictor"
        args:
          evictLocalStoragePods: false
          nodeFit: true
      - name: "RemovePodsViolatingTopologySpreadConstraint"
        args:
          constraints:
            - DoNotSchedule
    plugins:
      balance:
        enabled:
          - "RemovePodsViolatingTopologySpreadConstraint"

The nodeFit: true setting asks the evictor to check that an evicted Pod could plausibly fit elsewhere before evicting it. Verify argument names against the README for your release, because plugin arguments have evolved across versions.

Safety rails you should understand

The README states that Pods covered by a PodDisruptionBudget are not evicted if doing so would violate it, and that evictions go through the eviction subresource. By default, only Pods below the priority of system-cluster-critical are eligible, and you can set a threshold by class name or numeric value (not both). Critical system Pods are not evicted unless evictSystemCriticalPods is enabled, which you should leave off.

Pods without an owning controller are a different matter. A bare Pod that is evicted is simply gone, because nothing recreates it. The default evictor logic is conservative about this, but verify with a dry run (--dry-run or checking logs) before enabling any strategy in production, and annotate Pods that must never move.

The interplay with autoscalers

Consider what happens when HighNodeUtilization evicts Pods from a lightly loaded node. The replacements are scheduled by the default scheduler, which, under its default spreading-oriented scoring, might put them right back on the same lightly loaded node. If you want consolidation to actually work, you need the scheduler to bin-pack as well: the MostAllocated profile shown earlier, or a node autoscaler configured to consolidate.

Karpenter makes its own consolidation decisions: it simulates scheduling to decide whether nodes can be removed. Run Karpenter consolidation and a descheduler consolidation plugin against the same pool and the two controllers can fight, with one evicting Pods to empty a node and the other provisioning a node for the displaced Pods. Pick one owner for consolidation per node pool. Our Karpenter versus Cluster Autoscaler comparison goes into how each treats unschedulable Pods and disruption budgets.

For the Pending-driven direction, the contract is simpler. The scheduler marks a Pod unschedulable and writes an event and a PodScheduled=False condition with reason Unschedulable. Cluster Autoscaler and Karpenter watch for exactly that, simulate whether adding a node would let the Pod fit given its constraints, and provision accordingly. Consequently, every scheduling constraint is also a provisioning constraint: a topology spread with minDomains: 3 makes the autoscaler create nodes in a third zone, and a required anti-affinity makes it create one node per replica.

PDBs: three different uses of the same object

A PodDisruptionBudget limits voluntary disruption through the Eviction API. It affects three of the components above in different ways:

Descheduler: respected. A budget with minAvailable too close to the replica count simply makes the descheduler a no-op for that workload.
Node drain and autoscaler scale-down: respected, and the most common source of stuck drains.
Scheduler preemption: best effort only. A preemptor can violate a PDB if there is no alternative, as the upstream preemption page states.

So a PDB is a strong guard against the descheduler and drains but only a hint for preemption. If a workload must never be preempted, the lever is its priority class (high enough that others cannot evict it), not its PDB.

Scale, Tuning and Debugging Pending Pods

On a small cluster you will never notice scheduler cost. On a large one, three knobs matter.

percentageOfNodesToScore

To bound latency, the scheduler stops searching once it has found enough feasible nodes. percentageOfNodesToScore sets that threshold. Per the upstream tuning page: values from 0 to 100 are accepted, 0 means use the compiled-in default, and above 100 behaves as 100. If unset, Kubernetes uses a linear formula that yields 50 percent for a 100-node cluster and 10 percent for a 5,000-node cluster, with a lower bound of 5 percent. There is also a hardcoded minimum of 100 feasible nodes, and clusters with fewer than 100 feasible nodes check all of them.

The scheduler iterates nodes round-robin, continuing for each next Pod from where it stopped for the previous one. When nodes span zones, it alternates between zones: the docs’ example of six nodes in two zones (four and two) is evaluated in the order Node 1, Node 5, Node 2, Node 6, Node 3, Node 4.

What this means for you: in a large cluster the scheduler is not globally optimal. It picks the best of roughly the first N feasible nodes it meets. If your Score plugins matter (spread, bin-packing), a low percentage means they act on a sample. Setting the value to 100 gives full scoring at the cost of cycle time, and the docs advise against setting it very low.

For batch bursts, the v1.37 docs also describe Opportunistic Batching, a feature-gated mode (check its stage in your version) that reuses Filter and Score results between scheduling cycles for Pods with equivalent constraints, with a cache that expires after 0.5 seconds. It applies only to Pods without inter-Pod affinity, topology spread, or DRA claims, and requires disabling the default topology spread constraints. The takeaway is consistent with everything above: the more cross-Pod constraints you use, the less of the scheduler’s fast path you can use.

A debugging playbook for Pending Pods

Figure 4: Debugging a Pending Pod. Read the FailedScheduling event, map each reason to a plugin, then verify labels, taints, requests and spread arithmetic.

Start with the event, which condenses each plugin’s verdict:

kubectl describe pod web-7f9c5d6b8-abcde | sed -n '/Events:/,$p'
kubectl get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp
kubectl get pod web-7f9c5d6b8-abcde -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")]}'

A typical message looks like 0/12 nodes are available: 3 node(s) had untolerated taint, 4 Insufficient memory, 5 node(s) didn't match pod topology spread constraints. The counts sum to the node total, and each phrase maps to a plugin:

Insufficient cpu or memory (NodeResourcesFit): compare the Pod’s requests with kubectl describe node allocatable minus requested. Limits are irrelevant here, and so is actual usage.
didn't match Pod's node affinity/selector: check node labels with kubectl get nodes --show-labels.
had untolerated taint: list taints with kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints.
didn't match pod topology spread constraints: do the skew arithmetic by hand. Count matching Pods per domain with kubectl get pods -l app=web -o wide, map nodes to zones, compute the global minimum, and test count + 1 - min <= maxSkew for each domain. Check minDomains and the node inclusion policies.
didn't match pod affinity/anti-affinity rules: look for symmetric anti-affinity from existing Pods against yours.
didn't have free ports, volume node affinity conflict, unbound PersistentVolumeClaims: volume and port plugins; zonal PVs pin a Pod to the zone where the volume was created, which can contradict a spread constraint.

If there is no FailedScheduling event, suspect scheduling gates (kubectl get pod -o jsonpath='{.spec.schedulingGates}'), a different schedulerName with no scheduler running, or a scheduler that is down.

To see why the scheduler chose a node rather than why it rejected others, raise kube-scheduler verbosity (for example --v=10 temporarily on a non-production cluster) or inspect scheduler metrics such as pending-Pod counts per queue and scheduling attempt duration. The scheduler exposes Prometheus metrics, including queue-size gauges and per-extension-point durations; confirm exact metric names in the metrics reference for your version rather than relying on memory.

A final check that catches many “everything looks right” cases: kubectl get pod -o yaml and read status.nominatedNodeName. If it is set, the Pod is waiting on preemption victims to terminate, not stuck.

Trade-offs, Gotchas, and What Goes Wrong

Over-constraining. The most common failure is stacking hard rules: required node affinity, required anti-affinity, and DoNotSchedule spread. Each is individually reasonable, but their conjunction can match zero nodes, as the upstream conflicting-constraints example shows. Prefer one hard rule for the property you truly cannot compromise on and soft rules for the rest.

Spread is a placement-time promise. Skew guarantees hold at the moment of scheduling and erode through scale-down, node failure, spot reclamation and drains. If you advertise zonal redundancy as an SLO, you need the descheduler, a topology-aware autoscaler, or periodic verification.

Rolling updates and skew. Without matchLabelKeys: [pod-template-hash], old and new ReplicaSet Pods are counted together. During a rollout the new revision can legitimately end up lopsided because the old revision’s Pods mask the imbalance. With it, each revision is balanced alone. The docs caution against using matchLabelKeys with labels that might be edited directly on Pods, since the API server will not reflect such edits into the merged selector.

Affinity at scale. Heavy use of required inter-Pod anti-affinity is the classic reason a cluster that scheduled 50 Pods per second at 100 nodes crawls at 2,000 nodes. This is mechanism-level reasoning (each decision scans matching Pods across domains) rather than a benchmark; measure on your own cluster with the scheduler’s attempt-duration metrics before and after changes.

Preemption surprises. Preemption is best-effort on PDBs, delayed by termination grace, and can cascade: evicted lower-priority Pods return to the queue and may preempt still-lower ones. Assign a small number of well-named priority classes and avoid giving every team a high value.

Descheduler churn. An aggressive strategy plus a Pod that takes minutes to become ready causes capacity dips on every cycle. Evicting Pods whose replacements can only land in the same place (because of strong affinity, or because the autoscaler lags) just generates restarts. Use nodeFit, scope strategies with namespace and label selectors, and start in dry-run.

Stateful workloads. Zonal volumes pin StatefulSet Pods to a zone, so a spread constraint cannot move them. Spread zone-by-zone from the start, and expect to repair imbalance by recreating PVCs, not by evicting Pods.

Two controllers, one goal. Scheduler bin-packing, Karpenter consolidation, descheduler HighNodeUtilization and HPA scale-in can each be correct alone and oscillate together. Define a single owner per concern.

Practical Recommendations

Start from the failure you are preventing, then choose the weakest mechanism that prevents it. For distribution, use topology spread constraints. For isolation, use taints. For co-location, use preferred Pod affinity first. Reach for required Pod anti-affinity only for small, strict cases like a three-node quorum.

A defensible baseline for a stateless multi-zone service:

One zone-level constraint: maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: DoNotSchedule if zonal loss is an SLO violation, else ScheduleAnyway.
One hostname-level constraint with ScheduleAnyway so nodes are balanced without blocking.
matchLabelKeys: [pod-template-hash] to balance revisions independently.
minDomains equal to your target zone count, with the autoscaler allowed to create nodes in each zone.
A PodDisruptionBudget that leaves the descheduler and drains room to work.
A descheduler policy with RemovePodsViolatingTopologySpreadConstraint only, in a CronJob, dry-run first.

Checklist before shipping:

[ ] Every node has the topology labels your constraints reference (and no typos).
[ ] labelSelector matches the labels in the Pod template.
[ ] You have done the skew arithmetic for N-1 zones.
[ ] Priority classes are few, documented, and quota-limited.
[ ] Only one controller owns consolidation per node pool.
[ ] Scheduler attempt-duration and pending-queue metrics are on a dashboard.
[ ] You tested a node drain and a zone-loss drill, not just a clean rollout.

If you operate a very large cluster, test percentageOfNodesToScore and affinity cost with your real Pod mix, not a synthetic one. Unverified rules of thumb are no substitute for your own measurements.

Frequently Asked Questions

What does the kubernetes scheduler do?

The kube-scheduler watches for Pods without an assigned node and picks one in two steps. Filtering removes nodes that cannot run the Pod (insufficient resources, taints, affinity or spread violations). Scoring ranks the remaining nodes with weighted plugins, and the highest score wins, with ties broken randomly. It then writes a binding to the API server. It decides once per Pod and does not rebalance running Pods afterwards.

What is the difference between pod affinity and topology spread constraints?

Pod affinity and anti-affinity attract or repel a Pod relative to other Pods, using label selectors and a topology key. Topology spread constraints instead limit the numerical imbalance of matching Pods across domains through maxSkew. Spread is cheaper to evaluate at scale, tolerates any replica count, and is the better tool for distribution. Affinity is better for co-location, such as a cache beside its application. Upstream docs advise caution using inter-Pod affinity in clusters beyond several hundred nodes.

Why is my Pod Pending with “didn’t match pod topology spread constraints”?

Placing the Pod in every candidate domain would push its skew above maxSkew. Compute count plus one minus the global minimum for each domain. Common causes: minDomains exceeds the number of domains with nodes (global minimum treated as zero), a tainted or unreachable zone pinning the minimum, nodes lacking the topology label, or two constraints with no overlap. Raise maxSkew, switch to ScheduleAnyway, fix labels, or let the autoscaler add a node in the missing domain.

Does the descheduler replace the scheduler?

No. The descheduler never places Pods. It evicts those whose current placement violates a policy, such as broken spread, affinity or taints, or an underused node, and then relies on the default scheduler to place the replacements. It respects PodDisruptionBudgets because it evicts through the eviction subresource. Without a controller to recreate evicted Pods, eviction simply deletes them, so scope its strategies carefully and trial them in dry-run.

How does preemption interact with PodDisruptionBudgets?

Scheduler preemption treats PDBs as best effort. It tries to choose victims whose eviction does not violate a budget, but if no such set exists it evicts anyway. Victims also receive their full graceful-termination period before the preemptor can run, and the preemptor’s nominatedNodeName is only a hint. By contrast, the descheduler and node drains strictly honour PDBs. Protect critical Pods with priority, not with a PDB alone.

How many nodes does the scheduler score in a large cluster?

By default it uses an adaptive percentage: 50 percent at 100 nodes, falling linearly to 10 percent at 5,000 nodes, never below 5 percent, and never fewer than 100 feasible nodes. Set percentageOfNodesToScore in the scheduler configuration to override it; 0 means the default, and 100 scores every feasible node. Lower values cut scheduling latency but make plugins like spread and bin-packing act on a sample of nodes.