Karmada Graduates: A Multi-Cluster K8s ADR

On September 7-8, 2026, at KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China in Shanghai, the Cloud Native Computing Foundation announced the Karmada CNCF graduation — the project’s move to CNCF’s top maturity tier, alongside Kubernetes, Prometheus, and Envoy. Five years after its first commit, Karmada now counts 1,214+ contributors from 292 organizations and 5,600+ GitHub stars, with production deployments at Bloomberg, Alibaba Cloud, Bilibili, Huawei, and a dozen other named adopters. Graduation lands weeks after Karmada v1.19 shipped real engineering: priority-based scheduling turned on by default, a substantial cut in peak scheduler memory, and a fourth phase of multi-component workload placement built for distributed AI training. This is an architecture decision record, not a press release. It treats the graduation announcement as the trigger for the question platform teams actually need answered: does your organization need cross-cluster workload placement, cluster lifecycle management, or both — and if both, how do Karmada and Cluster API compose without one team’s automation fighting the other’s.

What this covers:

What actually changed in Karmada v1.19, with real before/after numbers
Karmada’s control-plane architecture and how PropagationPolicy and OverridePolicy drive placement
A decision matrix for choosing Cluster API, Karmada, both, or neither
Overflow cluster affinity scheduling as a pattern for regional-DC-plus-cloud-burst edge and IIoT fleets
How Karmada’s architecture compares to the deprecated kubefed model and to Rancher Fleet
Concrete recommendations and a pre-adoption checklist

Context and Background

Multi-cluster Kubernetes stopped being an edge case around 2023. Once a platform team runs more than one cluster — for blast-radius isolation, regulatory data residency, multi-region latency, or because one cluster ran out of headroom — two questions appear that a single kubeconfig cannot answer: how do new clusters get provisioned and upgraded, and how does a workload defined once end up running correctly across several of them. The Kubernetes ecosystem’s earliest attempt at the second question was Kubernetes Cluster Federation, known as kubefed, which tried to bolt a federation API onto existing cluster APIs. It never reached general availability, and development stalled well before Karmada’s own graduation. The lesson the ecosystem drew from that failure was specific: federation cannot be an afterthought layered onto cluster APIs designed for a single cluster. It needs a purpose-built control plane with its own resource model, its own scheduler, and its own opinions about propagation.

Karmada picked up that lesson directly. First committed in November 2020, the project entered CNCF Sandbox in September 2021 and reached Incubating status in December 2023. On September 7-8, 2026, the CNCF announced its graduation — see the official CNCF graduation announcement for the primary source and the full adopter list. That five-year arc mirrors how other graduated projects matured: sandbox status to prove the technical model, incubation to build a completed security review and a broad base of production adopters, graduation once governance, testing maturity, and real-world usage clear the CNCF’s bar.

Graduation by itself should not change your architecture. What should change it is whether your workloads genuinely need cross-cluster placement, which is a separable question from whether you need multi-cluster lifecycle management with Cluster API to provision the clusters in the first place. This ADR treats those as two independent decisions and gives you a matrix for both.

The adopter list behind the graduation announcement is worth reading closely, because it is unusually specific for a CNCF milestone post. Bloomberg, Wellhub, Alibaba Cloud, Bilibili, Huawei, iFLYTEK, JDCloud, Kuaishou, RedNote, SenseTime, Trip.com, Vivo, WPS, and ZTO are named as production users alongside DaoCloud, which has staffed a large share of the maintainer team. That is a mix of financial services, consumer internet, and enterprise software — not a single vertical betting on one pattern, which is part of what CNCF graduation is meant to signal about a project’s staying power. CNCF CTO Chris Aniszczyk tied the graduation directly to a present-day driver: GPU-constrained AI environments that need fleet-wide coordination across clusters rather than capacity siloed inside any single one. That framing matters for how you read the rest of this ADR — the engineering investment in v1.19 is not incidental polish, it is aimed squarely at multi-cluster AI training workloads, with edge and IIoT fleets picking up the same capability as a side effect rather than the primary target.

Reference Architecture: Two Control Planes, Two Jobs

Karmada and Cluster API solve different halves of multi-cluster Kubernetes. Cluster API provisions, upgrades, and deletes the clusters themselves through declarative Cluster and MachineDeployment resources. Karmada takes a set of already-running clusters and decides which workloads run where, propagating and overriding manifests through its own dedicated control plane. Neither replaces the other.

Figure 1: Karmada’s dedicated control plane — API server, controller manager, scheduler, and its own etcd — sits alongside host infrastructure and pushes workloads to member clusters through four internal controllers.

The diagram shows a client submitting a Deployment plus a PropagationPolicy to the Karmada API Server, which is backed by its own etcd instance rather than reusing any member cluster’s datastore. Four controllers read from that API server: a propagation controller that turns policies into ResourceBindings, a cluster lifecycle controller that tracks member cluster health and credentials, a policy binding controller that resolves which policy governs which resource, and an execution controller that pushes final manifests outward. The Karmada Scheduler consumes ResourceBindings and returns placement decisions before the execution controller fans work out to the member clusters at the bottom of the diagram.

Why Karmada runs its own control plane instead of extending one cluster

The design choice that separates Karmada from kubefed’s failed model is structural independence. Kubefed tried to make one cluster’s API server aware of federation, which meant every federation concept had to be squeezed into that cluster’s existing resource model and RBAC boundaries. Karmada instead stands up a parallel API server, its own controller manager, its own scheduler, and — critically — its own etcd instance. InfoQ’s coverage of the graduation describes this control-plane topology explicitly: a dedicated API Server, Controller Manager, Scheduler, and etcd sitting alongside or extending host clusters, rather than living inside one of them.

That independence has a direct operational consequence. A member cluster can be upgraded, rebooted, or lost entirely without taking down the placement decisions for every other cluster, because those decisions live in Karmada’s own etcd, not in any member’s. It also means Karmada’s blast radius for a control-plane failure is contained to placement and propagation — workloads already running in member clusters keep running under their own local kube-controller-manager and kubelets even if the Karmada control plane is temporarily unreachable. That is the same trade Kubernetes itself makes between the control plane and the kubelet’s local reconciliation loop, just one layer up the stack.

PropagationPolicy and OverridePolicy as the API surface

Karmada’s user-facing API surface is deliberately small: two custom resources do almost all of the work. A PropagationPolicy (or its cluster-scoped sibling, ClusterPropagationPolicy) selects which Kubernetes resources it governs and which member clusters are eligible targets, using label selectors, cluster affinity rules, and — as of v1.19 — a priority field. An OverridePolicy then patches the propagated manifest per target cluster: swapping an image registry for a region-local mirror, adjusting resource requests for a smaller edge cluster, or injecting cluster-specific environment variables. Between them, a single Deployment manifest authored once can land in three clusters with three different resource footprints without three copies of the YAML.

This is where the ResourceBinding and ClusterResourceBinding objects earn their keep. They are the intermediate representation between “here is a policy” and “here is a scheduling decision,” and as of v1.19 they carry a new spec.clusters[*].components field for workloads that are not a single Pod template but a graph of them — a distributed training job with a coordinator, parameter servers, and workers being the flagship example the InfoQ coverage highlights.

What v1.19 actually changed under the hood

Three changes in v1.19 matter more than the version bump suggests. Multi-Component Workload Scheduling reached “Phase IV,” adding the per-component placement field described above so that a single multi-pod-template workload can have its coordinator pinned to a low-latency cluster while its workers scatter across cheaper capacity. PriorityBasedScheduling graduated from alpha to Beta and is now enabled by default via spec.schedulePriority, meaning a critical workload can preempt a lower-priority one’s cluster placement without an operator writing a custom admission webhook to enforce it. Push-mode member clusters — the deployment model where Karmada reaches out to clusters it does not control the network path to — gained automatic bearer-token rotation, so long-lived informer watches survive a credential rotation instead of silently failing until someone notices stale data. Each of these closes a gap that previously needed a hand-rolled operator or a support ticket to work around; the deeper analysis section below quantifies the memory change alongside these behavioral ones. None of this replaces the scheduling that happens inside a single cluster — Karmada decides which cluster a workload lands in, and once it lands, that cluster’s own scheduler still handles bin-packing, gang scheduling, and queueing, which is the layer covered by our Kubernetes 1.37 gang scheduling comparison of Volcano and Kueue.

Decision Matrix: Cluster API, Karmada, or Both

The two most common mistakes in multi-cluster architecture are adopting Karmada to solve a provisioning problem, and adopting Cluster API to solve a placement problem. They fail differently: Karmada without a lifecycle tool leaves you hand-provisioning the clusters it schedules onto, while Cluster API without Karmada leaves every cross-cluster placement decision as a manual GitOps commit per cluster. Both assume you have already decided Kubernetes itself is the right orchestration layer for every cluster in the fleet; if some of those clusters are resource-constrained edge sites, it is worth checking that assumption against a Kubernetes vs Nomad edge decision matrix before building a Karmada topology on top of it.

Figure 2: A workload submission’s path through Karmada — from PropagationPolicy to ResourceBinding, through the scheduler’s priority and affinity rules, through OverridePolicy, to execution against member clusters.

Figure 2 traces the request path that Figure 1’s components implement. A user submits a Deployment with a PropagationPolicy attached. The API server creates a ResourceBinding, which the propagation controller hands to the scheduler. The scheduler applies priority and affinity rules — including, since v1.19, the schedulePriority field — and returns a cluster selection. That selection passes through the override controller, which applies any per-cluster OverridePolicy patches, and finally reaches the execution controller, which applies the resulting manifests to each target member cluster independently. Nothing in this path touches how those member clusters were created or how their control planes are patched; that is Cluster API’s job entirely.

Figure 3: A composition decision tree — start from whether you need cluster provisioning, workload placement, or both, and land on the right combination of Cluster API and Karmada.

The decision tree in Figure 3 is the practical output of this whole section. If your clusters are long-lived and manually curated but your workloads need to move between them — a common pattern for regulated industries running a fixed set of regional clusters — Karmada alone covers you. If your workloads are single-cluster but your cluster count and churn are high — think ephemeral preview environments or a managed-service provider spinning up a cluster per customer — Cluster API alone covers you. If both are true, which is increasingly the default for any organization running Kubernetes at fleet scale, they compose cleanly: Cluster API manages the Cluster and MachineDeployment objects that bring member clusters into existence, and once a cluster’s control plane is reachable, Karmada’s cluster lifecycle controller registers it as a member and starts scheduling onto it.

It is worth being precise about maturity claims here, because vendors on both sides get this wrong. Karmada holds a CNCF-graduated maturity level as of September 2026, backed by graduation’s completed security review and an OpenSSF Best Practices badge. Cluster API is a Kubernetes SIG subproject, not an independently CNCF-leveled project in the same sense, so the honest comparison is architectural, not a maturity-tier scoreboard: Cluster API answers “how do I provision and upgrade clusters,” Karmada answers “how do I place and propagate workloads once clusters exist,” and treating them as competing on the same axis misreads what each one is for.

Your need	Recommended stack	Why
Provisioning only — fixed workloads per cluster, no cross-cluster placement	Cluster API alone	Karmada adds a control plane and a policy model you would not use
Placement only — clusters are long-lived, manually curated, or provisioned by another team	Karmada alone	No lifecycle churn to automate; policies do the real work
Both — high cluster churn and workloads that move	Cluster API + Karmada	Cluster API creates clusters, Karmada’s lifecycle controller registers and schedules onto them
Neither — single cluster, single region	Neither	Multi-cluster tooling is pure overhead below this threshold

Failure modes and capacity planning

Karmada’s control plane is a single additional system to operate, which means it is also a single additional thing that can fail. A Karmada API server outage does not stop workloads already running in member clusters, but it does stop new placements, policy updates, and OverridePolicy patches from propagating until it recovers — plan for that window in your incident runbooks the same way you would plan for a CI/CD outage freezing deploys, not the same way you would plan for a production outage. The dedicated etcd instance is the component to size and back up most carefully, since ResourceBinding and PropagationPolicy state lives only there; losing it without a backup means re-deriving placement policy from source-controlled manifests, which is recoverable but slow under pressure.

Capacity planning gets more concrete with the v1.19 memory numbers. Distributing 20,000 Deployments across two member clusters previously peaked around 5 GB of Karmada control-plane memory; v1.19 brought that down to roughly 3.4 GB by reducing the managedFields metadata retained in dynamic informer caches. That is close to a one-third reduction, and it changes the sizing conversation for the Karmada control plane’s own host node from “reserve generously and hope” to “budget with headroom based on your actual object count.” Below 5,000 propagated objects the difference is largely academic; above 15,000 to 20,000, it is the difference between fitting comfortably on a mid-sized node and needing a dedicated larger one just for placement bookkeeping.

Cost planning should separate the Karmada control plane’s own footprint from the member clusters it schedules onto, because the two scale on completely different curves. The control plane’s memory and CPU track the number of propagated objects and the rate of policy churn, not the number or size of member clusters directly — a fleet of fifty small edge clusters running a modest, stable set of workloads can be cheaper to schedule for than five large clusters running thousands of frequently-updated Deployments. Budget the Karmada control-plane node independently of your member-cluster sizing, and re-benchmark whenever your propagated-object count crosses roughly an order of magnitude, since informer cache behavior does not scale linearly all the way up.

Trade-offs, Gotchas, and What Goes Wrong

The most common Karmada production mistake is treating PropagationPolicy selectors the way teams treat Kubernetes labels generally — loosely, with overlap tolerated. Two PropagationPolicies with overlapping selectors targeting the same resource produce ambiguous behavior that is genuinely confusing to debug at 2 a.m., because the symptom is “the workload landed somewhere I didn’t expect,” not an error message. Treat PropagationPolicy selectors with the same rigor as RBAC rules: audit for overlap before merging, not after an incident.

The second gotcha is OverridePolicy sprawl. It is tempting to use per-cluster overrides for everything that differs between environments, but every override is another place a manifest can diverge silently from what a kubectl diff against the source repository would show you. Reserve overrides for genuinely cluster-specific values — registry mirrors, node-pool-specific resource requests — and push everything else back into the base manifest or into per-environment PropagationPolicy targeting instead.

Compared with Rancher Fleet, the third option many teams evaluate alongside Karmada and Cluster API, the trade-off is topology and blast radius rather than feature parity. Fleet is fundamentally GitOps-first: it watches Git repositories and applies manifests to registered “downstream” clusters through a lightweight per-cluster agent, with no independent scheduler making cross-cluster placement decisions the way Karmada’s does. That makes Fleet simpler to reason about for teams already standardized on GitOps and wanting configuration fan-out rather than dynamic workload placement, but it also means Fleet has no equivalent to Karmada’s PriorityBasedScheduling or overflow affinity — placement is whatever your Git branch and target selectors say, computed once at commit time rather than continuously arbitrated against live cluster capacity. If your requirement is “deploy this config to these clusters,” Fleet’s simplicity wins. If your requirement is “place this workload on whichever cluster currently has capacity, and rebalance if that changes,” you need Karmada’s scheduler, not Fleet’s sync loop.

Push-mode clusters — the ones Karmada does not directly administer — deserve a specific warning even with v1.19’s bearer-token rotation fix. Rotation solves the credential-expiry failure mode, but push-mode clusters still depend on Karmada’s execution controller having outbound network reachability to each member’s API server; a firewall change or a NAT gateway swap on the member side breaks propagation with no signal from the Karmada side beyond an eventually-stale ResourceBinding status. Alert on ResourceBinding age, not just on Karmada component health.

Kubefed’s failure is worth restating precisely, because it explains a design choice that otherwise looks like unnecessary complexity: running an entire second control plane, including a second etcd, for something that “just” schedules workloads. Kubefed tried to add federation as an API extension on top of a designated host cluster, which meant federation state lived in that host cluster’s own etcd, competed for that cluster’s own API server capacity, and inherited that cluster’s own availability envelope. When the host cluster had a bad day, federation had a bad day with it, and there was no clean way to reason about federation’s blast radius separately from the host cluster’s. Karmada’s separate control plane looks heavier on a resource diagram, but it is the direct fix for that specific failure: placement logic gets its own availability envelope, its own upgrade cadence, and its own capacity planning, decoupled from any single member cluster’s fate. Teams evaluating Karmada’s control-plane overhead against Fleet’s lighter agent model should weigh it against this history rather than against an abstract preference for fewer moving parts.

Practical Recommendations

Start from workload behavior, not from tooling enthusiasm. If nobody on your platform team can describe a concrete scenario where a workload needs to move between clusters based on capacity, priority, or region, Karmada is solving a problem you do not have yet, and Cluster API alone — or no multi-cluster tooling at all — is the right call for now.

If you do adopt Karmada, adopt it on v1.19 or later specifically for the default-on PriorityBasedScheduling and the memory improvements; running an older Incubating-era release means manually enabling a feature gate that is now standard and living with the higher memory ceiling on the same object counts. Size the Karmada control plane’s node using the 20,000-object, 3.4 GB reference point as your floor, not your ceiling, and re-benchmark against your own object count rather than assuming linear scaling.

Treat PropagationPolicy and OverridePolicy the way you treat RBAC and NetworkPolicy: reviewed in pull requests, linted for selector overlap, and owned by a named team, not left to whoever wrote the first policy. Pair Karmada with Cluster API only when you have independently verified you need both provisioning automation and placement automation — bolting them together speculatively multiplies the surfaces you have to operate for no immediate benefit.

If you are migrating off an existing multi-cluster tool rather than adopting one for the first time, sequence the change carefully. Teams moving off a kubefed remnant should stand up Karmada’s control plane and register member clusters read-only first — letting the cluster lifecycle controller observe cluster health without any PropagationPolicy governing live workloads — before cutting any single Deployment over, so that a scheduler misconfiguration surfaces in a status field rather than in a production placement change. Teams moving from Fleet’s GitOps fan-out face the opposite risk: because Fleet computes placement once at commit time and Karmada computes it continuously against live capacity, the same Git commit can produce different placement results under each tool, so run them in parallel against a non-production workload class until the placement decisions converge before retiring Fleet’s sync loop for anything that matters.

Pre-adoption checklist:

[ ] Can you name a specific workload that needs to move between clusters, not just be deployed to several statically
[ ] Are you on Karmada v1.19 or later for default priority scheduling and the memory fix
[ ] Have you sized the Karmada control-plane node against your actual propagated-object count, not a rule of thumb
[ ] Do you have a PropagationPolicy review process that catches selector overlap before merge
[ ] Have you separately confirmed a need for cluster lifecycle automation before adding Cluster API alongside Karmada
[ ] Do your alerts cover ResourceBinding staleness, not just Karmada component uptime

Overflow Affinity and the Edge Fleet Case

Industrial and IoT fleets running regional data-center clusters with cloud burst capacity are a natural fit for a Karmada pattern introduced in the prior v1.18 release: Overflow Cluster Affinity Scheduling. It is background to v1.19 rather than new in this cycle, but it is the piece that makes the graduation-era feature set relevant to edge topologies specifically, so it earns its own diagram here.

Figure 4: Overflow cluster affinity scheduling fills a primary cluster group first and only bursts to a secondary group once primary capacity is exhausted, with the estimator feeding capacity state back to the scheduler.

The pattern fills a primary cluster group — typically an on-premises or regional data-center pool close to where IoT gateways and industrial controllers actually sit — before considering a secondary group at all. Only once the primary group’s capacity is exhausted does the scheduler burst new replicas to a secondary group, which is commonly a public cloud region kept for elasticity rather than steady-state load. For a fleet coordinating hundreds of edge sites feeding a central inference or aggregation layer, this maps directly onto the cost structure that matters: owned regional capacity is a sunk cost worth saturating, and cloud capacity is a marginal cost worth reserving for genuine peaks. An alpha-stage companion feature, Scheduling Overcommit Protection, adds an assume-and-deduct handshake between the scheduler and its capacity estimator so that a burst of scheduling requests cannot double-book the same headroom before the estimator catches up — it ships disabled by default, so treat it as a feature to pilot rather than to rely on in the first production rollout.

The CNCF’s own framing of the graduation ties directly back to this pattern. CNCF CTO Chris Aniszczyk connected Karmada’s graduation to the needs of GPU-constrained AI environments requiring fleet-wide coordination — the same underlying problem as an edge fleet coordinating scarce regional compute, just with GPUs standing in for industrial edge nodes. Multi-Component Workload Scheduling’s Phase IV, discussed earlier, is the AI-training-specific expression of the same idea: place the parts of a workload that need to be together on capacity that is actually together, and let the rest scatter to wherever capacity exists.

Frequently Asked Questions

Is Karmada a replacement for Cluster API?

No. Karmada schedules and propagates workloads across clusters that already exist; it has no concept of provisioning a cluster’s control plane or worker nodes. Cluster API owns that lifecycle through Cluster and MachineDeployment resources. Teams that need both typically run Cluster API to bring clusters into existence and Karmada’s cluster lifecycle controller to register those clusters as scheduling targets once their API servers are reachable. Treating them as alternatives usually means one team has not yet hit the problem the other tool solves.

What does CNCF graduation actually verify?

Graduation requires a completed third-party security review, an OpenSSF Best Practices badge, a documented and diverse set of committers across multiple organizations, and evidence of production adoption at scale — not just technical merit. For Karmada, the CNCF’s announcement cites 1,214+ contributors from 292 organizations and named production users including Bloomberg, Alibaba Cloud, and Huawei as part of that evidence. It is a governance and adoption bar, not a guarantee that the project fits your specific architecture.

Do I need Karmada if I only run two or three clusters?

Possibly not. Karmada’s value compounds with cluster count and workload churn — the more clusters and the more frequently workloads need to move between them based on capacity or priority, the more a dedicated scheduler pays for itself. At two or three long-lived clusters with stable workload-to-cluster assignments, a simpler GitOps fan-out tool, or even per-cluster manifests, may be lower operational overhead than standing up Karmada’s control plane.

How does PriorityBasedScheduling defaulting to on in v1.19 affect existing PropagationPolicies?

Existing policies without an explicit spec.schedulePriority value get a default priority, so the scheduler’s behavior does not change for workloads that never cared about relative priority. The practical effect appears once you start setting priorities on new or updated policies: those now genuinely compete for placement against unprioritized or lower-priority workloads. Audit your policies before upgrading if you have any priority-sensitive workloads that previously relied on manual scheduling ordering rather than an explicit field.

Is the 5 GB to 3.4 GB memory improvement guaranteed at my scale?

No — that figure is specific to the tested scenario of 20,000 Deployments across two member clusters, and it comes from reduced managedFields retention in dynamic informer caches, a change that scales with your object count and field-manager churn rather than a fixed percentage. Treat it as a directional benchmark for planning control-plane node sizing, and re-measure against your own object counts before finalizing capacity, especially if you run many field managers or heavy GitOps reconciliation that rewrites managedFields frequently.

How does Karmada compare to Rancher Fleet for a GitOps-first team?

Fleet is a lighter-weight, GitOps-native fan-out tool: it applies manifests from Git to registered clusters via a per-cluster agent with no independent placement scheduler. Karmada adds a real scheduler that makes live placement decisions based on priority, affinity, and capacity, at the cost of an additional control plane to operate. If your requirement is “sync this config to these named clusters,” Fleet’s simplicity is likely enough. If your requirement is “place this workload wherever capacity and priority say it should go, and rebalance as conditions change,” Fleet has no equivalent mechanism and Karmada is the appropriate tool.

Comments

Leave a Reply Cancel reply

Tag Cloud

Categories