Kubernetes Multi-Cluster Management With Cluster API: A 2026 Reference Architecture
One cluster is a computer. A hundred clusters is a distributed system whose nodes are themselves distributed systems — and that recursion is where most platform teams get hurt. Kubernetes multi-cluster management is the discipline of treating a whole fleet as one declarative surface: you describe the clusters you want, the add-ons they should carry, and the policies they must obey, and controllers make it true. Done badly, it becomes a spreadsheet of snowflake clusters and a pager that never sleeps. Done well, it is boring: a new region is a pull request, a Kubernetes upgrade is a rolling change with automated rollback, and blast radius stops at a cluster boundary instead of taking the company down.
This article is a working reference architecture, not a tour. It assumes you already run Kubernetes in production and are deciding how to run twenty, or two hundred, of them without linearly scaling your team.
What this covers: why fleets exist, the hub-and-spoke model, a Cluster API (CAPI) deep-dive, GitOps rollout with ApplicationSets and Flux, fleet policy, cross-cluster networking, identity, observability, and day-2 lifecycle to hundreds of clusters.
Context and Background
The single-cluster era had a comfortable lie built into it: scale up until you can’t, then figure out the rest later. Kubernetes itself set the ceiling. The project’s published scalability envelope is roughly 5,000 nodes, 150,000 pods, and 300,000 containers per cluster, with no more than 110 pods per node (Kubernetes scalability thresholds). Those are aspirational maxima under ideal conditions, not comfortable operating points. Well before them, etcd write latency, the API server’s watch cache, and controller-manager reconcile loops turn into the bottleneck.
But raw scale was never the main reason teams sharded. Blast radius was. A cluster is a single failure domain for its control plane, its CNI, its admission webhooks, and its cluster-wide CRDs. A bad CRD conversion webhook or a wedged etcd quorum takes down everything scheduled there. Splitting workloads across clusters converts a company-ending incident into a degraded-region incident.
Then come the non-negotiable drivers: data residency (EU data physically in the EU), regulatory isolation (a PCI cluster with its own admission and audit posture), tenancy (a noisy or hostile tenant that must never see another’s secrets or exhaust its nodes), and edge (hundreds of small clusters in stores, factories, or vehicles that are offline half the time). Our companion piece on Kubernetes multi-tenancy architecture covers the in-cluster alternative; this post is about when isolation must be a whole separate cluster.
The tooling arrived in waves. The first federation attempt, KubeFed (Federation v2), tried to teach one control plane to write objects into many and was archived after the model proved too rigid. What replaced it is not one tool but a layered stack: Cluster API for lifecycle, GitOps for configuration, a policy engine for guardrails, and a service mesh for connectivity. The rest of this article assembles those layers.
Reference Architecture
A production 2026 fleet is a hub-and-spoke system. One management cluster (the hub) runs the controllers that create, upgrade, and govern every workload cluster (the spokes); the spokes run only tenant workloads and the minimal agents needed to be governed. The hub is the single declarative surface; Git is its source of truth; spokes are cattle.
Figure 1: The management cluster consumes fleet config from Git, drives infrastructure providers to build regional and edge spokes, pushes policy from a central engine, and aggregates telemetry back to an observability hub.
Figure 1 shows the four responsibilities that live on the hub. Cluster API translates declarative Cluster objects into real infrastructure through cloud providers. A GitOps engine reconciles application and add-on manifests onto each spoke. A policy engine distributes admission rules fleet-wide. An observability layer pulls metrics, logs, and cost data back so the fleet is legible from one pane. Spokes hold workloads plus lightweight agents; if the hub disappears, spokes keep serving traffic — they simply stop changing.
The hub owns lifecycle, not runtime
The most important boundary in the whole design is that the hub is a control plane for clusters, not a data plane for workloads. No production traffic flows through it. This is what lets the hub be small, heavily guarded, and occasionally offline for maintenance without customer impact. A wedged hub means you cannot create or upgrade clusters for a while; it does not mean an outage. Treat the hub itself as a tier-0 service: back up its etcd, and ideally make it self-managed (a CAPI management cluster that can rebuild its own workload clusters from Git) so a hub loss is recoverable rather than catastrophic.
Spokes are declarative and disposable
A spoke should be reconstructible from its Cluster definition plus the Git repo that configures it. If reconstructing a cluster requires tribal knowledge or a runbook of manual kubectl steps, you have a snowflake, and snowflakes are what fleets exist to abolish. The test is blunt: could you delete a spoke and have it back, fully configured and serving, from Git alone? If yes, upgrades and disaster recovery become the same mechanism. If no, fix that before you scale the count.
Choose your fleet manager deliberately
Cluster API handles lifecycle, but “fleet management” — placement, propagation, and status aggregation across clusters — is a separate concern with real contenders. Karmada (a CNCF project) offers a Kubernetes-native API to schedule workloads across clusters with override policies. Open Cluster Management (OCM) uses a hub-agent pull model well suited to clusters behind firewalls. Rancher Fleet scales GitOps to large cluster counts. Cloud vendors ship their own: Azure Kubernetes Fleet Manager, GKE fleets (formerly Anthos, now under GKE Enterprise), and EKS’s cluster-management primitives. Pick based on your connectivity model (can the hub reach spokes, or must spokes call home?) and how much you want to own.
The connectivity model is the axis most teams underweight. A push hub, like Argo CD reaching out to registered clusters, needs network reachability and credentials to every spoke’s API server — clean in a single cloud VPC, awkward across accounts, and a non-starter for edge clusters behind NAT. A pull model, like OCM’s klusterlet agent or Flux running inside each spoke, inverts the trust: the spoke initiates an outbound connection home and reconciles locally, so the hub never needs an inbound route to the spoke. For fleets that mix cloud regions with hundreds of firewalled edge sites, a pull model is usually the only one that survives contact with real networks. You can also run both: push for cloud spokes you fully control, pull agents for everything at the edge.
Fleet observability and cost are hub concerns
A fleet you cannot see is a fleet you cannot operate. Each spoke has its own Prometheus-style metrics, its own logs, and its own API server events, and none of that is visible from the hub by default. The standard pattern is a two-tier telemetry pipeline: a lightweight agent on each spoke (a Prometheus agent, an OpenTelemetry collector, or a vendor equivalent) remote-writes metrics and forwards logs to a central store, tagged with a cluster label so every query can slice by spoke. Thanos or Mimir on the hub side gives a global query view without co-locating all storage. The non-obvious requirement is a fleet inventory: a live list of which clusters exist, their versions, their add-on sets, and their health, so drift and orphaned clusters surface instead of hiding. Cost follows the same shape — per-cluster and per-namespace attribution (via Kubecost, OpenCost, or cloud billing tags) rolled up to a fleet total, because the failure mode of fleets is not one expensive cluster but fifty slightly-over-provisioned ones nobody is watching.
Cluster API and Fleet Lifecycle
Cluster API is the SIG-Cluster-Lifecycle project that makes cluster creation a declarative Kubernetes operation. You install CAPI controllers on the management cluster, apply a Cluster custom resource, and controllers provision the infrastructure and bootstrap Kubernetes. The current stable contract is the v1beta1 API group. CAPI’s power is its provider model: the core controllers are cloud-agnostic and delegate the dirty work to pluggable providers.
Figure 2: A ClusterClass templates the topology; the Cluster object expresses desired state; controllers reconcile control plane and MachineDeployments down to individual Machines, each backed by an infrastructure VM and a bootstrap config, until nodes join the workload cluster.
Figure 2 traces the object hierarchy. There are three provider categories. Infrastructure providers create the actual compute — CAPA for AWS, CAPZ for Azure, CAPG for Google Cloud, CAPV for vSphere, and more. Bootstrap providers turn a machine into a Kubernetes node; the default, CABPK, generates cloud-init from kubeadm. Control-plane providers manage the control plane as a unit; KubeadmControlPlane (KCP) is the reference implementation, and managed variants defer to EKS, AKS, or GKE control planes.
The object model: Machine, MachineSet, MachineDeployment
CAPI deliberately mirrors the workload API you already know. A Machine is one node’s declarative record — the cluster equivalent of a Pod. A MachineSet guarantees N replicas of identical Machines, like a ReplicaSet. A MachineDeployment manages MachineSets to give you rolling, versioned updates of nodes, exactly as a Deployment does for Pods. Change the Kubernetes version or image in a MachineDeployment and CAPI rolls new Machines in and old ones out, respecting your surge and unavailability settings. The control plane has its own analog: KubeadmControlPlane rolls control-plane nodes one at a time, waiting for etcd health between each.
ClusterClass and managed topologies
Writing the full set of objects per cluster is how fleets rot. ClusterClass solves this: it is a reusable template that captures the shape of a cluster — control-plane class, worker classes, provider references, and variables. With managed topologies enabled, you then declare a Cluster that references a ClusterClass and sets a handful of values (Kubernetes version, replica counts, region), and CAPI generates and maintains all the underlying objects. This is the difference between templating a hundred clusters and hand-authoring a hundred clusters. A ClusterClass change can propagate a control-plane image bump to every cluster that references it, gated by your rollout controls.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: eu-west-prod-03
labels:
region: eu-west
tier: prod
spec:
topology:
class: standard-regional
version: v1.33.2
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: general-worker
name: md-0
replicas: 6
variables:
- name: instanceType
value: m6i.xlarge
The reconcile loop
Everything above is driven by the standard controller pattern: observe actual state, compare to desired state in the resource spec, act to close the gap, record the result in status, requeue. If an infrastructure VM dies, its Machine goes unhealthy; the MachineSet notices the replica shortfall and creates a replacement; the infra provider provisions a new VM; the bootstrap provider joins it. MachineHealthChecks formalize this by remediating nodes that fail health criteria for a configured duration. The loop never “finishes” — it continuously converges, which is precisely what makes a fleet self-healing rather than merely automated.
Day-2: upgrades and scaling to hundreds
Day-1 is provisioning; day-2 is where fleets are won or lost. A Kubernetes upgrade in CAPI is a declarative version bump — change spec.topology.version on the Cluster (or the ClusterClass) — after which KubeadmControlPlane rolls control-plane nodes one at a time, checking etcd and API-server health between each, and MachineDeployments perform a surge-and-drain rollout of workers. Crucially, this respects Kubernetes’ supported skew: control plane first, then nodes, never the reverse. The fleet discipline is to never bump every cluster at once. Ring the rollout: a canary cluster, then a small cohort, then the rest, each gate conditioned on real health signals and each reversible. Because the version lives in Git, a bad upgrade is a revert, and CAPI rolls back to the prior image the same way it rolled forward.
Scaling to hundreds of clusters stresses the hub, not the spokes. The management cluster’s own API server and etcd carry the CAPI object graph — potentially tens of objects per cluster — plus the GitOps controllers’ Application or Kustomization objects and the telemetry the fleet inventory needs. Past a few hundred clusters, the pragmatic move is to shard the hubs: one management cluster per region or per business unit, each a tier-0 service, with a thin aggregation layer for a global view. This keeps blast radius bounded on the control plane too — a wedged hub degrades one shard’s lifecycle operations, not the entire fleet’s.
Rolling configuration with GitOps
Lifecycle gets you empty clusters. Configuration fills them, and it must be pull-based to scale. Two engines dominate: Argo CD, whose ApplicationSet controller templates one Application per target cluster, and Flux, whose Kustomization and HelmRelease controllers reconcile from Git. The pattern in Figure 3 is fleet-wide fan-out: a commit updates the desired state, and every matching cluster converges to it on its own schedule.
Figure 3: A developer commits to the fleet repo; an ApplicationSet cluster generator produces one Application per registered spoke; each Application syncs to its target and reports health, giving a single rollout status across the fleet.
ApplicationSet’s cluster generator is the workhorse. It reads the clusters registered as Argo CD secrets and emits an Application per cluster, substituting per-cluster values (region, replica counts, endpoints) from cluster labels. Add a spoke, label it, and it inherits the whole add-on baseline with no new manifests. Flux achieves the same with per-cluster Kustomization overlays. Either way, “install the ingress controller and metrics stack on every prod cluster in the EU” becomes one declarative rule, not a hundred helm install commands. This is also the natural place to progressive-rollout: sync to a canary cluster, verify, then let the rest follow. Our Argo CD vs Flux decision record walks the trade-offs in depth.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: platform-baseline
spec:
generators:
- clusters:
selector:
matchLabels:
tier: prod
template:
metadata:
name: 'baseline-{{name}}'
spec:
project: platform
source:
repoURL: https://git.example.com/fleet/addons
targetRevision: main
path: 'baseline'
destination:
server: '{{server}}'
namespace: platform-system
syncPolicy:
automated:
prune: true
selfHeal: true
Policy and add-ons as fleet concerns
Add-ons (CNI, CSI, ingress, DNS, metrics) and policy travel with the same GitOps mechanism, but policy deserves its own engine. Kyverno and OPA Gatekeeper enforce admission rules — required labels, banned latest tags, forbidden host mounts, resource-limit floors — as CRDs synced to every cluster. The fleet win is uniformity: a single Kyverno policy bundle in Git guarantees identical guardrails everywhere, and a compliance auditor can read the enforced posture from one repo. Managers like OCM and Karmada add placement APIs on top, so “this policy applies only to PCI-labeled clusters” becomes declarative selection rather than manual application.
Trade-offs, Gotchas, and What Goes Wrong
Multi-cluster is not free isolation; it is a new distributed system with its own failure modes. The first is networking, which does not span clusters by default. Pods in cluster A cannot reach Services in cluster B without deliberate plumbing.
Figure 4: A global load balancer fronts two regional Cilium Cluster Meshes; each exports Services via the Multi-Cluster Services API so pods in one region can resolve and fail over to the other.
Three approaches compete. Cilium Cluster Mesh connects clusters at the CNI layer with global services and identity-aware policy. Submariner builds encrypted tunnels between cluster pod networks. The Multi-Cluster Services API (MCS, from SIG-Multicluster) standardizes ServiceExport and ServiceImport so a Service in one cluster becomes discoverable as svc.ns.svc.clusterset.local in others. The gotcha: cross-cluster calls cross real network boundaries with real latency and egress cost, and a naive mesh can turn a same-region hop into an inter-region round trip you pay for on every request.
Identity and secrets are the second trap. Each cluster is its own trust domain with its own service-account issuer and CA. There is no fleet-wide identity out of the box, which means a workload in cluster A has no inherent, verifiable identity that cluster B will accept. Teams close this three ways. SPIFFE/SPIRE issues short-lived, cryptographically verifiable workload identities (SVIDs) that a service mesh can trust across cluster boundaries, giving you mTLS between spokes without shared long-lived credentials. Alternatively, a central secrets manager — HashiCorp Vault or a cloud KMS — holds the source of truth, and each cluster authenticates to it using its own service-account tokens (Vault’s Kubernetes auth method validates the cluster’s OIDC issuer), so secrets are pulled at runtime, never stored in Git. The External Secrets Operator is the common glue: it runs in each spoke and syncs named secrets from the central store into native Kubernetes Secrets on demand.
The anti-pattern is copying a static secret into every cluster’s Git repo — even sealed or SOPS-encrypted — because rotation then means a fleet-wide commit and re-sync, and any single leaked decryption key compromises every cluster at once. Prefer per-cluster authentication to a central authority over per-cluster copies of a shared secret. The same logic applies to the credentials CAPI itself uses to talk to cloud providers: scope them per management cluster, rotate them, and never let a spoke hold the keys that could provision more infrastructure.
The third is version skew and drift. Kubernetes supports limited version skew between control plane and nodes (kubelets may run up to three minor versions behind the API server on recent releases). Across a fleet, an aborted or half-applied upgrade leaves clusters straddling versions, and a CRD that exists on some clusters but not others makes your “identical” GitOps manifests fail on a subset. Fleet observability that surfaces per-cluster version and add-on drift is not optional at scale — it is how you find the three clusters that silently diverged.
Practical Recommendations
Start with the hub and make it recoverable before you make it fancy. A self-managed CAPI management cluster whose own definition and every spoke live in Git means a hub loss is a rebuild, not a resume-writing event. Adopt ClusterClass from day one, even for two clusters, because retrofitting managed topologies onto hand-rolled clusters is painful. Keep the number of distinct cluster shapes small; variety is the cost driver, not count. A hundred identical clusters is cheaper to run than ten bespoke ones.
Push everything through GitOps, including the CAPI objects themselves, so cluster lifecycle and cluster configuration share one audit trail and one rollback mechanism. Treat cross-cluster networking as a deliberate, minimized surface — connect only the services that genuinely need it, and keep latency-sensitive calls same-cluster. Invest early in fleet observability and drift detection; you cannot manage what you cannot see across a hundred API servers.
Checklist:
- [ ] Management cluster is self-managed, etcd-backed-up, and rebuildable from Git.
- [ ] Every spoke reconstructs from its
Clusterobject plus its Git config — no snowflakes. - [ ] ClusterClass + managed topologies in use; distinct cluster shapes minimized.
- [ ] CAPI objects, add-ons, and policy all delivered via GitOps (ApplicationSet or Flux).
- [ ] Kyverno/Gatekeeper policy bundle enforced fleet-wide from one repo.
- [ ] Cross-cluster connectivity is explicit, encrypted, and latency-aware.
- [ ] Per-cluster version and add-on drift is monitored and alerted.
- [ ] Upgrades roll canary-first with automated health gating and rollback.
