Kafka KRaft in Production: Migrating Off ZooKeeper — A Hands-On Guide (2026)
If you are still running Apache Kafka on ZooKeeper in 2026, you are on a dead-end branch, and this guide walks you off it. A kafka kraft migration is no longer a modernization project you can defer to next quarter — ZooKeeper mode was fully removed in Kafka 4.0, released on 18 March 2025, so every cluster on 3.x must move to KRaft before it can ever run a 4.x binary. There is no in-place zookeeper.connect fallback left in the code. The only supported path is the online, dual-write migration introduced by KIP-866, executed on a bridge release, followed by a version upgrade.
This is a hands-on tutorial, not a marketing overview. You will get the real server.properties and controller.properties blocks, the exact kafka-storage.sh format invocations, the migration command sequence, controller quorum sizing math, the metrics that tell you the switch is safe, and the rollback rules that quietly stop applying at a specific point.
What this covers: KRaft architecture and the __cluster_metadata quorum, the KIP-866 migration procedure step by step, controller sizing, monitoring metadata lag, rollback constraints, and the gotchas that break production migrations.
Context and Background
For roughly a decade, every Kafka cluster shipped with a second distributed system bolted to its side: an Apache ZooKeeper ensemble that held cluster metadata — topic definitions, partition assignments, in-sync replica sets, ACLs, and configuration. Kafka brokers watched ZooKeeper znodes; a single elected controller broker translated ZooKeeper state into cluster actions. It worked, but it meant operators had to run, secure, patch, and reason about two consensus systems with different failure modes, and metadata operations were bounded by ZooKeeper’s write throughput and the O(partitions) controller failover.
KRaft (Kafka Raft) is the answer, defined in KIP-500 and hardened across the 3.x line. It moves metadata into an internal Kafka topic, __cluster_metadata, managed by a Raft-based controller quorum inside Kafka itself. No external ZooKeeper. Metadata becomes an event log the controllers replicate and brokers replay, which collapses controller failover from minutes to sub-second on large clusters and removes an entire operational dependency. The Confluent and Apache release notes confirm KRaft became the default in 4.0 and ZooKeeper support was deleted, not merely deprecated — see the Apache Kafka 4.0 upgrade notes. If you want the streaming-engine context around Kafka, our comparison of Flink vs Spark Streaming vs Kafka Streams frames where the broker sits in a modern pipeline. As of mid-2026 the current line is Kafka 4.x, with 4.2 having launched 17 February 2026 and later bugfix releases in the 4.3.x range.
KRaft Architecture: The Metadata Quorum
In KRaft, cluster metadata lives in a single-partition internal topic named __cluster_metadata, and a dedicated KRaft metadata quorum of controller nodes runs a Raft protocol over it. One controller is the active leader; the others are hot standbys replicating the log. Brokers are pure metadata followers: they fetch records from the quorum and apply them locally. There is no ZooKeeper, no controller election through an external system, and no O(partitions) metadata reload on failover — a standby simply already has the log.
Figure 1: The KRaft metadata quorum. The active controller appends records to __cluster_metadata, standby controllers replicate the log, and brokers receive incremental deltas.
Figure 1 shows the data flow: the active controller is the Raft leader; every metadata change (create topic, shrink ISR, alter config) becomes an ordered record appended to the log, replicated to standby controllers for durability, periodically compacted into a snapshot, and pushed to brokers as deltas rather than full reloads.
The metadata log and snapshots
__cluster_metadata is an append-only, single-partition log with a monotonic offset. Each record describes a discrete change — TopicRecord, PartitionChangeRecord, ConfigRecord, AccessControlEntryRecord, and so on. Because a log that only grows would be unbounded, KRaft periodically writes a snapshot: a compacted materialization of all state up to a given offset. On restart a controller or broker loads the latest snapshot, then replays only the records after it. This is why KRaft failover is fast — the new active controller does not rebuild state from thousands of ZooKeeper znodes; it already holds the log and snapshot in memory.
The Raft implementation here is Kafka’s own (KIP-595), not an embedded etcd or a ZooKeeper replacement library. Quorum membership is defined by controller.quorum.voters (static) or, on newer releases, managed dynamically via KIP-853. A write commits once a majority of voters have durably persisted it, so a 3-node quorum tolerates 1 failure and a 5-node quorum tolerates 2.
It helps to understand why snapshots matter for latency and not just disk. Under ZooKeeper, a newly elected controller had to enumerate every topic and partition znode and rebuild its in-memory view — an O(partitions) scan that, on a cluster with a million partitions, could take minutes during which no metadata operations could proceed. KRaft replaces that scan with a bounded operation: load the most recent snapshot (a single compacted file) into memory, then replay the small tail of records that arrived after the snapshot offset. The work is proportional to the log tail, not the whole cluster. Snapshotting is triggered by configurable thresholds — metadata.log.max.record.bytes.between.snapshots and a time bound — so the tail never grows unbounded. This is the concrete mechanism behind the “seconds not minutes” failover claim, and it is worth internalizing because it changes how you reason about controller capacity: controllers are memory- and disk-fsync-bound, not CPU-bound, and the metadata log’s fsync latency is the real ceiling on metadata throughput.
Active versus standby controllers
Only one controller is active at a time. It holds the Raft leadership lease and is the sole writer to the metadata log. Standby controllers replicate continuously and stand ready to win the next election if the leader’s lease lapses. Because standbys already have the full, up-to-date log in memory, promotion is near-instant — there is no cold load. This is the structural reason KRaft advertises controller failover measured in tens of milliseconds even on clusters with millions of partitions, versus the multi-minute reloads that plagued large ZooKeeper deployments.
Node roles: controller, broker, or both
Every KRaft node declares a process.roles value. Three shapes exist. process.roles=controller makes a node a pure metadata node that never serves client produce/fetch traffic. process.roles=broker makes it a pure data node. process.roles=broker,controller — combined mode — makes one JVM do both. Combined mode minimizes machine count for dev and small clusters; isolated (dedicated-controller) mode separates the two for blast-radius isolation and is the recommended production shape. Critically for migration, the KIP-866 path only targets isolated controllers — you cannot migrate directly into combined mode.
Migrating From ZooKeeper to KRaft: The Hands-On Path
The migration is online and reversible up to a point. In one sentence: you stand up a dedicated KRaft controller quorum formatted with your existing cluster ID, tell it to import ZooKeeper’s metadata, let it run in dual-write mode, roll your brokers into migration mode, then finalize by removing the migration flags. Availability is preserved throughout; clients keep producing and consuming. The mechanism is defined in KIP-866.
Figure 2: The five phases of a ZooKeeper-to-KRaft migration — provision controllers, enable migration, dual-write, migrate brokers, finalize.
Figure 2 lays out the phases. You never take a full outage; each phase is a rolling change, and the cluster runs on two metadata stores simultaneously during the dual-write window.
Prerequisite: get onto a bridge release
You cannot migrate from an arbitrary old version, and you cannot skip straight to 4.0. You must first be running a bridge release — a 3.x version that understands both ZooKeeper and KRaft and ships the migration tooling. Kafka 3.9 is the recommended bridge to 4.0. So the real sequence for a cluster on, say, 3.5 is: upgrade in-place to 3.9 (still on ZooKeeper), perform the KRaft migration on 3.9, then upgrade the now-KRaft cluster to 4.x. Attempting to run a 4.x binary against a ZooKeeper cluster simply fails — the code to talk to ZooKeeper is gone.
Step 1 — Format the KRaft controllers with your existing cluster ID
First, read your current cluster ID out of ZooKeeper so the new controllers adopt the same identity. Then provision an odd number of dedicated controller nodes (3 is the common choice). Each gets a controller.properties:
# controller.properties — dedicated KRaft controller (node 3000)
process.roles=controller
node.id=3000
controller.quorum.voters=3000@ctrl-1:9093,3001@ctrl-2:9093,3002@ctrl-3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
# Enable the migration bridge on the controller
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
# Match your existing broker security if in use
# controller.quorum.election.timeout.ms=1000
inter.broker.protocol.version=3.9
Format each controller’s storage using the existing cluster ID (do not generate a fresh one — that would create a new, disconnected cluster):
# Read the current cluster ID from ZooKeeper (run once)
CLUSTER_ID=$(bin/zookeeper-shell.sh zk-1:2181 get /cluster/id \
| grep -o '"id":"[^"]*"' | cut -d'"' -f4)
echo "Existing cluster id: $CLUSTER_ID"
# Format each controller with that same cluster ID
bin/kafka-storage.sh format \
--cluster-id "$CLUSTER_ID" \
--config /etc/kafka/controller.properties \
--ignore-formatted
Start the three controllers. At this point they form a Raft quorum but do not yet own the cluster — ZooKeeper is still authoritative and the brokers are untouched.
Step 2 — The controllers import ZooKeeper metadata (dual-write begins)
Because zookeeper.metadata.migration.enable=true and the controllers can reach ZooKeeper, the active controller reads the entire existing metadata set — topics, partitions, ISR, ACLs, dynamic configs — and materializes it into the __cluster_metadata log. From this moment the controller runs in dual-write mode: every new metadata change is written both to the KRaft log and back to ZooKeeper. The two stores stay in sync, which is exactly what makes rollback possible.
Figure 3: Dual-write migration sequence. The controller loads ZooKeeper metadata, brokers restart into migration mode, and the controller write-behind-syncs to ZooKeeper until the operator removes the migration flag.
Figure 3 shows the ordering. The write-behind sync to ZooKeeper is intentionally not in the blocking commit path of KRaft — KRaft commits at quorum speed and lazily catches ZooKeeper up, flushing any pending writes on a clean shutdown. That keeps latency low while preserving the ZooKeeper copy for rollback.
Step 3 — Roll the brokers into migration mode
Now restart each broker, one at a time, adding the KRaft controller connection and the migration flag to its server.properties. The brokers keep their existing broker.id as node.id and keep serving traffic:
# server.properties — existing broker, now in migration mode
broker.id=1
node.id=1
process.roles=broker # optional on the bridge release; brokers stay data-only
# Point brokers at the new KRaft controller quorum
controller.quorum.voters=3000@ctrl-1:9093,3001@ctrl-2:9093,3002@ctrl-3:9093
controller.listener.names=CONTROLLER
# Turn on migration on the broker side
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
inter.broker.listener.name=PLAINTEXT
listeners=PLAINTEXT://:9092
inter.broker.protocol.version=3.9
Do them one broker at a time, and after each restart confirm partitions are back in-sync before moving to the next. When every broker has restarted into migration mode, the cluster is fully in the hybrid/dual-write state: KRaft is the controller, brokers talk to it, and ZooKeeper is still being kept current as a safety net.
The reason to insist on one-broker-at-a-time here is not superstition. Each restart moves partition leadership off the restarting broker to its followers and then back once it rejoins the ISR. If you restart two brokers that happen to be leader and follower for the same partitions simultaneously, you can transiently drop below min.insync.replicas for those partitions, which will make acks=all producers block or error. Rolling one at a time, with an ISR check between each, guarantees you never remove two replicas of the same partition at once. A useful gate between restarts is to poll under-replicated partitions and refuse to proceed while any are non-zero:
# Between broker restarts: proceed only when 0 under-replicated partitions
bin/kafka-topics.sh --bootstrap-server broker-1:9092 \
--describe --under-replicated-partitions
# empty output == safe to restart the next broker
Watch for one important behavioral detail during this window: the brokers are now getting their metadata from the KRaft quorum via the controller listener, but ZooKeeper is still being written to behind the scenes. If a broker cannot reach the controller quorum on CONTROLLER://:9093 — a firewall rule, a wrong controller.quorum.voters entry, a missing controller.listener.names — it will fail to fetch metadata and effectively fall out of the cluster even though ZooKeeper looks fine. The most common migration incident is not data loss; it is a network/listener misconfiguration that only surfaces the moment brokers switch their metadata source to the quorum.
Step 4 — Finalize: remove ZooKeeper
Finalizing is the point of no return. You take the migration flags off the brokers first, then off the controllers, restarting each with a clean KRaft-only config. Once a broker restarts without zookeeper.metadata.migration.enable, it is a pure KRaft node:
# server.properties — finalized KRaft broker
node.id=1
process.roles=broker
controller.quorum.voters=3000@ctrl-1:9093,3001@ctrl-2:9093,3002@ctrl-3:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listeners=PLAINTEXT://:9092
# zookeeper.* lines are GONE
Then restart the controllers without zookeeper.metadata.migration.enable and without zookeeper.connect. The controller stops writing to ZooKeeper entirely. At this point you can decommission the ZooKeeper ensemble, and only now are you eligible to upgrade the cluster to Kafka 4.x. The Confluent migration runbook documents the equivalent steps for their distribution.
Order matters in finalization, and getting it wrong is the one mistake with no undo. Finalize the brokers first — restart them KRaft-only so none of them still expect a ZooKeeper connection — and only then finalize the controllers. If you flip the controllers to KRaft-only while brokers are still in migration mode expecting dual-write, the brokers can lose their metadata source mid-flight. Once the controllers are finalized, verify with kafka-metadata-quorum.sh describe --status that the migration state reports as complete and that there is exactly one active controller and the expected number of voters, before you touch ZooKeeper. Keep the ZooKeeper ensemble running (but now idle) for a few days after finalization as a psychological and forensic safety net; decommission it only once you have run a full traffic cycle and, ideally, at least one deliberate controller failover drill on pure KRaft. Nothing forces you to delete ZooKeeper the same day, and there is no downside to letting it sit idle briefly while you build confidence.
Controller Sizing, Combined vs Isolated, and Monitoring
Figure 4: Combined mode co-locates controller and broker in one JVM for low overhead; isolated mode runs dedicated controllers for production blast-radius isolation. Migration targets isolated only.
Figure 4 contrasts the two deployment shapes. The sizing and observability decisions below are what separate a migration that survives its first incident from one that does not.
How many controllers?
Controller count is a Raft quorum question, not a throughput one. A quorum of 2f+1 voters tolerates f failures. Use 3 controllers for the vast majority of production clusters — it tolerates one controller loss and one node down for patching simultaneously would leave you at risk, so schedule maintenance carefully. Use 5 controllers only for very large or extremely availability-sensitive fleets where tolerating two simultaneous controller failures is worth the extra Raft replication cost. Even-numbered quorums are pointless: 4 voters tolerate the same 1 failure as 3 but need a larger majority. Controllers are metadata-only and modest on CPU, but give them fast, dedicated disks — the metadata log fsync latency directly bounds how fast metadata operations commit.
Combined versus isolated in practice
Combined mode (process.roles=broker,controller) is legitimate for development, CI, and small single-digit-node clusters where running three extra machines is disproportionate. Its downside is coupling: a controller GC pause or a broker OOM now affects both roles in the same JVM, and a bad broker deploy can jeopardize the metadata quorum. Isolated mode costs three more small nodes but means a broker incident cannot take down the controllers and vice versa. Production should default to isolated — and since KIP-866 migration only lands you on isolated controllers anyway, migrating clusters get the right shape for free.
There is a subtler operational reason to prefer isolated controllers at scale: resource contention on the metadata path. A broker under heavy produce load is contending for page cache, disk bandwidth, and network buffers. If that same node is also the active controller, a burst of client traffic can slow the metadata log fsync, which delays every metadata commit cluster-wide — including the ISR shrink/expand records that keep replication healthy. In isolated mode the controllers are quiet, predictable machines whose only job is to commit metadata fast. For clusters past roughly ten brokers, or any cluster with a high rate of topic/partition churn (a common pattern in multi-tenant platforms that create and delete topics programmatically), isolated controllers are effectively mandatory. Combined mode also complicates rolling upgrades: you cannot restart a node to patch the broker without also bouncing a member of the controller quorum, which tightens your maintenance choreography.
Right-size the controller machines conservatively. A dedicated controller for a large cluster rarely needs more than a few CPU cores and 8–16 GB of heap, because its working set is the metadata log and snapshot, not message data. Where you must not economize is disk: put the metadata log on fast local NVMe with low fsync latency, isolated from any other I/O. A p99 fsync of a few milliseconds is fine; a p99 of tens of milliseconds on a slow network volume will show up as sluggish topic operations and delayed leader elections across the whole cluster.
Monitoring metadata lag
The single most important migration and steady-state metric is metadata lag — how far behind the metadata log a broker or standby controller is. Watch these JMX metrics:
kafka.controller:type=KafkaController,name=MetadataLastAppliedRecordOffsetand...LastCommittedRecordOffset— the gap between them shows apply lag.kafka.server:type=broker-metadata-metrics,name=last-applied-record-lag-ms— how stale a broker’s view of metadata is, in milliseconds. This should sit near zero; a growing value means a broker is falling behind the quorum.kafka.controller:type=KafkaController,name=ActiveControllerCount— must be exactly 1 across the quorum. 0 means no leader (outage); 2 would indicate split brain.- During migration specifically,
ZkMigrationState/ZkWriteBehindLagsurfaces how far the write-behind ZooKeeper sync trails KRaft. A large, growing write-behind lag means rollback would lose recent changes — do not finalize until it is small and stable.
A quick verification loop during the roll:
# Confirm exactly one active controller and inspect the quorum
bin/kafka-metadata-quorum.sh \
--bootstrap-controller ctrl-1:9093 describe --status
# Watch broker metadata lag via JMX (jmxterm or your metrics pipeline)
# last-applied-record-lag-ms should stay near 0 on every broker
For the broader storage picture once you are on KRaft, our write-up on Kafka tiered storage and KIP-405 covers how to keep broker disks small after the migration.
Trade-offs, Gotchas, and What Goes Wrong
Rollback has a hard deadline. You can roll back to ZooKeeper only while dual-write is active — that is, before you finalize. The instant you remove the migration flags and restart controllers in KRaft-only mode, ZooKeeper stops receiving writes and diverges; there is no supported path back. Treat finalization as a one-way door and only walk through it after a soak period where metadata lag and write-behind lag are both flat and near zero.
You cannot migrate into combined mode. KIP-866 only supports dedicated (isolated) KRaft controllers as the migration target. If your plan was “just add controller to process.roles on the existing brokers,” it will not work — you must stand up separate controller nodes.
Skipping the bridge release fails silently in the worst way. A 4.x binary has no ZooKeeper code. If you upgrade the JAR before migrating, the broker will not start against a ZooKeeper cluster. Always migrate on a bridge release (3.9) first, verify KRaft-only operation, then upgrade to 4.x.
Cluster ID mismatch creates a phantom cluster. Formatting controllers with kafka-storage.sh random-uuid instead of the existing ZooKeeper cluster ID produces a brand-new empty cluster that will not adopt your topics. Always read and reuse the existing ID.
Node ID collisions and listener misconfig. Controllers and brokers share one node-ID namespace in KRaft. Give controllers high IDs (3000+) so they never collide with broker IDs. And controller.listener.names plus a matching listeners entry must be present on both sides — a missing controller listener is the most common “brokers cannot reach the quorum” failure.
Under-provisioned controller disks. The metadata log fsync is in the commit path. Slow controller disks turn every topic create/alter and every ISR change into a latency spike. Give controllers fast local NVMe, separate from broker data disks.
Security config must match on the controller listener. If your cluster uses TLS or SASL between brokers, the controller listener also needs its security protocol, keystores, and SASL mechanism configured consistently. A frequent failure is standing up controllers with PLAINTEXT while the rest of the cluster is on SASL_SSL; the brokers cannot authenticate to the quorum and t
