Kafka KRaft Migration: Moving Off ZooKeeper in Production (2026)

Last Updated: November 2026

If your Kafka brokers still read their cluster state from ZooKeeper, you are running a mode the project no longer ships. A kafka kraft migration is the one-way, online procedure that moves topic definitions, partition assignments, ACLs and configuration from a ZooKeeper ensemble into a Raft quorum that lives inside Kafka itself. Apache Kafka 4.0 removed ZooKeeper mode entirely, so a 3.x cluster that never migrates can never run a 4.x binary, and the upgrade notes are blunt about it: ZooKeeper-mode clusters must migrate to KRaft before upgrading to 4.0.

That makes this a deadline-shaped problem, not a modernization wish. The good news is that the procedure, defined in KIP-866, is designed to be boring: brokers restart one at a time, clients keep producing, and you can retreat right up until the final step. The bad news is that the final step is irreversible, and several preconditions (version, log directories, ACL authorizer, node IDs) fail late and confusingly if you skip them.

You will leave with the architecture model, the five migration phases, the exact configuration changes per phase, controller sizing logic, the signals that say “safe to finalize”, the rollback rules by phase, and the failure modes that bite real clusters.

What this covers: how KRaft stores metadata, the KIP-866 dual-write design, the step-by-step runbook, quorum sizing, monitoring, rollback boundaries, post-migration cleanup, and an honest list of gotchas.

Context and Background

For about a decade every production Kafka cluster came with a second distributed system attached. An Apache ZooKeeper ensemble held the cluster’s metadata: topics, partitions, in-sync replica sets, ACLs, quotas and dynamic configuration. One broker won an election to become the controller, read that state, and pushed changes to the other brokers over RPC. The design worked, but it meant two consensus systems with different security models, different failure modes, different upgrade cadences and different on-call runbooks. It also capped scale: controller failover had to reload state proportional to the number of partitions, which on large clusters turned a controller crash into a long metadata brownout.

KRaft, short for Kafka Raft, was specified in KIP-500 and removes that dependency. Metadata becomes an event log, a single-partition internal topic named __cluster_metadata, replicated by a dedicated set of controller nodes using a Raft variant. Brokers consume that log like any other follower and rebuild their in-memory image from it. Because standby controllers already hold the log, failover no longer means rereading the world. The Apache upgrade notes say KRaft was considered production-ready from 3.3, which is also the floor for upgrading existing KRaft clusters straight to 4.0 (clusters older than that should first go to 3.9.x).

Timeline matters for planning. Kafka 4.0 shipped as KRaft-only, removing ZooKeeper mode rather than just deprecating it, along with a long list of removed APIs, Java 17 for brokers, and Log4j2. Kafka 4.2.0 followed on 17 February 2026 with production-ready share groups (KIP-932 queues) and new controller observability metrics, and the project’s documentation site now points at a later 4.x line still. None of those later releases bring ZooKeeper back, so the migration path is exactly one thing: migrate on the 3.x bridge release, then upgrade.

If you run Kafka as part of an event-driven backbone, this migration touches adjacent decisions. Our guide to Kafka exactly-once semantics, transactions and the idempotent producer covers client behavior that changes at the 4.0 boundary, and the Kafka queues versus RabbitMQ and SQS comparison explains why share groups make 4.2 interesting once you are through. The authoritative source for everything below is the Apache Kafka 4.0 upgrade notes and the KIP-866 design page.

What changed since the first version of this guide

An earlier version of this article described the same overall procedure but treated version numbers loosely. Three corrections are worth stating up front. First, the supported bridge is the 3.9 line, and the current 3.9 documentation tells you to upgrade brokers to 3.9.1 and set inter.broker.protocol.version to 3.9 before you start. Second, the controller bootstrap setting in current docs is controller.quorum.bootstrap.servers, with the older static controller.quorum.voters still existing for static quorums. Third, the 3.9 documentation lists newer known issues (a controller-epoch edge case on revert, a malformed /migration node left by the ZooKeeper security migration tool, and an AlterConfigPolicy inconsistency addressed in 3.9.2), so read the live 3.9 page the day you plan the change rather than relying on any blog, including this one.

How KRaft Replaces ZooKeeper: The Reference Architecture

In KRaft mode a small quorum of controller nodes runs Raft over the __cluster_metadata log. One controller is the active leader and appends every metadata change as a record. The others replicate it and stay hot. Brokers fetch the log, apply records to a local metadata image, and forward admin requests such as create-topic to the active controller. There is no ZooKeeper, no external election, and no full metadata reload on failover.

Figure 1: The KRaft target state. Controllers replicate __cluster_metadata; brokers are metadata followers that forward admin requests to the active controller.

The diagram shows the end state you are migrating toward. Three controllers sit in a quorum, one active and two standby. Every broker consumes the same log, so any broker can serve metadata reads from its local image, and any admin call lands on the active controller through forwarding. Think of the metadata log as the cluster’s write-ahead log, and the controller quorum as its sole writer.

Metadata as a log, not a tree

ZooKeeper stored metadata as a tree of znodes with watches. KRaft stores it as an ordered sequence of records, with periodic snapshots so the log can be truncated. This changes failure semantics in useful ways. A broker that falls behind simply fetches from its last applied offset. A new controller that joins replays a snapshot plus the tail. Ordering is total and explicit, and every consumer of the log agrees on what happened and in what order, which is exactly the property ZooKeeper-plus-watches only approximated.

For the migration this has a direct consequence. Because metadata is a log with offsets, the KRaft controller can record “I have written everything up to offset N back to ZooKeeper” in a single znode and use it as a fencing and progress marker. That mechanism is what makes dual-write safe, and we return to it below.

Controller roles and the quorum

A node’s role is set with process.roles: broker, controller, or both (combined mode). For migration, the design in KIP-866 supports dedicated controllers only; migrating directly into combined mode is listed as unsupported. Production guidance in the docs agrees: combined mode is acceptable for development but is not recommended for critical deployments, because you cannot roll or scale controllers independently of brokers, and a broker that is garbage-collecting or disk-bound shares a failure domain with the metadata quorum.

Sizing follows Raft arithmetic. To tolerate N simultaneous controller failures you need 2N + 1 voters, so three controllers survive one failure and five survive two. The docs state that 5 GB of memory and 5 GB of disk on the metadata log directory is sufficient for a typical cluster, which makes controllers cheap in hardware terms. Spread them across failure domains (racks or availability zones), because a quorum that loses majority stops accepting metadata changes, even though existing partitions keep serving produce and fetch traffic for a while.

Static versus dynamic quorum

Kafka 4.0 made dynamic quorum membership (KIP-853) available, so voters can be added and removed with tooling rather than by editing every config and restarting. Release 4.2 added a controller.quorum.auto.join.enable setting, default false, that lets controllers join the voter set automatically. For a first-time migration, the 3.9 documentation formats a single standalone controller with kafka-storage.sh format --standalone, and you grow the quorum afterward. If you prefer static membership, use controller.quorum.voters in the form id@host:port and keep it identical on every node. Whichever you choose, decide before migration, because changing quorum membership during the hybrid phase adds risk you do not need.

Why the migration is online at all

The designers rejected an offline migration because of downtime and rejected switching brokers to KRaft without restarts because of implementation risk. What they built instead is a controller that can speak both languages. It reads existing state from ZooKeeper once, copies it into the KRaft log inside one metadata transaction (KIP-868), takes over the controller role by overwriting /controller and /controller_epoch in ZooKeeper, and then keeps ZooKeeper updated in the background so ZooKeeper-mode brokers and a rollback path both keep working. That hybrid controller is the heart of the whole procedure.

The KIP-866 Migration Procedure Step by Step

Migration moves through five phases defined in the official docs: initial (all ZooKeeper), initial metadata load, hybrid (some brokers still ZooKeeper-mode while a KRaft controller runs), dual-write (all brokers KRaft but the controller still mirrors writes to ZooKeeper), and finalized (ZooKeeper no longer written). Rollback is possible until the last phase. The sequence below follows the official 3.9 documentation, with operational commentary.

Figure 2: The five migration phases. Everything left of Phase 5 can be reverted; Phase 5 cannot.

Step 0: Prepare the cluster

Upgrade every broker to 3.9.1 and set inter.broker.protocol.version=3.9. Do this as an ordinary rolling upgrade, soak it, and confirm it is healthy before touching anything related to KRaft. The docs warn that the metadata version must not change during migration or the cluster may break, so freeze version bumps for the duration of the project. Enable TRACE logging for org.apache.kafka.metadata.migration on the controllers and DEBUG on controllers and ZooKeeper-mode brokers, because the log is your primary observability channel during the move.

Run a feature audit at this point. The docs state that features not fully implemented in KRaft block migration if you use them, and brokers with multiple log directories are a documented trap: in ZooKeeper mode any log directory failure shuts the broker down, and a broker with a failed directory can only migrate after it is repaired (KAFKA-16431). Check every broker’s disks before you start, not halfway through.

Also check whether you ever ran the ZooKeeper security migration tool. The 3.9 docs flag that if it ran earlier it can leave a malformed /migration znode that stalls the process (KAFKA-19480), and the fix is to delete that node in zookeeper-shell.sh before provisioning controllers. Note that the tool itself was removed in 4.0.

Step 1: Provision the controller quorum

Controllers must carry the same cluster ID as the existing cluster. Read it from any broker’s meta.properties, or from ZooKeeper with bin/zookeeper-shell.sh localhost:2181 get /cluster/id. A mismatched cluster ID is the classic first failure, and it surfaces as controllers that refuse to talk to brokers.

A migration-time controller configuration, adapted from the official sample, looks like this:

# controller.properties (migration mode)
process.roles=controller
node.id=3000
controller.quorum.bootstrap.servers=ctrl1:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
listener.security.protocol.map=CONTROLLER:SSL
inter.broker.listener.name=PLAINTEXT

# migration flag and the ZooKeeper connection it reads from
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181

Node IDs deserve a rule: brokers and controllers share one node ID namespace, so a controller node.id must not collide with any existing broker.id. Many teams start controllers at 3000 or 9000 for that reason. A KRaft broker that reuses a ZooKeeper broker’s ID is explicitly rejected by the design.

Format and start the controller, using the latest metadata.version that kafka-storage.sh applies by default:

bin/kafka-storage.sh format --standalone \
  --cluster-id <zk-cluster-id> -c config/kraft/controller.properties
bin/kafka-server-start.sh config/kraft/controller.properties

In a three-node quorum you then add the other two voters per the dynamic-quorum tooling, or you format all three with a static voter list from the start. At this stage the controllers are running, in the PreMigration state, but nothing is migrating yet because no broker has been told about them.

Step 2: Enable migration on the brokers

Now restart brokers one at a time with additional configuration. Each broker needs a non-negative broker.id (even if automatic ID generation is enabled, and not above reserved.broker.max.id), the controller bootstrap setting, the controller listener name, the listener added to listener.security.protocol.map, and the migration flag:

# broker server.properties (hybrid phase)
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker1:9092
inter.broker.protocol.version=3.9

zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
zookeeper.metadata.migration.enable=true

controller.quorum.bootstrap.servers=ctrl1:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:SSL

Migration begins automatically once the last ZooKeeper broker has restarted with this configuration. The active controller logs “Completed migration of metadata from Zookeeper to KRaft” at INFO when the initial copy finishes. Until then, do not assume it is working; read that line.

The cluster is now in the hybrid state. The KRaft controller is in charge, yet your brokers are still ZooKeeper-mode processes. The controller sends them the same controller RPCs the old ZooKeeper-elected controller used (LeaderAndIsr, UpdateMetadata, StopReplica), flagged so brokers know the sender is a KRaft controller. Brokers continue to use ZooKeeper watches for ACLs, dynamic configs and delegation tokens, which dual-write keeps current.

Step 3: Migrate the brokers to KRaft

Take a deliberate pause here. You are in a stable, fully reversible state, and this is the right place to run a soak test. Then roll each broker again, this time converting it:

# broker server.properties (KRaft broker)
process.roles=broker
node.id=1                 # same number as the old broker.id
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker1:9092

controller.quorum.bootstrap.servers=ctrl1:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:SSL

# removed: broker.id, zookeeper.connect, zookeeper.metadata.migration.enable
# removed: inter.broker.protocol.version (metadata.version governs now)
# removed if present: control.plane.listener.name

Three details trip people up. The node ID must stay identical to the previous broker ID so replica assignments still resolve. inter.broker.protocol.version must be removed because KRaft uses the metadata.version feature level instead. And if you used the ZooKeeper-based authorizer, change authorizer.class.name from kafka.security.authorizer.AclAuthorizer to org.apache.kafka.metadata.authorizer.StandardAuthorizer, otherwise the broker cannot load ACLs.

After the last broker converts, you are in the dual-write phase: everything is KRaft-mode, yet the controller still mirrors every metadata change into ZooKeeper so that a rollback remains possible.

Step 4: Finalize

Remove zookeeper.metadata.migration.enable from the controller configuration and restart controllers one at a time. After that ZooKeeper receives no more writes, rollback is gone, and the ensemble can be decommissioned if nothing else uses it. Some teams wait a week or two in dual-write before finalizing to validate the new control plane under real load, and the official docs describe exactly that practice. Choose your waiting period against your change calendar: month-end batch jobs, partition reassignments and topic-creation storms are the workloads worth observing before you burn the bridge.

Figure 3: Dual-write. The controller commits a change to the metadata log, replies to the client, then mirrors it to ZooKeeper asynchronously.

The sequence matters for latency and safety. The client’s create-topic call is acknowledged after the KRaft commit, not after the ZooKeeper write, so a slow ensemble cannot slow the new control plane. The mirrored write is a ZooKeeper multi-op that also updates the /migration znode with the last KRaft offset written back, so the state in ZooKeeper is always internally consistent and attributable to a known log position.

Dual-Write Internals, Monitoring, and Capacity Planning

The migration’s safety comes from a handful of small mechanisms. Understanding them tells you what to watch and which symptoms are alarming versus merely noisy. This section goes below the runbook and into the design described on the KIP-866 page.

The write-behind model and fencing

Every metadata change goes into the KRaft log first. The ZooKeeper copy is write-behind and asynchronous, so Raft commit latency is independent of ZooKeeper latency. Each ZooKeeper write is a multi-op transaction combining the data change with a conditional update to the /migration znode, which records the last KRaft offset flushed. The condition is what fences a stale controller after a failover: if a deposed controller tries to write, the conditional update fails because the znode no longer matches what it expects, and controller-epoch checks add a second layer.

There is a deliberate back-pressure rule. If ZooKeeper writes fail, the controller stops appending to the metadata log rather than letting the unmirrored backlog grow without bound. Read that carefully, because it defines the one scenario where ZooKeeper health can hurt a KRaft-led cluster: an ensemble that is unavailable during dual-write can stall metadata changes. Reads, produces and fetches on existing partitions continue, but creating topics, moving leaders or changing configs will queue. Keep the ensemble healthy until you finalize.

The metrics that matter

The KIP defines specific MBeans. The official 3.9 migration page lists no metric names, so these come from the KIP page and are worth wiring into your dashboards before you start:

MBean	What it tells you
kafka.controller:type=KafkaController,name=ZkMigrationState	Reported by the active KRaft controller only; values are None, PreMigration, Migration, PostMigration
kafka.controller:type=KafkaController,name=MetadataType	Active controller mode: ZooKeeper=1, KRaft=2, Dual=3
kafka.server:type=KafkaServer,name=MetadataType	Per-broker mode: ZooKeeper=1, KRaft=2
kafka.controller:type=KafkaController,name=MigratingZkBrokerCount	ZooKeeper-mode brokers still registered with the quorum
kafka.controller:type=KafkaController,name=ZkWriteBehindLag	How far ZooKeeper trails the KRaft log
kafka.controller:type=KafkaController,name=ZkWriteSnapshotTimeMs and ZkWriteDeltaTimeMs	Time taken by full snapshot syncs and incremental writes

One historical trap lives in ZkMigrationState: the numeric values for PreMigration and Migration were swapped in 3.5 relative to 3.4. If you build alerts on raw numbers rather than state names, test them against the exact version you run.

A sensible readiness gate before you finalize is four conditions at once: MetadataType on the active controller reads Dual (3), MigratingZkBrokerCount is zero, ZkWriteBehindLag has been near zero for your whole soak period, and every broker reports KRaft (2). Pair those with ordinary cluster health signals: under-replicated partitions at zero, offline partitions at zero, request latency at baseline, and no controller election events other than ones you caused.

Kafka 4.2 adds metrics that make the post-migration state easier to reason about. KIP-1190 introduces a controller thread idle ratio, KIP-1229 a MetadataLoader idle ratio, and KIP-1180 generic feature-level metrics for finalized and supported levels. Put the idle ratios on your long-term dashboard, because a controller that is rarely idle is your earliest sign of a metadata hot spot such as mass partition reassignment.

Sizing the metadata plane with a worked example

Controller hardware is rarely the limiting factor, but you should still reason about it. Consider an illustrative cluster of nine brokers hosting 60,000 partition replicas across 4,000 topics. These numbers are for demonstration, not measurements. The metadata image for that cluster is a few tens of megabytes at most, which fits trivially inside the documented 5 GB memory and disk guidance. What grows the log is churn: every leader change, ISR shrink and topic edit appends a record, and snapshots bound replay time.

What the quorum does need is a fast, durable disk for the metadata log directory and low, predictable network latency between voters, because each commit needs a majority acknowledgement. A three-voter quorum spread across three availability zones pays one cross-zone round trip per commit, which is acceptable for metadata volumes. A five-voter quorum buys tolerance for two failures and costs two more nodes; the common reasoning is to take five only when you must survive a zone outage while another controller is under maintenance.

For the metadata log directory, give it dedicated storage separate from broker data volumes. The point is isolation: a broker volume filling up should never block the quorum, and the converse. Separate metadata.log.dir from log.dirs on controllers explicitly so the layout is intentional rather than accidental.

Security configuration during migration

Controller traffic needs the same security care as inter-broker traffic. Add a dedicated listener for controllers (the sample uses CONTROLLER), map it in listener.security.protocol.map, and use TLS or SASL on it. Brokers and controllers must also agree on the principal mapping so that inter-node requests authenticate as a user allowed to perform cluster actions. Because ACLs migrate with the metadata, the old AclAuthorizer has to be swapped for StandardAuthorizer at the broker-conversion step, and your super-user list must be carried across as well. Verify by running an authorization test from a non-admin principal both before and after the broker conversion.

SCRAM credentials and delegation tokens deserve explicit test cases. The KIP says ZooKeeper-mode brokers keep using watches for ACLs, dynamic configs and delegation tokens and that dual-write keeps these current, but Kafka 4.0 also removed delegation.token.master.key as a config, so token users need to read the upgrade notes for the replacement before the version jump rather than at it.

Operating the Migration: Real-World Walk-Through

A runbook is only as good as the sequencing around it. This section stitches the pieces together into a plan you can hand to a change board, with the gates that separate a calm migration from a long night.

Figure 4: Rollback decision flow. The first question is always whether the controllers are still in migration mode.

A realistic timeline

For an illustrative production cluster of a dozen brokers, a defensible plan runs over roughly three to five weeks, mostly waiting. These durations are planning suggestions, not benchmarks.

1. Week 0, rehearsal. Build a staging cluster with production-shaped topic counts, ACLs, quotas and connector set. Run the entire procedure, including a deliberate rollback. If your staging cluster has 20 topics and production has 20,000, you have learned nothing about the initial metadata copy.
2. Week 1, bridge upgrade. Rolling upgrade to 3.9.1 with inter.broker.protocol.version=3.9. Soak for several days.
3. Week 2, provision controllers and enter migration. Format controllers, start them, restart brokers into the hybrid state, wait for the completion log line. Soak at least a few days. This is where you inspect the ZooKeeper-to-KRaft copy for fidelity, for example by diffing kafka-topics.sh --describe output and ACL listings before and after.
4. Week 3, convert brokers. One broker per change window if you are risk-averse, or one rack at a time if your replication layout tolerates it. Check under-replicated partitions between each restart.
5. Weeks 4 to 5, dual-write soak, then finalize. Run through at least one full business cycle, then remove the migration flag from controllers and decommission ZooKeeper after another waiting period.
6. Afterwards, upgrade to 4.x. This is a separate rolling upgrade with its own preconditions, covered below.

Verifying the initial metadata copy

The initial load is the one step where silent data loss would be catastrophic, so verify it rather than trusting the log line. Capture three snapshots before entering migration and again after the hybrid phase begins: the full topic list with partition counts and replication factors, the full ACL listing, and the dynamic broker and topic config overrides. Compare them mechanically. Also compare consumer group state, since group offsets live in __consumer_offsets as ordinary data and are not part of cluster metadata, so they should be unaffected. A mismatch in topics, ACLs or configs is a reason to stop and revert while revert is cheap.

Client and ecosystem checks before the 4.x hop

The migration itself is invisible to clients, but the upgrade that follows is not. Per the 4.0 notes, clients including Kafka Streams and Connect must be at 2.1 or higher before brokers move to 4.0, and non-Apache clients may need separate review because old protocol API versions were removed (KIP-896). Java requirements also change: clients and Streams need Java 11 or higher, while brokers, Connect and tools need Java 17 or higher. Scala 2.12 support is gone, so any custom broker plugins built against it need rebuilding.

Several defaults shift as well. linger.ms changes from 0 to 5, which improves batching but nudges producer latency for unbatched workloads. message.timestamp.after.max.ms now defaults to one hour, so producers that stamp records in the future may see rejections. The DefaultPartitioner and UniformStickyPartitioner classes were removed, so code referencing them by name will fail. MirrorMaker 1 is removed, and logging moved from Log4j to Log4j2, which means broker log4j.properties files must be converted. Scan your fleet configs for each of these before the hop, because they surface as startup failures during a rolling upgrade, when you least want them.

The 4.x upgrade after migration

After finalization, upgrade the KRaft cluster to 4.x as a normal rolling upgrade, brokers and controllers one at a time, and read the metadata version rules. Each metadata version has a flag marking whether it introduces metadata changes, and when it does, a metadata downgrade is not supported. The 4.0 notes tell you to finalize the upgrade only after verifying cluster behavior, using bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.0, which also activates the KIP-848 consumer rebalance protocol and the KIP-890 transaction defenses. Keep the binary upgrade and the metadata.version finalization as separate, separately scheduled steps. That separation is your rollback window for the 4.x hop, in the same way dual-write was for the migration.

Moving on to 4.2 or a later 4.x line follows the same pattern. The 4.2 notes add little in the way of hard requirements beyond a share-group note for small clusters: with fewer than three brokers you must set share.coordinator.state.topic.replication.factor and share.coordinator.state.topic.min.isr to 1 before using share groups.

Trade-offs, Gotchas, and What Goes Wrong

The migration’s reputation for being smooth is mostly deserved, but the failures that do occur follow patterns. Here are the ones worth designing around.

The point of no return is real. Finalization deletes your rollback. The Apache docs are explicit that once finalized the cluster cannot return to ZooKeeper mode. The cost of waiting a few extra days in dual-write is a ZooKeeper ensemble you keep paying for and a moderate amount of change-freeze friction. The cost of finalizing too early is an unrecoverable surprise. Bias toward waiting.

Reverting is not symmetrical. Rollback after entering migration mode requires manual ZooKeeper surgery: stop the KRaft quorum cleanly, compare /controller_epoch with the kraft_controller_epoch field in /migration, raise the former if the KRaft epoch is higher, then delete /controller and /migration. The 3.9 docs also record a known issue (KAFKA-20488) where reverting can misbehave if the KRaft controller epoch exceeds the old ZooKeeper epoch, which is why the epoch comparison step exists. And the broker roll has a trap: on the first rollback roll you keep zookeeper.metadata.migration.enable=true and only remove it on the second roll. Do the rollback in a rehearsal so no one learns this at 3 a.m.

Multiple log directories. A ZooKeeper-mode broker with several log directories shuts down on any directory failure, and a broker with a broken directory can only migrate after repair. Treat disk health as a precondition and sweep for it.

Node ID collisions. Controller IDs and broker IDs share a namespace. A collision is rejected, but only at the moment the broker tries to register, which can be mid-migration.

Unsupported features. The docs warn that features not fully implemented in KRaft block migration if you rely on them. Do not assume your exotic configuration is supported; test it in staging.

Combined mode is not a migration target. The KIP excludes migrating into combined broker-controller mode. If you want combined mode for a small edge cluster, migrate to dedicated controllers first or rebuild the cluster fresh, and weigh whether the lost isolation is acceptable.

Changing versions mid-flight. Altering inter.broker.protocol.version during migration may break the cluster. Resist the urge to “just bump it while we’re here”.

Stale tooling and scripts. Any automation that talks to ZooKeeper directly, such as scripts that read /brokers/ids or call kafka-topics.sh --zookeeper, silently stops being useful. Search your repositories for --zookeeper, zookeeper.connect and ZooKeeper client libraries in monitoring exporters before decommissioning anything. Note too that ZooKeeper-only monitoring becomes irrelevant: alerts on ZooKeeper outstanding requests and session counts can be retired, while new ones on controller quorum health must be created before the old ones are deleted.

Shared ZooKeeper. Some organizations point multiple systems at one ensemble. The docs say ZooKeeper can be deprovisioned only if nothing else uses it. Confirm that with a connection audit rather than by assumption.

Operator-managed clusters. If Kafka runs under a Kubernetes operator or a vendor control plane, the operator may own the migration workflow and its own ordering. Follow that vendor’s procedure and version support matrix rather than hand-editing StatefulSets, and verify what it automates against the KIP-866 phases described here.

A final, broader trade-off: KRaft removes an operational dependency but concentrates risk. The controller quorum is now the single most critical component in the cluster, and its disks, network and certificate expiry deserve more attention than any individual broker. A team that gains simplicity but under-invests in controller monitoring has traded one fragility for another.

Practical Recommendations

Treat the migration as a project with explicit exit criteria, not an afternoon change. Start with rehearsal on a production-shaped staging cluster, because the ratio of cluster size to metadata volume is what determines how the initial load behaves. Pin the bridge version at 3.9.1 or later 3.9 patch, freeze inter.broker.protocol.version for the duration, and read the current 3.9 operations page for known-issue updates the week you execute.

Use dedicated controllers, three of them for most clusters and five where you must tolerate two simultaneous failures, spread across failure domains and equipped with their own fast disks. Provision monitoring first: the migration MBeans, controller logs at TRACE for the migration package, and the 4.2 idle-ratio metrics once you are on that line. Define the finalize gate in numbers: Dual metadata type, zero migrating ZooKeeper brokers, near-zero write-behind lag, zero under-replicated partitions, and a completed soak.

Keep finalization and the 4.x upgrade as separate changes, and keep the metadata.version bump separate from the binary upgrade. After finalization, decommission ZooKeeper only after a connection audit, and keep its data snapshot archived for a while.

A short pre-flight checklist:

• [ ] Brokers on 3.9.1 or newer 3.9.x, inter.broker.protocol.version=3.9, soaked for days.
• [ ] Cluster ID copied from meta.properties or /cluster/id.
• [ ] Controller node IDs disjoint from broker IDs; dedicated controllers, not combined.
• [ ] No broker has a failed log directory; multi-directory brokers checked.
• [ ] Authorizer plan: AclAuthorizer to StandardAuthorizer at broker conversion.
• [ ] Snapshots of topics, ACLs and configs captured for before and after diffing.
• [ ] Migration MBeans on dashboards and alerts tested against the running version.
• [ ] Rollback rehearsed, including the two-roll sequence and epoch check.
• [ ] Client floor (2.1+), Java 17 for brokers, Log4j2 conversion and removed configs scanned for the 4.x hop.
• [ ] Finalization scheduled as its own change with a named approver.

Frequently Asked Questions

Can I skip the migration and upgrade straight from Kafka 3.x with ZooKeeper to 4.x?

No. Kafka 4.0 supports only KRaft mode and ZooKeeper mode has been removed, so brokers in a ZooKeeper-mode cluster cannot be upgraded to 4.0 or later. The upgrade notes say ZooKeeper-mode clusters must migrate to KRaft first. Practically, that means moving to a 3.9.x bridge release, running the KIP-866 migration on it, finalizing, and only then upgrading binaries to 4.x. Starting a brand-new KRaft cluster and mirroring data is possible but is a different, heavier project.

Is a Kafka KRaft migration done online, without downtime?

Yes. KIP-866 specifies an online migration where brokers are restarted one at a time and a hybrid KRaft controller keeps both worlds in sync through dual-write. Clients continue to produce and consume throughout, although each rolling restart briefly moves partition leadership, so applications should tolerate normal leader changes. The offline alternative was explicitly rejected in the design because of downtime. Plan for the usual rolling-restart effects such as short latency bumps, not for an outage window.

Can I roll back after starting the migration?

You can until you finalize. Before controllers leave migration mode you can revert by restarting brokers in ZooKeeper mode, stopping the KRaft quorum cleanly, checking and, if necessary, raising /controller_epoch, and deleting the /controller and /migration znodes. The procedure differs depending on which phase you completed. Once you remove zookeeper.metadata.migration.enable from the controllers and restart them, the docs state that reverting to ZooKeeper is no longer possible.

How many KRaft controllers do I need, and can they share nodes with brokers?

Use three or five dedicated controllers. A quorum needs 2N + 1 voters to survive N simultaneous failures, so three tolerate one failure and five tolerate two. Combined mode, where one process is both broker and controller, is documented as fine for development but not recommended for critical deployments, and KIP-866 does not support migrating into it. The docs call 5 GB of memory and 5 GB of metadata disk sufficient for a typical cluster, so controllers can be small.

What happens to consumer offsets and topic data during the migration?

Nothing about your data moves. Topic data and the __consumer_offsets topic are ordinary partitions on the brokers and are not rewritten. Only cluster metadata, meaning topic definitions, partition assignments, ACLs and configs, is copied from ZooKeeper into the __cluster_metadata log. That is why verifying metadata fidelity with before-and-after diffs is the right check, while consumer group behavior should simply continue as before.

Do I need to change the authorizer when moving to KRaft?

Yes, if you use ACLs. The ZooKeeper-based kafka.security.authorizer.AclAuthorizer must be replaced with org.apache.kafka.metadata.authorizer.StandardAuthorizer when you convert each broker to KRaft, per the 3.9 migration docs. ACLs themselves are carried over with the metadata during the initial load and kept current by dual-write. Test authorization with a non-admin principal before and after the broker conversion, and confirm your super-user list is still applied.

Further Reading

Internal:

• Kafka exactly-once semantics, transactions and idempotent producers for client-side behavior that interacts with the 4.0 transaction changes.
• Kafka queues versus RabbitMQ and SQS for task queues for what share groups in 4.2 enable once you are on KRaft.
• Distributed locks, leases and fencing tokens: ZooKeeper and etcd correctness for the consensus background behind the fencing ideas used in dual-write.

External:

• Apache Kafka 4.0 upgrade notes
• Apache Kafka 3.9 KRaft operations and ZooKeeper migration
• KIP-866: ZooKeeper to KRaft Migration
• Apache Kafka 4.2.0 release announcement

By Riju — about