Kafka Exactly-Once Semantics: Idempotent Producers, Transactions and Their Limits (2026)
“Exactly once” is the most abused phrase in distributed systems. In a lossy network, a sender that does not hear an acknowledgement cannot know whether the receiver got the message, so it must either retry (risking duplicates) or give up (risking loss). Apache Kafka’s answer is not magic. It is a precise combination of sequence numbers, epoch fencing, a replicated transaction log and a consumer-side visibility rule. Get the details wrong and you ship duplicates to production while a dashboard proudly says “EOS enabled”.
This guide explains Kafka exactly once semantics from the wire protocol up: how the idempotent producer deduplicates retries, how transactions make a consume-transform-produce loop atomic, how zombie producers are fenced, what read_committed actually guarantees, and what changed with KIP-447 and KIP-890. Then it does the less popular part: the precise boundary where the guarantee ends, which is almost always at the first external side effect.
What this covers: delivery-semantics background, the producer ID/epoch/sequence mechanism, the transaction coordinator and log, control markers and the last stable offset, Java code for a transactional pipeline, Kafka Streams and Flink integration, failure modes, latency costs, and a practical checklist.
Context and Background
Messaging systems offer three delivery semantics. At-most-once never retries, so failures lose data. At-least-once retries until acknowledged, so failures create duplicates. Exactly-once promises that the effect of each message is applied once, even across retries and crashes. Strictly, no protocol can deliver a message exactly once over an unreliable channel; what real systems deliver is effectively once: at-least-once delivery combined with idempotent or atomic processing so duplicates have no visible effect. Kafka’s documentation uses “exactly-once semantics” (EOS) in exactly this sense, and scoped to data that lives inside Kafka.
Before EOS, the standard Kafka producer was at-least-once. A produce request could reach the leader, be written and replicated, and then the acknowledgement could be lost to a network blip. The producer, seeing a timeout, retried, and the partition now held the record twice. Worse, with several requests in flight, a retry of batch 1 could land after batch 2, reordering the stream. Consumers faced the mirror-image problem: process a record, crash before committing the offset, restart, and process it again.
EOS was introduced in Kafka 0.11 through KIP-98, which defined three capabilities: idempotent producers (no duplicates per partition per producer session), transactions (atomic multi-partition writes, including consumer offsets) and the message format v2 that carries the metadata to make both work. The design was motivated by stream processing, where a job reads from input topics, computes, and writes to output topics. Kafka Streams had to guarantee that a crash never produced a half-written result. KIP-98’s text is worth reading directly; the KIP-98 design document remains the clearest statement of the protocol.
Two things have changed since. KIP-447 (Kafka 2.6) removed the one-producer-per-input-partition scaling problem. KIP-890 hardened the transaction protocol against “hanging transactions” and, per the Kafka 4.x documentation, its strengthened protocol is controlled by the transaction.version feature flag and is enabled automatically on the server starting with Kafka 4.0. We cover both below.
A word on why this matters for IoT and industrial pipelines. Telemetry ingestion often tolerates duplicates because readings are idempotent by timestamp. But billing events, work orders, command acknowledgements and state-machine transitions are not, and duplicates there are expensive. If your pipeline writes events out of a database, pair this article with our guide to the transactional outbox pattern with CDC for exactly-once event publishing, which handles the other end of the boundary where Kafka’s guarantee stops.
Scope note: this article reflects the Kafka 4.x documentation. Where a detail is not confirmed in official documentation I say so rather than guess.
The Core Mechanism: How Kafka Gets to Exactly-Once
Kafka exactly-once semantics is built from two layers. The idempotent producer removes duplicates and reordering caused by retries within one partition and one producer session. Transactions then make writes to several partitions, plus the consumer’s offset commit, atomic, while read_committed consumers see only committed data.
Figure 1: Kafka exactly once semantics reference architecture. The producer talks to a partition leader (data) and to the transaction coordinator (state); the coordinator persists state in a replicated transaction log and writes COMMIT or ABORT markers into the data partitions that a read_committed consumer honours.
The diagram shows two planes. The data plane is ordinary Kafka: records go to partition leaders and are replicated. The control plane is new: a transaction coordinator, which is simply a broker that owns one partition of the internal transaction log, tracks each transaction’s state and ensures every participating partition eventually receives a commit or abort marker. The consumer closes the loop by deciding which records are visible.
Layer one: the idempotent producer
Every producer that enables idempotence is assigned a producer ID (PID) by the broker at initialization, via the InitProducerId request. Each record batch sent to a topic-partition carries the PID, a producer epoch, and a monotonically increasing sequence number that starts at zero for that PID and partition. According to KIP-98, the broker rejects a sequence number that skips ahead (a gap signals lost data) and ignores one that is lower than expected (a duplicate). The result is that a retried batch is acknowledged but not appended twice.
In message format v2 these fields live in the batch header: the PID is a 64-bit integer, the epoch a 16-bit integer and the first sequence a 32-bit integer. Because the sequence is per partition, the broker needs only a small amount of state per PID and partition: the last few sequence numbers and offsets for the most recent batches. That state is rebuilt from the log itself after a leader change, which is why idempotence survives failover: a new leader has seen the same batches via replication.
The practical defaults are friendlier than in the early days. In the current producer configuration reference, enable.idempotence defaults to true, acks defaults to all, retries defaults to 2147483647, and max.in.flight.requests.per.connection defaults to 5, with idempotence requiring a value of 5 or less. In other words, since Kafka 3.0 a stock producer is already idempotent. The original KIP-98 text required a single in-flight request; the broker was later extended to track several recent batches per PID, so ordering is preserved with up to five in flight. If you set acks=1 or retries=0 explicitly while also forcing idempotence on, the client throws a ConfigException rather than silently weakening the guarantee.
The idempotent producer’s boundaries matter, and people get them wrong constantly. The PID is not stable across producer sessions: restart the process and the producer gets a new PID, so dedupe state from the previous incarnation is useless. It only dedupes writes by one producer instance. Two different producers that send the same logical event produce two records. And it says nothing about atomicity across partitions: write to partitions A and B, crash between them, and A has the record while B does not.
Layer two: transactions
Transactions fix both gaps. A producer sets a transactional.id, a stable, application-chosen string, which implies idempotence. Per the producer configuration reference, with no transactional ID the producer is limited to idempotent delivery. The stable ID is what lets a restarted producer be recognized as the successor of the dead one.
At startup the producer calls initTransactions(). The coordinator for that transactional.id is found by hashing the ID onto a partition of the transaction log (the internal topic __transaction_state, which the KIP-98 page simply calls the “Transaction Log”); the leader of that partition is the coordinator. The coordinator maps the ID to a PID, bumps the epoch, and completes or aborts any transaction left over from the previous incarnation. After the bump, any older instance holding a lower epoch is a zombie: brokers reject its requests and it receives ProducerFencedException.
Then the application loops: beginTransaction(), several send() calls, optionally sendOffsetsToTransaction(), and finally commitTransaction() or abortTransaction(). Behind the scenes (classic protocol) the producer registers each new partition with the coordinator via AddPartitionsToTxn before first writing it, and registers the consumer group’s offsets partition via AddOffsetsToTxn. On commitTransaction(), the producer sends EndTxn. The coordinator first writes a PREPARE_COMMIT record to the transaction log, which is the point of no return: once that record is replicated, the transaction will commit even if the coordinator crashes, because the successor coordinator replays the log and finishes the job. It then sends WriteTxnMarkers to the leaders of every participating partition, which append a control record, COMMIT or ABORT, to the log of each. Finally it writes COMPLETE_COMMIT to the transaction log.
This is classic two-phase commit with the coordinator’s replicated log as the durable decision record. It is cheaper than the usual 2PC because participants (partition leaders) do not vote; they have already durably accepted the data, and the only remaining question is visibility. That is a very different failure model from a database 2PC, and it is why Kafka transactions avoid blocking on participant failures.
Layer three: what consumers see
Data written in a transaction is physically in the partition log before the commit. Visibility is a consumer-side decision controlled by isolation.level. With the default read_uncommitted, a consumer sees everything, including records from open and aborted transactions. With read_committed, the broker hands the consumer records only up to the last stable offset (LSO): the offset of the first record belonging to a still-open transaction (or the high watermark if none are open). The fetch response also carries a list of aborted transactions in that range so the client library can drop their records. The control records themselves are consumed by the client library and never surface to application code, although offsets are still consumed by them, which is why you will see apparent gaps in offsets of a transactional topic.
Because the LSO is bounded by the oldest open transaction on that partition, one long-running or stuck transaction stalls every read_committed consumer of the partition. This single property drives most of the operational advice later in the article.
Deeper Analysis: Walking Through a Transaction and Its Failures
Figure 2: One Kafka transaction end to end. Note the two places the coordinator persists state (PREPARE and COMPLETE) and that markers are written after the commit decision is durable.
Figure 2 follows a single committed transaction. The numbers matter less than the ordering: the decision is durable before any marker is written, and markers are written asynchronously with respect to the application’s commitTransaction() returning in some versions of the protocol. A consumer in read_committed mode may therefore briefly see the LSO lag the high watermark after the application has already seen the commit succeed.
The consume-transform-produce loop in Java
The canonical EOS use case reads from an input topic, transforms, writes to an output topic, and commits the consumer offsets in the same transaction. The consumer must have auto-commit disabled and isolation.level=read_committed, so it does not read other producers’ aborted output as input.
Properties cp = new Properties();
cp.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
cp.put(ConsumerConfig.GROUP_ID_CONFIG, "enricher");
cp.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
cp.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
cp.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
cp.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
Properties pp = new Properties();
pp.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
// Stable per logical worker; must NOT be shared by two live instances
pp.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "enricher-" + shardId);
pp.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 30_000);
pp.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
pp.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
try (KafkaConsumer<String,String> consumer = new KafkaConsumer<>(cp);
KafkaProducer<String,String> producer = new KafkaProducer<>(pp)) {
consumer.subscribe(List.of("raw-events"));
producer.initTransactions(); // fences older instances
while (true) {
ConsumerRecords<String,String> records = consumer.poll(Duration.ofMillis(500));
if (records.isEmpty()) continue;
try {
producer.beginTransaction();
Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
for (ConsumerRecord<String,String> r : records) {
producer.send(new ProducerRecord<>("enriched-events", r.key(), enrich(r.value())));
offsets.put(new TopicPartition(r.topic(), r.partition()),
new OffsetAndMetadata(r.offset() + 1)); // NEXT offset to read
}
// KIP-447 overload: passes group generation so the broker can fence zombies
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
} catch (ProducerFencedException | InvalidProducerEpochException | AuthorizationException e) {
throw e; // fatal: close and restart the worker
} catch (KafkaException e) {
producer.abortTransaction(); // retriable: roll back and rewind
rewindConsumerToCommitted(consumer);
}
}
}
Several details in this snippet are load-bearing. The committed offset is lastProcessed + 1, the next record to read; committing the processed offset itself re-reads one record. The offsets travel inside the producer’s transaction, not through consumer.commitSync(); that is how the output records and the input progress become one atomic unit. ProducerFencedException and its relatives are fatal: the instance has been superseded and must shut down, not retry. And after an abort the consumer’s in-memory position has moved past records whose output was rolled back, so you must seek() to the last committed offsets (the rewindConsumerToCommitted helper here) or the aborted batch is silently skipped.
KIP-447: why one producer per partition used to be needed
The first generation of EOS tied a transactional.id to an input partition. The reason was fencing: if partition P moves from consumer A to consumer B in a rebalance, B must be able to fence A’s in-flight transaction, and the only handle the coordinator had was a shared transactional.id. So Kafka Streams created one producer per task (per input partition), multiplying connections, buffers and concurrent transactions. KIP-447 states the root cause plainly: transaction coordinators had no knowledge of consumer group semantics.
KIP-447, adopted in Kafka 2.6.0 according to its page, fixed this by passing consumer group state into the transaction. The new sendOffsetsToTransaction(offsets, ConsumerGroupMetadata) overload carries the group ID, generation ID, member ID and group instance ID, and TxnOffsetCommitRequest gained the generation, member and instance fields. If they do not match the group coordinator’s current state, the offset commit is rejected and the producer is fenced, so a zombie that missed a rebalance cannot commit offsets for partitions it no longer owns. One producer per thread now suffices, and the transactional.id no longer has to map to a specific input partition. In Kafka Streams this arrived as exactly_once_beta and later exactly_once_v2.
KIP-447 also introduced a deliberate availability trade-off. When a consumer fetches committed offsets under read_committed and finds pending transactional offset commits, the broker returns a PENDING_TRANSACTION error and the client backs off and retries until those transactions complete or expire. The authors describe this as a trade between availability and correctness; the alternative would be to hand out a stale offset and risk processing duplicates. Broker requirements: non-Streams applications must upgrade brokers to 2.5 before switching to the new overload.
KIP-890: closing the hanging-transaction hole
Figure 3: Classic versus KIP-890 (transaction version 2) transactions. The new protocol bumps the epoch on every commit or abort and adds partitions implicitly on first produce, which closes the delayed-message window.
A classic weakness: a delayed produce request from an aborted transaction can arrive at a partition leader after the transaction ended and a new one has started under the same PID and epoch. The leader sees an apparently valid request, appends it, and now the new transaction contains a record that was supposed to be rolled back, or worse, the partition is in an open transaction that no coordinator knows about. That is a hanging transaction: it pins the LSO, blocks log compaction on compacted topics, and leaves read_committed consumers stuck.
KIP-890 addresses this in two ways. For the new client protocol, the producer epoch is bumped on every commit or abort, so each transaction is identified by a unique (PID, epoch) pair and stragglers from an earlier transaction are fenced by epoch comparison. The first produce to a partition within a transaction adds it to the transaction implicitly, so clients no longer send AddPartitionsToTxn. For older clients the broker performs verification: before appending, it checks through AddPartitionsToTxn with a verify-only flag that the partition really is in an ongoing transaction, returning INVALID_TXN_STATE otherwise. Verification is on by default and has a config switch.
The strengthened protocol is gated by the transaction.version feature flag; the Kafka 4.x docs state that transaction.version=2 enables it, that it is enabled automatically on the server starting with Kafka 4.0, and that producers of version 4.0 or later use it as long as it is enabled. Producers need not be restarted: they switch at the start of the next transaction after reconnecting. Downgrade is described as safe. One visible side effect: because endTransaction is asynchronous, the next transaction’s partition additions can hit CONCURRENT_TRANSACTIONS while markers are still being written, and with the new protocol the server retries internally, which can raise the reported produce latency without raising end-to-end transaction time. Treat this as a documented claim, not a number you can assume for your workload.
Hedge: I could not confirm from the official pages I fetched exactly which minor release shipped each part of KIP-890; check the release notes for your version before relying on a specific behaviour.
Kafka Streams and the processing guarantee
Most teams never call the transactional API directly; they set one property in Kafka Streams. processing.guarantee=exactly_once_v2 makes every task commit its state-store changelog writes, its output records and its input offsets in a single transaction. The changelog is the subtle part: Streams state stores are backed by Kafka topics, so atomically committing output and changelog together means that after a crash the state store restores to exactly the state consistent with the committed offsets. Without that, a rebuilt store could include updates for records whose output was never emitted.
The exactly_once_v2 mode is the productized form of KIP-447’s single-producer-per-thread design. The older exactly_once (one producer per task) and exactly_once_beta values were deprecated in favour of it; I did not verify in which release they were removed, so check your version’s migration notes before upgrading. The official guidance in KIP-447 for the transitional upgrade was two rolling bounces, the first enabling generation-aware fencing and the second switching mode.
The commit interval is the practical knob. In EOS mode Streams commits on commit.interval.ms, and the documented default drops to 100 ms when EOS is on (versus 30 seconds for at-least-once); verify against your version’s reference. Every commit is a transaction, so a tiny interval means many small transactions and heavy marker traffic, while a large interval means downstream read_committed consumers see data in large delayed bursts. KIP-447 also pushed Streams to a 10-second default transaction.timeout.ms and shortened the broker’s abort-cleanup interval from one minute to 10 seconds, so stuck transactions clear faster.
Flink and the two-phase commit sink
Flink’s Kafka sink shows how an external engine layers its own checkpointing on top of Kafka transactions. With DeliveryGuarantee.EXACTLY_ONCE, the sink writes records inside a Kafka transaction and commits that transaction only when the Flink checkpoint completes. The Flink Kafka connector documentation states that checkpointing is required, that downstream consumers must use read_committed to avoid duplicates after a restart, and that records become visible only after a checkpoint, so the checkpoint interval sets your end-to-end latency floor.
Two operational rules from that page deserve emphasis. First, each application needs a unique transactionalIdPrefix, because the sink derives transactional IDs from it and two jobs sharing a prefix will fence each other. Second, the producer’s transaction.timeout.ms must be at least the maximum checkpoint duration plus the maximum restart duration; if it is shorter, Kafka may expire an uncommitted transaction the job intended to commit after recovery, and data may be lost. The page adds that a ProducerFencedException usually means the transaction timed out on the broker. The broker caps what the client can ask for via transaction.max.timeout.ms, which defaults to 900,000 ms (15 minutes) in the Kafka 4.2 broker reference, so a job with a 20-minute worst-case recovery needs a broker config change as well.
If you are choosing a stream processor for materialised views over telemetry, the same sink semantics show up in comparisons such as our RisingWave vs Materialize vs Flink streaming SQL analysis for IoT telemetry; the exactly-once story differs by engine, and “supports EOS” in a feature table rarely says which boundary it covers.
Numbers: what transactions cost
Transactions are not free, and the costs are structural rather than mysterious. The following is an illustrative model, not a benchmark; plug in your own measurements.
Suppose a pipeline processes 10,000 records per second on one task, committing every 100 ms. That is 10 transactions per second, each carrying 1,000 records. Per transaction, the classic protocol costs roughly: one AddPartitionsToTxn round trip per newly touched partition, one AddOffsetsToTxn, one TxnOffsetCommit, one EndTxn, then the coordinator’s two log writes (PREPARE and COMPLETE) plus one marker write per participating partition. If the output touches 12 partitions, that is on the order of 12 marker records per commit, or 120 control records per second per task, written at the cost of an extra replicated append on each partition leader. Against 10,000 data records per second that is about 1.2 percent extra records; the real cost is not bytes but round trips and latency.
Latency is the bigger effect. A record written in a transaction is invisible to read_committed consumers until the commit marker lands, so the visible end-to-end latency is the commit interval plus marker propagation plus consumer fetch delay. With a 100 ms interval, expect on the order of a few hundred milliseconds at the tail under healthy conditions; with Flink checkpointing every 60 seconds, expect minute-scale latency. The knob trade is simple: shorter intervals reduce latency and increase transaction overhead and coordinator load; longer intervals do the reverse and enlarge the blast radius of an abort, since the whole interval’s work is redone.
Throughput impact is workload dependent. Confluent and community write-ups typically describe a modest penalty for large batches and a large relative penalty for tiny transactions, but I am not citing a specific percentage because published figures depend heavily on batch size, partition count and hardware. The honest guidance: benchmark with your real transaction size, and remember the cost curve is dominated by transaction count, not record count.
Sizing the coordinator and the transaction log
The transaction log is an ordinary replicated compacted topic with 50 partitions by default (transaction.state.log.num.partitions; verify in your version). Broker defaults in the 4.2 reference include transaction.state.log.replication.factor of 3 and transaction.state.log.min.isr of 2, and the producer reference notes that transactions require a cluster of at least three brokers by default unless you lower the replication factor for development. Each transactional.id consumes a small record in the log, and the broker expires idle IDs after transactional.id.expiration.ms, 7 days by default. A fleet that mints a fresh random transactional.id per process start will leak state for a week; use stable, bounded IDs.
Coordinator placement is a hash, so a skewed set of IDs can pile load on one broker, although in practice the coordinator work per transaction is small. The more common bottleneck is the marker fan-out: a transaction that touches 500 partitions forces 500 marker writes across many brokers, which is why wide-fanout producers are the usual source of slow commits.
What Goes Wrong: Trade-offs, Gotchas and Failure Modes
Figure 4: Where Kafka exactly once semantics ends. Everything inside the Kafka boundary is covered; the database write, HTTP call or email on the right is not.
The most important sentence in this article is the boundary: Kafka’s exactly-once guarantee covers the effects that are themselves Kafka writes. Reads from Kafka, processing in your code, and writes to Kafka topics and consumer offsets are atomic. Anything else your code does is outside the transaction.
If the transform step calls a payment API, inserts into PostgreSQL, sends an email or publishes to RabbitMQ, a crash and abort will re-run that step, and the side effect happens twice. The transaction abort rolls back Kafka records; it cannot un-send an email. The fixes are well known and none of them is “turn on more EOS”: make the external operation idempotent with a deterministic key (for example, the topic-partition-offset of the source record as an idempotency key in the database), or move the side effect to a sink that participates in two-phase commit (Flink’s TwoPhaseCommitSinkFunction pattern), or write the result to Kafka and let a separate, idempotent consumer perform the effect. When the source is a database rather than Kafka, the transactional outbox is the mirror image of the same idea.
Zombies, timeouts and fencing in practice
A zombie is an instance that the cluster has superseded but that does not know it yet: a process paused by a long GC, a VM suspended during live migration, or a container cut off from the network. When it wakes, it tries to continue a transaction. Two mechanisms stop it: the producer epoch bump on initTransactions() (the new incarnation fences the old), and the consumer-group generation check from KIP-447 (the group coordinator rejects offset commits from a stale member). The zombie’s application code sees ProducerFencedException; correct handling is to stop, not retry.
A second source of fencing is the coordinator itself. If a transaction stays open longer than transaction.timeout.ms (default 60,000 ms in the producer reference; the timer starts when the first partition is added), the coordinator proactively aborts it and bumps the epoch. The slow-but-alive producer then discovers it has been fenced on its next call. The usual trigger is a processing step that sometimes takes longer than the timeout: a slow enrichment lookup, a rebalance storm or a stop-the-world pause. Set the timeout above your worst realistic transaction duration, but never above what you can tolerate as a stalled LSO, since a stuck transaction delays every read_committed consumer until it expires.
Stuck LSO and the invisible-data problem
If one producer begins a transaction and then dies without the timeout firing quickly, the LSO of every partition it touched is pinned at the first record of that transaction. Consumers using read_committed stop advancing there. Symptoms include consumer lag that grows while the producer side looks healthy, and tools that report the high watermark moving while consumers report zero throughput. Monitor the gap between high watermark and LSO per partition; a growing gap is the early warning. Tools such as kafka-transactions.sh (shipped with recent Kafka releases) can list and, for hanging transactions, abort; check its availability on your version.
Hanging transactions caused by delayed messages are exactly what KIP-890 targets, which is the best argument for being on a recent client and broker combination. Mixed-version fleets are the risky case: the server’s verification protects older clients, but the full guarantee needs both sides on the new protocol.
Other sharp edges
- Sharing a
transactional.id. Two live instances with the same ID will fence each other in a ping-pong. Derive the ID from a stable, unique worker identity, such as a StatefulSet ordinal, not from a hostname that can be reused. - Consumer offsets must be in the transaction. Calling
commitSync()alongside a transactional producer breaks atomicity: you can commit input progress without committing output. read_uncommittedreaders downstream. EOS is only visible to consumers that opt in. A dashboard consumer with the default isolation level reads aborted data.- Compacted topics and aborted data. Aborted records remain in the log until cleaned, and hanging transactions block cleaning.
- Retention and aborts. If retention deletes an input range before a retried transaction re-reads it, the retry cannot reproduce the original input. Keep retention comfortably above worst-case recovery time.
- Idempotence is per partition per session. Resending the same business event from an application-level retry creates a new sequence number and therefore a duplicate; that is the application’s problem to deduplicate with a business key.
- Cross-cluster replication. MirrorMaker 2 and similar replicators copy records, not transactions or PIDs; downstream clusters get at-least-once unless the replicator itself is carefully designed. Treat the exactly-once boundary as one cluster.
- Brokers below 2.5. The KIP-447 API requires brokers at 2.5 or newer; older brokers make the client fail.
When not to use it
EOS buys you correctness at a price in latency, complexity and operational risk. For high-volume telemetry where consumers tolerate duplicates or dedupe on (device_id, timestamp), plain idempotent production (already on by default) plus an idempotent sink is simpler and faster. For fan-out to many independent consumers where each does its own external work, EOS on the shared topic does not remove the need for consumer-side idempotency. If you compare Kafka against queue-style alternatives for task processing, where the unit of work is a job and not a stream, our analysis of Kafka queues versus RabbitMQ versus SQS for task queues helps decide whether you need streaming semantics at all.
Practical Recommendations
Start from the failure you are actually trying to prevent, not from the feature name. If it is duplicate records caused by producer retries, you probably already have the fix: the default producer is idempotent. If it is a stream job that must never emit half a result, you need transactions, and in practice that means Kafka Streams exactly_once_v2 or Flink’s EXACTLY_ONCE sink, not hand-rolled transactional code. If it is a side effect in another system, no Kafka setting will save you; design the idempotency key.
When you do roll your own transactional loop, treat the code in this article as a skeleton and add what production needs: rewind logic after aborts, fatal handling of fenced exceptions, graceful shutdown that aborts an open transaction, and metrics on transaction duration, abort rate and the high-watermark-to-LSO gap. Run it on a recent client and broker combination so you benefit from KIP-890; confirm the transaction.version finalized on your cluster rather than assuming it.
Checklist before you declare a pipeline “exactly once”:
- Brokers are at least three, with
transaction.state.log.replication.factor=3andmin.isr=2, and clients and brokers are on a version where KIP-447 and KIP-890 behaviour is available. - Every transactional worker has a stable, unique
transactional.id(or FlinktransactionalIdPrefix); none is shared by two live instances. - Consumer
enable.auto.commit=false,isolation.level=read_committed, and offsets are sent withsendOffsetsToTransaction(offsets, consumer.groupMetadata()). transaction.timeout.msexceeds worst-case processing (and, for Flink, checkpoint plus restart time) yet stays withintransaction.max.timeout.ms.- Fenced exceptions are fatal; other Kafka exceptions abort and rewind the consumer.
- Every external side effect has an idempotency key or sits behind a two-phase-commit sink.
- All downstream consumers that must see EOS use
read_committed. - Alerts exist for abort rate, transaction duration, LSO lag and coordinator load.
- Retention on input topics exceeds the longest realistic recovery window.
- You have chaos-tested it: kill workers mid-transaction, pause them with SIGSTOP past the timeout, and partition them from the coordinator, then diff inputs against outputs.
For FinTech or billing pipelines, this article is a systems-architecture discussion only and is not financial, legal or compliance advice.
Frequently Asked Questions
Does Kafka really guarantee exactly-once delivery?
Within Kafka, effectively yes: the idempotent producer prevents retry duplicates per partition and session, and transactions make multi-partition writes plus offset commits atomic for read_committed readers. It is not exactly-once delivery over the network, because retries still happen; the broker simply makes them harmless. The guarantee stops at the Kafka boundary, so database writes, API calls and emails triggered by processing can still repeat after a failure unless you make them idempotent.
Is the idempotent producer enabled by default?
In the current Kafka producer reference, enable.idempotence defaults to true, acks to all, retries to 2147483647 and max.in.flight.requests.per.connection to 5. Idempotence turns on automatically when no conflicting settings exist, and an explicit conflict raises a ConfigException. Older client versions had different defaults, so check what your producer library actually resolves rather than assuming; logging the effective config at startup settles it.
What is the difference between transactional.id and the producer ID?
The producer ID (PID) is an internal identifier the broker assigns per producer session; applications never see it and it changes on restart. The transactional.id is a stable string you choose. The coordinator maps it to a PID and bumps the epoch each time a new instance initializes, which fences the old one. Stability of the transactional.id is what lets a restarted worker pick up where a dead one left off.
Why is my read_committed consumer stuck?
Almost certainly an open transaction is pinning the last stable offset. A read_committed consumer cannot read past the first record of the oldest open transaction on a partition. Check for a producer that crashed or paused mid-transaction, compare the high watermark with the LSO, and wait for transaction.timeout.ms to expire or abort the hanging transaction with the transaction tooling. KIP-890 reduces the chance of hanging transactions on newer versions.
Does exactly-once work with external databases?
Not by itself. The Kafka transaction cannot include a PostgreSQL commit. Common patterns are an idempotent upsert keyed on the source topic, partition and offset; storing consumed offsets in the same database transaction as the result and seeking on startup; a two-phase-commit sink such as Flink’s; or the transactional outbox pattern when the database is the source. Each is a different trade between complexity and latency.
What is the performance cost of Kafka transactions?
The cost is dominated by transaction count, not record count: each transaction adds coordinator round trips, log writes and per-partition markers, and it delays visibility to read_committed consumers until commit. Large batches amortise this well; tiny, frequent transactions amplify it. I do not quote a universal percentage because published results vary with batch size, partitions and hardware. Measure with your own transaction size and commit interval.
Further Reading
- Transactional outbox pattern with CDC for exactly-once event publishing: the database-to-Kafka boundary that Kafka transactions do not cover.
- RisingWave vs Materialize vs Flink for streaming SQL on IoT telemetry: how streaming engines consume and emit Kafka topics.
- Apache Pulsar geo-replication for industrial IoT telemetry: an alternative log system and its delivery guarantees across regions.
- Kafka queues vs RabbitMQ vs SQS for task queues: when you need job-queue semantics rather than stream semantics.
- KIP-98: Exactly Once Delivery and Transactional Messaging and KIP-447: Producer scalability for exactly once semantics.
- Apache Kafka producer configuration reference and transaction protocol documentation.
By Riju — about
