Kafka Queues (KIP-932) vs RabbitMQ vs SQS: Picking a Task Queue After Kafka 4.2

Apache Kafka 4.2 shipped on 17 February 2026 and, with it, Queues for Kafka reached production-ready status. Within a week the internal Slack argument started at half the companies I know: we already run Kafka, can we finally delete RabbitMQ? The honest answer is that share groups close most of the functional gap and leave one specific hole that will decide the question for you. Kafka’s own KIP is blunt about it — KIP-932 “does not add the concept of a ‘queue’ to Kafka per se.” It adds cooperative consumption over a log, which is a different object with different failure modes. This post works through exactly where that difference bites, with a decision matrix, a worked capacity and cost model, and the redelivery semantics that determine whether a migration is boring or a six-month incident.

What this covers: the share-group mechanism, a ten-dimension decision matrix against RabbitMQ quorum queues and Amazon SQS, a 50-million-task-per-day capacity and cost walkthrough, and the failure modes that do not appear in any vendor comparison table.

Context and Background

For fifteen years the queue-versus-log distinction was a clean architectural fork. You used RabbitMQ, ActiveMQ or SQS when work items were independent, individually retryable and needed a pool of workers sized by latency rather than topology. You used Kafka when you needed ordered, replayable, multi-subscriber event history. Teams that needed both ran both, and paid the tax twice: two clusters, two client libraries, two on-call rotations, two sets of dashboards nobody quite trusted.

Kafka’s consumer group model is the reason. A partition is assigned to at most one consumer in a group, which gives you per-partition ordering and clean offset bookkeeping — but it hard-couples worker count to partition count. KIP-932 names the consequence directly: users “often have to ‘over-partition’ simply to ensure they can have sufficient parallel consumption to cope with peak loads.” Over-partitioning is not free. Every partition costs controller metadata, open file handles, produce-side batching efficiency and rebalance time. A topic with 600 partitions to support 600 seasonal workers is a topic you will regret in January.

Queues for Kafka breaks that coupling through a new construct called a share group. Multiple consumers can process records from the same partition concurrently, each record is acknowledged individually, and the broker counts delivery attempts so poison messages stop circulating. The feature landed as early access in Kafka 4.0 (March 2025), moved to preview in 4.1, and the 4.2.0 release announcement declares it production-ready. Kafka 4.3 followed on 22 May 2026 with KIP-1240, adding further share-group configuration knobs.

Meanwhile the incumbents did not stand still. RabbitMQ 4.0 removed classic queue mirroring entirely and made quorum queues the default choice for replicated durable queues, with a delivery limit now enabled by default. RabbitMQ 4.3 added delayed retry with linear back-off. Amazon SQS raised the maximum message size to 1 MiB and added fair queues to standard queues via MessageGroupId. If you last evaluated these systems in 2023, all three have moved. For the broader cluster-shape question, see our Kafka vs Redpanda vs WarpStream decision record.

What Queues for Kafka Actually Is

Queues for Kafka is a cooperative consumption mode, not a new storage primitive. A share group lets many consumers read the same partitions at once, each acquiring individual records under a time-limited lock, acknowledging them one by one, with the broker counting delivery attempts. The underlying topic remains an ordinary, replayable, retention-governed Kafka log. Nothing is removed on acknowledgement.

That last sentence is the whole argument in miniature, and it is worth sitting with before you plan a migration.

Figure 1: the share-partition record state machine defined by KIP-932.

Every record in a share-partition occupies one of four delivery states. Available records are eligible for delivery. When the share-partition leader hands a record to a consumer it moves to Acquired, bumps the delivery count, and starts an acquisition lock. From Acquired, the consumer can accept it (Acknowledged), release it back to Available for another attempt, reject it (straight to Archived), or simply do nothing and let the lock expire — which also returns it to Available. Records that exhaust their delivery budget land in Archived and are never redelivered to that group. Two pointers bound the in-flight window: the share-partition start offset and the share-partition end offset.

The acquisition lock is a visibility timeout with a different name

Leases are how Queues for Kafka guarantees progress without exclusive partition ownership. The default lock duration is 30 seconds, governed by the group configuration share.record.lock.duration.ms with a documented minimum of 1,000 ms. Functionally this is SQS’s visibility timeout: a lease that guarantees forward progress when a consumer dies mid-task, at the cost of duplicate delivery when a consumer is merely slow.

The failure mode is identical in all three systems and catches everyone at least once. If your p99 task takes 45 seconds and your lock is 30 seconds, roughly one task in a hundred gets executed twice — concurrently — while the first attempt is still running. Kafka 4.2 added a mitigation that SQS has had for years: KIP-1222 introduced a RENEW acknowledgement type, letting a consumer in explicit acknowledgement mode extend the lock on a record it is still working. It is enabled by default via share.renew.acknowledge.enable. SQS’s equivalent is ChangeMessageVisibility; RabbitMQ’s is per-consumer x-consumer-timeout tuning against a default consumer_timeout of 30 minutes.

Acknowledgement is per record, but the mode matters

Share consumers run in one of two acknowledgement modes, set by the client property share.acknowledgement.mode. The default is implicit: you call poll() and every delivered record is marked processed, with the commit flushed asynchronously on the next poll or synchronously via commitSync(). In explicit mode you must call acknowledge(record, type) for every record in the batch, choosing ACCEPT, RELEASE or REJECT, and the acknowledgements are only sent to the share-partition leader when you commit.

Explicit mode is the one that makes Queues for Kafka behave like a queue. Implicit mode acknowledges the whole batch on the next poll, which means a mid-batch crash re-delivers records you already processed — the classic at-least-once batch problem, not a per-record one. Teams migrating from RabbitMQ’s basic.ack(delivery_tag) expect per-record semantics and get batch semantics unless they explicitly opt in. Set share.acknowledgement.mode=explicit on day one; the ergonomic cost is small and the correctness difference is not.

Delivery counting replaces requeue-forever

Every acquisition increments a per-record delivery count. When it reaches share.delivery.count.limit — default 5, with a documented valid range starting at 2 — the record transitions to Archived rather than being delivered again. This is Kafka’s poison-message circuit breaker, and it is the same idea as RabbitMQ’s delivery-limit and SQS’s maxReceiveCount.

The critical difference is what happens next, and it is the single largest gap in Queues for Kafka as shipped. We will come back to it in the trade-offs section, because it deserves more than a bullet.

Taken together, those three mechanisms — lease, per-record acknowledgement, delivery counting — are the entirety of what Queues for Kafka adds. Everything else about the topic is unchanged Kafka.

The Decision Matrix

Below is the comparison across the dimensions that actually change an architecture. The Kafka column describes share groups on Apache Kafka 4.2 and 4.3; the RabbitMQ column describes quorum queues on RabbitMQ 4.x; the SQS column covers both queue types where they differ.

Dimension	Kafka share groups (4.2+)	RabbitMQ quorum queues	Amazon SQS
Delivery semantics	At-least-once, per-record acknowledgement. Transactional reads via `share.isolation.level`, default `read_uncommitted`	At-least-once with publisher confirms and consumer acks. At-least-once dead-lettering is opt-in	At-least-once on standard; exactly-once processing on FIFO within the 5-minute deduplication interval
Per-key ordering	None. KIP-932 states records “can be delivered out of order… in particular when redeliveries occur”. Only a single fetched batch is offset-ordered	FIFO per queue, but broken by redelivery and by any prefetch above 1. Single Active Consumer restores strict order at one consumer	Best-effort on standard; strict FIFO per `MessageGroupId` on FIFO queues
Consumers vs partitions	Decoupled. Consumer count may exceed partition count; assignment is cooperative via `SimpleAssignor`	Fully decoupled — consumers attach to a queue, not to a shard	Fully decoupled on standard. On FIFO, effective parallelism equals the number of distinct message groups
Redelivery / lease	Acquisition lock, `share.record.lock.duration.ms`, default 30 s, min 1 s. `RENEW` extends it (KIP-1222)	`consumer_timeout`, default 30 min; per-consumer `x-consumer-timeout`. Delayed retry with linear back-off from 4.3	Visibility timeout, default 30 s, max 12 h. `ChangeMessageVisibility` extends per message
DLQ story	No built-in DLQ. Records past the limit are Archived in place. KIP-1191 is Accepted but unshipped as of 4.3, gated on `share.version=2`	Mature. `delivery-limit` defaults to 20 since 4.0; overflow routes to a DLX, with at-least-once dead-lettering available	Mature. `RedrivePolicy` with `maxReceiveCount`; DLQ must match source queue type; console redrive built in
Max in-flight / prefetch	`share.partition.max.record.locks`, default 2,000 per share-partition, minimum 100. Fetch shaping via `ShareAcquireMode` (KIP-1206)	Per-consumer QoS prefetch. Global per-channel prefetch is not supported on quorum queues	~120,000 in-flight messages per queue for both standard and FIFO; `OverLimit` on short polling
Fan-out	Native and cheap. Many independent share groups read the same topic; each gets its own delivery state	Requires an exchange fan-out to N queues, so N copies of every message on disk	Requires SNS-to-SQS fan-out or duplicate publishes; one queue is one consumer population
Retention & replay	Full log replay. Retention is time or size based; `share.auto.offset.reset` defaults to `latest`, and a group’s start offset can be reset to a point in time	Destructive read — acknowledged messages are gone. Streams give replay, but streams are not queues	Destructive read. Retention 4 days by default, 60 s minimum, 14 days maximum. No replay after delete
Operational surface	Highest. Brokers, controllers, the `__share_group_state` internal topic, share coordinators, the `share.version` feature flag	Medium. Raft quorums per queue, memory and disk alarms, policy management, upgrade coordination	Lowest. No servers, no upgrades, no capacity planning beyond quota increases
Cost model	Infrastructure-priced: brokers, replicated disk, network, plus engineering time. Marginal cost near zero if the cluster exists	Infrastructure-priced: nodes, disk, ops. Fan-out multiplies storage	Per request. $0.40 per million standard, $0.50 per million FIFO, first million free each month, billed per 64 KB payload chunk

Two rows deserve emphasis because they are the ones people skip. The ordering row is not a footnote — it is the defining property of the mechanism. And the DLQ row describes a gap that exists today in released code, not a roadmap quibble.

It is also worth comparing share groups against the Kafka construct they sit beside, because the choice is frequently within Kafka rather than across vendors.

Property	Consumer group	Share group
Partition assignment	Exclusive, one consumer per partition per group	Shared, many consumers per partition
Max useful consumers	Partition count	Unbounded by partitions; bounded by record locks
Progress tracking	Single committed offset per partition	Per-record state persisted by the share coordinator
Ordering	Strict per partition	None across batches
Poison handling	Application’s problem	Delivery count and limit, enforced by the broker
Rebalance cost	Stop-the-world or cooperative reassignment	Assignment changes do not revoke in-flight locks

Choosing Between Them in Practice

Figure 2: a decision path that starts from ordering and replay requirements rather than from an existing vendor relationship.

The flowchart encodes a deliberate ordering of questions. Strict per-key ordering is the first gate because it eliminates share groups outright — no amount of configuration recovers it. Replay comes second because it is the only capability Kafka has that the queues structurally cannot offer. Existing operational footprint comes third, not first, which is where most real decisions go wrong.

When Queues for Kafka is the right answer

The archetype is a workload that is already produced to Kafka, whose work items are genuinely independent, and where worker count must scale far past a sensible partition count. Webhook delivery, document enrichment, thumbnail generation, LLM inference fan-out, batch notification sends. In each case the payload is already in a topic, another consumer group is probably doing analytics on the same data, and the queue-shaped consumer is one more reader.

The second archetype is subtler and more valuable: workloads where you want queue semantics and the ability to reprocess. Queues for Kafka persists per-record delivery state separately from the log, so you can create a second share group, reset its start offset to a timestamp, and reprocess two days of work items through a fixed worker while the original group keeps running. No queue system in this comparison can do that, because in a queue the acknowledged message is gone. When a bad deploy silently mis-processed 40 hours of tasks, this is the difference between a replay and a reconstruction project from your database’s audit tables.

When RabbitMQ still wins

Three situations, all concrete. First, when you need rich routing — topic exchanges, headers exchanges, per-message priority. Quorum queues implement 32 priority levels from 0 to 31, with unprioritised messages treated as priority 4. Kafka has no concept of priority at all, and Queues for Kafka does not add one; simulating priority means separate topics and separate groups, with all the starvation management that implies.

Second, when you need back-off between retries. RabbitMQ 4.3 added delayed retry on quorum queues with a linear formula, delay = min(min_delay × delivery_count, max_delay), configured through x-delayed-retry-type and friends. Queues for Kafka has no retry back-off: a released record becomes Available immediately and will be re-acquired as fast as consumers fetch. Building back-off on Kafka means either sleeping in the consumer — which burns a record lock — or hopping to a delay topic.

Third, when the workload is not already in Kafka. Standing up a Kafka cluster to get a work queue is a poor trade against a three-node RabbitMQ quorum. The operational surface row in the matrix is not decoration.

When SQS wins on arithmetic

SQS wins whenever the request volume is modest and engineering time is the scarce resource, which describes more systems than architects like to admit. There are no brokers, no upgrades, no rebalances, no disk alarms. The $0.40-per-million standard-queue price only becomes uncomfortable at sustained high volume, and long before that point the salary cost of a Kafka operator dominates.

SQS FIFO is also the only option in this comparison that gives you strict per-key ordering and elastic consumer scaling in one product. Each MessageGroupId is processed in order by exactly one consumer at a time, and parallelism comes from the number of distinct groups. Its cost is throughput shape: non-high-throughput FIFO is limited to 300 transactions per second per API action per partition, or 3,000 messages per second with ten-message batching. High-throughput mode raises that ceiling substantially — the AWS documentation quotes up to 70,000 non-batched TPS in the largest regions, and 2,400 TPS as the default elsewhere — but the regional variance is real and needs checking before you design around it.

A Worked Capacity and Cost Model

Assume a concrete workload. 50 million task messages per day, mean payload 8 KB, mean processing time 1.2 seconds with a p99 of 4 seconds, peak rate three times the daily mean. That is 579 messages per second on average and roughly 1,750 per second at peak.

In-flight capacity

The binding constraint for every one of these systems is Little’s Law applied to the lease window: sustained throughput cannot exceed in-flight capacity divided by processing time.

Kafka share groups. share.partition.max.record.locks defaults to 2,000 per share-partition. At a 4-second p99, a single partition sustains at most 2,000 ÷ 4 = 500 messages per second. To carry 1,750 per second you need at least four partitions of lock headroom, and you want margin, so eight to twelve partitions is the sensible range. Note what changed: you no longer size partitions to worker count, but you do still size them to in-flight record capacity. A twelve-partition topic gives 24,000 concurrent locks and a 6,000 messages-per-second ceiling. That is a far better scaling story than one-consumer-per-partition, but it is not the unbounded elasticity the marketing suggests.

RabbitMQ. In-flight is consumers × prefetch. Two hundred consumers at a prefetch of 20 gives 4,000 unacknowledged messages, so 1,000 per second at 4 seconds each — not enough. Raise prefetch to 50 and you get 10,000 in flight and 2,500 per second. Remember that quorum queues reject global per-channel QoS, so this must be per-consumer prefetch, and that high prefetch directly worsens ordering and memory behaviour on redelivery.

SQS. The in-flight ceiling is roughly 120,000 messages per queue, which at 4 seconds is a 30,000-per-second processing ceiling — an order of magnitude above this workload. Standard SQS is not the bottleneck here; your worker fleet is.

Storage

At 8 KB per message, 50 million messages per day is 400 GB of raw payload per day. With replication factor 3 and 7-day retention, a Kafka topic holding this workload needs about 8.4 TB of replicated disk before overheads — and that storage exists whether or not anyone replays it. This is the honest price of the replay capability in the matrix. Tiered storage under KIP-405 moves the cold portion to object storage and changes this calculation materially, which is worth modelling before you assume the disk bill is fixed.

RabbitMQ holds only unacknowledged and queued messages, so steady-state storage is a function of backlog, not of volume. SQS charges nothing for storage inside the retention window at all.

Request cost

SQS billing is per request, with each 64 KB chunk of payload counted as one request. Unbatched, each message costs one SendMessage, one ReceiveMessage and one DeleteMessage — three billable requests at 8 KB each. That is 150 million requests per day, 4.5 billion per month, and at $0.40 per million, roughly $1,800 per month.

Batch at ten and the picture changes. Ten 8 KB messages is 80 KB, which bills as two chunks: two requests to send, two to receive, and one for the delete batch, whose payload is only receipt handles. Five billable requests per ten messages is 25 million per day, 750 million per month, about $300 per month. Batching is not a micro-optimisation here; it is a six-fold cost reduction, and it is the first thing to check in any SQS bill that looks wrong.

Add idle polling. One hundred long-polling workers at a 20-second wait issue three ReceiveMessage calls per minute each when the queue is empty: 432,000 requests per day, around 13 million per month, roughly $5. Negligible — but only because long polling is switched on. At the default WaitTimeSeconds of zero, the same fleet can generate hundreds of millions of empty receives.

The break-even that actually matters

A Kafka cluster sized for this workload — say six brokers with 8 vCPUs each plus about 9 TB of replicated block storage — will not come in under $300 a month at any cloud provider’s pricing. I am deliberately not quoting instance prices, because they move and vary by region, but the conclusion is robust to the input: if you do not already run Kafka, Queues for Kafka is not the cheap option for this workload. It is not close.

If you do already run Kafka and the data is already in a topic, the marginal cost of adding a share group is the share-coordinator state topic and some broker CPU. Then it is close to free, and the comparison flips entirely. This is why the decision is so often about existing footprint — but that should be the conclusion of the analysis, not its premise.

Where the Log-Versus-Queue Distinction Bites

Figure 3: the redelivery path that breaks offset monotonicity in a share group.

KIP-932 documents this scenario explicitly, and it is worth tracing because it dissolves a whole class of assumptions. Two consumers share a single-partition topic. Consumer A fetches offsets 100 to 109 and crashes. Consumer B fetches, processes and acknowledges 110 to 119. When B fetches again it receives 100 to 109 with delivery count 2. The work all completes, but B observed offsets in the order 110–119 then 100–109.

Three consequences follow, and each has bitten a real system.

Consumer-side ordering logic is invalid. Any code that assumes monotonically increasing offsets — deduplication by “highest offset seen”, watermarking, late-arrival detection — is wrong in a share group. The KIP’s guarantee is narrow: records in a single fetched batch for a share-partition arrive in increasing offset order, with no guarantee between batches.

Compacted topics and share groups interact badly. If the topic is compacted, a record acquired but not yet acknowledged can be removed by compaction before redelivery. The share-partition leader does not maintain a cache of fetched records and may have to re-fetch for redelivery. Treat compacted topics as unsupported for task-queue semantics until you have tested the interaction yourself.

Transactional producers need a decision. share.isolation.level defaults to read_uncommitted, which means a share group will deliver records from aborted transactions unless you change it. If any producer to the topic uses transactions, set it to read_committed at group creation. This is a group-level setting, not per consumer — you cannot have one consumer in the group reading committed data and another reading everything.

The operational surface nobody mentions

Queues for Kafka introduces a new internal topic, __share_group_state, created automatically on first use, and a share coordinator alongside the existing group coordinator. Like Kafka’s other internal topics it defaults to three replicas. On a cluster with fewer than three brokers you must set share.coordinator.state.topic.replication.factor and share.coordinator.state.topic.min.isr to 1 before first use — a footgun that reliably bites development and staging environments.

The feature itself is gated behind the share.version cluster feature, finalised with kafka-features.sh. That gate is a genuine operational benefit: you can upgrade brokers to 4.2 and enable share groups later, and the feature-level metrics added by KIP-1180 let you see exactly where every node sits.

Kafka 4.2 also shipped the observability that a queue workload actually needs. KIP-1226 added share-partition lag persistence and retrieval, so you can finally answer “how far behind is this worker pool” the way you would with ApproximateNumberOfMessagesVisible on SQS or queue depth in RabbitMQ. Before 4.2 that metric did not exist, which made share groups very hard to operate seriously. KIP-1206 added ShareAcquireMode, offering batch_optimized as a soft limit on fetched records and record_limit for strict enforcement — the latter matters when per-record processing is expensive and you need hard control over in-flight work. KIP-1224 removed the 5 ms latency floor in the share coordinator through adaptive batching.

Trade-offs, Gotchas, and What Goes Wrong

Figure 4: what happens to a record that exhausts its delivery budget, in each system.

The missing dead-letter queue is the headline gap in Queues for Kafka. When a share-group record hits share.delivery.count.limit, it transitions to Archived. Nothing is written anywhere. The record still sits in the log at its original offset, but no consumer in that share group will ever see it again, and nothing in the group’s state tells an operator which records were abandoned. KIP-932 provides the circuit breaker without the diagnostic.

KIP-1191 fixes this properly — a group-level errors.deadletterqueue.topic.name, an Archiving state, context headers under __dlq.errors.*, and broker metrics such as DeadLetterQueueRecordCount. Its status on the Apache wiki is Accepted, gated behind share.version=2, and it did not ship in 4.2 or in 4.3. Until it does, a production share group needs application-level poison handling: catch the failure, REJECT the record, and produce it yourself to a dlq. topic with enough context to reconstruct the failure. That is exactly the code both RabbitMQ and SQS let you delete, and it is the strongest single argument for not migrating a queue workload to Kafka today.

Pathological cases in delivery counting. The delivery count increments on acquisition, not on failure. A consumer that fetches, holds a lock, and is killed by an orchestrator for an unrelated reason burns an attempt. With the default limit of 5, five unlucky pod evictions retire a perfectly good task permanently. RabbitMQ’s default of 20 is far more forgiving, and its delivery count only increments on genuine failures. If you deploy frequently on Kubernetes, raise share.delivery.count.limit above 5 and make your shutdown path release in-flight records explicitly.

Record lock exhaustion looks like a stall, not an error. When a share-partition hits share.partition.max.record.locks, fetches return no records until locks release naturally. There is no exception and no obvious signal on the client. A slow downstream dependency causes consumers to hold locks longer, locks hit the cap, fetches go empty, and the pool looks idle while the backlog grows. Alert on share-partition lag rising while fetch rate falls — that combination is the fingerprint.

Prefetch is an ordering decision in RabbitMQ. Any prefetch above 1 means several messages are outstanding per consumer, so a redelivery after a crash arrives behind messages published later. Quorum queues will also refuse a global per-channel QoS setting outright, which breaks older client code silently on migration from classic queues.

SQS DLQ type matching is a deploy-time failure. A FIFO queue’s dead-letter queue must itself be FIFO, and a standard queue’s must be standard. Mismatches fail at configuration time — cheap to fix, but a recurring cause of failed infrastructure-as-code deploys.

The message-size cliff. SQS now accepts 1 MiB directly, but billing is per 64 KB chunk: a 1 MiB message bills as 16 requests per API action. A payload-size regression can multiply an SQS bill by an order of magnitude with no change in message rate. Put large payloads in object storage and pass references — in every one of these systems.

Practical Recommendations

Start from the ordering requirement, because it is the only property that cannot be configured back in. If any consumer depends on per-key order, share groups are out; choose keyed Kafka consumer groups, SQS FIFO, or RabbitMQ with Single Active Consumer, and accept the throughput ceiling that comes with each.

If ordering is genuinely not required, ask whether you need to reprocess consumed work. That is the one capability Queues for Kafka has and the queues do not, and it is worth real money in regulated or reconciliation-heavy domains. If you need it and already run Kafka, Queues for Kafka is a strong choice — with the DLQ caveat handled in application code.

If you need neither ordering nor replay, pick on operational cost, and be honest about what a broker fleet costs in engineering attention rather than instance-hours. Most teams that do this arithmetic properly end up on SQS or a small RabbitMQ cluster, and are right to. Our NATS JetStream versus Kafka comparison for edge telemetry applies the same reasoning to a different constraint set.

Before running a share group in production:

Set share.acknowledgement.mode=explicit and acknowledge per record.
Size share.record.lock.duration.ms to p99 processing time plus headroom, and use RENEW for genuinely long tasks.
Raise share.delivery.count.limit above the default of 5 if your platform evicts pods frequently.
Verify partition count against share.partition.max.record.locks divided by processing time, not against worker count.
Set share.isolation.level=read_committed if any producer to the topic is transactional.
Build application-level dead-lettering now; adopt KIP-1191 when it ships behind share.version=2.
Pre-create __share_group_state settings on clusters with fewer than three brokers.
Alert on share-partition lag from KIP-1226, cross-referenced with fetch rate.
Never run share groups against a compacted topic without testing redelivery after compaction.

Frequently Asked Questions

Is Kafka a message queue now?

Not in the strict sense. Queues for Kafka, generally available since 4.2, adds queue-like consumption — per-record acknowledgement, delivery counting, consumers decoupled from partitions — on top of an append-only log. KIP-932 is explicit that it “does not add the concept of a queue to Kafka per se”. Messages are not removed on acknowledgement, there is no maximum queue depth, and there is no ordering guarantee across fetched batches. It is closest to a durable shared subscription.

No. KIP-932 states that records in a share-partition “can be delivered out of order to a consumer, in particular when redeliveries occur”. The only guarantee is that records within a single fetched batch for one share-partition arrive in increasing offset order. If you need per-key ordering, use a Kafka consumer group with keyed partitions, SQS FIFO with MessageGroupId, or RabbitMQ with Single Active Consumer instead.

Yes — that is the core motivation for KIP-932. A share group’s consumer count may exceed the partition count, because partitions are shared rather than exclusively assigned. Practical parallelism is still bounded, though, by share.partition.max.record.locks (default 2,000 per share-partition) divided by your processing time. Size partitions for in-flight record capacity rather than for worker count.

No — Queues for Kafka ships without one in Apache Kafka 4.2 or 4.3. A record that hits share.delivery.count.limit moves to the Archived state with nothing written anywhere. KIP-1191 adds a proper DLQ topic with context headers and broker metrics; its wiki status is Accepted and it is gated behind the share.version=2 feature level, but it has not shipped in a release. Until then, reject failing records explicitly and produce them to your own DLQ topic.

Should I replace RabbitMQ with Queues for Kafka?

Only if the data already flows through Kafka, ordering is not required, and you value replay. Keep RabbitMQ when you need message priorities (quorum queues offer 32 levels), retry back-off (delayed retry landed in 4.3), rich exchange-based routing, or a built-in dead-letter path today. Standing up a Kafka cluster purely to obtain a work queue is rarely a good trade against a three-node quorum.

How does the acquisition lock compare to an SQS visibility timeout?

They are the same mechanism with different defaults and ceilings. Kafka’s lock defaults to 30 seconds with a 1-second minimum and is extended with the RENEW acknowledgement type added in KIP-1222. SQS’s visibility timeout also defaults to 30 seconds, ranges from 0 seconds to 12 hours, and is extended with ChangeMessageVisibility. Both exist so a crashed consumer cannot strand work, and both cause duplicate processing when a consumer is slow rather than dead.

Kafka Queues (KIP-932) vs RabbitMQ vs SQS: Picking a Task Queue After Kafka 4.2

Kafka Queues (KIP-932) vs RabbitMQ vs SQS: Picking a Task Queue After Kafka 4.2

Context and Background

What Queues for Kafka Actually Is

The acquisition lock is a visibility timeout with a different name

Acknowledgement is per record, but the mode matters

Delivery counting replaces requeue-forever

The Decision Matrix

Choosing Between Them in Practice