Transactional Outbox Pattern: Reliable Event Publishing with CDC

Every team that builds event-driven services eventually ships the same bug. A handler writes a row to its database, then publishes an event to Kafka, and for months it works. Then a deploy lands between the two steps, or the broker blips, and the order exists but nobody downstream ever hears about it. This is the dual write problem, and no amount of retry logic inside the handler fixes it, because the two systems cannot commit together.

The transactional outbox pattern removes the second write from the request path. The service records the event in an outbox table inside the same local transaction as the business change, and a separate relay moves it to the broker. What you get is atomic intent and at-least-once delivery. What you must still build is consumer-side deduplication, because “exactly-once” is a property you assemble, not one you buy.

You will leave with the failure analysis of dual writes, a production-grade outbox schema, two relay designs (polling and log-based CDC), working Debezium configuration, an inbox table for idempotent consumers, and an honest list of what goes wrong.

What this covers: the dual write problem, outbox table design, polling versus Debezium relays, ordering and partitioning, idempotent consumers, Kafka transactions and the real meaning of exactly-once, failure modes such as stuck replication slots, and when to skip the pattern entirely.

Context and Background

Microservices split state across databases, and events are how those databases stay loosely consistent. A service owning orders emits OrderPlaced; billing, shipping and analytics react. The architecture is sound, but it depends on a guarantee that is easy to state and hard to deliver: if the order commits, the event must eventually be published, and if the order rolls back, the event must never be published.

Distributed transactions across a database and a message broker would give that guarantee directly. In practice they are not available. Kafka does not participate in XA two-phase commit, and the Debezium team’s own write-up of the pattern states this as the root cause of the dual write problem. Even where a broker offers XA, it couples availability of the service to availability of both systems and adds lock-holding latency that most teams reject on throughput grounds.

The accepted answer, popularised by Chris Richardson’s microservices.io catalogue and documented in depth on the Debezium blog, is to change only one resource synchronously. The database is the system of record, so the event becomes data in that database. A relay then does the asynchronous second hop. You can read the Debezium team’s explanation in Reliable Microservices Data Exchange With the Outbox Pattern.

This pattern sits next to several neighbours that readers of this site will know. Change data capture with Debezium is the transport most production outboxes use. Event sourcing makes events the primary state, which removes the need for an outbox but demands a different storage model. And the saga pattern relies on exactly the reliable publication the outbox supplies: a saga step that cannot reliably emit its next command is not a saga, it is a hope.

A word on terminology. “Outbox” is the table; “relay” (also called message relay, publisher or forwarder) is whatever moves rows to the broker; “inbox” is the consumer-side table recording processed event ids. Keeping those three nouns distinct avoids most design-review confusion.

Why the Dual Write Problem Cannot Be Retried Away

A dual write is any operation that must update two independent systems with no shared transaction coordinator. The classic form is db.commit(); producer.send(). There are three orderings and each one fails differently.

If you publish first and commit second, a rollback leaves a phantom event: consumers act on an order that never existed. If you commit first and publish second, a crash or broker outage leaves a lost event: the order exists and nobody knows. If you wrap both in the same handler and retry the publish on failure, the retry happens in a process that may itself die, so the guarantee is only as good as process uptime.

Figure 1: The dual write problem. Between the database commit and the broker acknowledgement there is a window in which a crash, deploy or network partition silently loses the event.

Figure 1 shows the loss window. It is narrow, which is exactly why it survives testing. Suppose the window is five milliseconds of a fifty-millisecond request, and your service handles one million writes a day with a monthly pod restart rate that happens to land mid-request for 0.01 percent of requests. That is about a hundred inconsistent records per day. These numbers are illustrative, but the shape is real: rare per request, constant at scale, and almost impossible to reproduce on a developer machine.

Retrying does not close the window because the retry state lives in memory. Persisting the retry intent to disk is precisely the outbox. The pattern is therefore not an exotic trick; it is the observation that a durable queue of “things I still owe the broker” must live somewhere, and the only place that is atomic with the business write is the business database.

Why Two-Phase Commit Is Not the Answer

XA transactions across a relational database and a broker are theoretically possible on some platforms. They require a transaction manager, recovery logs on both sides, and heuristic-outcome handling when the coordinator dies between prepare and commit. Prepared transactions hold row locks for the duration of the coordinator’s slowest participant. Cloud-managed Kafka services do not expose XA at all. Most teams that evaluate it conclude, correctly, that they are trading a rare inconsistency for a common availability problem.

What the Outbox Actually Guarantees

State the guarantee precisely, because the rest of the design follows from it. The outbox provides: an event row exists if and only if the business change committed. It does not provide that the event reaches the broker exactly once, nor that consumers process it exactly once. It provides at-least-once publication, in per-aggregate commit order if you design the key correctly. Everything about exactly-once effects is built downstream.

Reference Architecture: Outbox Table Plus Log-Based CDC

Direct answer: In the transactional outbox pattern, a service writes its business row and an event row into an outbox table in one ACID transaction. A relay, usually Debezium reading the database write-ahead log, publishes each outbox row to Kafka at least once. Consumers deduplicate by event id, producing effectively-once processing.

Figure 2: Reference architecture for the transactional outbox pattern with Debezium. The only synchronous dependency of the request path is the local database.

Figure 2 shows the reference topology. The request path touches one system. The Postgres write-ahead log (WAL) is the durable, ordered record of every committed change. Debezium attaches through a logical replication slot, decodes the outbox inserts, applies the Outbox Event Router single message transform (SMT), and produces to Kafka. The service never talks to the broker, so a broker outage degrades freshness, not correctness or request availability.

Designing the Outbox Table

The Debezium documentation for the Outbox Event Router lists a default table shape: id (uuid, not null), aggregatetype (varchar 255, not null), aggregateid (varchar 255, not null), type (varchar 255, not null) and payload (jsonb, nullable). Those defaults are a good starting point because the router’s option defaults line up with them: table.field.event.id=id, table.field.event.key=aggregateid, table.field.event.payload=payload, and route.by.field=aggregatetype. Each column does a specific job.

id is the idempotency key. It travels to consumers as a Kafka header and is the primary key of their inbox. Generate it in the application, ideally as a UUIDv7 so it is roughly time-ordered and index-friendly.
aggregatetype names the entity class (order, shipment) and, by default, drives topic routing.
aggregateid is the Kafka message key. All events for one aggregate hash to one partition, which is the only ordering guarantee Kafka gives you.
type is the event name (OrderPlaced), carried so consumers can dispatch without parsing the payload.
payload is the event body. Treat it as a public contract, not a database row dump.

Beyond the defaults, add a created_at timestamptz not null default now() column for the polling relay, lag metrics and retention, and optionally a trace context column. The router supports tracing through a field named tracingspancontext by default, configurable with tracing.span.context.field, so a span started in the request can continue through Kafka.

CREATE TABLE outbox (
  id            uuid        PRIMARY KEY,
  aggregatetype varchar(255) NOT NULL,
  aggregateid   varchar(255) NOT NULL,
  type          varchar(255) NOT NULL,
  payload       jsonb,
  created_at    timestamptz  NOT NULL DEFAULT now()
);

-- Business change and event in ONE transaction
BEGIN;
  INSERT INTO orders (id, customer_id, total_cents, status)
  VALUES ('5f6e...', 'c-77', 12900, 'PLACED');

  INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
  VALUES (gen_random_uuid(), 'order', '5f6e...', 'OrderPlaced',
          '{"orderId":"5f6e...","totalCents":12900,"currency":"EUR"}');
COMMIT;

The Payload Is an API, Not a Row

The most common design mistake is serialising the internal entity into the payload. The moment you do, every column rename in the service database becomes a breaking change for unknown consumers. Define the event as its own versioned schema, put a schema version in the envelope (the router can set the Kafka Connect schema version from a column via table.field.event.schema.version), and evolve it with additive changes only. Use a schema registry with compatibility checking where you can, because the outbox row is written long before a consumer reads it, and the two may be deployed weeks apart.

A related decision is thin versus fat events. A thin event carries ids and forces consumers to call back; a fat event carries the state consumers need. Callbacks reintroduce the synchronous coupling the architecture was avoiding and can read state newer than the event. Fat events cost payload size but keep consumers autonomous. The usual compromise is event-carried state transfer for the fields that define the business fact, and ids for everything else.

Partitioning and Ordering Keys

Ordering is where the pattern is most often oversold. The relay can only preserve the order in which rows were committed. Kafka only preserves order within a partition. So the sequence you can promise is: events for the same aggregateid appear in commit order in one partition. Across aggregates there is no order, and across topics there is none either.

Two traps follow. First, if two transactions update the same aggregate concurrently, commit order is decided by the database, not by event timestamps; use optimistic locking on the aggregate row so concurrent writers serialise. Second, if you route by aggregatetype into one topic with a coarse key, such as a tenant id, you create hot partitions. Choose the finest key that still contains the ordering you actually need.

Relay Options: Polling Publisher Versus Log Tailing

Two relay families exist. Both achieve at-least-once publication. They differ in latency, database load, operational burden and failure surface.

Polling Publisher

A polling publisher is a scheduled loop in your application or a sidecar. It selects unsent rows, publishes them, and marks or deletes them. The correct query uses row locking so multiple instances do not collide.

-- Claim a batch without blocking other relays
WITH batch AS (
  SELECT id FROM outbox
  WHERE published_at IS NULL
  ORDER BY created_at, id
  LIMIT 200
  FOR UPDATE SKIP LOCKED
)
UPDATE outbox o SET published_at = now()
FROM batch WHERE o.id = batch.id
RETURNING o.id, o.aggregatetype, o.aggregateid, o.type, o.payload;

This needs a published_at column. SKIP LOCKED is the key: it lets competing relays take disjoint batches. The cost is that disjoint batches destroy ordering, because relay B may publish aggregate X’s second event while relay A still holds its first. The remedies are a single relay instance, sharding relays by hash of aggregateid, or ordering by (aggregateid, created_at) within a claim. Polling also marks rows after sending, so a crash between send and mark republishes. That is acceptable, because consumers deduplicate, but it must be a conscious choice.

Polling’s strengths are simplicity and portability: it needs no database replication privileges and works on managed databases that restrict logical decoding. Its weaknesses are latency bounded by the poll interval, read load on the primary that scales with outbox size, and index bloat from the update-then-delete churn.

Log-Based CDC with Debezium

Log tailing reads the commit log instead of querying the table. For Postgres that is logical decoding through a replication slot; for MySQL it is the binlog. Debezium reads the log, emits a change event per row insert, and the Outbox Event Router transforms each into a business event. There is no polling query, no marker column and no ordering ambiguity: the log is already a total order of commits, and Debezium preserves it per source.

This is the design the Debezium team describes, and it has a trick worth understanding. Because the connector reads the log rather than the table, the application can insert the outbox row and delete it in the same transaction. The log still contains the INSERT, which Debezium forwards; the connector ignores the DELETE. The table stays empty, so there is no growth and no cleanup job. The router also filters deletes automatically, and by default warns on updates through table.op.invalid.behavior (warn, error or fatal), because outbox rows are meant to be immutable.

The cost is operational. You now run Kafka Connect, manage a replication slot on the primary, and own the failure modes in the next sections. For teams already running Debezium for other purposes, as covered in our CDC streaming architecture guide, the marginal cost is low. For a team with one service and no Kafka Connect, polling is often the honest choice.

Comparison

Dimension	Polling publisher	Debezium log tailing
Typical latency	Poll interval plus processing, often 100 ms to seconds (estimate)	Tens to low hundreds of ms in a healthy pipeline (estimate)
Load on primary	Repeated SELECT and UPDATE on outbox	Reads WAL stream; one slot per connector
Ordering	Needs sharding or single relay	Commit order preserved per source
Cleanup	Required (delete or partition drop)	Optional if insert-then-delete
Database privileges	Ordinary read/write	Replication role, logical decoding enabled
Biggest risk	Duplicates, lost ordering, bloat	Stuck slot fills disk, connector ops
Works on restricted managed DBs	Yes	Depends on provider support

The latency figures are estimates for orientation, not measurements; benchmark on your hardware.

Configuring the Outbox Event Router

A minimal Postgres connector, expressed as Kafka Connect JSON, ties the pieces together. Property names below for the router come from the Debezium reference; the Postgres connector keys are standard Debezium options, and you should confirm exact names against the documentation for your Debezium version before deploying.

{
  "name": "orders-outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "orders-db",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${file:/secrets/db.properties:password}",
    "database.dbname": "orders",
    "topic.prefix": "orders",
    "slot.name": "orders_outbox_slot",
    "publication.name": "orders_outbox_pub",
    "table.include.list": "public.outbox",
    "heartbeat.interval.ms": "10000",

    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.id": "id",
    "transforms.outbox.table.field.event.key": "aggregateid",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.table.expand.json.payload": "true",
    "transforms.outbox.table.fields.additional.placement": "type:header:eventType",
    "transforms.outbox.route.by.field": "aggregatetype",
    "transforms.outbox.route.topic.replacement": "outbox.event.${routedByValue}",

    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

Several choices here are deliberate. table.include.list restricts capture to the outbox so business tables never leak to Kafka. table.expand.json.payload=true turns the jsonb string into real JSON and requires a JSON-capable value converter, as the docs note. The additional-field syntax is column:placement[:alias] with placement header or envelope, so type:header:eventType publishes the event name as a header. The routing defaults (route.by.field=aggregatetype, replacement outbox.event.${routedByValue}) already produce topics such as outbox.event.order, which I have spelled out for clarity. Using the Postgres built-in pgoutput plug-in avoids installing server-side libraries; the Debezium docs describe it as always present in PostgreSQL 10 and later.

The Debezium documentation states that the SMT should apply only to outbox events, via a predicate or route.topic.regex, and that multiple outbox tables are supported only when they share the same structure. If the connector captures other tables, add a predicate so the router does not try to interpret them.

The Publish Side in Application Code

Application code should make the outbox write hard to forget. A thin repository method that accepts the domain event and writes it inside the ambient transaction does the job. Here is a Java sketch with Spring and JDBC.

@Service
public class OrderService {
  private final JdbcTemplate jdbc;
  private final ObjectMapper json;

  @Transactional
  public void placeOrder(Order o) throws JsonProcessingException {
    jdbc.update("INSERT INTO orders(id, customer_id, total_cents, status) VALUES (?,?,?,?)",
        o.id(), o.customerId(), o.totalCents(), "PLACED");

    var event = new OrderPlaced(o.id(), o.totalCents(), "EUR");
    jdbc.update("""
        INSERT INTO outbox(id, aggregatetype, aggregateid, type, payload)
        VALUES (?,?,?,?,?::jsonb)""",
        UUID.randomUUID(), "order", o.id().toString(),
        "OrderPlaced", json.writeValueAsString(event));
  }
}

The annotation is the entire point: both statements join one transaction, so they commit or roll back together. A unit test that forces the second insert to fail and asserts the first was rolled back is cheap insurance against someone later moving the outbox write outside the transaction boundary.

Deeper Analysis: Idempotent Consumers and What Exactly-Once Really Means

Once the relay delivers at least once, the burden shifts to consumers. The outbox cannot prevent duplicates, and it should not try; it makes loss impossible, and consumers make duplication harmless. That division of labour is the core insight of the whole design.

Where Duplicates Come From

Duplicates are not an edge case, they are a routine event with at least four independent sources. The Debezium connector resumes from its last committed offset after a restart, so any events produced but not yet recorded are produced again. The Postgres documentation says a logical slot persists its position only at checkpoint, so after a crash it may return to an earlier position and resend recent changes, and that clients must handle duplicates. A producer retry after a lost acknowledgement can double-write unless the idempotent producer is enabled. And a consumer that processes a record then crashes before committing its offset will see the record again on restart.

Any one of these is enough. A consumer design that assumes unique delivery is simply wrong, and the failure will appear during the first rolling restart.

The Inbox Table

The robust consumer-side counterpart is an inbox (also called a processed-messages or message-log table). Before applying an event the consumer checks whether the event id has been seen. Crucially, it records the id in the same local transaction as the business effect. If the transaction rolls back, the id is un-recorded and a retry will reprocess; if it commits, the id is durable and any redelivery is skipped. The Debezium write-up describes this same approach, carrying the UUID as a Kafka header and checking a local message log.

Figure 3: Idempotent consumer with an inbox table. The inbox insert and the business change commit atomically, so a crash before the offset commit only causes a harmless skip on redelivery.

CREATE TABLE inbox (
  event_id    uuid        PRIMARY KEY,
  consumer    text        NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now()
);

A compact Python consumer using confluent-kafka and psycopg shows the control flow. The unique primary key does the real work: a concurrent duplicate cannot slip through a check-then-insert race, because the insert itself is the check.

from confluent_kafka import Consumer
import psycopg, json

c = Consumer({"bootstrap.servers": "kafka:9092",
              "group.id": "billing",
              "enable.auto.commit": False,
              "auto.offset.reset": "earliest"})
c.subscribe(["outbox.event.order"])

def header(msg, name):
    return dict(msg.headers() or []).get(name, b"").decode()

with psycopg.connect("dbname=billing") as db:
    while True:
        msg = c.poll(1.0)
        if msg is None or msg.error():
            continue
        event_id = header(msg, "id")
        body = json.loads(msg.value())
        with db.transaction():
            cur = db.execute(
                "INSERT INTO inbox(event_id, consumer) VALUES (%s, 'billing') "
                "ON CONFLICT (event_id) DO NOTHING", (event_id,))
            if cur.rowcount == 1:          # first time we see this event
                db.execute("INSERT INTO invoices(order_id, amount_cents) "
                           "VALUES (%s, %s)", (body["orderId"], body["totalCents"]))
        c.commit(message=msg, asynchronous=False)   # offset after DB commit

Note the order: database commit first, offset commit second. A crash between them redelivers the message, the ON CONFLICT hits, rowcount is zero, and the handler does nothing. Reversing the order would convert duplicates into losses, which is strictly worse. The header named id is what the router emits for table.field.event.id; verify the header name in your deployment.

Natural Idempotency Versus the Inbox

An inbox is general but adds a write per event and a table to prune. Where the effect is naturally idempotent, you can skip it. Setting a customer’s status to a value is idempotent; incrementing a counter is not. Upserting by a business key is idempotent; appending to a ledger is not. Prefer natural idempotency wherever the operation allows, and fall back to the inbox for the rest. Our guide to idempotent API design covers the same reasoning at the HTTP boundary: an idempotency key is the same concept as the event id, applied one hop earlier.

Prune inbox rows after a retention window longer than the maximum redelivery horizon. If Kafka retains topics for seven days and a consumer group can rewind, a thirty-day inbox is a defensible default. Rewinding beyond the inbox window reprocesses events, which is a decision to make with eyes open.

Kafka Transactions and Exactly-Once

Teams hear “Kafka supports exactly-once” and conclude the problem is solved. It is solved inside a narrow boundary. Kafka’s idempotent producer, which assigns a producer id and per-partition sequence numbers so the broker can discard retried duplicates, closes the producer-retry source of duplicates within one partition and session. Kafka transactions extend this so a consumer-process-produce loop can atomically commit output records and consumed offsets together, and consumers set to isolation.level=read_committed see only committed transactional output. These behaviours were introduced by KIP-98 and are described in the Kafka design documentation; I did not re-verify the current default values for this article, so check the client documentation for your version.

The boundary is the key. Those guarantees cover Kafka-to-Kafka pipelines: read from a topic, write to a topic, commit offsets, all in one Kafka transaction. The moment the consumer’s effect is a write to an external database, an HTTP call or an email, the Kafka transaction cannot include it. The consumer must make the external effect idempotent itself, which is exactly what the inbox does. Debezium’s own connectors deliver at least once; the transactional outbox does not change that, it makes it safe.

So what does “exactly-once delivery” mean in a real system? It is a marketing phrase for exactly-once processing effects, achieved as at-least-once delivery plus idempotent handling. The honest sentence for a design review is: “The pipeline delivers at least once and the effects are applied once.” If someone claims the transport itself delivers each message exactly once across a network and a crash, ask where the deduplication state lives.

Combining the Outbox with Kafka Transactions

There is one place Kafka transactions pair naturally with the outbox: stream processors that consume outbox topics and produce derived topics. In a Kafka Streams or transactional-producer service, enabling exactly-once processing gives you atomic consume-transform-produce with no inbox needed, since the output and the offsets are in one Kafka transaction. Use that for topic-to-topic enrichment. Use the inbox for topic-to-database consumers. Mixing the two models in a single consumer is where subtle bugs live.

Worked Numbers: Latency and Cost (Estimates)

These figures are illustrative planning numbers, not benchmarks. Assume an order service at 500 writes per second, each event payload about 1 KB. The outbox adds one insert per transaction, roughly 500 inserts per second and about 43 GB of WAL-visible insert data per day before compression if you keep rows for a day (500 × 1 KB × 86,400 s ≈ 43 GB). With insert-then-delete the table stays tiny, but the WAL still records both operations, so WAL volume roughly doubles for that table’s traffic.

On latency, the end-to-end path has four hops: commit, WAL decode, Connect transform and produce, consumer poll. A healthy pipeline may deliver in the order of tens to a couple of hundred milliseconds, while a polling relay at a one-second interval adds half a second on average. Measure your own with an event timestamp on the outbox row and a consumer that records now() - created_at; that single metric, end-to-end outbox lag, is the most useful health signal you can have.

Trade-offs, Gotchas, and What Goes Wrong

The pattern removes one failure class and introduces several operational ones. Know them before you adopt it.

Figure 4: The most dangerous outbox failure. A stopped connector leaves its replication slot pinned, WAL accumulates, and the primary’s disk fills. Heartbeats, a retention cap and slot lag alerts break the chain.

Stuck Replication Slots and WAL Bloat

This is the incident that pages people at night. The PostgreSQL documentation is blunt: replication slots persist across crashes, know nothing about the state of their consumers, and prevent removal of required WAL even when no connection is using them. Neither required WAL nor required catalog rows can be vacuumed away while a slot needs them, and in extreme cases the database can shut down to prevent transaction ID wraparound. The documented remedy is that an unneeded slot should be dropped.

For an outbox the risk is real in both directions. If Debezium stops, WAL accumulates. And on a low-traffic database whose captured tables rarely change, the slot may not advance even while the connector is healthy, because changes in other tables generate WAL the slot must still retain. Debezium offers heartbeat events (heartbeat.interval.ms, with an optional heartbeat.action.query that writes to a small table) so the connector acknowledges progress regularly; confirm the details in the connector’s “WAL disk space consumption” section for your version. On Postgres 13 and later, max_slot_wal_keep_size caps retention, at the cost of invalidating a slot that falls too far behind, which then forces a resnapshot. Choose that trade deliberately and alert well before the cap.

-- Monitor retained WAL per slot
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed
FROM pg_replication_slots;

Poison Events

An event that a consumer cannot process, because of a malformed payload or a bug, blocks its partition if the consumer retries forever. Route such records to a dead-letter topic after bounded retries and alert on it. Resist skipping silently: in an outbox world each event is a business fact, and losing one reintroduces the exact inconsistency you built the pattern to avoid. Pair the dead-letter queue with a replay tool.

Schema Evolution

Events outlive deployments. A consumer on version one will read events written by version three. Additive, optional fields keep compatibility; renames and type changes break it. Enforce compatibility in CI against the registry, version the event type name when a change is genuinely breaking, and run old and new consumers in parallel until traffic drains. Also remember that outbox rows already in flight were serialised under the old schema.

Relay Crashes and Resnapshots

When the Debezium connector restarts it resumes from its stored offset, which produces duplicates by design. Worse, if the slot was dropped or invalidated, a snapshot can replay the table. Because the outbox is usually empty or short, a snapshot of it is cheap, but the events it emits may already have been delivered. This is why idempotent consumers are non-optional rather than a nice-to-have.

Other Gotchas

Large payloads inflate WAL and Kafka message size; keep events to a few kilobytes and reference blobs externally. Long-running transactions delay when their outbox rows become visible in the log, so events can appear late though still in commit order. Multi-tenant databases with a shared outbox need a tenant column and care with keys. Finally, the outbox adds a write to every transaction, which on very hot paths is measurable; test it.

When Not to Use the Pattern

Skip the outbox when a lost or duplicate event is cheap, such as telemetry, click streams and cache invalidations where a periodic reconciliation exists. Skip it when you adopt true event sourcing, where the event store is already the source of truth. Skip it when the producer has no transactional database, for example a stateless gateway. And skip the CDC relay, but not the pattern, when your platform forbids logical replication: a polling publisher on the same table gives the same atomicity guarantee with higher latency.

Practical Recommendations

Start from the guarantee you need and work backwards. If a lost event creates a financial or legal inconsistency, use the outbox. Then pick the relay by operational capacity: Debezium if you already run Kafka Connect, polling with FOR UPDATE SKIP LOCKED otherwise, sharded by aggregate key if order matters.

Design the contract carefully. Make the payload a versioned public schema, key by aggregate id, generate event ids in the application, and keep outbox rows immutable. Build the inbox or natural idempotency into every consumer from the first day; retrofitting it after the first duplicate incident is painful, and the failure it prevents is silent.

Operate it like a database feature, not a message-queue feature. Monitor slot lag and retained WAL, outbox end-to-end lag, dead-letter volume and consumer group lag. Rehearse a connector restart and a slot invalidation in staging. For consumers in a multi-step workflow, tie this into your saga design, and size the Kafka side using the guidance in our KRaft production tutorial.

Checklist before go-live:

Business write and outbox insert share one transaction, covered by a rollback test.
Outbox payload is a versioned contract with compatibility checks in CI.
Message key is the aggregate id; ordering needs are documented per topic.
Every consumer is idempotent through an inbox or a naturally idempotent operation.
Offsets commit only after the database commit.
Heartbeats configured; alerts on retained WAL and slot lag; retention cap decided.
Dead-letter topic and replay procedure exist and have been tested.
Inbox and outbox retention are defined and automated.

Frequently Asked Questions

What is the transactional outbox pattern?

It is a reliability pattern for event-driven systems. Instead of writing to the database and then publishing to a broker, the service inserts the event into an outbox table in the same local transaction as the business change. A separate relay, often Debezium reading the write-ahead log, later publishes those rows to the broker. This guarantees an event exists exactly when the state change committed, with at-least-once publication.

How does the outbox pattern solve the dual write problem?

The dual write problem arises because a database and a message broker cannot commit atomically, so a crash can leave one updated and the other not. The outbox reduces the synchronous work to a single resource, the database, where one ACID transaction covers both the state change and the event record. The second hop becomes an asynchronous, retryable copy from durable storage, which can be repeated until it succeeds.

Does the outbox pattern give exactly-once delivery?

No. It gives at-least-once publication, and duplicates are normal after restarts, slot rewinds and producer retries. Exactly-once effects come from combining that with idempotent consumers, typically an inbox table keyed by event id and written in the same transaction as the business change. Kafka transactions provide exactly-once only inside Kafka pipelines, not for external databases or APIs.

Should I use Debezium or a polling publisher for the outbox?

Use Debezium when you already operate Kafka Connect, need low latency, and want commit-ordered delivery without extra coordination. Choose polling when you lack replication privileges, run a database that restricts logical decoding, or want minimal infrastructure. Polling needs row locking, a published marker, cleanup and careful sharding to preserve order, and it adds latency bounded by its interval.

How do I clean up the outbox table?

With log-based CDC you can insert and delete the row in the same transaction; the connector reads the insert from the log and ignores the delete, so the table stays empty. With a polling publisher, delete or archive rows after publication, or partition the table by time and drop old partitions, which avoids vacuum overhead. Keep a short retention if you want replay for debugging.

What happens if Debezium goes down?

Events keep accumulating safely in the database, so no data is lost, and the connector resumes from its stored offset when restarted, redelivering anything not yet acknowledged. The danger is the replication slot: while the connector is down, Postgres retains WAL for it, which can fill the disk. Alert on slot lag, set a retention cap deliberately, and drop slots you no longer need.