The Dual-Write Problem & the Outbox Pattern

Here is a piece of code that appears in nearly every microservice ever written, and is almost always subtly broken:

saveOrderToDatabase(order)          // step 1
publishEvent("OrderCreated", order) // step 2

You wrote to your database and you published an event so other services find out. Two writes, two systems. The trouble is there is no way to make those two steps atomic — to guarantee that either both happen or neither does. This is the dual-write problem, and it is one of the most common sources of data drift in event-driven systems. It’s the operational shadow of everything in Event-Driven Architecture: the moment you tell other services about a change by emitting an event, you’ve signed up for this problem.

Why it’s genuinely impossible, not just hard

The database and the message broker are two separate systems with two separate transaction domains. Your local database transaction can guarantee atomicity within the database. It cannot reach across the network into Kafka or RabbitMQ and enroll that send in the same all-or-nothing unit. So whichever order you choose, a crash in the gap between the two steps leaves you inconsistent:

DB first, then publish:                Publish first, then DB:
  ✓ save order                           ✓ publish "OrderCreated"
  ✗ CRASH before publish                 ✗ CRASH before save
  → order exists, NO event               → event exists, NO order
  → downstream never hears about it      → ghost event for an order
     (lost notification, no shipment)       that doesn't exist

A natural instinct is “wrap them in a distributed transaction (two-phase commit, XA).” You can, and it’s a cure worse than the disease: 2PC couples the availability of your database to the availability of your broker (if either is down, neither write proceeds), it holds locks across a network round trip, it scales poorly, and many modern brokers don’t support it well anyway. You’d trade a rare inconsistency for chronic unavailability and latency.

The retry trap

“Just retry the publish if it fails” doesn’t save you. If the process crashes between the two steps, there’s no code left running to do the retry — the intent to publish died with the process. And if you retry naively you risk publishing twice. Retries help with transient broker errors; they do nothing for the atomicity gap.

The insight: make it one write, not two

The dual-write problem is unsolvable as stated because it’s two writes to two systems. So we refuse to do two writes. The transactional outbox pattern turns the event into a row in the same database as the business data, written in the same local transaction:

BEGIN TRANSACTION
    INSERT INTO orders        (...)            -- the business change
    INSERT INTO outbox_events (type, payload)  -- the "I should publish this" record
COMMIT

Now there is exactly one atomic write. Either both rows commit or neither does — the database’s own transaction guarantees it. There is no longer a gap where the order exists but the event doesn’t, because the event is in the order’s transaction. The outbox_events table is just an ordinary table holding “messages I intend to send, but haven’t yet.”

        ┌─────────────────── ONE local transaction ───────────────────┐
        │  orders table  ◄── business row                              │
        │  outbox table  ◄── event row (type, payload, created_at)     │
        └──────────────────────────────────────────────────────────────┘
                                   │
                                   ▼   (separate, asynchronous process)
                         relay reads new outbox rows
                                   │
                                   ▼
                         publish to Kafka / RabbitMQ
                                   │
                                   ▼
                         mark row as sent (or delete it)

Getting the event out: polling vs CDC

The event now sits safely in the outbox. A separate message relay moves it to the broker. Two ways to build the relay:

1. Polling publisher. A background loop runs SELECT … FROM outbox WHERE sent = false ORDER BY id, publishes each, and marks it sent. Dead simple, works on any database. Costs: polling latency and constant query load; you tune the interval to trade freshness against database pressure.

2. Change Data Capture (CDC). Instead of polling the table, tail the database’s write-ahead log — the append-only record of every committed change the DB already keeps for crash recovery. A tool like Debezium reads the log, sees new outbox rows the instant they commit, and streams them to Kafka. No polling, near-real-time, near-zero added load on the database, and it reads committed data only — so it never sees a row that rolled back.

WAL (write-ahead log):  ...│order#42│outbox#42│order#43│outbox#43│...
                                       │
                              Debezium tails the log
                                       │  (only committed entries)
                                       ▼
                                  Kafka topic

Outbox makes delivery at-least-once

The relay publishes, then marks the row sent. If it crashes in between, on restart it re-reads the unmarked row and publishes again. That’s a feature, not a bug — it guarantees the event is never lost (at-least-once). It does mean consumers can see duplicates, which is exactly why the next page, Exactly-Once Semantics, exists: consumers must be idempotent. Outbox guarantees delivery; idempotency guarantees correct processing.

What it buys, what it costs

The outbox pattern buys you the one thing 2PC couldn’t without wrecking availability: atomic, guaranteed publication of an event alongside the data change that caused it. No more orders without events, no more ghost events for orders that don’t exist. Your service stays available even when the broker is down — events simply queue in the outbox table until the relay can drain them.

The costs are honest and worth naming. You add a table, a relay process (or a CDC pipeline to operate and monitor), and end-to-end latency — the event reaches the broker a beat after the commit, not synchronously. You inherit at-least-once delivery, pushing the dedup burden onto consumers. And the outbox table needs housekeeping (delete or archive sent rows so it doesn’t grow unbounded). For almost any system that emits events about its own state changes, that’s a price well paid: you’ve converted an impossible atomicity problem into a routine, observable pipeline.

The CDC footgun — a stuck replication slot

CDC’s “near-zero load” pitch has a sharp operational edge worth knowing before you reach for it. To tail Postgres’s write-ahead log, Debezium holds a logical replication slot, and Postgres will not recycle WAL segments that a slot hasn’t confirmed it consumed. So if your CDC consumer stalls or dies and nobody notices, the database dutifully retains every WAL segment since the slot fell behind — pg_wal grows and grows until it fills the disk and the primary stops accepting writes. The very mechanism that makes CDC cheap (the DB already keeps the WAL) becomes a way to take down the source of truth. The lesson fits this page’s ledger: the outbox + CDC pipeline is worth it, but “another pipeline to operate and monitor” is a real line item — and monitoring replication-slot lag is non-negotiable, not optional.

The architect’s lens

The outbox is the pragmatic answer to an impossible atomicity problem. Run it through the five questions:

• Why does it exist? Because writing to your database and publishing an event are two writes to two separate transaction domains, and no local transaction can reach across the network to enroll a Kafka/RabbitMQ send — so a crash in the gap leaves an order with no event, or a ghost event with no order.
• What problem does it solve? Atomic, guaranteed publication of an event alongside the change that caused it. By writing the event as a row in the same database in the same local transaction, there’s exactly one atomic write — both rows commit or neither does — and the service stays available even when the broker is down (events queue in the table).
• What are the trade-offs? You add a table, a relay (polling or a CDC pipeline to operate and monitor), and end-to-end latency — the event reaches the broker a beat after commit, not synchronously. And you inherit at-least-once delivery, pushing dedup onto idempotent consumers.
• When should I avoid it? When nothing else needs to hear about the change (no event, no problem). Don’t reach for 2PC/XA instead — it couples DB and broker availability and holds locks across the network. And weigh CDC’s footgun: a stuck replication slot retains WAL until the disk fills and the primary stops accepting writes.
• What breaks if I remove it? You’re back to the naive dual write and silent data drift — lost notifications, ghost events — that no retry fixes (a crashed process has nothing left to retry). The outbox converts that impossible atomicity gap into a routine, observable pipeline.

Check your understanding

1. Explain precisely why a database write and a message-broker publish cannot be made atomic, in terms of transaction domains.
2. Show the failure outcome for both orderings (DB-first and publish-first) when the process crashes between the two steps.
3. Why is two-phase commit (XA) a poor fix here? Name at least two concrete downsides.
4. What is the single conceptual move that makes the outbox pattern work, and why does it eliminate the atomicity gap?
5. Compare polling vs CDC for the relay. Why does CDC add almost no load and never publish an uncommitted row?

Show answers

1. The database and the message broker are two separate systems with two separate transaction domains. A local DB transaction can guarantee atomicity within the database, but it cannot reach across the network and enroll a Kafka/RabbitMQ send in the same all-or-nothing unit — so the two writes can never be made atomic.
2. DB-first: save the order, then crash before publishing → the order exists but no event fires, so downstream never hears about it (lost notification, no shipment). Publish-first: publish OrderCreated, then crash before saving → a ghost event for an order that doesn’t exist.
3. 2PC/XA is “a cure worse than the disease”: it couples the availability of the DB to the broker (if either is down, neither write proceeds), it holds locks across a network round trip and scales poorly, and many modern brokers support it badly — trading a rare inconsistency for chronic unavailability and latency.
4. Make it one write, not two: write the event as a row in the same database, inside the same local transaction as the business change (the transactional outbox). The database’s own transaction guarantees both rows commit or neither does, so there’s no gap where the order exists but the event doesn’t — the event is in the order’s transaction.
5. Polling runs SELECT … WHERE sent = false on a loop — simple and DB-agnostic but adds constant query load and polling latency. CDC tails the database’s write-ahead log (the append-only record the DB already keeps for crash recovery), so it adds almost no extra load and is near-real-time; and because the WAL contains only committed entries, it never publishes a row that rolled back.