The transactional outbox pattern: solving the dual-write problem
Distributed Systems Fundamentals covered why a transaction spanning two separate systems is hard in general. Here's the single most common place that exact problem shows up in everyday event-driven code: saving a row to a database and publishing an event about it are two separate operations, and one of them can fail without the other.
Advanced
3 min read
The dual-write problem, concretely
def place_order(order): db.save(order) # write #1: the database message_queue.publish("OrderPlaced", order) # write #2: the queue
These two writes are not atomic with each other. If the process crashes (or the queue is briefly unreachable) between the two lines, the order is saved but no event was ever published — every downstream consumer (fulfillment, email, analytics) never finds out the order exists. Reversing the order of operations just moves the failure mode: publish the event first and the database write can still fail afterward, leaving consumers acting on an order that was never actually persisted. There is no ordering of these two lines that makes the pair atomic on its own.
Why this is a smaller, more common version of what lesson 10 (Distributed Systems Fundamentals) covered
Distributed Systems Fundamentals' two-phase-commit and saga lessons covered coordinating a transaction across multiple services. The dual-write problem is the same fundamental issue — two systems that can't commit as one atomic unit — showing up in a much smaller, far more common shape: a single service's own database and a single message broker, which is a problem nearly every event-publishing service runs into, not just ones doing cross-service sagas.
The fix: write the event to the same database, in the same transaction
def place_order(order): with db.transaction(): db.save(order) # write #1 db.save_to_outbox_table("OrderPlaced", order) # write #2 — same DB, same transaction # A separate process reads the outbox table and publishes to the real queue
The transactional outbox pattern sidesteps the dual-write problem entirely by turning it into a single-write problem: instead of writing to the database and the queue separately, the event is written to an outbox table in the same database, as part of the same transaction as the actual business data. A relational database's own transaction guarantee — both rows commit, or neither does — is now doing the work that no cross-system coordination mechanism could provide for free.
The relay: turning outbox rows into real messages
A separate, simple process (a polling loop, or — in more advanced setups — a change-data-capture tool like Debezium reading the database's own write-ahead log) reads unpublished rows from the outbox table and publishes them to the actual message queue, then marks them published. This relay can itself fail and retry without the dual-write problem recurring — publishing the same outbox row twice just means a consumer sees a duplicate message, which is the ordinary at-least-once delivery reality every lesson so far in this domain has already assumed consumers handle via idempotency.
The trade-off: added latency and an extra moving part
Events no longer reach the queue the instant the transaction commits — they reach it whenever the relay next polls (or the CDC tool next processes new log entries), typically milliseconds to low seconds later. For nearly every real use case this delay is invisible; for something genuinely requiring the queue to see an event the instant a transaction commits, the outbox pattern's small guaranteed delay is a real, deliberate cost paid in exchange for the correctness guarantee.