architecture 4 min read

The Outbox Pattern: how to save data and tell other services about it, without losing either

A microservice that writes to its database and then publishes an event can lose one of the two in a crash. The Outbox Pattern fixes that, and here's how it works.

Let me introduce you to a really cool microservice design pattern today: the Outbox Pattern! 📤

It’s a small idea that solves a very real problem. When a service changes its own data and also needs to tell other services about that change, it has to do two writes: one to its database and one to a message broker. Those two writes can’t share a transaction, and that’s where things go wrong.

The problem: one change, two writes

Say a customer places an order. The order service saves the order to its database, and then publishes an OrderPlaced event so inventory, billing and email can do their part.

Now crash the service, or lose the network, between those two steps. You get one of these:

Two hand-drawn panels. Left, save then publish: the order service inserts the order into the orders DB, then crashes before publishing to the broker, so the order is saved but no event goes out. Right, publish then save: the service publishes OrderPlaced to the broker, then crashes before the insert, so everyone hears about an order that never happened.

  • Save, then publish: the order exists, but nobody downstream ever hears about it. No stock reserved, no invoice, no email.
  • Publish, then save: everyone hears about an order that was never saved. Inventory reserves stock for a ghost.

Retries don’t fully save you here either, because the crash can happen right after a write succeeded and before you recorded that it did. This is usually called the dual write problem, and in a microservices setup, where each service has its own database, you hit it all the time.

The fix: put the message in the database too

The Outbox Pattern’s trick is to stop doing two writes in the moment. Instead, the service writes the message into an outbox table in its own database, in the same transaction as the business data. Either both rows are committed, or neither is. A separate process then reads the outbox and does the publishing.

A hand-drawn flow. A "place order" request hits the order service, which writes a row to the orders table and a row to the outbox table inside one DB transaction. An outbox relay reads unsent rows from the outbox, publishes them to the broker, and marks them sent. The broker fans the event out to Inventory, Billing and Email, and each one updates its own database.

Here’s how that plays out for an order sale:

  1. When an order is placed, the order data is stored in the database.
  2. In the same transaction, a message with the order data goes into the outbox table.
  3. A background process (the relay) periodically polls the outbox table for unsent messages.
  4. When it finds one, it publishes it to the other microservices that need to know about the order.
  5. It marks the message as sent, and the other microservices update their own databases with the order data.

In SQL, the write side is just a normal transaction:

BEGIN;

INSERT INTO orders (id, customer_id, total)
VALUES (1042, 'c-311', 1499.00);

INSERT INTO outbox (id, aggregate_id, type, payload)
VALUES ('e-77', 1042, 'OrderPlaced', '{"orderId": 1042, "items": 2, "total": 1499.00}');

COMMIT;

And the relay is a loop along these lines:

SELECT id, type, payload FROM outbox
WHERE sent_at IS NULL
ORDER BY created_at
LIMIT 100;

-- publish each one to the broker, then:
UPDATE outbox SET sent_at = now() WHERE id = 'e-77';

If the service crashes or loses connectivity, nothing is lost: the message is sitting in the outbox, and the relay picks it up when things come back.

The catch: “at least once”, not “exactly once”

One thing worth being precise about. What the outbox makes atomic is saving the order and saving its message. The delivery itself is at least once.

Look at the relay again. If it publishes e-77 and then crashes before it marks it as sent, it will publish e-77 again after a restart. The broker can’t tell that it’s a duplicate.

A hand-drawn diagram. The outbox relay publishes event e-77 to the broker, crashes before marking it sent, and publishes e-77 again after a restart. Both copies reach Billing, which checks a processed table that already has e-77 and skips the duplicate. Caption: at-least-once delivery plus an idempotent consumer equals effectively once.

So the consumers have to be idempotent: processing the same event twice must have the same effect as processing it once. The simplest way is to remember which event IDs you’ve already handled, in the same transaction as the work itself:

BEGIN;

INSERT INTO processed_events (event_id) VALUES ('e-77')
ON CONFLICT DO NOTHING;
-- if that inserted nothing, we've seen e-77 already: stop here.

INSERT INTO invoices (order_id, amount) VALUES (1042, 1499.00);

COMMIT;

At-least-once delivery plus an idempotent consumer gets you “effectively once”, which is what you actually wanted for something like order sale transactions, where it’s essential that the numbers come out right.

Polling, or tailing the log

The relay in my example polls the table. That’s simple and works with any database, but it adds a little latency and some load from the queries.

The other option is change data capture (CDC): instead of polling, a tool like Debezium reads the database’s own transaction log (the Postgres WAL or the MySQL binlog) and turns new outbox rows into events. It’s faster and puts no query load on the database, but it’s one more piece of infrastructure to run. Debezium even ships an outbox event router for exactly this.

A few things to keep in mind

  • Ordering: if consumers care about order, publish events for the same entity in order. Using the entity ID (here, the order ID) as the message key keeps them on the same partition.
  • Clean up the outbox: sent rows pile up fast. Delete or archive them on a schedule.
  • Keep messages self-contained: put what consumers need in the payload, so they don’t have to call back into your service to make sense of the event.
  • You may not need it: if everything lives in one database, or nobody downstream cares about a lost event, a plain transaction is enough. The outbox earns its keep when other services depend on hearing about every change.

That’s the Outbox Pattern: write the message where your data lives, in the same transaction, and let a relay deliver it. Your service never ends up with “saved but never announced” or “announced but never saved” again. 🙌

Comments

    Supports `code`, ```blocks```, **bold**, *italics* and links.

    Keep reading