The Transactional Outbox Pattern: Stop Losing Events Between Systems

Learn how the transactional outbox pattern prevents lost events when an application must update its database and notify another system.

A payment can be committed to your database while its confirmation event never reaches the message broker. Both operations succeeded in testing; only one survived in production. This is the dual-write problem, and retrying harder does not solve it. The transactional outbox pattern does.

Why Two Successful Writes Are Not Atomic

Imagine an order service that must save an order and publish OrderCreated. If it saves first, the process can crash before publishing. If it publishes first, the database transaction can fail afterward. Either ordering creates a state that lies to another system.

A distributed transaction could coordinate both resources, but it adds operational complexity and is often unsupported by modern brokers or managed services. The outbox uses something applications already handle well: a local database transaction.

Do not try to make two separate writes perfectly synchronized. Turn them into one write, then deliver the second result asynchronously.

How the Outbox Works

Create an outbox table beside your business tables. In the same database transaction, update the business record and insert an event into that table. A separate relay process reads pending rows and publishes them to the broker or external API.

A useful outbox row typically contains:

• A unique event ID
• The aggregate type and ID, such as order and ord_123
• An event type and schema version
• The serialized payload
• Creation time and delivery status
• Attempt count and next retry time

Because the business update and outbox insert share one commit, they either both exist or neither does. The relay can use polling with row locking, or database change-data capture when scale justifies the added infrastructure.

Design for At-Least-Once Delivery

The relay may publish an event and crash before marking it delivered. On restart, it will publish the same event again. That is not a bug; it is the unavoidable edge between your database and the destination.

Consumers must therefore be idempotent. Give every event a stable ID, and have each consumer record IDs it has processed under a unique constraint. For operations with a natural business key, such as sending one receipt per payment, enforce that uniqueness too.

Keep ordering requirements narrow. If events for one order must remain ordered, partition or select them by aggregate ID and include a sequence number. Requiring global ordering reduces throughput and usually solves a problem the product does not have.

Make Failure Visible

An outbox replaces silent data loss with a queue you own, so operate it like one. Monitor:

• Age of the oldest undelivered row
• Pending row count and relay throughput
• Attempts, permanent failures, and dead-lettered events
• Time from database commit to successful publication

Use exponential backoff with jitter for temporary failures. Set a retention policy for delivered rows, but keep enough history for debugging and audits. Avoid deleting failed rows automatically; preserve the payload and error so an operator can inspect and replay them safely.

The transactional outbox is not glamorous infrastructure. It is a small, explicit boundary that makes failure recoverable. When one user action must update local state and trigger work elsewhere, commit the intent locally, deliver it repeatedly, and make every receiver safe to retry.