Lesson 2 of 2
The Outbox and Idempotency
Two primitives that every distributed write depends on: making a state change and its message atomic, and making a repeated message harmless.
Before any saga or compensation logic, two mechanisms have to be in place. Without them, distributed flows lose messages and duplicate work, and every higher-level pattern built on top inherits the defect. They are unglamorous and they are the foundation.
The dual-write problem, and the outbox
A service that changes its state and then publishes a message is performing two writes to two systems with no transaction between them. If the database commits and the broker publish fails, the rest of the system never learns what happened. If the publish succeeds and the database rolls back, consumers act on something that did not occur. Reordering the two operations changes which failure you get, not whether you get one.
// BROKEN: two systems, no shared transaction. Both orderings have a hole.
await db.order.update({ where: { id }, data: { status: 'PLACED' } });
await broker.publish('order.placed', { orderId: id }); // may never happen
// OUTBOX: the message is a row in the same transaction as the state change,
// so either both are durable or neither is.
await db.$transaction(async (tx) => {
await tx.order.update({ where: { id }, data: { status: 'PLACED' } });
await tx.outbox.create({
data: {
id: messageId, // stable, so a redelivery is detectable
type: 'order.placed',
payload: { orderId: id },
createdAt: new Date(),
},
});
});
// A separate relay reads unpublished outbox rows, publishes them, and marks
// them sent. If it crashes after publishing but before marking, the message
// is delivered twice — which is exactly why consumers must be idempotent.Idempotency, and where to put it
An operation is idempotent when performing it twice has the same effect as performing it once. Some operations are naturally idempotent — setting a status to `SHIPPED`, storing a value at a key. Most interesting ones are not: charging a card, decrementing stock, sending an email. For those, idempotency must be constructed, and the construction is always the same shape: a stable identifier for the operation plus a record of what has already been done with it.
export async function handleOrderPlaced(message: Message): Promise<void> {
await db.$transaction(async (tx) => {
try {
// The unique constraint is the actual guard. An if-then check would
// race: two concurrent deliveries can both read "not processed".
await tx.processedMessage.create({
data: { messageId: message.id, consumer: 'billing', at: new Date() },
});
} catch (error) {
if (isUniqueViolation(error)) return; // already handled; safe to drop
throw error;
}
await tx.invoice.create({ data: invoiceFrom(message.payload) });
});
}
// Note what makes this correct: the marker and the effect are written in the
// same transaction. Marking first and acting afterwards loses the work when
// the process dies in between.| Situation | Key | Retention |
|---|---|---|
| Consuming a message | The message identifier assigned by the producer | Longer than the maximum possible redelivery window |
| A client retrying an HTTP request | An idempotency key the client generates and resends | Long enough to cover client retry behaviour, typically 24 hours |
| Calling an external provider | A key you generate and store before the call | Until the operation is confirmed settled, often days |
| A scheduled job that may overlap | The period it is processing, such as the date | As long as the job may be re-run for that period |
In practice
Duplicate charges from a correct-looking consumer
A subscription platform charged customers from a message consumer. During a broker failover, a batch of messages was redelivered and 1,400 customers were charged twice. The consumer had a processed-message check.
Constraints
- Broker redelivered roughly 6,000 messages during failover
- The consumer ran with eight concurrent workers
- The check read a `processed_messages` table before charging
- The payment provider supported idempotency keys, unused
Decision
Replace the check-then-act with an insert protected by a unique constraint in the same transaction as the charge record, and pass the message identifier to the payment provider as its idempotency key so the provider deduplicates as well.
Why
The check was a race, and eight concurrent workers made it a race that fires reliably during a redelivery burst. Two independent defences were chosen deliberately: the local constraint prevents duplicate work, and the provider key means even a defect in the local logic cannot produce a second charge. For money, one layer of protection was judged insufficient.
What it cost
Refunding 1,400 customers cost about £9,000 in fees and a week of support effort, and the failure damaged trust more than the money did. The team also accepted a permanent obligation: the `processed_messages` table now grows continuously and needs a retention policy set longer than the broker's maximum redelivery window — which they had to look up rather than guess, since guessing is how the same class of bug returns.
Reviewing a new message consumer
As a developer
Checks the business logic, the error handling and the tests, and confirms that a failure results in the message being retried rather than lost.
As an architect
Asks what happens when this exact message arrives twice, because it will — during a failover, a redeployment, or a broker restart. Then asks where the deduplication marker is written relative to the effect, since a marker outside the effect's transaction is either a race or a way to lose work. Neither question is about this consumer specifically; they are the two questions that apply to every consumer, which is why they belong in a checklist rather than in one review.
Your outbox relay publishes a message, then crashes before marking the row as sent. On restart it publishes the same message again. Is this a bug in the outbox?RevealHide
No — it is the outbox working as designed, and the design is deliberate. The relay has two options when it cannot be sure: publish again, risking a duplicate, or skip, risking a lost message. It chooses duplication because a duplicate is recoverable by an idempotent consumer while a lost message is silently and permanently wrong. This is why at-least-once delivery is the guarantee the whole pattern is built on and why consumer idempotency is not optional but a structural requirement. If duplicates are causing problems, the fault is in a consumer that is not idempotent, not in the relay.
Key takeaways
- A state change plus a message publish is a dual write with no shared transaction; the outbox makes it one write.
- A relay publishes outbox rows and may publish twice — at-least-once is the guarantee you actually have.
- Idempotency is a stable operation key plus a durable record, enforced by a uniqueness constraint rather than a check.
- Write the deduplication marker in the same transaction as the effect, never before or after it.
- For money, defend twice: locally and with the provider's own idempotency key.