Why switching the order of two calls cannot fix it
The first design is simple: commit the order, then publish the event. Crash after the commit, and the event never reaches the broker.
So someone suggests publishing first. That moves the failure. The broker may accept OrderCreated, and then the database transaction may fail. The warehouse hears about an order that does not exist.
Publishing while the database transaction is open does not give the broker the ability to roll back alongside the database. The two systems still have different commit boundaries. This is the dual-write problem.
“Retry the publish” sounds sensible, but after a process restart, what tells the new process which events were owed? We need to record the obligation before we depend on memory or a network call.
Commit the obligation with the order
In the same database transaction that accepts o481, insert an outbox row for event e901. The row carries its stable event ID, order ID, order version, event type, and the committed facts that a consumer needs. It starts as pending.
One database transaction
order: o481 accepted
outbox: e901 OrderCreated for o481, version 1
COMMIT
The commit creates an order and durable evidence that publication is still required, or neither. It does not atomically commit the broker. A relay must now turn this stored obligation into a broker message.
For a first implementation, a polling relay can claim pending rows briefly, send to the broker outside the database transaction, then record publication only after the required broker acknowledgment. A claim lease or generation allows other relays to recover abandoned work without allowing a stale relay to overwrite newer bookkeeping.
Now picture the uncomfortable failure:
Relay sends e901
Broker accepts and acknowledges e901
Relay crashes before recording success
Another relay later sends e901 again
The outbox still says pending. It cannot know whether the previous send was accepted. Retrying is the correct recovery action under this design, and the broker may receive the same event twice. If we mark it published before sending, a crash can lose the event instead. The outbox gives us recoverable publication, not a universal exactly-once effect.
Do not regenerate the event ID for a retry. e901 must remain e901, because the receiver needs a way to recognize repeated delivery. A broker acknowledgment also needs a defined durability setting. Putting the message into a producer's local buffer is not the same as durable broker acceptance.
Why the message key is not an ordering protocol
Suppose an order is created at version 1 and cancelled at version 2. Two relays race. Version 2 reaches the broker first. Both messages share the same order ID as their partition key.
The broker preserves the order it received, 2 then 1. It cannot reconstruct the order of the original database commits merely from the key.
If consumers require ordered transitions per order, the publication path must preserve that sequence. One approach is to assign versions under the order's database transaction and prevent publication of version 2 until version 1 is resolved. Different orders can publish in parallel. A consumer also needs to reject or repair stale and missing versions because duplicate publications can outlive the original attempt.
Notice how specific that guarantee is. We need order for one order, not a single global order for all customers. A global sequence would add coordination with no benefit to this business rule.
The part operations will notice
The queue can be empty while the outbox has a million pending rows. Monitoring broker lag alone would miss the problem entirely. I would watch the age of the oldest unpublished event, pending count, relay success rate, permanent rejection reasons, and end-to-end time from order acceptance to warehouse acknowledgment.
Suppose checkout accepts 2,000 orders per second while the broker is unavailable for 15 minutes. That is 1.8 million outstanding events. When the broker returns, the relay handles 5,000 per second but another 2,000 arrive each second. The net drain rate is 3,000 per second, so clearing the old backlog takes 10 minutes under those assumptions.
That figure describes time to empty the backlog, not the wait experienced by every event. It also assumes the broker, database and downstream consumers can support the recovery load. If the outbox has limited disk space, admission must eventually slow or stop new work rather than accept more obligations than can be stored.
Change data capture is a valid alternative to polling. It reads committed database changes and publishes them. It brings its own progress and log-retention risks. The lesson is not “always poll” or “always use CDC.” It is to keep the publication obligation durable and understand the relay's recovery boundary.
We can now make it very unlikely that an accepted event is forgotten, provided the data survives and recovery eventually makes progress. We cannot promise it arrives only once. The warehouse receives e901, reserves a lamp, and crashes before saving its broker offset.
What should happen when it reads e901 again?