The local transaction was not the broken part
In A, one database transaction committed the checkout identity, order o481, and outbox event e901. It became durable on A's local storage. The API acknowledged the checkout. The relay published e901, and the warehouse committed a reservation.
Before that database log reached B, A became unreachable. B had none of those three records. A retry with c781 found no matching checkout in B, so B created o902 and another event. The warehouse saw a different event ID and possibly a different fulfillment identity.
A: c781 -> o481 -> e901 -> stock reserved
replication not yet at B
A unreachable
B promoted: c781 not found
B: c781 -> o902 -> e1202
The original transaction did not partially commit. Its records disappeared together from the promoted history. Database atomicity was intact. The deployment had only promised durability on A before acknowledging the client. It had not guaranteed that every acknowledged decision would survive losing that region.
If the warehouse also stores c781 as a durable business identity, it might reject the second reservation. That is useful defense in depth. It does not restore the missing order, settle which order ID clients should trust, or repair divergent histories elsewhere.
Decide which guarantee matters before choosing a topology
I would ask product and operations a precise question: after losing one region, must every order previously acknowledged as accepted still be recoverable?
Suppose the answer is yes. Then the contract is not “the API usually responds quickly.” It is this: an accepted checkout maps to one durable decision that remains recoverable after any single supported region loss. If the surviving system cannot establish that decision or safely establish write authority, it may return an unavailable or recovery-pending result. It must not reinterpret an uncertain checkout as definitely new.
Three decisions follow. Which copy must store the decision before success is returned? Who can be the writer during a partition? And when can that decision cause irreversible downstream action? Those choices must agree.
Move the acknowledgment boundary
One option is synchronous cross-region replication, configured so a successful transaction has reached a durable copy outside A under the chosen failure model. But the failover policy must promote a history that contains those acknowledged transactions. Waiting for a remote acknowledgment is useless if an automated failover can later choose an older replica.
Another option is consensus-backed replication. For example, three full voting replicas placed in three regions can require two votes to commit. If one region fails and the other two can communicate, a majority may remain. The tradeoff is additional coordination, wide-area latency, and the possibility of refusing writes when quorum is unavailable. A leader-election service does not provide this protection if the order database still acknowledges writes only on its local disk.
We should not describe either approach as simply “strong consistency” and stop there. Which participants record the transaction? Which failure domains are independent? What exactly does the acknowledgment prove? How are changes to membership handled? What happens when an entire region becomes slow instead of fully unavailable?
There is also no requirement to impose this cost on every kind of data. An accepted purchase may need stronger survival guarantees than a recommendation impression. The API should make the difference understandable instead of silently changing durability settings in an incident.
Promotion needs to revoke old authority
A network timeout does not prove region A is dead. A may still be reachable by some clients. Merely changing DNS or a load balancer does not stop its existing writers.
The promotion protocol must prevent the old primary from continuing to accept consequential writes. Depending on the design, that may require a consensus term, an enforced fencing epoch, or verified isolation of the old primary. The enforcement point matters. Writing a higher generation number in a control-plane log does not stop a stale process unless the storage system or action destination rejects its work.
Data survival and single-writer authority are separate properties. Preserving every log entry does not help if two primaries continue producing competing decisions.
Do not publish work before the decision is recoverable
Now revisit the outbox. Suppose the order service has not acknowledged the checkout because it is still waiting for remote durability, but a local relay can already publish its event. The warehouse acts. A fails and the valid replacement lacks the order.
Even though the client never saw success, an external effect has escaped. For this strict contract, the outbox relay must not publish consequential work until the underlying order decision satisfies the same supported recovery boundary as the accepted-order contract. A locally visible row alone is not enough evidence of that condition.
The warehouse needs its own durable identity for physical fulfillment too. A shipment cannot be rolled back by editing a database record. If the warehouse loses its fulfillment receipt during its own failover, the order system's stronger guarantee does not automatically fix it.
Recover the incident without inventing another history
For the incident in progress, stop unsafe checkout decisions and consequential publication for the affected scope. Preserve A's logs and all surviving copies. Establish valid write authority first, then determine the actual gap in recoverable transaction history. A two-second lag figure does not identify precisely which operations were lost.
Correlate checkout IDs, order IDs, outbox event IDs, warehouse reservations, and provider receipts. Keep uncertain requests in a recovery state until their outcomes are established. If A comes back, do not merge the two histories by taking whichever timestamp is newer. Recover and reconcile under a defined authority procedure, including explicit handling of any physical side effects.
In an interview, the strongest answer is not “use three regions.” It specifies which acknowledged decisions survive, who may still write, and which effects are permitted to escape. Then it explains the latency, availability, capacity, and operational cost of making those promises.
You could reuse the same problem for a build platform that publishes artifacts to a registry. If the registry publication survives but the build platform restores an older database snapshot, a retry is not automatically safe merely because the original database transaction was atomic. The external result has a history of its own.