Distributed Reliability · Principal
Database and workflow store are restored to different minutes. Which agent actions still exist?
The question
Interview question
A regional failure forces a restore. The application database is recovered through 10:04, but the workflow store through 10:02. At 10:03 an agent submitted a customer refund. The payment provider may have accepted it, while the recovered workflow has no record of the completion. Can we simply resume all workflows from 10:02?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
No. The workflow store, application database and payment provider now disagree about history. Replaying a workflow from 10:02 can issue a second refund if the external effect happened at 10:03. It can also miss a database change that survived to 10:04. A durable workflow does not give exactly-once effects across independent systems merely because its steps can be retried. Temporal's activity guidance recommends idempotent activities for precisely this kind of retry risk.
First, fence the old writers and pause new effectful work. Record the restore coordinates for each store and what is known about the lost interval. Preserve the backups and logs, including provider request IDs, outbox rows, audit entries and any durable receipts. If the old region might still be alive, an epoch or lease check must prevent it from writing while the restored region is reconciling. Do not treat one store's 10:04 timestamp as proof it contains every effect from the other store's 10:02 history. Clocks and commit boundaries differ.
Build a ledger keyed by a stable business operation ID. For each pending or possibly executed refund, compare the restored workflow history, database intent and outbox, and the provider's authoritative status under the original idempotency key. Classify it as definitely not submitted, submitted and completed, submitted with unknown outcome, or rejected. Query the provider for unknowns before any retry. A timeout is not evidence of failure. If the provider cannot look up a request and offers no idempotency guarantee, hold that operation for manual reconciliation. A compensating refund or cancellation is another business action, not a magic undo of an uncertain payment.
Choose a recovery cut that the system can substantiate, then rebuild forward from durable event sources and provider receipts. You may need to roll the application view back to a common checkpoint and replay changes, or replay missing workflow events into a new generation. The exact mechanism depends on which logs survived and whether they were atomically linked to each store. Do not copy database rows into a workflow history as if they were genuine workflow events. Where a causal gap cannot be filled, mark the task for explicit operator review and start a replacement workflow with linked lineage if necessary.
Before reopening traffic, verify a sample of operations on both sides of the restore gap, including a submitted refund with lost completion, a database update without corresponding workflow state, and an action that truly never left the system. Make recovery progress observable by unresolved operation count and oldest unknown effect. Then release safe classes of work gradually. If someone says the provider makes refund calls idempotent, that helps only if we reused the same stable key and the provider's retention period still covers this replay. Restoring yesterday's search index resurrects deleted documents restores one search index and guards deleted documents. This is a cross-store causal recovery problem with external effects that may outlive both restored stores.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →