Agent Architecture · Principal
The refund succeeded but the agent crashed. What happens next?
The question
Interview question
A support agent is authorized to issue a refund. It calls the payment provider. The provider commits the refund, but the network response is lost. The worker crashes before recording success. Another worker resumes from the last checkpoint. How do you recover without issuing a second refund? Now assume the outcome remains unresolved for days and the provider's idempotency key expires after a day.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
A timeout gives the agent one fact: it did not receive a usable response. It does not tell us whether money moved.
So I would never let the resumed model decide “the tool failed, try again.” The runtime owns this step. The model can propose a refund. Code checks the user's authority, the order, amount, currency, prior refunds, and any approval threshold. Once approved, the runtime creates one durable operation for that particular intended refund. The operation gets an ID before the provider call. Every attempt to carry out the same intent refers to that ID.
I would persist the operation with its business key, user and approval identity, target payment, amount, currency, provider account, request fingerprint, and current state. A transactionally recorded command or outbox can make the local intent survive a crash. It cannot create a single atomic transaction with a remote payment provider. That gap is why the provider request also needs a stable idempotency key, normally derived from the durable operation ID in a way the provider accepts.
The worker claims the operation and checks the approval still covers this exact action. It durably records a send attempt with the stable key, then calls the provider. If it gets a confirmed refund ID, it records success. If the response is lost, it records UNKNOWN. It may also crash after recording the attempt but before any bytes reach the provider. On restart, either case is treated as unknown until the provider's state is checked. The ledger cannot tell which side of the network call the crash happened on. The replacement does not create a new operation because the checkpoint did not record success.
Within the provider's documented idempotency window, retrying the same request with the same key and parameters may be safe. I would still reconcile if the returned result is ambiguous. An idempotent retry can repeat a stored error. Stripe's API, for example, returns the first saved status and body for a key, including a 500. A repeated 500 does not establish that no refund happened.
The expiry probe changes the answer. Stripe's published contract says keys may be pruned after at least 24 hours, and reuse after pruning can create a new request. We cannot say “we have an idempotency key, so retry forever.” After the provider's safety window, the runtime looks up the refund by provider reference or a durable operation marker if that provider supports it. It compares amount, target, status, and creation time. If the provider has no reliable lookup and the outcome remains uncertain, the operation stops for a human to resolve. That is less convenient than an automatic retry. It is still better than refunding twice.
Two workers may wake at once. An internal lease or compare-and-swap transition makes only one the current owner of the local operation. A fencing token can reject stale updates to our ledger. It cannot stop a stale worker from creating an external refund unless the provider accepts that token or both calls share a still-valid provider idempotency key. That distinction is easy to miss. Local exactly-once state transitions do not imply exactly-once effects in another system.
Approval is bound to the exact operation fingerprint. If someone changes the amount or payment ID while the job waits, the old approval no longer applies. Right before sending, the runtime checks current authorization and the approval state. It also checks whether this payment already has an equivalent refund operation. The model's reasoning can explain why the action was proposed. It is never the authority to spend money.
I would record an event for each transition: proposed, policy checked, approved, claimed, sent, provider result, reconciled, resolved. Logs need the run, step, operation, attempt, approval, and external refund IDs. The alert I care about most is not simply “agent failed.” It is an UNKNOWN financial operation that has not been reconciled within its target window.
This is one of those cases where a workflow engine helps a lot, but it does not remove the external effect problem. Temporal's own documentation recommends idempotent Activities because attempts can retry. I would use durable execution for the state and scheduling, and keep the operation identity and reconciliation logic explicit.
The next two pushes change the recovery path
The interviewer removes provider idempotency keys. Now the local ledger can prevent a new intent, but it cannot undo a duplicate external request if two workers race. I would serialize attempts locally, fence ledger updates, and use the provider's refund lookup to resolve an unknown result before any further call. If the provider also has no reliable lookup, automatic recovery is impossible for this effect. I would stop and send the operation to a person with the payment record and attempted request details.
Then the interviewer says the first worker lost its lease while its call was in flight. A new owner sees the unconfirmed attempt and must treat it as potentially committed. The stale worker can still reach the provider. The lease only protects our own state unless the provider participates in fencing or deduplication. This is why the external contract determines what recovery can safely automate.
Reversing a refund is another action, with its own policy and failure modes. I would only propose it if the provider and the business actually support a reliable reversal. It is not a generic compensation step.
Primary-source notes: Temporal Activity retry and idempotency guidance, Stripe's current idempotent request behavior. Stripe is an example contract. Other providers need their own verification.
Related ArchCrux reading: Designing Long-Running AI Agents That Survive Failures.
Continue practicing
Related questions
Read beyond the question
Explore more agent architecture
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →