Skip to content
ArchCrux

Reliability & Failure Recovery · Senior

The customer clicked once. Why are there two orders?

One order, five failure boundaries · Part 1 of 5 · 7 min read

Read the worked explanation ↓

Interview question. Your checkout API creates an order in a relational database and returns its ID. A customer clicks Place order once. The request times out, the client retries, and two orders appear. The button was disabled after the first click. Requests can reach different application instances. What happened, and how would you stop it?

For this exercise, all order data lives in one authoritative database. A committed transaction survives the application crashes we are discussing. Payments and regional failover will come later.

Think first

A customer clicks Place order once. The server commits, but the response is lost. The app retries and creates another order. Design a contract that works when retries hit different instances.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Two requests, one intention

Imagine a customer buying a desk lamp. The first request reaches the order service, which inserts order o481 and commits. The server sends a response, but it never reaches the phone. After five seconds, the phone retries. Another server receives the second request, sees a perfectly normal create operation, and inserts o482.

12:00:00  Client sends checkout c781
12:00:01  Server commits order o481
12:00:01  Response gets lost
12:00:05  Client times out and retries
12:00:06  Another server commits order o482

Nothing in that sequence requires two clicks. A timeout means the caller did not receive an answer before its deadline. It does not prove that the server failed. TCP retransmission is not the issue either. The client has sent a second application request after losing the outcome of the first.

Before discussing Redis locks or database isolation levels, I would state the rule we need to protect: one checkout operation may create at most one order for a given tenant. A customer can still buy two identical lamps intentionally. Same request data does not necessarily mean same intention.

Make the operation recognizable

The client should create an operation ID when the checkout starts, before the first network attempt. Every retry carries that same ID. A genuinely new checkout gets a different ID. In our example, that ID is c781.

It helps to distinguish three identities. A trace ID identifies one request attempt for observability. The checkout ID identifies one business operation across attempts. The order ID identifies the resulting resource. One checkout might involve three HTTP requests with three trace IDs, but it should still resolve to one order ID.

The server scopes c781 to the authenticated tenant and operation type. It also keeps the meaningful request parameters, such as cart version, currency, and quoted amount, or a fingerprint of those parameters. If the same ID arrives with a different basket, that is not a harmless retry. Return a conflict rather than quietly changing the accepted order.

The database must resolve the race

A common implementation first queries for c781, then inserts a new order when nothing is found. That is not safe. Two application instances can both query before either inserts. Both then decide to create an order.

I would store the operation identity alongside the order in the same relational database, with a unique constraint on (tenant_id, operation_type, operation_id) and all three identity columns declared NOT NULL. In one short transaction, claim that identity, create the order and its items, and record the original creation result. If another request already owns the identity, retrieve its recorded outcome after the competing transaction resolves.

BEGIN
  Claim unique checkout identity c781
  If new, create o481 and record its acceptance result
COMMIT

On a retry, resolve c781 to o481

This is a sketch of the transaction boundary, not ready-to-run SQL. The database constraint settles concurrent attempts, not an application-side check and not a process-local mutex. The order and its operation identity either commit together or neither commits.

Now stop the server at different times. If it crashes before commit, a retry can create the order. If it crashes immediately after commit, the retry finds o481. If it crashes after responding, another retry still finds o481. In all three cases, the result is consistent with the stored decision.

There are two easily missed details. A competing transaction may still be in flight, so a retry may need to wait. If the caller's deadline expires during that wait, keep the same operation ID and report an unresolved outcome rather than inventing another checkout. Also, returning an existing result still requires authorization. Knowing an operation ID must not be enough to see someone else's order.

What does the client show?

Suppose the first request timed out and the app displays “Order failed. Try buying again.” That wording invites the customer to start a new operation even though the first may have committed. The API may be correct while the product still creates duplicates.

I would preserve the pending checkout identity across retries and display something like “Checking your order” until the result is known. The retry returns the original acceptance result and order ID. A separate order endpoint can provide its current fulfillment state, which may have progressed since the first response.

There is also a limit to the guarantee. If the operation record is deleted after a day and the same identity can be accepted again, that later request may be treated as new. Retention therefore belongs in the API contract. You cannot promise lifetime deduplication while forgetting the identity.

In an interview, I would finish by distinguishing the operation identity from the database enforcement. A key in the request alone does not solve concurrent creates. The unique operation claim and the business mutation need one atomic boundary, and the client must preserve that identity when the response is unknown.

The order is now created once. But the warehouse does not read the order database. It waits for an event. What happens when the order commits and the application dies before publishing it?

Go deeper

If you want to go deeper

The uniqueness check needs database semantics, not hopeful SQL

In PostgreSQL, a practical implementation can attempt an insert under a unique constraint. But INSERT ... ON CONFLICT DO NOTHING RETURNING returns newly inserted rows. It does not return a previously committed conflicting row. Under Read Committed, the insert may also encounter a concurrent row that was not visible to the statement's original snapshot. Handle the losing request with a subsequent authoritative read after the winning transaction resolves. Bound the wait and preserve the operation ID if the client deadline expires.

Do not turn the sketch into SELECT followed by an unprotected INSERT. That reintroduces the original race. Nor should the server store a committed PROCESSING marker before creating the order unless it also implements recovery for an abandoned in-progress state. Keeping the work short and within one local transaction avoids that extra state machine for this example.

A separate idempotency table is convenient when the API needs to record a response or share one operation identity across several tables. For a simpler entity creation operation, a unique client_operation_id directly on the resource can be enough. The meaningful requirement is that the identity and effect share the same enforcing authority.

Which response should the server replay?

Returning the original order ID is essential. Returning the original creation status is a useful convention, while current order state can live on a separate GET /orders/{id} resource. A retry of an accepted checkout should not become a new request just because shipping has started. The server must still authorize access to the existing result.

Matching payloads are not the identity. A customer may intentionally place two identical orders. Conversely, reusing an operation ID with different material parameters should produce a conflict rather than mutate the earlier accepted operation. Define those material fields precisely. A changed tracing header is not a new checkout, but a changed amount or cart version may be.

The deduplication record has a lifetime

If a service retains operation IDs for 24 hours, it needs to say what happens to retries after that boundary. Some APIs document that an expired key can be treated as new. For a purchase, it may be safer to keep a durable business identity longer, make late retries consult an order lookup, or refuse a request whose identity is no longer trustworthy. Those are product choices, not properties supplied by HTTP.

Finally, a safe retry is not a free retry. Thousands of retrying clients can exhaust connections and database capacity. The client should use deadlines, bounded retries, backoff, and jitter. Idempotency protects correctness while backoff protects the service from a storm.

Check your reasoning

  • Calls the timeout an unknown result, not proof of failure
  • Preserves one client operation ID across attempts and separates it from tracing and order IDs
  • Uses a unique identity enforced in the database with the order created in the same transaction
  • Handles two concurrent attempts and a crash after commit
  • Explains parameter mismatches, authorization, retention, and the client response

Common mistake: Checking whether the ID exists before the insert does not resolve two concurrent requests.

Two application instances both check whether checkout c781 exists. They both see nothing. Which change actually prevents two orders?

Read the answer explanation

Enforce a unique checkout operation ID in the same database transaction as the order — The database constraint arbitrates competing transactions on the shared authority. UI controls and local caches do not enforce that cross-instance invariant.

Apply it elsewhere

A cloud platform provisions virtual machines. The first provision call times out after the VM is created. How do you avoid provisioning a second VM when the user retries?

Compare your approach

Carry a caller-provided provisioning operation ID across attempts. Enforce it at the authority that owns VM creation or durable provisioning state. Return the existing VM or pending operation on retry. Treat provider timeouts as uncertain, and define retention and reconciliation if the provider is outside your transaction.

One more constraint

The original operation ID expires after one day. An offline mobile client retries two days later. What contract do you need?

Quick revision

The answer in an interview

A timeout does not tell the client whether the server committed an order. I would give each checkout a stable operation ID created before its first request and reuse that ID across retries. The order service would enforce uniqueness for that tenant and operation in the authoritative database. The operation identity, order and recorded acceptance result would commit in one transaction, so two application instances cannot each create a different order for one checkout.

If the first attempt commits but its response is lost, the retry retrieves the same order ID. If the first attempt aborts, the retry can complete it. Reusing the operation ID with changed material parameters returns a conflict. The client must preserve the identity while the outcome is unknown, and the service must define how long it retains the mapping. A safe retry also needs a deadline and backoff.

Remember: Stable intent, database-enforced uniqueness, atomic creation, bounded retention.

Trap: Identical baskets can represent two intentional purchases. Payload equality alone is not identity.

References

  • Making retries safe with idempotent APIs · Amazon Builders' Library

    Unknown outcomes, caller-provided operation identity, equivalent intent, and atomic recording of identity with effects.

  • Idempotent requests · Stripe API documentation

    Parameter comparison, replayed results, and behavior after key pruning. This is a Stripe-specific example, not a universal API contract.

  • Constraints · PostgreSQL 18 documentation

    Composite uniqueness and non-null constraints used to arbitrate concurrent attempts.

  • Transactions · PostgreSQL 18 documentation

    All-or-nothing local transactions. The lessons state their durability assumptions separately.

  • INSERT · PostgreSQL 18 documentation

    ON CONFLICT behavior and the rows returned by RETURNING.

  • Transaction Isolation · PostgreSQL 18 documentation

    Read Committed snapshots and conflicts with concurrent inserts.

  • Timeouts, retries, and backoff with jitter · Amazon Builders' Library

    Bounded retries and overload control. The authoring work verified the original AWS address and its redirect.

  • RFC 9293: Transmission Control Protocol · IETF / RFC Editor

    Reliable, ordered byte-stream service and the distinction from application-level repeated operations.

Read across the boundary