The first question is which finish line matters. Does 800 ms mean no first token yet, no complete answer yet, or a pause in the middle of a stream? A second request cannot race an already displayed first half of an answer without changing the user experience. I would consider hedging only before the response is committed to the client, and first measure where the delay sits. If the request is waiting in our gateway queue, sending a second copy to the same provider just increases the offered load.

For a read only, side effect free model call, a delayed hedge can reduce a long tail when the two routes fail independently enough. The Tail at Scale paper explains why a slow subrequest can dominate an entire user request and why a delayed duplicate can help. The qualification matters. If both calls share a saturated quota, model pool, or region, their delays can be correlated. A duplicate may make both queues worse. If the alternate route uses a different model, we also need evidence that its answer satisfies the same quality and policy contract.

The 900 ms follow-up is a good sanity check. At 800 ms, the duplicate still needs transport, admission, prefill, and enough decode to beat the original's remaining 100 ms. On the frequent requests where the original ends at 900 ms, the hedge likely pays for another attempt without improving the answer time. I would estimate the benefit from production traces: among requests still unfinished at 800 ms, how many have much longer remaining times, and how fast is the independent route when launched at that moment? Replay paired timing under live load. Do not choose the threshold from an idle provider p95 and assume it remains right under an incident.

There is also cost and cancellation. Both copies may incur input token charges. Cancellation after one wins may not erase billed prefill or generated tokens on the loser, and the cancellation may arrive after it has nearly completed. Instrument attempted, started, generated, canceled, and billed work. Compare p99 improvement against incremental provider spend and against the effect on everyone else's queue. A fleet with tight headroom can make a few hedged users faster while raising p99 for the rest.

I would only accept the first completed output if both routes are permitted for the tenant and the task. Two stochastic samples can disagree. “Fastest wins” can select a concise but unsupported answer ahead of a slower grounded one. For a docs answer, both candidates must still pass the same evidence and citation checks. For a tool using agent, do not hedge the step that can perform a write. Duplicating the model proposal may produce two different actions, and idempotency at the provider API does not make the business effect safe. A read only planning call can sometimes be hedged, but the runtime must execute at most one authorized proposal.

Would I ever use 800 ms? Possibly for a small high value class whose measured conditional tail has a long residual after 800 ms, with independent spare capacity and a hard hedge budget. Perhaps hedge only a fraction of still waiting calls, with one extra attempt maximum and a deadline that leaves room for the second call. During a shared provider overload I would often disable hedging, because it behaves like a retry surge.

The decision is based on conditional benefit, not the overall latency histogram. If an original often finishes at 900 ms and both are billed, 800 ms sounds too early for broad deployment. I would first fix the queue or route that causes the long tail, then canary a bounded hedge where its saved user time is worth the full marginal work and it does not lower task quality.