I would first ask whether the three percent is a stable identifiable group or a tail across all traffic. Which tenant, query family, document filter, region, index generation, prompt length, and route are involved? Is p99 computed over completed requests only, or did the timeout and cancellation rate also change? An apparent stable median can coexist with an unbounded queue for a small slice. It tells us very little about what those users experience.

Take representative slow and fast traces with the same query shape. The RAG path has retrieval fanout, candidate merge, reranking, context assembly, model queue, prefill, decode, and sometimes citation lookup. Compare wall time and waiting time at each boundary. Do not call the model slow because the final span is labeled “answer request.” It may be waiting on the last shard before model admission. If model TTFT is flat after retrieval completes, focus the investigation upstream. If search is flat and TTFT rises only for large context, inspect prompt construction and model queue instead.

Tail amplification is easy to miss in a fanout. If an answer waits for all twenty shard responses, one slow shard can dominate the whole search. As a thought experiment, if each shard independently has a one percent chance of being slow, the chance that at least one of twenty is slow is 1 - 0.99^20, about 18 percent. Real shard delays are correlated, and many systems can stop early, so that number is not a prediction. It shows why a small per shard tail can become a much larger query tail. The Tail at Scale paper develops this general fanout problem.

The probe points to one shard. I would inspect its queue age, search thread pool, CPU, heap or cache misses, segment and index state, replica placement, and query plan. Are many affected requests from one tenant routed there? Is a broad filter or vector search scanning an expensive candidate space? Did indexing or a merge start during the regression? Elasticsearch's search profile documentation can expose work inside each shard, but explicitly does not include queue wait, network, or coordinating node merge time. A fast profile on an idle shard does not explain a slow loaded request.

I would not immediately add a timeout to the shard call. If that shard holds the only authorized supporting passage, returning partial results and a confident answer changes the product's correctness contract. We can perhaps return an incomplete search indicator, abstain, or retry a read on a healthy replica if it serves the same current index view. We need to know whether a replica is actually available, whether it is independently loaded, and whether the extra attempt increases pressure. A hedge to the same saturated node is only more work.

Suppose a tenant's query always fans into the hot shard. The short term fix may be query admission, routing to a less loaded replica, reducing unnecessary fanout, or stopping a backfill competing for its resources. The long term fix might require partitioning that tenant's large corpus, changing the filter/index strategy, or adding read replicas. Each has a quality or coordination cost. More partitions can lower work per shard but increase the number of shards every query waits for. Test the whole query at the actual load, not only a shard benchmark.

I would replay the slow slice and record p50, p95, p99, errors, and missing evidence before and after the fix. A p99 improvement that comes from silently dropping the slow shard is not an improvement for an answer that needed its document. The question is where the latency was introduced and what promise we preserve while removing it.