I would not start by adding shards. The low load result tells me the query plan can be fast when queues are short. At 10,000 QPS we need to know whether the hot shard receives more searches, does more work per search, shares a node with another hot workload, or is slow because of indexing, cache misses, and segment churn. Record the tenant, query family, filters, lexical and vector branches, routed shard set, queue time, execution time, merge time, and reranker time for each traced request. The p99 could be the slowest shard of a fanout, a coordinating node, or the model based reranker after retrieval.

Compare actual search traffic and physical placement. If all popular queries for a large tenant route to one primary and its replicas, the demand distribution is skewed even if the total shard count looks healthy. A replica can spread read requests when placement and routing allow it. It does not make a single expensive query cheaper, and a replica on the same saturated node will not rescue that node. Look at search thread pools, queued and rejected tasks, CPU, heap, disk, and per shard timings. Elastic's hot spotting guide suggests comparing shard and node statistics rather than inferring balance from cluster totals.

Hybrid search adds two work profiles. Exact API names and error strings can be served by the lexical path, while fuzzy conceptual queries need a vector path. Does every request run both branches across every shard, even when an exact filter could narrow the search? Are the candidate limits and reranker batch size bounded under load? Preserve quality when optimizing. Turning off the lexical branch may lower latency and quietly lose the exact identifier query that matters to a user. Turning off ACL filters for speed is not an option.

The tenant's corpus no longer fits one shard. Introduce stable virtual partitions within that tenant, with a routing catalog that maps source document IDs to physical lexical and vector partitions for a particular index generation. A document's chunks and permission metadata need a coherent identity across both paths. Partition by a collection or other selective boundary when queries often name one. If not, hash documents to distribute writes and reads but expect cross partition query fanout. Search only the tenant's permitted relevant partitions, merge candidates with a defined rank fusion or score calibration, then rerank under a fixed candidate and time budget. Raw vector and lexical scores from different branches should not be compared as if they were the same unit.

More partitions can make p99 worse if every query waits for every partition. I would model query fanout and the top relevant document's location before choosing a split. A request that touches forty shards may create a new straggler problem after the old hot shard is cured. Elastic's shard sizing guidance warns that many shards can exhaust search thread pools. The right partition size comes from our tenant distribution, update rate, candidate recall, and loaded query traces, not one vendor size rule.

During migration, build new partitions from immutable source versions and keep the old search generation available. Dual read a sample, compare authorized evidence recall, exact identifier recall, citations, and p99 at load, then switch the generation for a request consistently. Do not merge candidates from old and new chunk layouts without version labels. If the new routing catalog is wrong, a fast query can silently miss the document. Rollback needs the old catalog and index, not just the old code.

There is a short term control path too. Limit simultaneous expensive queries for the hot tenant, move safe traffic to loaded replicas, pause a competing backfill, and reject searches whose deadline is already impossible. A timeout that simply returns partial shards must mark incomplete evidence or abstain for questions requiring a missing source. I would call the design successful only when the 10,000 QPS workload meets its tail target and the restricted, versioned, exact queries still find their permitted evidence.