First, what is a “hit”? It might mean at least one block matched in a request, a fraction of queried blocks, or a fraction of input tokens whose prefill was avoided. Those denominators give very different pictures. A short cached system prompt on every request can make a request hit rate look impressive while most of the prompt is a new 50,000 token document. Inspect computed and reused tokens per request, by tenant and prompt length, then trace where the p99 time was spent.

Exact prefix reuse does save work on matching blocks. It does not make the unique suffix free. The vLLM prefix caching design uses block hashes linked to the earlier prefix and describes eviction of cached blocks. It also supports cache salts for isolation. If tenant instructions, tool definitions, or document order change near the start, later text can look identical to a human but fail exact prefix matching. The dashboard may still count a small common header as a hit. I would compare rendered token IDs and the useful length of each matching prefix, not just the template names.

Second, retained cache blocks and live KV use the same memory budget, but the eviction rule matters. In vLLM, unused cached blocks can sit on a free queue and be evicted when active work needs those blocks. So a high cache occupancy number alone does not prove that live slots shrank. In another configuration, a reservation or slower eviction can coincide with more preemption, recomputation, and a longer admission queue. Check active and cached blocks, free blocks, evictions, preemptions, waiting tokens and queue age before and after enablement. The p99 may belong to the few long prompts that miss and wait while the common prefix traffic gets cheaper.

Third, routing can spoil locality. One replica has the hot prefix and is overloaded. Another has spare GPU capacity but a cold cache. Sticky routing to maximize hits can increase queue time more than it saves in prefill. Or requests are spread evenly, so every replica repeatedly warms the same blocks and evicts them. I would compare routing by queue estimate plus expected reused tokens against pure affinity, with a bound on how much extra waiting we accept for a hit. Cache reuse is valuable only if it reduces the entire request's wait plus compute.

The long unique suffix is the probe that forces the decision. If 2,000 shared tokens are followed by 50,000 unique ones, the cached prefix can be real and still have little effect on first token latency. Reducing unnecessary context or sending it through a different path may matter more. If the long suffix is necessary, tune chunked prefill and admission for that slice, or isolate it from short interactive requests. Do not weaken tenant cache isolation or remove a policy revision from the prefix to make the hit chart prettier.

I would A/B the feature on identical traffic and model versions, partitioned by prompt length and tenant. Record queue, prefill compute, hit tokens, eviction, preemption, and p50/p99 first token time. Run a replay that fixes routing to separate cache behavior from load balance. If p99 rises only under contention, the mechanism is probably capacity or placement. If it rises even in an isolated worker with the same prompt, inspect engine overhead and the actual cached block path. The rollout criterion is latency and cost at the required quality and privacy boundary, not a single hit percentage.