What is actually in the KV cache, and why does it fill up?

Take one attention layer while a model generates a token. It computes a query for the current position and compares it with keys from positions it is allowed to attend to. The resulting weights combine the corresponding values. Without a cache, the model would keep recomputing the old keys and values for the same prefix at every generation step. The KV cache retains those per layer attention states so the next token can use them. It is not the prompt text, not a summary of the conversation, and not model weights. Hugging Face's cache explanation shows the key and value tensors and their sequence dimension.

For a simple full attention decoder, an approximate memory calculation is 2 × layers × KV heads × head dimension × bytes per element × live tokens. The 2 is keys and values. For a hypothetical 32 layer model with eight KV heads, head dimension 128, and two bytes per element, that is 131,072 bytes, or 128 KiB per token. An 8,192 token live sequence is about 1 GiB of KV alone. Sixteen such sequences are about 16 GiB before weights, runtime workspace, fragmentation, and any reserved blocks. This is an illustration, not a universal model footprint. Multi query or grouped query attention changes the KV head count, compressed attention changes the representation, and a sliding window may stop some layers' cache from growing with the full history.

Both input and generated tokens matter. Prefill creates KV for the prompt. Each decode step extends the live sequence with newly generated state. A request with a short prompt and a long answer can become expensive after a quick first token. A request with a huge prompt can occupy memory before it streams anything. This is why total request count is a poor predictor of available slots. The server also allocates in blocks or pages, so the physically reserved amount can differ from the simple live token calculation. The PagedAttention paper explains why block management and sharing matter for serving throughput.

Now prefix caching is enabled. If two requests truly begin with the same token sequence under a compatible model and cache configuration, the second may reuse completed KV blocks from the first rather than repeating prefill for that prefix. It then computes its own suffix. An unchanged system prompt is a common example. vLLM's prefix cache design describes a hash of prompt blocks and earlier prefix state to find reusable blocks. Exact reuse depends on the engine's block and identity rules, not on two prompts being semantically similar.

What if the common prefix contains tenant private text? Sharing physical computation does not by itself reveal that text, but the platform must prevent cross tenant cache identities, timing disclosures, and stale reuse after a policy change. Include the relevant tenant and policy scope in cache identity or isolate cache pools where needed. Do not call a cached answer safe just because a KV block was safe to reuse. The generator still needs current authorization for retrieved evidence, and a changed instruction or document version must produce a different effective token prefix or invalidate its cached state.

Does prefix caching halve KV memory for two conversations? Only for the blocks they genuinely share and retain. If both start with 4,000 identical tokens and then each runs for another 4,000 different tokens, the common part can be shared while the two suffixes remain separate. A cache hit also requires the useful prefix blocks to remain resident. Keeping old blocks for potential future hits competes with KV for active sequences, so eviction policy matters. Prefix caching can reduce repeated prefill substantially without making live decode memory disappear.

I would measure prompt tokens, computed versus reused prefix tokens, active and cached blocks, per request KV residency, eviction, and preemption along with TTFT. Then test long streams and multiple tenants. A high cache hit rate is helpful only if the retained blocks do not crowd out requests that need memory now.