KV Cache Explained: The Memory Problem Behind Fast LLM Inference

A team raises the context limit on its self-hosted engineering assistant from 8,000 to 32,000 tokens. The model and GPUs are unchanged, and a few longer prompts complete successfully during testing. Later that day, several engineers start attaching incident logs and runbooks at once. Requests queue, first-token latency rises, and the engine begins reporting preemptions. An operator sees the growing waiting queue and increases max_num_seqs, expecting the engine to run more requests concurrently and clear the backlog. Instead, the active requests compete for even more KV memory, and latency gets worse.

The model weights still fit. The problem is that each request brings an attention history into a finite GPU KV cache pool. Some blocks are used by active sequences, some are shared between them, and some belong to completed requests whose prefixes might be reused. The allocator can reclaim the last group easily. When the pool is filled with blocks still referenced by active work, another request can force waiting or preemption, and rebuilding discarded KV adds work to an already busy GPU.

The calculations below use one illustrative Llama 3.1 8B-style configuration with BF16 KV, grouped-query attention, and a 32 GiB usable KV pool. A real engine's cache layout and accounting depend on the model, runtime, and parallelism. I want to follow that one pool from allocation to eviction to the moment it cannot provide one more block. The same ownership question appears in distributed serving when a remote worker expects to read KV whose source lease may have expired.

What the KV cache actually stores#Copy link

In a decoder-only Transformer, an attention layer projects token representations into queries, keys, and values. When the model processes a new position, its query attends to the keys and values of earlier positions that causal attention allows it to see. Those earlier keys and values do not need to be computed from scratch on every generation step.

During prefill, the model processes the prompt and produces the KV state for the relevant attention layers. During decode, it extends that state as it generates output, usually one token per active sequence in an iteration. The cache is numerical attention state, not a copy of the conversation text or a set of completed model answers.

Caching saves computation but does not eliminate memory traffic. With ordinary full attention, the retained KV grows approximately linearly with the number of tokens in the sequence, and the decode attention kernel must use the earlier state on subsequent steps. That creates two different constraints. One is capacity, the bytes of state that need somewhere to live. The other is bandwidth, the cost of using that state as new tokens are produced.

Prefill can exploit large matrix operations, while decode often does much less arithmetic per sequence per step and can be sensitive to memory traffic. The exact balance changes with batch size, context length, attention implementation, and hardware. I would not infer it from the phrase decode is memory-bound without measuring the actual workload.

How much GPU memory does the KV cache use?#Copy link

For an ordinary full-attention decoder that stores keys and values in the same precision and shape, a useful starting point is:

KV bytes per token
  = 2 × attention layers × KV heads × head dimension × bytes per element

The factor of two is for keys and values. Use KV heads, not query heads. In grouped-query attention, several query heads share one KV head, which reduces stored state. Also, a model with quantized weights does not automatically use quantized KV tensors.

Take a Llama 3.1 8B-style configuration with 32 layers, 32 query heads, eight KV heads, head dimension 128, and BF16 KV state:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes
                            = 128 KiB per token

A sequence's retained length includes generated tokens as well as its prompt. The engine may allocate memory incrementally, but a request with a 24,000-token prompt and a long answer can continue acquiring cache blocks well after prefill.

Suppose the serving instance has 32 GiB of usable KV capacity after the model and other runtime reservations. In the simplified calculation, that pool can hold the state for eight unrelated 32,768-token histories, or 32 unrelated 8,192-token histories. Those are raw capacity ceilings. Block rounding, shared prefixes, request growth, and scheduling headroom determine safe admission. The pool size already accounts for weights, temporary buffers, CUDA graphs, and runtime reservations, so an 80 GB GPU might expose far less than 80 GB for KV.

For contrast, the same attention dimensions with 32 KV heads instead of eight would store 512 KiB per token and consume 16 GiB for one 32,768-token history. The model's attention architecture can therefore change KV requirements by a large factor without a comparable change in its headline parameter count.

Multi-head latent attention, sliding-window layers, recurrent or state-space components, and hybrid attention layouts have different state representations and retention rules. Tensor parallelism can shard or replicate parts of KV state depending on the engine and model. Capacity planning has to use the deployed model configuration and the engine's per-device allocation, not just the parameter count.

Why PagedAttention allocates KV in blocks#Copy link

If every request reserved a contiguous region for its maximum possible context, short answers would leave large unused allocations. Requests would finish at different times and leave gaps that were awkward to reuse. The PagedAttention paper, which introduced the technique used in vLLM, addressed this by mapping logical token blocks to physical KV blocks rather than demanding one contiguous cache tensor for each sequence.

Assume a logical block holds 16 tokens. At 128 KiB per token, that represents 2 MiB across the layers in our simplified model. A 32,768-token sequence needs 2,048 logical blocks. Its block table points to physical storage that can be allocated from different locations in the pool. A final partially filled block causes some internal waste, but the engine does not need to reserve an entire maximum-length contiguous region from the start. An engine may place the layer slices in separate physical allocations. The 2 MiB is the total KV represented by that logical block across layers.

The allocator now needs to manage more than addresses. It tracks which physical blocks a request references, whether a completed block has a reusable prefix identity, and whether an unused cached block may be reclaimed. In the vLLM V1 prefix-cache design, the block pool is initialized up front. A free-block queue and a hash-to-block lookup allow the scheduler to find reusable computed blocks and allocate fresh slots. A block can carry a cache hash and still be available for eviction when no running request references it.

This distinction matters during eviction. Cached does not necessarily mean pinned. A full block retained for a possible prefix hit may be available for reuse. A block referenced by a running request must remain available to that request until the runtime deliberately preempts or finishes it.

Follow the 32 GiB pool until it cannot allocate one more block#Copy link

Keep the same hypothetical model and imagine six unrelated requests already holding 24 GiB of active KV. Another 6 GiB contains completed prefixes kept for possible reuse, and 2 GiB is unused. Each 16-token block represents 2 MiB, so the complete 32 GiB pool contains 16,384 logical block equivalents in this simplified accounting.

A new request arrives with a 32,768-token prompt and needs 4 GiB of KV. The allocator can use the 2 GiB that is empty and reclaim 2 GiB of old, unreferenced cached prefixes. None of the six existing requests has to stop. Their blocks remain pinned, and the only cost of reclaiming those old prefixes is that a later request might have to compute them again.

Now suppose the running requests collectively produce another 4 GiB of retained history before any of them finishes. The allocator can reclaim the remaining idle cached state. All 32 GiB is now referenced by active work. When the next request needs even one new block, there is no evictable prefix left.

At this point the engine has several possible policies. It can leave further work waiting until a request finishes or releases memory, or preempt an active request and later reconstruct state. In vLLM V1, recomputation is the documented default preemption approach. It is expensive because the same prefix may be processed again merely to recover memory that was discarded under pressure. The vLLM tuning documentation explains this behavior.

Real requests grow in blocks, some share prefixes, and engines may count inactive cached blocks as free capacity. An admission policy can also stop before the pool is entirely pinned. What matters is the transition: while reclaimable blocks exist, eviction can satisfy allocation without interrupting active work. Once the relevant capacity is pinned, LRU eviction is no longer a solution.

That is how the opening incident can have high GPU utilization and poor useful throughput at the same time. The accelerator may be recomputing attention state for preempted requests instead of progressing on new output. Raising the number of concurrent requests makes the active working set larger and can turn a capacity problem into repeated recomputation.

Prefix reuse saves prefill, while decode still needs the history#Copy link

An engineering agent may send several requests beginning with the same system instructions, tool definitions, and conversation history. When those tokens exactly match a compatible cached prefix, the engine can reuse previously computed KV blocks. That avoids prefill work for the shared prefix and can lower time to first token.

During subsequent decode, the model still attends to the relevant keys and values from the retained history. Reusing 16,000 prompt tokens does not make a 16,000-token history disappear from attention. Prefix caching may sharply improve first-token latency without producing a comparable reduction in inter-token latency, especially for long outputs.

There is also a memory saving when two concurrent requests share physical prefix blocks. Take two sequences of 24,576 tokens each, sharing their first 16,384. Separate copies occupy 6 GiB in our BF16 example. With one physical copy of the common prefix and two 8,192-token suffixes, the unique KV state occupies 4 GiB, a 2 GiB saving. That requires actual block sharing on a compatible engine. Two replicas with identical prefixes can still hold separate copies.

The vLLM automatic prefix-cache design uses block identities derived from token IDs, their preceding prefix, and relevant extra data. Reused blocks gain active references. When the requests finish, those references fall, and the same blocks may remain as inactive cached state until pressure causes eviction. This explains why a high prefix-hit rate and high pinned KV usage can occur together.

A long-running agent makes that trade-off visible. Its next turn may benefit from a warm prefix, but an inactive cached block can be reclaimed before the next turn arrives. The useful diagnostic is both how many prefill tokens were reused and how long unused cached state sits before eviction.

Cache identity is a correctness boundary#Copy link

Matching visible text is insufficient to establish that two KV histories are interchangeable. The token IDs, model revision, position configuration, adapter state, and any multimodal input identity must agree. A placeholder such as [image] cannot distinguish one image from another. The cache-key construction needs to represent those differences.

An incorrect match can make a request use attention state from a different input. That is a correctness failure and, on shared infrastructure, a possible confidentiality issue. vLLM documents SHA-256-based hashing options and warns about collision risk with non-cryptographic hashes. The key still needs all the right identity fields, regardless of the hash function.

There is also a cross-tenant timing signal. A caller might infer that another tenant used a guessed prefix by comparing hit and miss latency. vLLM provides cache_salt to separate reuse domains. A trusted gateway should assign the salt, and deployments requiring stronger isolation may need separate serving instances. The vLLM security guidance describes both the protection and its limits.

What changes when you use FP8 KV#Copy link

The calculation above uses two bytes for each BF16 KV element. In an otherwise comparable layout, FP8 brings the raw element storage to one byte. The same pool can then hold approximately twice as many KV elements, subject to scaling metadata, alignment, and the engine's actual allocation scheme. Whether that increases completed requests per second depends on how much of the workload was constrained by KV capacity.

Quantized values need scales. The vLLM quantized KV documentation describes configurations using default scales and calibrated scales, including per-tensor and backend-dependent per-head support. Calibration on representative data is particularly important when quality matters. The attention kernel, accelerator, and runtime version determine which precision paths are efficient.

I would not sign off on FP8 using a few short-chat examples. Compare it with BF16 on the same long-context workload, including exact-reference and multi-step reasoning cases. Measure answer quality, first-token latency, per-token decode latency, usable cache capacity, and preemptions. A larger effective cache is valuable when KV pressure is actually causing queues or recomputation. Its value is smaller when the model is limited by another part of the execution path.

Models with sliding-window or hybrid attention, multi-query and grouped-query attention, and multi-head latent attention need their own memory accounting. The vLLM hybrid KV cache manager handles layer groups with different storage and reuse rules.

What changes when KV moves off the GPU#Copy link

A completed 32,768-token prefix holds about 4 GiB of BF16 KV in our example. Keeping it in HBM preserves a fast future hit but uses space that could serve active work. An offload tier can hold the state in host RAM and bring it back when the next matching request arrives.

That choice depends on transfer cost versus recomputation. At a hypothetical effective 50 GB/s, transferring 4 GiB has a byte-transfer floor of about 0.09 seconds. At 10 GB/s, the floor is about 0.43 seconds. Real transfers also incur staging, connector, and contention costs. The full history must be accessible to the decode attention kernel at the appropriate memory tier before ordinary decode can proceed efficiently.

At fleet scale, the cache also affects placement. The llm-d project, founded by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA and accepted as a CNCF Sandbox project, released version 1.0 on October 20, 2026. Its work on prefix-aware routing, tiered KV caching, and prefill/decode separation illustrates how block locality extends beyond one worker. Google's llm-d 1.0 announcement reports workload-specific latency and throughput improvements, which would need fresh benchmarking on any other deployment.

For the broader placement and scheduler trade-offs, see LLM Inference Is a Scheduler Problem. Here, the harder question is how a destination knows it has received valid state that still belongs to the request.

The lease race when KV crosses workers#Copy link

In a disaggregated system, a prefill worker computes KV and holds the source blocks until a decode worker can load them. vLLM's NIXL connector uses a lease for that lifetime, initially 30 seconds by default. When a request reaches the decoder, heartbeat tracking begins even if it is queued. Batched heartbeats renew the lease as the decoder makes progress through its forward loop. Ordinary queue waiting is therefore covered by the protocol.

The interesting failure is a live decoder that stops sending heartbeats. vLLM sends heartbeat notifications from start_load_kv(), which is called during forward steps, rather than from an independent timer thread. A prolonged forward step, host scheduling stall, or communication interruption can keep the decoder alive but stop renewals long enough for the prefiller's lease to expire. The source releases the blocks and may allocate the same physical buffers to another request. A delayed transfer completion or notification for the old request now refers to a lease that is no longer valid.

The destination must treat expired or mismatched transfers as failed, even if a late notification arrives. A safe connector needs source-buffer lifetime protection, request or transfer generation checks, and an acknowledgement that the destination copy is ready before decode uses it. When expiry wins the race, the request needs a fresh valid transfer or recomputation. vLLM documents this mechanism in its NIXL lease design and exposes counters for expired requests and notifications after expiry. Its warning is specific: some late notifications may follow reads of blocks that have already been reused.

An allocator's reuse decision remains a correctness decision even when the blocks are transferred between machines.

Diagnose the 8K-to-32K rollout from the block pool#Copy link

Before a load test, inspect the capacity reported when the vLLM engine starts. Depending on the release and model layout, startup output can include lines such as GPU KV cache size: ... tokens and Maximum concurrency for ... tokens per request: .... The latter is a theoretical capacity estimate for that configured request length. The engine also exposes per-data-parallel-group fields such as kv_cache_size_tokens and kv_cache_max_concurrency, with special accounting for hybrid attention models. These numbers let you estimate how many 32K histories fit before the first engineer runs a benchmark. They do not include a promise about output growth or latency. See vLLM cache configuration.

Then inspect the live allocation picture. A generic GPU-memory chart may remain almost flat because the engine already reserved its KV pool. What matters is how many of those blocks are referenced by active sequences, how much is reclaimable, and how often the engine preempts or recomputes. Use the metrics exported by the deployed vLLM version, since names and sampled block-residency diagnostics change across releases.

Replay short prompts, long prompts with short outputs, long generated outputs, and multi-turn agent requests with reusable prefixes. Sweep concurrency while holding model revision, KV precision, prompt template, and workload trace fixed. Record the first preemption, useful output tokens per second, p95/p99 latency, queue time, and the number of prefill tokens actually reused. A throughput graph that counts recomputation as progress can hide the problem.

The vLLM production metrics guide covers the counters available for KV usage, prefix hits, preemptions, and NIXL connectors. Some cache-block residency and idle-before-eviction metrics are sampled and require configuration. A separate reuse-distance approach can estimate the value of retaining more old prefixes, as explored in the September 2026 preprint The KV Cache Working Set. That evaluates the cache's reusable working set, which is distinct from memory pinned by active requests.

What I would change before scaling out#Copy link

For this rollout, I would first compare the engine's reported 8K and 32K capacity, reproduce the point where preemptions begin, and set prompt and output limits against the usable block pool. Then I would undo the concurrency increase that made the pinned working set larger, and verify that the engine recovers useful throughput and tail latency. Once local capacity is stable, I would measure whether prefix caching saves enough repeated prefill to justify retaining idle blocks.

FP8, offloading, locality-aware routing, and separate prefill/decode workers are reasonable later experiments when a trace identifies the specific limitation they address. Each changes a different part of the cost or ownership problem. None changes the need to know where the next allocation will come from.

Which blocks are pinned, and what happens when the next one is needed?

Primary references#Copy link

• PagedAttention and vLLM, SOSP 2023
• vLLM automatic prefix caching and block manager
• vLLM optimization and preemption
• vLLM cache identity, hashing, and tenant-salting security
• vLLM quantized KV cache
• vLLM hybrid KV cache manager
• vLLM NIXL KV cache lease design
• vLLM NIXL connector metrics and expiry
• vLLM cache capacity and concurrency configuration
• vLLM production metrics
• llm-d v1.0 release, project maintainers
• Google's October 20, 2026 llm-d 1.0 announcement
• The KV Cache Working Set, September 2026 preprint