The cache contains attention keys and values for tokens processed so far. If a running sequence loses those blocks, it cannot continue at the same state without restoring them or processing its history again. Keeping everything in GPU memory would avoid the problem but limits concurrency. The PagedAttention paper describes eviction and the swap versus recompute trade-off. That is an architectural choice, not a promise that a particular current engine offers live-sequence swapping. In vLLM V1, the documented default preemption mode is recompute, with lower overhead than swap in that architecture. vLLM's current KV offloading guide describes offloading completed prefix cache blocks, which is a different workload from preserving a live paused generation. I would check the exact engine and version before proposing a flag.

For a live sequence, compare the cost of copying its KV blocks out and back with the cost of regenerating those blocks. The bytes depend on layers, KV heads, head size, precision, and sequence length. Transfer time depends on actual usable host link bandwidth and contention, not the marketing peak. Recompute cost depends on prompt length, current prefix reuse, GPU batch shape, and whether the evicted history must be processed repeatedly. A long prefix with expensive recompute and rare preemption may justify host preservation. A short sequence, a busy host link, or a cache that churns back into GPU memory repeatedly can make swapping worse. CPU capacity also has a limit, and pinned memory and concurrent transfers cost something.

Here the service already misses inter-token targets. I would first ask whether the misses are caused by preempted streams, prefill interference, or a slow decode step with no preemption. Get per-request preemption counts, bytes moved, recovery time, KV occupancy, queue age, and inter-token gaps. If active streams are being paused repeatedly, adding a swap path may trade a big recompute pause for a transfer pause. It does not make the pause disappear. Reserve memory for the interactive class, reduce unsafe admissions, cap extreme output lengths, or move batch work away before optimizing eviction. Keep the comparison under the same useful completion rate and p99 contract.

What if the CPU link has spare bandwidth but most evicted requests never resume because clients cancel? Swap out did real work for no user. What if many requests share a cached prefix? Evicting one request's unique tail is different from evicting a shared block needed by others. The policy must account for reference counts and projected reuse, and a performance experiment should include cancellation and cache churn. The right answer is a measured policy for this traffic, not “CPU RAM is cheaper than recomputation.”