Model and Inference Engineering · Principal
Prefill and decode disagree about the KV cache
The question
Interview question
A disaggregated serving pool rolls prefill workers to model revision B while decode workers still run revision A. The KV transfer succeeds because tensor dimensions match. Answers degrade without a clean crash. Define the compatibility contract and a safe rollout. What if both sides use B but only the prefill side changes its FP8 scaling mode?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The cache is not a generic bag of vectors. It is the keys and values computed by a particular model for a particular token prefix, position scheme, attention layout and cache representation. Matching byte lengths proves very little. Revision B can have identical tensor shapes and different weights. Feeding its KV into revision A means the next token attends to a past that A did not compute. Checksums can tell us the bytes arrived intact, not that the bytes have the right meaning.
I would make the router bind a request to an immutable compatibility group before prefill. The group includes the exact model and adapter revision, tokenizer and chat template identity, attention and position configuration, KV dtype and scaling scheme, serving engine or connector compatibility, and the tenant and prefix scope. The transfer carries a request identity, prompt token count, cache layout and version fingerprint. Prefill and decode check the contract before any KV is consumed. The documented vLLM NixlConnector compatibility check includes engine and connector versions, model geometry and dtype, attention backend, KV dtype and transfer mode. Application-level tokenizer, template and authorization identity still need their own checks. Do not pretend that the connector's hash covers them automatically.
For a rollout, create B-to-B prefill and decode capacity, warm it, then route a small new-request cohort there. Keep A-to-A for existing streams. Do not transfer an A live cache into B or casually retag a prompt prefix. If B decode capacity is unavailable, queue new B requests within a deadline, route eligible ones entirely through A, or refuse them. Recompute prefill on a compatible B worker if that is cheaper and safe than waiting. A mixed request must never be declared successful because it produced grammatical text.
The FP8 probe is subtle. In the current NixlConnector documentation, matching KV dtypes are required, static scales from the checkpoint are supported, and dynamic per-block scales are not supported because those scales are not transferred with the cache blocks. Both sides saying “FP8” is insufficient for that configuration. This is an implementation-specific limit to recheck when the engine version changes. Test handshake rejection, transfer timeout, retries, partial cache reuse, wrong tenant, tokenizer change, and a mid-rollout decode drain. Compare token-level outputs on fixed prefixes against a nondisaggregated B baseline. A compatibility failure should be visible before generation, not discovered by a customer's wrong answer.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →