The fundamental unit here is the actual prefix the model consumes, not the name we gave a prompt template. Prefix caching reuses work only for compatible, identical input tokens at the relevant boundary. If a tenant rule changes from “never summarize private tickets” to “summarize for approved support staff,” the old prefill cannot stand in for the new one. More importantly, the application must send the new rule on the next request whether or not the serving engine would have a cache hit.

I would split stable common instructions from tenant and request material, with the common prefix first when the model's input format allows it. Keep tenant instructions in a versioned policy bundle rendered after that prefix. The request should pin the bundle revision and take a consistent snapshot of its contents. A logical key for our own cache bookkeeping includes model identity and tokenizer or prompt serialization version, system-template hash, tool-schema revision, tenant identity where tenant data enters, policy revision, and any privacy partition or cache salt required by the serving platform. The serving engine may hash token blocks itself. We should not assume we can choose its key merely by supplying a string.

This sounds like too many fields until you see the failure: an application cache says system-v4 while the prompt renderer silently inserts an updated tenant instruction. If it returns old rendered tokens under that key, the model sees stale policy. Hash the rendered, ordered content, or use an immutable revision that guarantees those bytes. Bind request logging to the revision actually sent. A policy update should atomically change the active revision for new requests, while a run already in progress needs an explicit rule for whether to pin its starting revision or reauthorize before a sensitive action. Prompt freshness and action authorization are related, but the latter cannot be delegated to a cached sentence.

What is worth caching? The long common prefix can be shared when provider isolation and data handling allow it. A tenant-specific suffix may also get repeat hits within that tenant. A per-request document or secret near the front destroys shared prefix reuse for everything after it. Reordering content for cache performance must not move a trust boundary or change instruction precedence. If tenants have different tools, languages, or dynamic safety rules, their token prefixes are different even if the marketing name of the “system prompt” is the same.

On a model version change, I expect a miss for model-specific prefill, even if the visible prompt is unchanged. Different tokenization, hidden message framing, adapter, or KV representation can make prior cached computation incompatible. vLLM's prefix caching design hashes block content with prior prefix and extra context and documents cache salts for isolation. OpenAI's prompt caching guide describes reuse of matching prefixes for its API. Neither makes a customer-controlled invalidation API universal. For a hosted provider, our enforceable action is to send the correct prompt and pin or route to the correct model. We verify the provider's cache scope and reported hit behavior, rather than claiming we purged its internal cache.

The interviewer may ask about an urgent tenant-rule revocation during a long run. I would block the next sensitive step against the current authoritative rule, even if the model was prefetched with an earlier prompt. If a changed rule affects the answer being generated now, cancel or restart that generation with the new snapshot before showing output. A cache miss is not the safety mechanism. A separate policy check is.

Measure saved prefill time and spend per tenant, hit rate by common and tenant prefix, and policy-staleness incidents. In a test, change a tenant rule between two otherwise identical requests and assert the new rendered tokens and resulting action gate reflect the new revision. Then switch model version and verify no application-level cache object is reused as though its computed prefix were compatible.