Model and Inference Engineering · Principal
The tenant deleted its adapter. Why can the serving fleet still use it?
The question
Interview question
A tenant asks to delete its fine-tuned LoRA adapter. The control plane removes the object store entry and marks the adapter deleted. Thirty seconds later one GPU worker answers a request using the old adapter. Other workers may hold a memory copy, local file cache, and prefix or KV cache for active requests. Define the deletion contract and recovery. The tenant also asks whether an answer generated before deletion must disappear from audit.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The first question is what “deleted” was promised to mean. Removing one object store key is not the same as preventing the next inference from using the weights. If a router can still select the adapter by tenant name, a worker has a resident copy, or a resolver can reload it from a cache, the control plane changed metadata while the serving path remained eligible.
I would give each adapter a tenant-bound immutable version and a monotonic eligibility epoch. The deletion request first makes new admission impossible at the router and worker boundary. Every request carries the authorized adapter version and epoch, not an arbitrary client-provided adapter name. Workers reject a request when its authorization is older than the current deny epoch. They must get that deny decision through a path whose propagation and partition behavior meet the promised deletion time. If a worker cannot verify current eligibility, it cannot serve that adapter. A control-plane flag alone is not enough if the worker keeps accepting previously queued requests.
Then handle requests already admitted. The contract could allow them to finish, or require cancellation before the next generated token. State which one applies and expose the effective time to the tenant. For an immediate stop promise, fence decode continuation at a safe scheduling boundary and fail closed on an unreachable worker. We cannot retract tokens already delivered. For a drain promise, show when the last in-flight use ended. A worker that times out during deletion is not assumed clean. Quarantine it from tenant traffic until it rejoins, proves the new epoch, and unloads or is restarted. vLLM documents dynamic LoRA load and unload endpoints, but calling an unload endpoint on one process does not establish a fleet-wide deletion contract. Its security guidance also warns against exposing dynamic loading to untrusted clients.
The memory and disk inventory matters. Remove or make unreachable the resident adapter, local resolver copy, prepared kernels or merged weights where used, object versions and backups according to the retention contract. A prefix or KV cache created under that adapter may hold computation derived from it. Do not reuse it under another tenant or adapter version, and retire it when the old request ends or is cancelled. Whether a particular cache representation contains recoverable tenant data is implementation-specific, so the design cannot declare it safe by calling it “just vectors.” Track what is stored and how long it persists. The old adapter must also be unable to return after a restore from backup or an autoscaled worker snapshot.
The proof is a fleet acknowledgement with a coverage boundary: every routable worker and resolver either confirms the deny epoch and cache disposal or is excluded from serving. Test a disconnected worker, an in-flight generation, a queued request, an autoscaler restoring a stale snapshot, and a deletion followed by a new adapter with the same display name. Reuse of the name must not reuse the old internal identity or cache namespace.
About the audit answer generated yesterday, deletion of the adapter is not automatically deletion of every historical answer. The tenant's contract may separately require removal of prompts, outputs, traces, and training artifacts, subject to permitted audit retention. Keep that decision explicit. Do not tell the tenant “all copies are gone” when the request only covered serving weights, and do not keep a secret copy of weights under the label of audit. The incident here is that an old adapter served after the point at which we claimed new use had stopped.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →