Memory over time is not the same as memory at the moment a request arrives. If customers alternate among three models whose weights cannot coexist, each change may fetch weights from a slower tier, allocate runtime buffers, initialize execution paths and displace a warm model. The next request for the displaced model pays again. A 60 percent average can include long idle periods and short bursts of constant eviction. LRU may faithfully evict exactly the model that comes back next when the working set is larger than the pool's resident capacity.

I would reconstruct a request timeline by model and worker. Record routing, resident set before admission, load and unload duration, bytes moved from each storage tier, graph or kernel warmup, queue time, prefill and first token. Track model switches per worker, reloads per minute, and the requests delayed behind a load. Include KV cache and active requests in the memory budget. A model's weights fitting in free bytes is not enough if loading it evicts running KV state or leaves no safe headroom. NVIDIA Triton documents explicit model load and unload controls. The specific loading cost and residency behavior depend on the engine, model format, disk or network tier, and GPU layout.

Then size the concurrent working set. How many models receive interactive requests inside a latency window, and what is their burst pattern by region and tenant? If model A has tight interactive latency, reserve enough always-resident replicas for its peak and failure headroom. Put B and C on separate lanes if their switching would make A miss its deadline. A long-tail model with infrequent demand might use a cold lane with an honest startup time or an asynchronous contract. Route by model affinity so a request reaches an already warm worker, but avoid packing all of one model onto one overloaded worker while another compatible replica sits idle. Capacity planning needs both residency and queueing.

There is a trade. Pinning every model everywhere wastes GPU memory and may increase fleet cost. Replicating only the hot model can starve a rare but important customer. Test several placements against the actual arrival sequence, not a shuffled histogram that removes bursts. Report p50 and p99 by model, load events, successful requests per GPU-hour and the spillover one model causes to another. During rollout or zone loss, verify that the remaining cells can still keep the required resident set. Do not promise an interactive SLO for a model that has to be loaded from storage on every unpredictable request.

Suppose the interviewer says a bigger GPU has enough VRAM for all three models. That might solve the eviction loop, if it also has room for KV, runtime buffers and the target concurrency. Compare the cost and fault-domain implications against dedicated pools or a bounded hot set. Hundreds of tenants share one base model and many LoRA adapters is about hundreds of tenant adapters over a base model. This is whole base-model churn, where average free memory hides the repeated cost of switching the actual weights.