Model and Inference Engineering · Principal
A rolling model deploy runs out of GPU capacity
The question
Interview question
Eight GPUs run eight replicas of a model and meet the latency SLO. A new checkpoint needs more weight memory and less KV headroom. A rolling deployment either cannot place the new replica or removes an old one and overloads the rest. Design a safe rollout and rollback. There is no spare GPU.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The rollout has a temporary capacity requirement that the steady-state plan omitted. A new process has to load weights, allocate runtime and KV memory, warm kernels, and prove readiness while the old model still serves traffic. Even if the new weights fit one GPU, fewer live sequence slots may lower completed requests at the latency target. Kubernetes rolling updates bound extra and unavailable Pods through maxSurge and maxUnavailable. Those counts do not manufacture a GPU or guarantee that a loaded model has enough KV capacity.
I would first measure the new checkpoint on the actual traffic shape. Record model load and warmup time, memory after weights and workspace, usable KV blocks, time to first token, per-token latency and SLO goodput at each concurrency point. Keep old and new versions as separate serving groups. Pin each request to one model version for its full generation. A pod becomes routable only after weights, tokenizer and serving configuration match the expected revision and a small readiness inference succeeds. It is not ready just because the HTTP port opened.
With no spare GPU, there is no free zero-loss rollout. I would create room deliberately. Pause elastic batch traffic, shed or defer low-priority work, and reserve enough headroom for one new group. Then move a bounded slice to the new model, watch quality and per-version latency, and release the old capacity in measured steps. If the interactive SLO still needs all eight old replicas at peak, schedule a lower-demand window or acquire capacity. If neither is possible, do not claim a seamless update. An explicit temporary admission limit is more honest than letting the queue silently grow.
Rollback needs room too. Keep the old checkpoint and runtime configuration available, and retain at least a viable old serving group until the new model's quality and capacity are established. If the new checkpoint changes tokenization or tool format, do not migrate an in-flight stream or its KV by relabeling it. Drain old requests under a bounded deadline and report the ones that cannot finish. Suppose the new version has better offline quality but 25 percent lower SLO goodput. That is a product and capacity decision, not a green deployment status. Either fund the extra fleet, reduce admitted load, or decline the rollout.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →