I would start by refusing to call the whole fleet one pool. A GPU with model A's weights loaded is not immediately spare capacity for model B. Loading and warming a model takes time, and the space left after weights and runtime overhead determines how many live sequences that worker can hold. The real supply is capacity by model and configuration, under an input and output length mix, at the required latency. An aggregate GPU utilization number cannot answer whether an interactive request for model B will get its first token on time.

Give each interactive model a baseline of warm capacity sized from a production shaped load test. Keep headroom for bursts and failures, then let batch work borrow the measured slack. I would rather account for the price of that idle headroom than sell a p95 that depends on batch jobs politely finishing at the right moment. A job with a 50,000 token prompt can consume prefill time. A long output can occupy decode slots and KV memory for much longer. One request is not one unit of capacity.

At admission, estimate input tokens from the actual tokenizer, reserve a bounded output budget, and check the deadline against queue work and available KV slots. The estimate will be wrong sometimes. Update the reservation as generation proceeds and cap or renegotiate unusually large outputs. Separate queue and concurrency budgets for interactive and batch within each model pool. If a request cannot plausibly begin before its deadline, reject it quickly or give a product approved alternative instead of hiding it in a queue that already missed the promise.

What should the scheduler do inside a busy pool? Protect interactive queue age and time to first token. Batch can be scheduled in smaller chunks or paused at a supported boundary, but preemption is not free. Depending on the engine it may retain, transfer, or recompute KV state. Repeatedly evicting a nearly finished batch sequence to save every arriving interactive prompt may burn more compute than reserving enough headroom in the first place. I would measure actual pause and resume cost, not write “preempt batch” as if it were a zero cost operation. vLLM's production metrics expose queue, prefill, decode, and preemption observations that can help distinguish these effects on that engine.

I would not let batch starvation be an invisible side effect. Batch jobs get a completion objective, but a looser one than interactive. Publish admission limits, queue age, projected completion, and the share that batch is actually receiving. Batch may borrow interactive slack when present. Interactive cannot quietly borrow every batch reservation forever if the product also promised a nightly completion time. When a model's interactive load grows, add capacity, reduce accepted batch work, or revise that promise explicitly.

Now the rollout halves available KV memory on some workers. That could be because the new weights occupy more memory, its runtime reserves more space, or its attention cache has a different footprint. I would verify the cause and calculate how many live token slots remain at our actual prompt lengths. The old admission model is no longer valid. Stop sending batch to the new pool, bound context and output sizes for the canary, drain or migrate only work that has a safe checkpoint, and keep enough old workers to serve pinned traffic. If interactive p95 or preemptions move, pause the rollout. Halving KV capacity can make the queue fail even with flat QPS and GPU compute utilization.

Could we just run the new model on every GPU and add more GPUs later? Only if the new model's memory and latency have been tested with the concurrent workload and the extra capacity is already available when traffic moves. Model residency, KV capacity, and request mix all change the packing problem. I would canary by worker pool and traffic class, compare completed good responses per GPU hour at the p95 target, and hold a route back to the old snapshot. A fleet that is technically processing more tokens but violating the interactive promise is not a successful rollout.

The decision I want to see from a candidate is where admission happens and who has a claim on scarce capacity. The implementation can use one serving engine or several. Its contract must still explain what happens to a late interactive request, an already running batch job, and a model that no longer fits the memory plan.