Every generated token extends a live sequence's history and can consume more KV capacity. We know the prompt size, an allowed maximum, and perhaps a task-specific length distribution. We do not know the actual stop point. An uncertainty-aware serving research paper studies this exact reservation tension. The basic choices are clear even without adopting its algorithm: reserve the full maximum and underuse the worker, reserve only current KV and risk a stall, or reserve a calibrated amount and grow it as the request progresses. max_tokens is a safety and product ceiling, not a prediction that the request will use all those tokens. vLLM exposes a generation maximum and configurable scheduler limits, but a product admission policy may live above the engine.

I would start with two budgets. The first is an admission budget for the known prompt plus a conservative initial output allowance by task and service class. The second is a small shared headroom pool for extensions. Charge actual KV occupancy and decode work as generation proceeds. Before crossing a grant boundary, the scheduler asks whether it can extend without pushing active streams below their latency floor. Interactive requests get protected headroom. Batch generations can wait, checkpoint where supported, or be preempted, with their deadline and recompute cost visible. A request must never exceed its hard model and product maximum even if headroom exists.

Predictions help only if calibrated on the current workload. Track underestimation by percentile, prompt type, customer, model, sampling settings, and stop behavior. A single “average output” gives no protection against the tail. Large initial reservations are also not free: they can strand memory while requests stop early. If an interactive output becomes much longer than its predicted class, we can extend in bounded increments or ask the application to continue in a new request, depending on whether truncation is acceptable. Quietly cutting off a legal answer because the scheduler guessed low is a correctness bug.

Client abandonment changes the accounting. Detect disconnects through the serving path, propagate cancellation to the engine, and release KV only when it confirms the work stopped. A disconnected transport may not imply cancellation for a durable task, so the application must say whether the generation is still needed. Measure useful completed tasks, wasted decoded tokens, preemptions, and inter-token stalls under the actual abandonment pattern. Reserving for all possible output and continuing every abandoned decode would be doubly wasteful.

The sharp follow-up is a synchronized wave of long interactive outputs. No clever forecast creates capacity. Bound admissions, preserve headroom for in-flight promises, and return overload to new callers before filling a queue that cannot meet deadlines. Fairness is part of that decision, but the mechanism here is the growing KV obligation of each admitted sequence.