When can continuous batching improve throughput and make a short request slower?

Take one short request from arrival to first token. It waits to be admitted. Its prompt has to be processed in prefill. Only then can it enter decode and produce the first output token. Continuous batching keeps adding and removing requests at scheduling steps so the GPU spends less time idle. Higher aggregate throughput says the GPU is doing more useful work. It does not say each class of request gets into the next batch promptly.

The long requests matter in two ways. Their prefills consume a lot of token budget and compute. Once admitted, their long generations remain active across many decode steps and occupy KV cache. A scheduler may protect the inter-token latency of these active streams by taking their decode work first. If it uses the remaining budget for prefills, a newly arrived short request can wait through many iterations even though its own prefill would be cheap.

Chunked prefill helps prevent one huge prompt from monopolizing a whole scheduling step. It does not create unlimited prefill capacity. In vLLM's documented V1 policy, pending decodes are batched first and prefills use the remaining token budget. Its tuning guide says a smaller max_num_batched_tokens tends to favor inter-token latency, while a larger value can improve time to first token by letting more prefill work into a batch. That is a useful example of the trade, not a universal knob value for every engine.

“Decode priority is the only reason our streaming feels smooth. Are you going to remove it?” No. I would first find how much prefill capacity is left after decode and whether an aging or reserved budget can admit short work while preserving the streaming target. If the fleet is genuinely saturated, scheduling cannot manufacture capacity. We need to reject, queue with an honest estimate, move work, or add capacity.

I would split the short request's latency into queue time, prefill time, and time until its first decode output. Compare each distribution before and after batching, by prompt length, expected output length, tenant, model, and serving pool. I would also inspect active sequences, waiting requests, token budget utilization, KV cache occupancy, cache preemptions, and prefix hit rate. A p99 regression caused by waiting needs a different fix from one caused by more expensive prefill.

Here is a possible trace, with invented numbers only to show the mechanism:

The short prompt did not become harder to compute. It waited while the fleet was busy with a different shape of work. If queue time stayed at 30 ms and prefill rose instead, I would investigate prompt construction, prefix caching, kernel behavior, and context length rather than blame decode priority.

What would I change? First, protect the product class that has a strict interactive latency objective. A separate pool for long analysis work is simple to reason about if capacity allows it. Otherwise use admission by estimated live tokens and a scheduler policy that gives short prefills a bounded opportunity without starving active streams. Limit excessively long prompts or output budgets where the product permits it. Test a few max_num_batched_tokens settings on a traffic replay, looking at p95 and p99 for both classes, completed tokens per second, KV preemptions, and cost per completed task. A change that improves short TTFT while causing frequent decode stalls may be the wrong trade.

Then they reveal that the short requests and long requests belong to different paying tenants. Now a single global p99 is especially misleading. The admission and service class contract should stop one tenant's long generations from consuming another tenant's reserved interactive capacity. That is a platform fairness decision as much as a kernel tuning decision.

The vLLM optimization guide documents its current chunked prefill and decode priority behavior. The numbers in the trace are illustrative.