I would ask which tokens are counted. Prompt tokens and generated tokens do different work. An aggregate output token rate at a saturated batch says how much work the fleet completed in that benchmark. It does not say how long a person waited to see the first token or how often a streaming response stalled. A benchmark can increase batch size, keep the GPU busy, and make one request wait longer. It can also use uniform prompt lengths, one model, warm prefixes, and no failed or cancelled requests while production has none of those conditions.

For a request, separate arrival to admission, queue, prefill, first token, each subsequent token, and completion. Time to first token includes waiting and prefill, not just GPU execution. A reported time per output token is often averaged over a request, while a user sees individual inter-token gaps. vLLM's metrics design distinguishes a gap between streamed events from a per request average over the whole output. They are not interchangeable, especially when one event bundles multiple tokens or a request pauses and catches up later.

The product success metric should count requests that meet both latency conditions and produce an acceptable result. Call that SLO goodput if the denominator and quality check are explicit. The DistServe paper uses TTFT and per output token targets to discuss goodput, but our success definition also needs errors, cancellations, and task quality. A raw token rate can be high when a small set of long batch jobs dominates the output, while short interactive requests miss their first token deadline. Break the data out by prompt size, output length, tenant, model, cache hit, and traffic class.

I would rerun the benchmark with the production arrival pattern, realistic length distributions and stopping rules, multiple concurrency levels, and a warmup that does not hide cold capacity if cold starts occur in production. Report completed good requests per second and per GPU hour under the stated p95 or p99 requirements, along with aggregate input and output tokens, queue age, KV occupancy, preemptions, and cost. Identify whether the 40 percent failures were late to first token, slow between tokens, errors, or low quality. Those have different fixes.

Now outputs double at the same arrival rate. Even if prompt work stays constant, live sequences last longer, generate more tokens, and hold KV longer. As a simplified illustration, 100 requests per second averaging 1,000 output tokens require 100,000 output tokens per second. At 2,000 tokens they require 200,000. If the pool could deliver 120,000 tokens per second at the required latency for the old mix, the new demand is above that simplified capacity and queue debt grows. Real token capacity is not a fixed scalar because sequence length and batch shape change execution, so measure the new mix. The arithmetic is enough to reject “QPS stayed flat” as an argument that supply is adequate.

The immediate response might be tighter output budgets, a different model route for tasks where quality permits it, extra warm replicas, or admission that tells callers to retry later. A queue with no bounded age makes the apparent throughput look good while new requests wait beyond usefulness. If a team proposes larger batches to recover token rate, check whether that moves the first token tail or per token stalls in the wrong direction. The answer is not one optimized number. It is a workload and a latency contract that the fleet can actually meet.