I would ask what was quantized. Weights, activations, and KV cache are different memory and error surfaces. A change to weight precision can free room for more KV cache without changing the KV representation at all. Some schemes need calibration, and some kernels make a low precision format faster on one GPU but not another. vLLM's quantization documentation lists multiple formats and hardware paths. “Quantized” is not one reproducible treatment. Pin the base checkpoint, tokenizer, quantization recipe, serving engine, kernels, and hardware before comparing.

Then I want the failures, not only a mean score. Are the enterprise cases about exact product versions, long tables, rare language, code identifiers, or tool argument selection? Did the new model choose the wrong answer, lose a qualifier, or merely use different wording? If a query about a payment rule becomes confidently wrong, the slice may be rare in traffic but high in consequence. A two point aggregate win in latency does not make that error acceptable. I would collect independent examples around the mechanism and have a reviewer label the needed evidence and decision, while keeping a held out set that the quantization team cannot tune to repeatedly.

The experiment should isolate quantization from capacity. Run the original and quantized versions on the same prompts and evidence at low load to find a model behavior difference. Then load test both with the same arrival trace, prompt and output lengths, streaming clients, and target p95. At high load the smaller version may have fewer queue delays, so the user visible outcome can improve even if isolated answer quality falls a little. Report both effects. Do not label a queueing gain as proof that the quantized model reasons equally well.

I would set a gate before the rollout. For example, no new unsupported claims on high consequence policy and finance cases, a bounded degradation on ordinary tasks after human adjudication, and a measured improvement in completed correct tasks per GPU hour at the required latency. Those bounds are product decisions, not universal percentages. If the rare slice cannot pass, keep it on the original model or give it a verified fallback. That route has its own capacity and latency cost and needs a reliable way to identify the slice before the answer is generated.

Twice as many concurrent sequences is useful, but it is not twice the throughput. The model still performs computation and reads weights and KV data for each generated token. A larger live batch can increase useful hardware occupancy until memory bandwidth, compute, or scheduler overhead becomes the next wall. It can also make each stream slower. I would measure good completions per second and GPU hour at the interactive p95, preemption rate, time to first token, output speed, and energy or rental cost. Count the cost of a wrong answer, review, or fallback as part of successful task cost, not as an invisible product problem.

What if the regression shows up only after thousands of long turns? Then a short benchmark did not test the condition that matters. Trace a few full conversations with fixed source versions and compare where the first decision diverges. Check whether it is weight precision, KV precision if changed, context packing, or stochastic decoding. A single sampling run per case is weak evidence. Use repeated paired runs where generation is nondeterministic and keep the prompts identical. If the issue is a broken kernel or configuration, fix that before deciding that lower precision itself is unacceptable.

I would ship a narrow canary only after the consequential slice passes its gate. Start with traffic where the behavior is understood, monitor answer support and user outcomes with sampled review, and keep the original route warm for rollback. If the quantized model buys real capacity but still fails a critical slice, the answer is a conditional deployment with explicit routing or no deployment for that workflow. The extra sequences are a benefit to spend, not permission to spend correctness.