Those virtual users are a closed loop. When responses stall, they stop sending new requests. The test reduces offered load precisely when the service is weakest, so it may hide the queue that a steady arrival stream would build. Its latency histogram also sees only requests that were actually sent and completed inside the measurement window. Scheduled demand that never became a request, timeouts omitted from the histogram, and unfinished work can vanish from the p99. This is a form of coordinated omission. Gil Tene's wrk2 documentation describes constant-throughput generation and latency measured from intended send time to address it.

The arithmetic makes the gap visible. If production demand is 100 requests per second and the gateway serves none for five seconds, roughly 500 arrivals accumulate, before considering what was already queued. A closed loop with 100 clients has at most 100 outstanding requests and does not generate the other planned arrivals while each client waits. On a long test with tens of thousands of completions, those 100 stalled requests can also occupy less than one percent of the completed histogram, leaving its p99 near the target even if their own latency is awful. This is illustrative arithmetic under a steady arrival assumption, not a claim about the actual benchmark's exact histogram.

I would test at a controlled offered rate independent of completions, with a generator that has enough capacity to keep its schedule and records when it cannot. Measure from the intended arrival time through admission, queue, prefill, decode, and client delivery. Count rejected and timed-out requests in the denominator for SLO goodput and report their separate latency or deadline outcomes. Keep a histogram of successful end-to-end latency plus queue age and unfinished requests. A p99 of successes with 20 percent errors is not a healthy p99.

Use the real workload mix: prompt length, output length, model routes, tool calls, tenants, prefix reuse, and burst shape. Run below, near, and above sustainable capacity, then test recovery after a stall. The goal is not one maximum QPS number. It is the highest offered load at which the chosen quality, latency, and error objectives hold, with queue debt clearing within an acceptable time. HdrHistogram's recording guidance can help account for expected intervals, but a mathematical correction cannot reconstruct missing server traces or prove the load generator offered the intended requests. Instrument the schedule and actual sends.

What if the real product is genuinely closed loop, such as one user who waits for each answer before asking another? Then a closed-loop test is useful for that interaction model. It still needs to report long waits and incomplete sessions. If thousands of independent users arrive while others are waiting, the system's aggregate demand is not bounded by one blocked client. Test the production arrival process instead of choosing the model that makes the chart look good.

If the generator itself cannot keep up at the intended rate, that invalidates the target-rate run. Scale or distribute the generator, verify its clocks and network, and show offered versus achieved load. Never call the lower achieved rate the service's capacity. I would reproduce the five-second pause deliberately and verify that the test reports a backlog, degraded goodput, and the later recovery tail.