The assigned request is not isolated from the other arm. B changes how much memory and scheduler time is left for A. If B evicts A's KV, causes preemption, or fills the shared queue with longer decodes, A's observed latency is partly an outcome of B's presence. A request-level difference in that mixed pool does not estimate what would happen if the fleet moved entirely to B. Even if the assignment itself is random, the shared machine is a channel for interference.

I would first verify the mechanics. Pin actual model and scheduler versions, request length distribution and routing. Trace queue wait, prefill, decode, KV occupancy and eviction, preemption, batch composition and worker assignment by arm. Check whether B's dashboard starts its clock at admission while A starts at gateway entry, or whether one arm loses failed and timed-out requests. Compare all assigned requests and completed useful work, not only successes. A better first token for B can coexist with worse per-token latency and less goodput for the service.

There are two valid questions to ask, and they need different tests. If the product plans to serve a 50/50 mixture indefinitely, measure that mixture as the treatment system against the existing all-A system. If the decision is a full rollout to B, estimate an all-B system. Randomizing individual requests inside one saturated pool measures neither cleanly. Assign comparable worker pools or capacity cells to configurations, with enough independent cells to estimate uncertainty, and keep a buffer so cross-pool admission, cache and autoscaling do not leak between arms. Balance hardware, region, time of day and tenant workload. If complete isolation is impossible, quantify the residual sharing and run controlled load sweeps at several B proportions, including the proposed steady state.

The cost objection is real. Dedicated pools can temporarily underutilize GPUs. That is the price of learning the correct system-level effect. A shorter, smaller experiment may be cheaper than rolling out a version that degrades everyone's p99. We can use paired time blocks or switchback periods on the same cells if workloads and caches can be reset or modeled, but beware carryover from warmed weights, prefix cache and queued work. A before-and-after comparison during changing traffic is weaker. Google research on experimental interference explains why clustering can help when one unit affects another. The specific GPU-cell design here is the proposed engineering response, not a claim that one statistical method automatically solves every shared scheduler.

A rollout decision should report p95 and p99 user latency at the gateway, time to useful completion, quality and safety, tokens and completed tasks per GPU-hour, GPU memory headroom, failures and queue debt under representative concurrency. Show the result for B users and the spillover to A users. If B is valuable for short prompts but harms long ones, a routed policy might outperform either uniform fleet, but that is a new system to evaluate under its own mix and capacity constraints.

Suppose the experiment is run at low load and B wins cleanly. That does not answer peak behavior, where KV contention appears. Test the actual target QPS and burst distribution with enough run time for caches and queues to reach representative states. Do not claim B is faster from one arm's median first token while the shared platform finishes fewer useful requests.