An MoE layer does not run every expert for every token. A router chooses a small set of experts and sends token activations to them. The serving cost depends on where those experts live, how many tokens choose each one, the communication needed to move activations, and how long the slowest participating group takes. Forty percent average GPU use can hide one saturated expert rank that every decode step waits for.

I would trace a slow request through routing and execution: tokens assigned per expert and per rank, queue age at the expert, all to all or other transfer time, compute time, and the waiting time of the other ranks. Compare with the old traffic mix and separate prompt prefill from decode. The culprit might be a popular expert on one rank, a link with poor placement, a changed batch shape, or a fallback kernel. “The model is sparse” is not enough to tell which. vLLM's expert parallel deployment guide documents expert distribution and load balancing, including the possibility of skewed routing under real traffic. Its specific controls are implementation details, not universal MoE behavior.

If routing counts show the same expert chosen much more often, I would first ask whether the mix shift is legitimate. Did one tenant start sending repetitive code tasks? Did a prompt template change? Did the router's inputs change because of a model or tokenizer rollout? A serving rebalance can fix placement, but it should not hide a bad rollout. Slice goodput and output quality for the new traffic. If experts are hot for valid tasks, place or replicate frequently used expert weights where capacity permits, or rebalance the expert to rank mapping with enough headroom for transition. Keep the model's router behavior and output contract fixed while changing where an expert's computation runs.

The probe rules out retraining. Do not propose sending tokens to a different expert merely because it is idle. That changes the function the model computes and can damage quality. Replicating the same expert and distributing its assigned tokens can preserve the intended weights if the implementation supports it. It costs memory, may reduce KV capacity, and may add routing or transfer overhead. A dynamic load balancer can chase a temporary spike and cause extra movement, so use a measurement window and rate limit remapping. Test the distribution and per-token latency before and after on the actual workload.

What if the bottleneck is the network rather than expert arithmetic? Moving weights to another rank across the same congested link may make p99 worse. Measure topology and the size and frequency of token dispatch. Group or place hot experts to reduce costly traffic if the model implementation permits it. Under immediate overload, cap admission for the new slice or route it to an existing replica with available expert capacity, keeping data and model version constraints. Do not report a general capacity win from mean GPU utilization.

I would accept a rebalanced map only if the p99 per output token and completed requests within SLO improve under the new mix, without a regression in output quality or a collapse in KV headroom. Then watch whether the hot experts migrate with the traffic. The key first principle is that sparse compute can still have a serial bottleneck at the most demanded expert and its communication path.