Should prefill and decode run on different GPUs?

Prefill processes the prompt and builds the attention state. Decode generates one new token at a time while reading previous state. Those phases ask for different things from the hardware and scheduler. A large prefill can occupy compute and make active decode requests wait, so their token gaps grow. But separation adds another queue, a routing decision, and a transfer of the prompt's KV state. I would not call it a win until the transfer and placement cost are in the same latency accounting as the saved interference.

Measure the baseline first. For each request, record prompt length, prefill queue time, prefill time, decode scheduling gaps, time to first token, time per output token, KV occupancy, and preemption. Slice by long input and long output. If active streams stall only when large prefills begin, the hypothesis is credible. If the delay is a front door queue or a congested output transport, a prefill pool may leave the symptom untouched. Try chunked prefill and a prefill budget in the colocated scheduler as a cheaper control, with an actual quality and latency comparison.

In a split design, a prefill worker accepts a routed request, computes the prompt state, and transfers compatible KV blocks to a decode worker that has reserved memory for them. Both sides must agree on model weights, tokenizer and prompt rendering, attention layout, cache dtype, and any adapter that changes the state. The coordinator tracks request identity and placement, and it must handle the prefill worker dying after computing but before transfer, a partial transfer, and the decode worker dying after accepting state. These are recompute or failover decisions, not a chance to attach another request's cache.

The slower link is the decisive probe. Estimate bytes transferred from the real model's KV shape and prompt length, then measure effective link throughput and p99 transfer latency under simultaneous traffic. In an illustrative calculation, 8 GiB of state over an effective 2 GiB/s link takes at least four seconds before queueing and protocol costs. That is already too large for a one second first token target. Compression or a different cache format may reduce bytes, but it adds compute and compatibility work. Placement near the decode pool or keeping shorter requests colocated may be better than sending every request across the slow link.

I would compare three policies on the same load trace: all colocated with tuned chunking, all split, and split only the long prefill slice that actually causes interference. Report goodput under both first token and per token SLOs, cost per successful request, wasted transfer, and p99 by prompt length. A split architecture may let prefill and decode use different parallelism or capacity ratios as the workload changes. It may also strand decode GPUs when prefill or the network is the bottleneck. The DistServe paper studied the benefit and bandwidth placement trade-off, while vLLM's disaggregated prefill documentation describes a concrete, currently experimental implementation. Neither benchmark proves the same win on this link.

If the long input share grows, revisit the ratio of prefill to decode capacity. If long outputs grow instead, the decode pool may become the limiter and extra prefill workers just build KV that waits for space. The phase boundary is useful only when it is scheduled and measured as one end to end request, with an honest fallback for transfer failure.