Eight GPUs are available. How should a model be split?

“Eight GPUs” is not enough information to choose. I need the model's weights at serving precision, runtime workspace, peak KV per live sequence, target prompt and output lengths, concurrency, and the memory and link topology. A model that barely fits across two GPUs can still fail at the first long prompt because weights are only one part of the memory budget. A placement that fits may miss first token or per token latency because each decode step crosses a slow link.

Here is the mental model I use:

If the model plus a useful KV budget fits on one GPU, start with replicas. Each can schedule independent requests and a slow link between nodes need not carry every token's intermediate state. If it does not fit, use the minimum parallel group that gives safe headroom and acceptable latency. Tensor parallelism within a fast connected node is often a reasonable first experiment for a model too large for one card. The collective communication cost repeats across layers and generation steps, so increasing tensor parallel width across a slow inter-node link can make decode slower even as per-device memory falls.

Pipeline parallelism places groups of layers on different devices or nodes. Inter-stage activations cross the boundary, but every request still has to traverse all stages. With a small number of active requests, stages wait for each other and the pipeline can have a large idle fraction. With enough concurrent sequences or microbatches it can fill better, at the cost of queueing and per-request latency. For two four-GPU nodes, a candidate might use tensor parallelism inside each node and two pipeline stages across nodes. That is a candidate to benchmark, not a formula that always wins. vLLM's parallelism guide documents these serving modes and their combinations.

The slow link changes the decision. Measure latency and utilization for a single stream, small interactive batch, and loaded batch. Capture cross-node bytes and collective time by layer or stage, per-stage idle time, KV headroom, time to first token, inter-token time, and completed requests within SLO. If a model fits in four GPUs with a smaller KV budget, two four-GPU replicas may beat one eight-GPU parallel group on this topology. If it only fits across all eight, quantify the unavoidable inter-node cost and whether the target can still be met. Quantization or a different model could be a better product decision, but it must pass a quality gate.

There is a follow-up about doubling QPS. Adding another eight-GPU group can add independent serving capacity. Doubling the parallel width of one group mostly changes how one request is executed. It does not automatically double useful throughput. Keep data parallel groups separate from the degree of model splitting. The vLLM data parallel deployment guide makes this distinction in its configuration, but the choice of group sizes should come from our memory and latency measurements.

If someone says “pipeline is for training,” I would correct that too. Pipeline and tensor splitting can both serve inference, though their scheduling costs differ from training. The final answer is a measured placement with memory headroom and a failure plan for one GPU or node, not a slogan that one parallelism type is always faster.