Tensor parallelism makes the ranks one serving unit. Each layer partitions work, then exchanges partial results. The next token cannot finish when seven ranks are ready and the eighth is late. So collective time on all ranks is a symptom, not eight independent network faults. A rank delayed by a slow kernel, CPU launch jitter, memory pressure, or a bad link can leave peers waiting inside the collective. Average GPU utilization hides the critical path of each decode step.

I would align a single request's token steps across ranks. At each step, record when work was scheduled, when the local kernels began and ended, when the collective started and ended, and when the token became available. The rank that enters a collective last may be the source of the wait. If all enter together and exit late, investigate the actual transport, topology and collective size. Compare a healthy replica on the same model and batch shape, then compare within-node and cross-node placement. Look at link counters, retransmits, GPU health, CPU affinity, synchronization and queueing. NVIDIA's NCCL diagnostics can validate intended paths at communicator initialization, but a clean startup test does not prove that a loaded link stays fast under production traffic.

Do not jump from “collective time is high” to changing an NCCL knob. A late rank can make everybody's collective timer high. Conversely, a shared fabric problem can make even the fastest rank wait. Use a controlled replay with the same batch and prompt/output mix on another topology. If the issue follows a GPU, investigate that host. If it follows a link or placement, keep tensor-parallel groups inside a faster failure and communication domain where memory permits, or pay the measured cross-node cost knowingly.

During the incident I would stop sending new work to the bad group and move traffic to healthy capacity with admission limits. An in-flight generation holds rank-local KV and scheduler state, so replacing one rank with a fresh process is not a transparent continuation unless the engine actually supports compatible state recovery. Mark interrupted attempts honestly and restart them as new attempts when appropriate. Watch per-token deadline success and useful completed requests after the move, not only aggregate tokens per second. Adding replicas on the same faulty topology can make this incident larger.