The GPU can only prefill requests that reached it. Before that, the service may authenticate, apply a chat template, count and tokenize input, assemble multimodal pieces, admit a request, and schedule a GPU batch. After a token is produced, CPU work detokenizes, formats and streams it. A dashboard that starts timing at model execution excludes the part where the user is waiting. Current vLLM optimization guidance identifies CPU input processing, scheduler work and output processing as possible bottlenecks when GPU utilization is low. That is a reason to measure them here, not proof that tokenization alone is the culprit.

Put a trace clock at request receipt, template completion, tokenization completion, engine admission, prefill start, first token, detokenization and client flush. Compare time distributions by prompt tokens, raw bytes, template version, tenant, CPU host, and arrival burst. A long code file can be expensive to tokenize even if the eventual token count looks manageable. A single-threaded template or a shared process pool can produce a queue while GPUs wait. Check CPU saturation, cgroup limits, run queues, allocator or GC pauses, and scheduling time inside the engine. Measure offered demand and waiting requests, including ones that time out before model admission.

To isolate the stage, replay a permitted sample with a fixed pretokenized input in a test environment and compare it with the ordinary frontend path. Preserve the exact tokenizer and template version so this comparison does not silently change the request. If the frontend is the limit, add appropriately sized CPU workers, keep them near their GPU or NIC where locality matters, and bound the intake queue. Cache only deterministic preprocessing whose key includes the prompt and policy-relevant version. Extra GPUs behind the same saturated frontend will mostly sit idle.

The stream pause changes the investigation. Check per-token engine scheduling, detokenization, serialization, network flush, and client backpressure. It might be the same CPU pool that was slow before prefill, or a separate decode or transport fault. One trace per token can tell us where the first growing gap appears. If model steps become slow under larger batches, the original “unchanged forward-pass time” aggregate was hiding a workload slice. Fix the measured critical path and retest completed answers within both first-token and inter-token targets.