First I would verify that a fallback really explains the tail. Record the execution path chosen for each request alongside actual prompt length, decode length, batch at each scheduling step, modality, cache state and time in queue, prefill and decode. A request can enter with an ordinary length and later be batched with an unusual mix. Long output might make decode dominate. A graph miss can also be caused by a different feature or operator rather than sequence length. The engine version and its capture configuration matter. For example, vLLM documents different CUDA graph modes and capture sizes and describes an eager fallback when a multimodal vision input exceeds its captured budgets. That is evidence of concrete paths in that engine, not a rule that all engines fall back the same way.

The benchmark should replay the joint shape distribution, not just a histogram of input tokens. Measure request rate, concurrency, prompt and output length together, image or tool modes if supported, prefix-cache hit rate, and the scheduler's resulting batch shapes. Report latency and work by path: graph replay, compilation or capture miss, and normal eager execution. A fallback that affects two percent of requests may account for most SLO misses if those requests occupy the GPU longer and delay neighbors. A p99 chart alone cannot tell whether the rare requests are slow or whether they slow the common ones behind them.

Then choose a bounded response. If a few high-volume shape classes miss a fast path, add capture sizes or buckets and warm those paths before admission. Check the extra memory and startup time because graph capture is not free. If the generic path is valid but costly, route that class to a separate capacity pool or give it a different deadline and truthful estimate. If an operation is unsupported by the optimized path, fix or specialize it only when the traffic and quality benefit justify the work. A generic path can be the right compatibility behavior. Blindly rejecting unusual shapes may discard the users whose tasks most need the service.

I would test with a production-weighted replay, then hold out recent traffic so the newly chosen buckets are not just tuned to yesterday's sample. Include canary traces that prove each intended path actually runs. Protect the fleet while new shapes appear with admission headroom and an overload policy. If the interviewer says “GPU utilization is only 55 percent,” ask over what interval and whether the stalled request is waiting on a CPU preprocessing step, compilation, memory allocation or a serialized GPU path. Average utilization does not disprove a shape-specific p99 problem. A tokens per second benchmark looks great. What did it measure? asks what a tokens-per-second number represents. This answer connects a missed execution path to actual traffic shapes and a capacity decision.