“Ready” may mean the process loaded weights, or that it completed one specific warmup request. It cannot mean every possible prompt shape has been compiled and every cache populated. Start with a timeline for the slow request: queue wait, tokenization and image preprocessing, prompt rendering, compilation or graph capture, memory allocation, prefill, and first output. Compare the first and second request with identical inputs on the same replica, then across fresh replicas. A one-time cost tied to an input shape points somewhere different from general overload.

NVIDIA Triton's model warmup documentation says configured warmup requests run before an instance serves traffic. The effect varies by backend. Completing a short text request does not warm a long multimodal path. Some engines compile shape-specialized kernels or capture graphs lazily, while other bottlenecks come from image decoding, tokenizer initialization or tool-template processing. Verify the actual engine and version rather than assuming the mechanism from the latency pattern.

For a release, choose a bounded set of representative warmup classes from production distributions: short and long text, relevant image resolutions, tool schemas, output modes and batch sizes. Run them on every new replica before advertising full capability. Do not warm every combinatorial shape. That could make deployment take longer than its benefit and consume the very GPU capacity needed for a rolling update. If some rare class still pays a cold-path cost, decide whether to pin a ready pool for that class, prewarm on forecast demand, or offer a truthful asynchronous or degraded path. The readiness contract can be class-specific instead of one boolean for all traffic.

The first bad request may also happen when routing moves it to a replica that never saw that class. A canary on one warmed replica is not evidence that the other hundred are warm. Emit per-replica cold-path counters and latency by input class. If warming is expensive, stagger replicas and hold serving capacity while they prepare. Compare cost of warmup and fleet size with the actual tail benefit.

If the interviewer asks why not simply route the first request and let it pay the cost, the answer depends on the contract. For an internal batch queue, that may be fine. For an interactive SLO, hiding the first user behind “readiness passed” is inaccurate. Scale to zero makes the first model request miss its deadline covers a scale-to-zero request that waits for the whole model to start. This model is loaded and useful for common traffic. Its readiness probe sampled too little of the execution surface.