Model and Inference Engineering · Staff
The new model is ready. Why does the first unusual prompt take 30 seconds?
The question
Interview question
An inference server loads a new model and passes its readiness check. Short English prompts meet the first-token SLO. The first long prompt with images and a tool schema takes 30 seconds, while later similar requests are fast. The team says warmup ran successfully. Where do you look, and what should readiness mean for this service?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
“Ready” may mean the process loaded weights, or that it completed one specific warmup request. It cannot mean every possible prompt shape has been compiled and every cache populated. Start with a timeline for the slow request: queue wait, tokenization and image preprocessing, prompt rendering, compilation or graph capture, memory allocation, prefill, and first output. Compare the first and second request with identical inputs on the same replica, then across fresh replicas. A one-time cost tied to an input shape points somewhere different from general overload.
NVIDIA Triton's model warmup documentation says configured warmup requests run before an instance serves traffic. The effect varies by backend. Completing a short text request does not warm a long multimodal path. Some engines compile shape-specialized kernels or capture graphs lazily, while other bottlenecks come from image decoding, tokenizer initialization or tool-template processing. Verify the actual engine and version rather than assuming the mechanism from the latency pattern.
For a release, choose a bounded set of representative warmup classes from production distributions: short and long text, relevant image resolutions, tool schemas, output modes and batch sizes. Run them on every new replica before advertising full capability. Do not warm every combinatorial shape. That could make deployment take longer than its benefit and consume the very GPU capacity needed for a rolling update. If some rare class still pays a cold-path cost, decide whether to pin a ready pool for that class, prewarm on forecast demand, or offer a truthful asynchronous or degraded path. The readiness contract can be class-specific instead of one boolean for all traffic.
The first bad request may also happen when routing moves it to a replica that never saw that class. A canary on one warmed replica is not evidence that the other hundred are warm. Emit per-replica cold-path counters and latency by input class. If warming is expensive, stagger replicas and hold serving capacity while they prepare. Compare cost of warmup and fleet size with the actual tail benefit.
If the interviewer asks why not simply route the first request and let it pay the cost, the answer depends on the contract. For an internal batch queue, that may be fine. For an interactive SLO, hiding the first user behind “readiness passed” is inaccurate. Scale to zero makes the first model request miss its deadline covers a scale-to-zero request that waits for the whole model to start. This model is loaded and useful for common traffic. Its readiness probe sampled too little of the execution surface.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →