Put the cold path on a timeline before deciding. An arrival may wait for autoscaler observation, GPU allocation, container and runtime start, model weight fetch, GPU weight load, KV allocation, compilation or graph warmup, and finally its own prefill. The durations vary by checkpoint locality, model size, storage bandwidth, GPU availability, and runtime. A warm pod or cached file does not imply weights are ready in GPU memory. The ServerlessLLM paper treats checkpoint loading and locality as first-class scheduling costs. KServe's autoscaling documentation shows scale-to-zero and a concrete cold-start effect for its example, but the example's seconds are not a promise for this model.

If measured worst-case cold start exceeds the two-second budget, zero ready capacity cannot honestly satisfy an always-on SLO for an unannounced request. Keep at least one verified ready replica for the contracted traffic, or change the product contract to an asynchronous or delayed path at low traffic. A sleep or standby state may save some GPU memory while keeping host weights available, but wake time and reserved host resources still count. A smaller eligible model can take first traffic only if its quality and data policies meet the use case. A queue behind a waking large model is a choice to defer service, not a latency fix.

The burst needs its own plan. Autoscale based on leading signals such as arrival rate, pending token work, and queue age, with cooldown to avoid oscillation. Start loading extra replicas early enough and route only when they pass a readiness check that includes a real inference, not merely a listening port. Set a bounded queue and prioritize deadlines. If requests arrive faster than capacity becomes ready, reject or route through an approved degraded mode while preserving user expectations. Overadmitting work during load only makes the first healthy replica miss more deadlines.

I would compare total cost and goodput under three actual traces: warm minimum, scheduled prewarming, and zero with cold starts. Include late arrivals, model updates, and region failover, not just an overnight mean. If the SLO covers 99 percent of requests over a month and the first request after idle is rare, a team may consciously accept that miss, but only after calculating the error budget and agreeing with product. “Two seconds at any hour” as stated requires a ready path, not hope that an autoscaler can move model weights instantaneously.