Checkout is 99.9% reliable. Why did one bad hour consume so much budget?

Interview question. A checkout API has a 99.9% availability SLO over a rolling thirty-day window. The on-call team gets either no alert for a serious short outage or constant alerts for tiny fluctuations. Explain error budgets and design multi-window burn-rate alerting that pages for real danger without paging for noise.

At a planning meeting, somebody hears "99.9%" and says that sounds nearly perfect. Then an incident takes ten minutes and customers cannot place orders. You tell them the monthly target is still mostly intact. Technically, maybe. Operationally, that ten-minute outage spent a meaningful part of the time budget, and an ongoing outage could spend the rest faster than anyone realizes.

Start with the SLI. For checkout, I might define an eligible request as a user-initiated attempt that reaches our edge, and a good event as one that produces the expected checkout result within a latency deadline. Valid card declines should not be failures of availability merely because they produce a business rejection. Internal errors, timeouts and broken sessions may be bad events. We should explicitly handle duplicate submissions, monitoring probes, and requests where the gateway never observed a client connection. A vague "HTTP 200 ratio" is not a complete SLI.

Suppose the target is 99.9% good events. The allowed bad-event fraction is 0.001, or 0.1%. Over thirty days, if request volume is roughly stable, this corresponds to 43.2 minutes of equivalent full outage. For a request-based SLO the actual budget is measured in eligible requests, not literal downtime minutes. The time equivalence is only a helpful approximation under steady load. This distinction matters when a payment incident happens during a huge seasonal sale.

Now define burn rate. It is the observed bad-event fraction divided by the allowed fraction. If 1% of eligible checkout attempts are bad, a service with a 0.1% budget is burning at 0.01 / 0.001 = 10 times the sustainable rate. If all requests fail, the instantaneous burn is 1,000 times the target allowance. The SLO might still be technically passing over thirty days when the incident begins, yet waiting for it to fail before alerting would be reckless.

A naive five-minute error-rate alert sees every brief hiccup and pages on-call constantly. A naive thirty-day-window alert reacts so slowly that the budget can vanish before someone is notified. Google's SRE Workbook presents multi-window, multi-burn-rate alerting as one reasonable compromise. For a thirty-day 99.9% SLO, a fast page condition often combines a one-hour window with a five-minute confirmation window, both above 14.4x burn. A slower page uses six hours and thirty minutes above 6x. The short window helps the alert clear after an incident instead of remaining active only because old bad requests are still inside a longer window.

Where do 14.4 and 6 come from? At 14.4x for one hour, budget consumption is 14.4 × 1 / 720 = 0.02, or 2% of a thirty-day budget. At 6x for six hours, it is 6 × 6 / 720 = 0.05, or 5%. These are policy choices, not constants of nature. The combined long-and-short check asks whether the problem is material over the long window and still happening recently. Since an alert rule evaluates over rolling windows, real firing timing also depends on scrape intervals, traffic, and the for duration if configured.

Imagine ten minutes of 100% failure followed by complete recovery. In a simple stable-volume model, that would consume 10 / 43.2, about 23% of the month's equivalent time budget. The fast-window condition will likely detect the outage, but not instantly at the first bad request. The long-window average has to cross its threshold. We should test rule evaluation against synthetic incidents of different lengths and traffic levels rather than assume the chosen thresholds meet the team's desired detection latency.

What if a low-volume service has only three requests a minute? A handful of failed attempts can produce enormous ratios and noisy alerts. Use minimum sample-volume safeguards, longer windows or explicit critical-event signals suited to the product. Those mechanisms trade detection speed for statistical confidence. Do not silently hide a severe event just because aggregate traffic was small. Checkout payment capture failures may deserve a dedicated correctness alarm regardless of aggregate SLO burn.

I also distinguish page from ticket. Rapid consumption of the remaining budget justifies waking somebody who can mitigate customer harm. A slow leak may justify scheduled engineering work. An error budget is a tool for deciding how aggressively to release and operate a system, not a promise that being under target makes every incident unimportant. If budget consumption persists across a series of releases, the organization may slow risky deployments until corrective work restores room for change.

In an interview, the strong answer isn't a memorized PromQL snippet. It includes a defensible SLI, a numerator and denominator you can actually measure, a measured error budget, short and long alert windows, low-traffic concerns, and a rehearsal with recorded incidents. Otherwise the team has invented an attractive percentage, not an operational reliability program.