The recommendation service is taking five seconds. Why are we still calling it?

Interview question. An online store's product endpoint normally calls recommendations in 40 ms. During a sale, that dependency slows to five seconds but keeps returning successful responses. Product-page p99 rises, request threads accumulate, and a team proposes a circuit breaker. Design one that actually helps, and explain when it should not open.

A product page has three jobs. It must show the product, the price, and whether the item can be bought. It also displays recommendations. Someone reports that checkout is slow because the recommendation engine is slow. At first this sounds like an easy fix. Stop calling recommendations when it fails. It answers every request, eventually. By then the customer's request deadline has passed and the frontend has spent seconds holding sockets, memory, and possibly a worker thread for an answer nobody can use.

The first design choice is not the breaker threshold. It is the dependency contract. Recommendations are optional for displaying a product. Inventory authorization is not. The product page should have a deadline for its entire operation, allocate a smaller explicit sub-budget to recommendations, and know what to render if that sub-budget expires. A breaker is another layer: it prevents repeated calls into a dependency whose observed behavior suggests that more calls are unlikely to help. A timeout limits one attempt, the breaker changes whether future attempts are made at all.

Consider a process serving 300 product requests each second. Normally recommendations take 40 ms, so under steady conditions the mean number of these requests in flight is approximately 300 × 0.040 = 12. At five seconds it becomes 1,500 if all 300/s are still completing and admitted. That is only an illustrative Little's Law comparison, not a claim that the system can actually sustain 1,500 operations. In practice an earlier concurrency bound, connection pool, or request deadline may stop it. What matters is the direction: slow successes can consume capacity just as reliably as errors. Set a 120 ms recommendation deadline in this hypothetical, based on actual product latency objectives and measured healthy tails, not because 120 is a universal best practice.

Now decide the scope of observation. Suppose the breaker watches the last 100 completed calls, requires at least 40 samples, counts timeouts and qualifying slow calls, and opens when at least half are bad. A ten-second wall-clock window behaves differently from a 100-call window during a quiet night. Make the rules explicit, inspect per-dependency and per-operation metrics, and avoid mixing /recommendations with a critical payment authorization that happens to use the same host.

In the closed state, calls proceed and the breaker records outcomes. In the open state, optional calls fail fast without occupying the recommendation pool. A local product renderer can omit the widget or use a known-safe snapshot with documented freshness. It should not silently substitute unrelated recommendations that might violate merchandising or privacy rules. After an open interval, a half-open state permits a small number of controlled probes. If the probes satisfy the required success and latency policy, close. Otherwise return to open. They are not proof that the remote service is healthy or unhealthy for every client.

The half-open step is where many designs create their second outage. Imagine 600 product-serving instances all observed the same slowdown and opened at 10:00:00. At 10:00:30 every instance lets ten requests through. That produces 6,000 nearly simultaneous probes just as the recommendation fleet tries to recover. Use staggered reopen times, very small probe budgets, and preferably a shared recovery capacity signal when the fleet is large. Note the tension: a purely local breaker scales and has no coordinator dependency, but the collection of local decisions can collectively overwhelm a recovering backend. Distributed coordination is optional and brings its own failure modes.

Count actual transport failures, explicit timeouts, and selected overload responses differently from ordinary customer-level outcomes. A valid 404 for a missing recommendation model or a deliberate empty recommendation set should not automatically trip the circuit as an infrastructure failure. A 429 might signal policy rejection or load protection. An application error from malformed input should be fixed at the caller, not suppressed behind a breaker. Define whether cancellation after the caller's deadline counts as a dependency error, so an over-eager client does not make a healthy backend look broken.

Where does the breaker live? Close to the call site, with isolation per meaningful dependency and operation. A breaker only at the edge cannot protect a service's internal expensive calls if other callers bypass that gateway. The natural unit is a caller, operation class, and target failure domain, with cardinality controlled so that the telemetry and state do not explode.

Test the unhealthy-but-successful scenario, not merely a server that returns 500. Inject five-second responses and confirm that product pages still return within their contract, that the recommendation concurrency budget stays bounded, and that the system records fast fallbacks rather than reporting full-feature success. Then restore a healthy dependency while there is live traffic. Ensure gradual probe admission, no stampede, a return to normal recommendations, and observability that distinguishes breaker_open from dependency_timeout.

In an interview I would close with the boundary of the pattern. A circuit breaker cannot provide spare capacity, repair the dependency, or make a mandatory operation optional. An open circuit deliberately trades one function for survival of another. If every request absolutely needs the downstream result, failing fast may be the correct response but the product is still unavailable. The engineering skill is deciding which capability can be omitted, which calls are allowed to test recovery, and how to avoid turning a useful local protection into a fleet-wide oscillation.