Distributed Reliability · Staff
The circuit opens, then every instance probes at once
The question
Interview question
A circuit breaker opens on high error rate. When its cooldown expires, all application instances call the dependency together and knock it down again. Fix recovery. The dependency is shared across regions.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
A local circuit breaker can protect one process while still hurting a shared service. If 500 instances each allow ten half open requests after the same 30 second timer, recovery can produce 5,000 calls in a moment. Each instance thinks it is cautious. The dependency sees a synchronized load test. Resilience4j's breaker documentation describes a bounded number of half open calls per breaker. That local bound does not become a global bound just because every process uses the same library.
During open state, stop ordinary attempts for that dependency and return the product's allowed degraded behavior, a bounded queue, or a clear failure. The breaker does not replace admission. If clients retry the immediate failure without a budget, the gateway can become the new queue. A 429 with a provider retry signal should also shape the next attempt time. An error rate breaker must distinguish local network failure, a bad credential or request, and genuine provider overload. Probing with malformed requests tells us nothing about whether useful traffic can resume.
For half open, I would put a small global probe budget at the service boundary that owns the provider quota. Let a few representative calls through, measure their latency as well as success, and grow the allowance gradually. That can be a shared token allocation with short leases or an explicit recovery controller. If the controller is unavailable, only predesignated probe owners may spend tiny, preallocated regional shares. Ordinary instances stay open. The budget must be chosen for the provider's recovery capacity, not the number of our instances. Keep probe timeouts and retry behavior fixed so the probes themselves do not multiply.
Success on five tiny requests does not prove a long context workload is healthy. Ramp from safe probes into traffic classes with measured token and concurrency budgets. Do not send one probe from each of 200 tenants unless the upstream has 200 independent quotas. Keep both failure rate and saturation signals: 200 responses can look successful while TTFT and queue age climb toward another trip. Reopen in steps and stop increasing if the dependency or our own backlog deteriorates.
Now the provider is shared across regions. A per region breaker could still send two full recovery waves into one global quota. Coordinate the aggregate attempt budget across regions when they hit the same capacity pool, while retaining regional health observations. If region A has a local network problem, region B should not automatically be declared broken. Conversely, moving A's traffic to B without checking shared provider headroom can cause a second incident. Model the failure domain correctly: region, credential, model, endpoint, or provider wide quota.
What if the shared coordinator dies during recovery? This is a failure policy, not a reason to skip coordination. We can divide a tiny known recovery budget among regions ahead of time, each with randomized probes, and refuse to ramp until control returns. The allocations may leave some capacity unused, which is cheaper than another collapse. A single elected probe can help, but it must not be the only path to recovery if its owner disappears. Short lease expiry and bounded local fallback matter.
I would test it with hundreds of instances whose timers expire together, plus a regional outage and a provider that can serve only ten percent of normal traffic. Watch global attempts per second, retry volume, probe latency, accepted useful work, and each region's queue. A breaker is recovered when the admitted load has been ramped to a sustainable level. Flipping the state from half open to closed on one green probe is merely a state change.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →