Distributed Reliability · Staff
One tenant's quota errors open a breaker for everyone
The question
Interview question
A provider returns 429 for tenant A's project key. The gateway's shared circuit breaker counts those responses as provider failures and opens for all tenants, including B and C whose calls would succeed. Fix the signal and scope. The provider later starts returning 503 for every project.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The breaker is answering “is this dependency usable for the calls behind me?” A tenant quota error says something narrower than “all projects cannot use the provider.” First inspect the actual response contract, key, model, region, and rate-limit headers or body. A 429 can mean a project-specific quota, a model pool limit, or a broader provider throttle. Do not decide its scope from the HTTP status alone. Track error and latency by the credential or project, route, model, region, and failure class. If A has exhausted its allotment, throttle A's eligible calls and give it a retry time or different approved route. Continue to test B and C on their own allowed routes.
Breakers and bulkheads should match actual failure domains. A breaker per provider cluster is useful for a real cluster outage, but it is too coarse for a tenant key quota. A breaker per individual request would never gather enough evidence or reduce load. Choose a small set of stable dimensions the gateway owns, with bounded cardinality and explicit state expiry. Envoy's circuit breaking documentation describes limits tracked by upstream cluster and priority. That illustrates scoping as a design choice, not a guarantee that Envoy automatically understands our tenant quota semantics. Keep per-tenant admission and shared provider-health controls distinct.
Now the provider returns 503 for every project. The aggregate health signal should catch that common failure quickly. Suppress or shed calls across the affected route, preserve deadline and retry budgets, and test a bounded recovery path. If there are permitted alternatives, route only work whose quality, residency, and customer policy allow them. Do not let a sea of project-specific 429s hide the later common 503, or let a healthy result from one tiny slice declare all models and regions restored.
I would replay the incident with A saturating its quota while B and C send ordinary traffic, then inject a genuine shared 503. The pass condition is isolation in the first case and shared protection in the second. Watch accepted useful work per tenant, false opens, backlog, 429 classification, and provider load during recovery. A circuit breaker is an automated policy decision. Give it a denominator and a failure domain it can actually observe.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →