The policy service is down. Should the gateway let everyone through?

Interview question. A gateway depends on authentication, an external rate limiter, fraud scoring and feature flags. At 10:15 the shared policy service becomes unreachable. Some requests could continue without it, while others could create financial loss or expose private data. Choose fail-open or fail-closed behavior per decision, and design recovery without turning one dependency failure into a platform outage.

The outage begins with a chart that seems simple: policy RPC success falls to zero. An engineer proposes a single emergency switch, skip_policy_checks=true. That would make most of the API available again. It could also allow unauthorized account reads, let an attacker bypass abusive-traffic limits, or approve payments the risk team would never have permitted. Saying “availability is important” does not decide whether an unknown security decision should be treated as allowed. We need to name each policy, the harm of a wrong decision, and what evidence remains trustworthy when its dependency fails.

Start with authentication. If the gateway already possesses a trusted public key and enough verified claims to validate a signed access token locally, an identity service outage need not prevent every request. Local verification still checks the signature, expiry, issuer, intended audience and relevant claims. It does not automatically know about a token revoked five seconds ago, an emergency account suspension, or a missing authorization rule. The documented contract may permit cached verification for a short interval, especially for low-risk reads, while high-risk changes require a fresher decision. If no trustworthy credential or policy material is available, an unauthenticated write must not be admitted by default.

Rate limiting is different. If the global limiter is unavailable, refusing every harmless API read could take down a service whose business logic is healthy. Some routes can continue under a conservative local token bucket with a per-identity cap. Expensive exports might be denied or given a very small allowance. Account-login and password-reset flows need stronger protection against abuse, possibly with local emergency rules or edge mitigation that does not rely on the failed service. Calling that “fail open” hides the actual design. It is usually a bounded degraded policy with explicit blast-radius limits.

Fraud checks deserve their own decision. Suppose a retailer handles fifty thousand checkout requests a minute during a major sale. Fraud scoring is unreachable. Allowing every high-value order could turn a resilience incident into irreversible financial loss. Blocking every small purchase might create unacceptable customer harm, too. The product and risk teams should decide beforehand which transaction amounts, identities, regions and payment methods may continue under a known fallback. For some transactions, authorize but hold fulfillment, or route to manual review. For others, stop. The fallback must respect regulatory requirements, and it must not invent a positive fraud score merely because the scoring service timed out.

Feature flags are another category. For a cosmetic layout choice, a locally cached value or a safe default may be adequate. For a kill switch disabling a vulnerable money-transfer path, an expired cached “on” value could be dangerous. Treat safety-critical flags as policy controls with freshness requirements, versioning and explicit invalidation behavior. A flag service is a control plane. The data plane should not need a fresh remote lookup for every harmless branch, but stale configuration cannot grant arbitrary permission forever.

Now design the request path. Tag each policy evaluation with a type, risk class, maximum tolerated staleness and failure mode. Evaluation has a bounded deadline and a separate result for ALLOW, DENY and UNAVAILABLE. Do not collapse UNAVAILABLE into DENY or ALLOW without an explicit rule. If multiple policies apply, a local allow from one cannot override a mandatory deny from another. Keep authenticated identity binding on policy caches so that a cached result for tenant A cannot be reused for tenant B. Do not trust caller-provided headers to select the relaxed policy class.

The HTTP response should reflect what actually happened. A user failing authorization may get 403, or 401 if valid authentication is missing. A user who exceeded a known quota may receive 429. If the server cannot perform a required policy decision because an internal dependency is unavailable, an appropriate 503 may be more honest than pretending the user exceeded their quota. Avoid leaking details of the security policy in public error messages. For permitted degraded execution, record that the request passed under the specific fallback policy version so later audits can distinguish it from a fully checked decision.

Failover also creates recovery risks. When the global rate limiter returns, don't replace every locally spent allowance with a fresh full bucket. Reconcile metering where possible and keep explicit uncertainty in accounting. When fraud scoring returns, process held decisions before shipping goods, according to the product's stated contractual obligations. When revocation data becomes fresh again, invalidate affected cache entries and recheck sensitive sessions. Policy versions should roll out atomically enough that half the fleet does not enforce contradictory emergency rules.

I would rehearse this outage in a staging environment and in narrowly scoped production experiments. Disconnect the policy service while sending legitimate read traffic, a revoked token, expensive write bursts, suspicious high-value payments and emergency feature-flag updates. Verify which operations remain available, which stop, how fallback allowance decays, and whether the system recovers without synchronized retries. Measure allowed-by-fallback decisions, denied-by-risk decisions, policy staleness, revalidation work, audit completion and customer-visible errors.

The strongest interview answer refuses the false binary. Fail open and fail closed are meaningful shorthand only after naming what is being allowed, whose risk matters and which safeguards remain. One policy service can fail, but it should not be allowed to decide, by accident, the security and availability posture of every product operation.