The edge lost its control plane. Why did it send users to a retired origin?

Interview question. A global edge network serves requests from hundreds of locations. Its control plane publishes origin routing, access policy and feature configuration. During a two-hour connectivity failure, several edge locations keep accepting requests using the last configuration they received. Some send traffic to a retired origin. Define what an offline edge may safely decide and how configuration recovery should work.

This looks like a routing incident until you ask a more precise question: what did the edge believe to be true? The edge has a fast data plane that forwards requests, and a slower control plane that decides where requests may go. If every request had to synchronously ask a central authority which backend to use, losing control-plane connectivity could take the whole edge down. So edges usually need some local configuration. The problem begins when we call that local copy current without defining how long it may remain authoritative.

Suppose version 41 routes /media to origin A. At 10:00, version 42 retires A and routes to B. Edge locations in one region cannot download version 42. Their health probes still show the edge process listening on port 443, so traffic continues arriving. Requests routed to A now fail or, worse, reach a machine that should no longer handle customer data. Nothing about the edge having working CPU and network connections establishes that it has a safe routing decision.

I would partition configuration into risk classes rather than install one global offline policy. Static public-image routing can often continue from a previously validated snapshot with explicit limits and alternate origins. A rule that grants account access or authorizes a revoked identity may require fresh control-plane knowledge or a separately maintained revocation path. A change that disables a compromised origin may need stronger enforcement than a change that updates an advertising label. The correct answer depends on the consequences of an outdated decision.

Configuration should arrive as a coherent snapshot, not a bag of independent key updates. Imagine version 42 contains an origin address change and an authentication audience change. If an edge applies one but not the other, it may route correctly while accepting tokens intended for the retired environment. Publish a signed or otherwise authenticated manifest of the complete snapshot, validate checksums and schema compatibility, then switch the active pointer atomically. Keep the prior known-good snapshot for controlled recovery if the new version is invalid.

A version number alone is insufficient. Edge nodes also need an issuance time, a policy-specific maximum age, and a monotonic generation or trusted sequence to prevent an old snapshot from silently replacing a newer one. Treat local time carefully, including clock skew and restart behavior. If a stale rule has exceeded its permitted age, make an explicit decision: use a conservative alternate route, serve only public immutable content, or fail the affected operation. What is not acceptable is quietly extending every permission because the control plane happens to be down.

Now consider the critical question from an interviewer: what if the edge never learns that a token was revoked? You cannot simultaneously promise unlimited offline operation and immediate globally enforced revocation unless another independent trusted mechanism delivers that revocation. Short credential lifetimes, online checks for high-risk operations, local deny lists with independent distribution, and a defined emergency fail-closed posture are possible tradeoffs. Say which contract the product actually offers. In some services, bounded stale policy is acceptable. In others, refusing an action is the safer result.

Recovery has its own trap. When connectivity returns, 500 edge nodes fetch new configuration at once. A poorly designed controller may receive a spike of identical expensive reads, while some edges apply version 44 before version 43's compatibility transition has finished. Use efficient snapshot distribution, jittered polling, bounded retries and a monotonic activation rule. If a new snapshot is incompatible with the running edge binary, do not partially activate it. Roll forward only after validating the schema, or fall back to the previously permitted snapshot.

During rollout, record which configuration version each edge is using and how far behind the controller it is. A dashboard that shows 99.99% availability but no age or version skew is misleading. Track the number of requests served with expired policy, last-good snapshot age, activation errors, fallback origin selection, and whether the affected rule was security-critical. Synthetic probes should test a request that exercises the exact policy transition, not merely a request to a static health endpoint.

A useful design exercise is to take four edge decisions and ask which can be made offline. Deliver an already public immutable photograph, probably yes for a bounded period. Route to a previously validated healthy origin, perhaps, subject to retirement and health policy. Approve a payment refund, probably not without fresh authorization and durable coordination. Enforce an emergency access revocation, only if the edge has a trusted way to know about it or denies the action while disconnected.

The engineering insight is that local availability and current authority are different properties. A resilient edge keeps serving the operations that remain safe under bounded stale knowledge. It does not confuse a working data plane with permission to make every decision using yesterday's configuration.