One service went down. Why did the other eleven follow?

Interview question. A subscription platform has twelve synchronous microservices on the request path. A minor profile dependency starts returning errors for only 10% of calls. Within minutes, checkout and login fail across the region. Trace the cascade, identify its feedback loops, and propose an architecture that keeps a small failure small.

At 09:40 the profile service loses one replica. Capacity falls slightly. The system could tolerate that. At 09:41 its p99 rises, a client retries, the next service retries that retry, and the gateway retries the entire request. By 09:45 the authentication team sees its CPU at 100%. Auth was not broken. It was trying to survive a request rate that now included many duplicate attempts.

Map the synchronous request graph, not just the organization chart. A single customer action might call checkout, pricing, identity, profile, promotions and inventory, with some calls in parallel and others serial. For each edge, write down the deadline, retry ownership, retry count, retryable errors, concurrency limit, and whether the data is genuinely required before a customer response. A harmless optional dependency becomes dangerous when the whole graph treats its response as mandatory, holds work while waiting, and then reacts to timeout by adding yet more work.

Try a simple calculation. If the gateway, checkout service and profile SDK each allow three total attempts for the same operation, a pathological request could induce up to 3 × 3 × 3 = 27 downstream attempts, subject to which layers actually retry. The exact number is not automatically 27 in every call graph because some attempts fail early and retry policies differ. But multiplicative attempt trees are real when layers are independent. A 10% failure at one endpoint can become much more expensive than a 10% loss of customer requests. Worse, the failures may occur under overload, when additional work is least likely to help.

The second feedback loop is deadline amplification. Profile slows from 50 to 900 ms. Checkout waits, holds a worker and a database connection, then times out after 700 ms. Profile's operation might continue consuming backend CPU after the caller has abandoned it if cancellation is ineffective. When checkout retries, the abandoned operation and new attempt overlap. Resource use goes up while successful useful responses go down.

Now consider load balancing. As profile replicas fail health checks, the gateway pushes traffic to surviving replicas. Their load rises, their latency rises, and they fail too. A naive regional fallback may route to a second cluster that never had spare headroom for the extra work. The apparent high-availability design carries the overload to every healthy region. Google SRE describes these positive-feedback cascades and the importance of rejecting excess work before it consumes scarce resources.

Containment starts at each boundary. Choose one layer to own retries for a user operation, and set a small global attempt or retry budget rather than independent retries at every hop. Use randomized backoff for suitable retriable failures. Apply end-to-end deadlines so a downstream service cannot spend another second on a response the caller no longer needs. Most importantly, gate expensive work with bounded concurrency and queue admission. Overloaded services should be able to reject cheaply and preserve a core rate of useful completions. If a dependency is optional, a carefully designed degraded result avoids the call altogether. If it is mandatory, fail honestly instead of manufacturing data.

A team proposes circuit breakers at every edge. They help suppress calls to known unhealthy dependencies, but they do not solve the graph. Suppose every request requires twelve services to be available, each with independent 99.9% availability. Under an oversimplified independent-failure model, the path's availability is 0.999^12, approximately 98.81%. Real correlated failures make multiplication an inadequate full availability forecast, but the toy model teaches something: a long synchronous critical path creates many opportunities for unavailability. The stronger design removes unnecessary synchronous dependencies and makes noncritical side effects asynchronous with explicit completion and failure semantics.

Dependency importance needs to be an explicit contract. Can the user view an order without current recommendations? Usually yes. Can the user confirm an inventory reservation without any authority on stock? Usually no. Can the user start an asynchronous export while export status has a transient delay? Perhaps, if acceptance is durably recorded and the UI shows pending state. These choices affect system behavior more than installing the same breaker library everywhere. The interview is testing whether you can turn a dependency graph into a business correctness graph.

During the incident, do not randomly turn up all timeouts. Longer timeouts often increase in-flight occupancy and worsen the cascade. Instead, compare original customer request rate, total downstream attempt rate, completed useful operations, queue age, rejected work and each dependency's saturation. Disable unnecessary retries, shed optional expensive work, and route to proven spare capacity only when measured. Recovery is incomplete until the ratio of total attempts to successful customer operations normalizes and the remaining queues drain without creating a fresh wave.

Then rehearse the event. Terminate one profile replica, inject a 900 ms slow-success mode, disable one network path and replay realistic offered traffic. Watch whether the failure remains within profile and optional enrichment, or whether shared thread pools, database connections and regional routing still spread it. Record the first resource that saturates and the first feedback loop that becomes self-reinforcing. Your prevention plan should target those facts, not a generic checklist of microservice resilience patterns.

I would finish the interview by drawing two graphs: the call graph showing where work travels and the dependency graph showing which resources and guarantees are shared. A cascading failure follows both. Breaking the causal path might mean a shorter critical route, bounded retries, admission control, capacity reservations or explicit fallback. The principal-level skill is being able to prove where the failure stops.