Learning series · 5 episodes · Senior / Staff / Principal
The Failure Should Have Stopped There
Six connected engineering decisions about circuit breakers, bulkheads, cascading failures, cells, shuffle sharding and metastable recovery.
Start series →Follow the decisions in order
- Part 1 of 6
Senior
The recommendation service is taking five seconds. Why are we still calling it?Use failure and slow-call signals, timeouts and bounded recovery probes to protect an optional dependency without inventing mandatory business outcomes.
- Part 2 of 6
Principal · AI Engineering reading
We gave each agent its own queue. Why did one slow tool still stall everyone?Bound independent request, connection and execution resources so a slow dependency cannot starve unrelated request paths.
- Part 3 of 6
Staff
One service went down. Why did the other eleven follow?Follow cascading failure propagation through nested attempts, deadlines, abandoned work and redistributing failed backends, then cut the causal path.
- Part 4 of 6
Principal
One tenant's import crashed a shard. Why did half the company go offline?Define a self-contained application and state cell with safe local routing, bounded cell capacity, controlled deployments and ownership-aware migration.
- Part 5 of 6
Staff
Every customer has three workers. Why do the same noisy tenants keep colliding?Derive shuffle-shard full-collision and partial-overlap probabilities, and use those to evaluate shared-pool noisy-neighbor protection.
- Part 6 of 6
Principal
The database recovered an hour ago. Why is the platform still down?Diagnose self-sustaining failure after a trigger ends and stage recovery using goodput, reduced admission, retry control and safe backlog draining.