The database recovered an hour ago. Why is the platform still down?

Interview question. A ticketing platform suffered a ten-minute database slowdown during a ticket release. The database has been healthy for an hour and original customer traffic is back to normal, yet API success remains near zero. Queues, retries and restarted workers are now the dominant load. Diagnose a metastable failure and design a safe exit.

The incident timeline doesn't make sense if you expect every failure to end when its initial cause ends. At 14:00 database latency spikes. At 14:10 the storage team fixes the underlying problem. At 14:30 the platform still cannot process a normal purchase. Engineers add instances. They also see more timeouts. But the system may now be failing for a reason that no longer requires the database problem.

A metastable failure is a system state sustained by its own internal positive feedback. A temporary disturbance pushes the system out of a healthy operating region. Retries, retained backlog, cache misses, connection storms or repeated restarts keep it overloaded even after the original disturbance disappears. Research on metastable distributed-system failures describes how useful work can collapse and how returning to a previously safe external arrival rate may be insufficient for recovery. The key sign is hysteresis: what load the system can tolerate depends on the condition it is already in.

Start with counters that separate new customer demand from total attempted internal work. Suppose new user traffic is 8,000 operations per second. A retry loop adds 5,000 attempted operations per second, while delayed background tasks contribute another 4,000 expensive operations per second. The backend now sees 17,000 attempted operations per second against a healthy capacity of 10,000 for this simplified workload. Reporting only new inbound RPS makes the load appear safe when the actual work offered is not.

There is a second problem: queue age. Imagine 600,000 queued tasks accumulated during the disruption. At a sustained processing rate of 10,000 tasks/s with 8,000 new tasks/s, net drain would be 2,000 tasks/s and clearing the old backlog would take 300 seconds, assuming uniform task cost, no retries and no expirations. But if the retry and background traffic consumes the spare capacity, the backlog does not drain. Expired tasks may still occupy workers and database connections even though their customer deadlines passed.

Now look at worker restarts. A saturated process misses health checks, is killed, and restarts with empty caches and cold connection pools. Healthy throughput from that instance is initially lower than steady state. The load balancer sends pending work to it as soon as it appears ready, it becomes saturated again, and another restart follows. Replacing instances or autoscaling can paradoxically decrease the amount of useful work completed.

How do you recover? First protect the source of truth and prevent irreversible duplicate effects. Establish a trusted picture of accepted versus completed operations using durable IDs, authoritative order state and a clear cutoff for deadlines. Then sharply reduce offered work to a level the currently healthy capacity can sustain. Temporarily suspend optional imports, recommendations, search-index backfills and nonurgent batch tasks. Gate new purchases at admission, preferably before allocating expensive downstream resources. Disable or bound retry layers that are multiplying attempts. This may mean deliberately returning fast errors for some new requests to restore a smaller core of correct completions.

Recovery may need phases rather than one switch. Phase one: protect payments, ticket ownership and inventory invariants. Phase two: let the database, caches and workers warm under a small, controlled load. Phase three: process retained legitimate work at a rate below measured spare capacity. Phase four: ramp fresh traffic, watching whether useful completion rate increases without a renewed queue-age rise. Do not simply dump the backlog on the newly recovered database. If tasks are now meaningless or expired, apply an explicit audited expiry or cancellation policy rather than silently deleting them.

The word goodput matters. A process can report 20,000 handled requests per second while completing only 2,000 distinct customer operations because most handled requests are retries, short-lived probes or calls doomed by expired deadlines. Count successful unique purchases, valid state transitions and useful completed background work. Track duplicate-attempt ratio, oldest queue age, in-flight work, startup churn, database write cost and per-tier admission. The recovery graph should show not just lower CPU but rising useful completions and falling residual backlog.

This lesson differs from the earlier cascade. A cascading failure describes how distress spreads from one part of a service graph into others. A metastable failure asks why the broken state persists after the trigger vanishes. Both may involve retries, but the diagnostic test is different. For the metastable case you should be able to construct a timeline in which the original defect is demonstrably gone while feedback loops continue to suppress goodput. Simply identifying the first failing dependency is not enough.

The long-term fix should remove the vulnerable operating mode. Give every queue a purpose, capacity bound, age policy and admission signal. Use end-to-end deadlines and propagate cancellation so abandoned work does not continue invisibly. Cap retries globally, not just per SDK. Preserve spare resources needed for startup, credential refresh and control operations. Validate readiness using the ability to process real useful work, not merely answer /health. Design controls to reduce traffic quickly, including safety rails for reopening.

In the interview, I would not promise that the database returning healthy means the incident is over. I would ask, “What work is the system actually doing now, where did it come from, and can we lower that work long enough to recover useful throughput?” That moves incident response from chasing the original trigger to changing the state the system is trapped in.