A regional product API handles 100,000 reads per second. Normally, its shared cache serves 98% of them. The underlying database can sustainably handle 5,000 uncached reads per second with room for required writes. After a configuration incident, every cache node restarts empty. Engineers add application servers and allow the entire fleet to resume traffic immediately. Latency and database errors climb instead of recovering.
Interview question. Explain why the larger application fleet may worsen the outage. Design a recovery plan that protects the source of truth while the cache is cold. Decide what you should prewarm, what you should reject or degrade, and how you know the system is safe to open fully again.
Think first
A 100k RPS API normally hits cache 98% of the time. Its 5k RPS origin cannot serve the empty-cache load. Design recovery admission, prewarming, traffic ramp and operational exit criteria without making the healthy API fleet amplify origin load.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
Warm capacity is not cold capacity
Before the incident, 2% of 100,000 requests missed the cache. Under the simplified one-miss-one-query model, the database saw approximately 2,000 reads per second, within its 5,000-read capacity.
After the restart, nearly every first request misses. Potential origin demand approaches 100,000 reads per second. That is fifty times the previous read demand and twenty times the database's stated capacity.
A warm cache keeps origin load below capacity, while an empty cache requires controlled rewarming
The application servers are not the scarce resource in this incident. They are a source of work for the scarce resource. Adding more of them can increase concurrent database requests, thread-pool occupancy, connection contention, and retries. Healthy app CPU alongside failing user requests is therefore not surprising.
Google's SRE material distinguishes a latency cache from a capacity cache. With a latency cache, the origin can still handle expected traffic when caching disappears, albeit more slowly. With a capacity cache, it cannot. This service is using the cache as a capacity component. Its cold-start state is an availability risk that architecture reviews must acknowledge.
Recovery starts by limiting work, not maximizing traffic
I would establish a global origin budget smaller than the database's sustainable read capacity, leaving space for writes, health checks, and existing internal consumers. For example, if the database supports 5,000 such reads per second in a healthy benchmark, an initial cap of 3,000 might be an illustrative emergency setting, not a universal tuning rule.
The cap should apply across all application instances. A 3,000-per-second limit independently configured on 100 instances is not a 3,000-per-second limit. Use a coordinated budget or allocations that sum to the intended fleet-wide ceiling. If central coordination fails, instances should have conservative local defaults rather than each assuming it owns the whole budget.
Above that budget, prioritize essential reads and reject or degrade lower-priority work with explicit signals. Some catalog metadata may safely be served from a bounded stale copy. A price confirmation or authorization decision should not silently use stale values simply because the origin is busy. The response contract determines which requests can be delayed, which can degrade, and which must fail closed.
I would temporarily disable broad speculative prefetching. Warming one million keys in the background can compete directly with foreground misses. A cache warming job must consume the same finite origin budget as live traffic, or it becomes another source of overload.
Warm the right keys, not every key
The target is a useful working set, not a photograph of the entire database. Start with measured hot keys whose values are small, frequently reused, and safe under the chosen freshness policy. Do not guess the hot set entirely from yesterday's averages if the outage coincides with a live event or shift in traffic.
Warm one partition or tenant cohort at a time, watch origin capacity, and expand deliberately. It may be better to route a small fraction of users to a warm cache pool while other pools are prepared. Stagger cache and application starts so every instance does not independently fetch the same hot entries. A healthy readiness check should distinguish “cache process is alive” from “this instance can serve its expected share without collapsing its dependency.”
Request coalescing still helps individual hot keys. But it cannot collapse a million unrelated misses into a single database query. That is why the key-expiry solution from Part 2 of 6 does not by itself solve a regional cold start.
Track both cache population and customer outcomes. A high fill count is not evidence of useful progress if those entries are immediately evicted. Monitor hit rate by cohort, actual database reads per second, database queueing and p99, admission rejections, expired-stale use, and the age of the most delayed accepted request. If backend latency starts rising toward its overload knee, stop ramping and reduce fill pressure.
Avoid the second collapse
Suppose traffic is restored once the cache reaches 90% hits. At 100,000 requests per second, ten percent misses still imply 10,000 origin reads per second under the simple model. That is twice the stated capacity. A hit rate that sounds “mostly warm” is not enough.
If the service needs origin reads below 4,000 per second to leave headroom, the effective hit rate at 100,000 requests per second must exceed 96%, assuming similar query cost and no coalescing. That target must be verified against live origin traffic rather than inferred only from cache counters.
I would complete recovery through a controlled ramp, not a binary all-clear. Run failure drills that empty a cache pool and measure whether the service remains inside its origin budget. Test a recovery where one replica is already degraded, because warmup traffic can compete with replication and repair just when the system has the least spare capacity.
Interview takeaway: A cold cache changes the capacity of the entire service. Recovery needs origin admission control, selective warming, protected business-critical traffic, and measurable ramp-up gates. More frontend capacity does not create more database capacity.
Go deeper
In severe incidents, a cold-starting cache can create a metastable failure. Requests miss, origin latency rises, waiting work accumulates, and retries add load. Even after the triggering cache restart ends, the system may remain trapped in overload until admission is reduced enough for the queue to drain.
Use explicit recovery budgets by tenant or priority so one hot customer does not consume every origin read permit. Cache nodes should report when they are healthy but cold, and load balancers should be able to limit their ramp-up. Some systems also preserve a small independently maintained warm tier, but that tier is another operational dependency that needs testing.
Separate minimum recoverable availability from full freshness. A useful low-cost degraded mode may permit reading public product descriptions while refusing new checkout commitments. Explain that decision to product teams before an incident rather than improvising it during one.
Check your reasoning
Calculate warm 2,000 origin reads/sec versus potential cold 100,000 reads/sec.
Explain that a capacity cache changes availability when it disappears.
Set a fleet-wide read budget with spare capacity for important writes and internal workloads.
Prioritize hot-set warmup, distinguish permitted stale reads from strong decisions, and avoid unbounded prefetch.
Define ramp gates based on actual origin load and customer outcomes rather than cache process health.
Common mistake: Adding more application instances or releasing traffic at 90% hit rate without checking backend capacity.
Read the answer explanation
More than 96% to stay below the 4k budget. — Four percent of 100k is 4k. Stay below that by exceeding 96% effective hits, plus headroom.
Apply it elsewhere
A recommendation platform loses its embedding cache during a product launch. Do you warm every embedding or limit the cold path first?
Compare your approach
Protect expensive inference and embedding backends with admission, prioritization and a high-value warm set, not an unbounded preload.
One more constraint
Which resilience test proves you can survive a cache reset while one database replica is already slow?
Quick revision
An empty cache can turn 100,000 API requests/sec into nearly 100,000 origin reads/sec. At a warm 98% hit rate, the same traffic required only 2,000 origin reads/sec. If origin sustainable capacity is 5,000, the cache is a capacity dependency, not merely a latency optimization. Limit origin work across the fleet, reserve capacity for critical operations, prewarm a measured hot set, and ramp user traffic against actual origin latency and queueing. More frontends without origin admission can deepen the outage.
