One key expired. Why did database reads surge?

A miss can multiply one query into thousands

At a steady 24,000 incoming requests per second, about 24,000 × 0.200 = 4,800 requests arrive during an initial 200-millisecond miss window. If the key remains missing throughout that window and each arrival immediately starts one database read, that means roughly 4,800 queries started—not 24,000 queries and not necessarily 4,800 queries executing simultaneously. Earlier refill, admission limits, queueing or coalescing can reduce the number actually started. Measure incoming requests, origin queries started and active query concurrency separately.

A naive retry makes matters worse. A database saturated by a flood of reads slows each miss. Slower responses leave more requests waiting, which can cause clients and gateways to time out and retry. The cache may appear healthy while the origin collapses.

In one process, I would use a per-key in-flight operation. The first caller starts the load. Concurrent callers for the same key await its result instead of executing identical queries. Go's singleflight implements that general idea, but it shares overlapping calls rather than permanently storing the result. We still need an actual cache and an expiry rule.

With 40 application instances, process-local coalescing permits approximately one origin fetch per active instance for this key, assuming overlapping requests and correct implementations. That is much better than thousands, but not necessarily one across the fleet. A shared cache with coordinated refresh, a bounded distributed claim, or a designated refresh owner can reduce duplicate work further.

Who should wait, and for how long?

The first caller should not hold everyone hostage indefinitely. Give the origin fetch a deadline. Other requests wait only inside their own budgets. If a waiter cancels, that does not automatically mean the shared origin fetch should be canceled for all remaining callers.

If the refresh fails, choose one explicit response strategy. For product description, a recent expired value may be useful. Keep a separately bounded stale representation, serve it while one refresher works, and stop serving it after the allowed stale window. HTTP's stale-while-revalidate and stale-if-error express related policies for compliant HTTP caches. They are not a blanket instruction to reuse arbitrary old application data.

For a price display or inventory decrement, serving stale may violate the user-facing contract. Under those requirements, waiting briefly, returning a degraded response, or rejecting admission is better than inventing availability. A database query for inventory authority cannot be replaced by copying yesterday's stock count into more caches.

An important edge case is the elected refresher crashing. A distributed claim must expire or be recoverable, or the key could remain frozen forever. But an expired claim does not cancel a query already in flight. A replacement refresh can race with the earlier one. Protect publication with a generation or conditional rule if data freshness requires it, rather than treating a lease as proof that all older work vanished.

Expiry is a load-shaping decision

A thousand keys expiring at one exact deployment timestamp can create a fleet-wide herd, even if no individual key is exceptional. I would spread expiry times with jitter where the business semantics allow it. For frequently used keys, probabilistic early refresh or background revalidation can smooth requests before the deadline.

Neither approach removes the need for origin admission control. If every cache disappears during a restart, nearly every key can miss at once. A system designed only to coalesce one hot key is not necessarily able to rebuild its entire working set. We will treat that separately in the final episode.

Metrics should distinguish the number of user misses from the number of actual origin fetches. A cache miss counter might show 24,000 per second while process-local coalescing permits roughly one overlapping origin fetch per participating instance. Compare misses and actual query starts over the same measurement window rather than equating request rate with a query count. Without both signals, on-call engineers may incorrectly conclude the database is overloaded or that the improvement did nothing.

I would instrument active refreshes per key, waiters per refresh, successful and failed origin fetches, stale responses by age, deadline rejections, and the fraction of requests served by a coalesced result. A single shared error can fan out to many users even while origin query counts look small, so successful user outcomes also matter.

Try this in an interview: The first fix is not “more Redis nodes.” It is to decide how concurrent misses share work and what callers do when the shared work cannot finish. For a bounded-stale description, one controlled refresh plus short-lived stale responses may work. For an authoritative operation, stale reuse is not an acceptable shortcut.

At the next deployment, the cache does not lose one key. The overall hit rate drops while request volume stays constant. We need to understand how much traffic that seemingly small percentage change sends to storage.