Hit rate fell from 95% to 80%. Why did the database suddenly struggle?

Work out the miss rate, not just the hit rate

At 95% hits, five percent of 50,000 requests miss. That is 2,500 misses per second. At 80% hits, twenty percent miss. That is 10,000 misses per second.

The hit rate fell by 15 percentage points. The implied database read rate rose fourfold. This is a ratio of misses, not a claim that database CPU automatically rises fourfold. Some misses may be coalesced, involve multiple queries, or hit a different downstream cache. Some cached operations may have far more expensive origin work than others.

The first question I ask is therefore not whether 80% is a good number. It is whether 10,000 origin reads per second is inside the database's stable capacity at the observed query mix.

If the database sustainably handles 6,000 such reads per second, the excess demand must queue, be rejected, or time out. Raising an application timeout can increase the amount of waiting work without raising capacity. This is an overload problem triggered by a workload change that looks small on a dashboard.

Look at the keys before changing the cache size

Several explanations can produce the same average metric.

A new release might accidentally add a random request ID to the cache key. Every request creates a new entry and reuse disappears. Increasing memory would not solve a key that is never requested twice.

A marketing campaign might bring in millions of one-time visitors. The working set grows. A cache that comfortably held last week's hot products now evicts them to make room. In that case memory pressure, evictions, and key popularity are relevant.

A deployment might shorten TTLs, causing entries to expire more often. Or a fleet-wide restart might temporarily replace warm instances with empty local caches. Those cases suggest expiration and rollout timing, not necessarily a bigger permanent working set.

A routing change might direct each customer to a different region whose cache has never seen them. One region could be healthy while another suffers misses. A global hit rate averaged across them hides the difference.

Finally, the cache can be reporting ordinary GET misses while the database pain comes from an entirely different path, such as a new synchronous write or a slow query plan. Correlation between a falling hit rate and rising CPU is a hypothesis, not proof of causation.

What I would measure in the first ten minutes

Plot application requests, cache hits and misses, origin read attempts, origin success, and latency on the same timeline, broken down by cache cluster, endpoint, tenant tier, and deployment version. Redis exposes keyspace hit and miss counters through INFO, but raw counts require rates and context. Redis also exposes eviction and memory information useful for narrowing the hypothesis.

I would sample key cardinality and normalized key patterns rather than log every raw key. A sudden explosion in distinct keys is a strong clue that key composition changed. Privacy matters too, since raw keys can include customer identifiers or secrets.

Then investigate bytes and cost, not just requests. A 90% hit rate for tiny product metadata can coexist with a low hit rate for expensive recommendation queries. Conversely, a cache with 99% hits can be a serious problem if the remaining one percent consists of very expensive misses. One blended success rate cannot express the consequence of a miss.

The fastest mitigation depends on what the evidence shows. Revert a broken key namespace, restore previous TTL settings, isolate a newly expensive query, increase cache capacity for a genuinely larger working set, or apply admission control while warming. Changing every TTL to an hour could hide the symptom while making price and policy data unacceptably stale.

Turn the number into a useful design contract

I would define the maximum affordable origin miss rate for each important operation. The cache hit rate then becomes a capacity assumption we monitor. If the database can handle 5,000 product reads per second and the API receives 50,000, the required effective hit rate is at least 90% under the simplifying one-miss-one-read model, with additional safety headroom for failures.

The contract should identify what happens below that threshold. Should lower-priority reads be throttled? Can certain content use bounded stale data? Can requests be coalesced? Which reads must bypass cache because correctness is more important? We do not need the same answer for every endpoint.

Interview takeaway: A cache hit ratio is a workload transformation. Convert percentages into origin traffic, verify that the traffic actually reaches the origin, and diagnose why the distribution changed before tuning eviction or buying hardware.

There is another way to produce enormous miss traffic even with plenty of memory. Clients can ask for keys that never existed in the first place.