We put a CDN in front of storage. Why is the origin still receiving thousands of requests?

Interview question. A global image site has 100 edge locations and an object store in one region. One newly popular image is requested by 500 users at each edge during the same minute. The origin sees a surprising amount of traffic even though every edge claims caching is enabled. Explain CDN cache layers, key design, origin shielding, and the failure modes of the proposed fix.

If you draw one browser, one CDN box and one origin, a cache miss seems harmless. The browser requests /photo/73.webp, the CDN retrieves it once and holds it for later visitors. But a CDN is not one cache. It has many geographically distributed locations, sometimes additional regional tiers, each with its own admission and eviction decisions. If a new file is absent from 100 independent edge caches, the origin may see 100 fills before they all settle. A sudden popularity spike also races the first fill inside each location.

That example has 50,000 viewer requests, but it does not imply 50,000 origin requests. With ideal request coalescing per edge and sufficient cache capacity, there might be roughly 100 origin fetches. With missed coalescing, key fragmentation, expiration or error retries, there can be more. The only honest way to estimate origin load is to state which layers actually combine requests. A superficial hit-rate figure does not tell you how many objects were fetched from the origin under a burst.

A common architecture adds an upper-tier cache or origin shield. Edge caches first ask the regional tier, which asks a designated shield, which asks object storage only when necessary. If the same immutable object is requested by all 100 edges, the shield may reduce repeated object-store fetches toward one, conditional on correct cache keys and coalescing. The origin can be protected while users still receive content from a nearby edge on later hits. This is why a cache hierarchy changes origin economics, not merely user latency.

The next question should be about identity. Suppose requests are for image.webp?utm_source=a, image.webp?utm_source=b, and image.webp?utm_source=c. If the CDN includes the entire query string in its key, identical bytes may occupy three entries. If you simply ignore all query parameters, a genuine width=320 variant may be confused with width=640. Define a canonical key from the fields that change the response representation. Preserve security and request-routing semantics where required. The exact defaults and permitted customizations vary by CDN provider, so test the effective key rather than trusting a sketch of the URL.

Likewise, Vary and representation negotiation matter. A client accepting AVIF may receive different bytes from a client accepting JPEG. A transformation gateway can normalize those inputs into separate stable URLs. Without that normalization, two requests with different headers may either fragment the cache unnecessarily or, in a badly configured system, share an incompatible representation. Do not assume Accept-Encoding or Accept is handled uniformly across all CDN configurations.

Now consider a content deployment. The old file lives at /app.js for one hour. Replacing the bytes behind the same URL asks every cache and browser to converge quickly. Purging a worldwide cache may be slow, expensive or incomplete at the exact instant of release. A safer static-asset design publishes /app.abcd1234.js, keeps its bytes immutable, and atomically updates the short-lived HTML or manifest that references it. The CDN can cache hashed assets for a long time without making emergency invalidation part of every ordinary deployment.

An origin shield introduces its own failure domain. If a shared upper tier becomes unavailable, lower tiers may have cached content but miss traffic could stall or surge toward alternative paths. Design a supported fallback hierarchy, keep finite origin connection pools, and admit uncached requests at a rate the origin can serve. Do not promise that a single shield makes the origin impossible to overwhelm. It concentrates misses as well as cacheable reuse. In a multi-origin architecture, choose shield placement according to connectivity, latency and egress costs, not simply nearest geography to end users.

Measure hits separately at the browser, edge, regional tier, shield and origin. Also record normalized cache-key cardinality, byte hit ratio, origin fetches per unique asset, conditional revalidations, and origin p99 under bursts. A single CDN hit ratio can hide a costly minority of giant misses. A 99% request hit ratio may still transfer most bytes from the origin if the missed objects are enormous.

The interview judgment here is about topology and identity. Global delivery improves when equivalent requests meet at the right cache tier, and when different representations stay distinct. The lesson is not to put yet another cache in front of every cache. It is to know what each layer protects, which requests it can safely merge, and what happens to the origin when a tier fails.