Skip to content

One software release is 2 GB. How do you deliver it to 100 million devices?

The File Left the Origin. Was It Safe to Deliver? · Part 6 of 6 · 5 min read

Read the worked explanation ↓

At 5% adoption during the first hour, calculate total payload and the average delivery rate over that hour, then state what extra arrival-rate information would bound peak demand if every participating device downloads the full 2 GB installer. Design the distribution and emergency response if the canary shows corrupt downloads in one region.

Think first

At 5% adoption during the first hour, calculate total payload and the average delivery rate over that hour, then state what extra arrival-rate information would bound peak demand if every participating device downloads the full 2 GB installer. Design the distribution and emergency response if the canary shows corrupt downloads in one region.

Take a few minutes to form an approach, then compare it with the worked explanation.

Check the exercise numbers

Five percent of 100 million is 5 million devices. At 2 GB each, that is 10 PB of full-installer payload. Spread uniformly over one hour, it averages about 2.78 TB/s, or 22.2 Tb/s, before retries and overhead. This is not a peak estimate: synchronized starts or bursty adoption can require much higher capacity. A 100 MB delta for the same devices would be 0.5 PB before full-package fallbacks.

Reveal the worked explanation

Interview question. Your company ships a 2 GB signed desktop application update to 100 million devices worldwide. The first launch overloaded origin storage and forced customers to restart downloads. Design content distribution, resumability, release control and client verification so the next rollout is fast without making a compromised mirror a path to malicious code.

The capacity calculation should happen before drawing boxes. Two gigabytes multiplied by 100 million devices is 200 petabytes of transfer, using decimal units. If everyone downloaded the full package during one day, the average delivery rate would be about 2.31 terabytes per second across the distribution fleet, before retries and protocol overhead. Peak demand may be far higher. This is not an application-server endpoint you solve by adding a bigger load balancer. Most bytes need to come from geographically distributed caches or peers under a verified content protocol.

The obvious architecture starts with immutable, content-addressed package objects in durable object storage, delivered through multiple CDN regions. A small update metadata service tells clients what version is available, what platform and architecture it applies to, and which artifact hashes and sizes to expect. Clients download the bytes from a nearby delivery location, not from the metadata API. Origin shielding and warmup can reduce how often a newly published package reaches the object store. But the first global rollout still creates real edge-fill traffic, so coordinate release timing and warm likely regions ahead of demand.

A major cost lever is not downloading unchanged bytes. If a carefully engineered binary delta averages 100 MB instead of 2 GB, the theoretical total is 10 PB rather than 200 PB, a twentyfold reduction in payload. Deltas are conditional, though. A client with an unexpected base version, damaged installation or unsupported patch format must be able to fetch a complete verified artifact. Patching also consumes CPU, memory and temporary disk. A slow patch on a low-end laptop can be worse for users than a somewhat larger download. Compare complete transfer, patch success, verification and install time, not bytes alone.

A staged rollout distributes immutable signed artifacts through regional caches, while clients verify metadata and content before installation.
A staged rollout distributes immutable signed artifacts through regional caches, while clients verify metadata and content before installation.

Now the unreliable-network case. A commuter downloads 1.7 GB and loses Wi-Fi. A client that always restarts at byte zero wastes almost an entire package. HTTP range requests can fetch missing ranges from an immutable resource. Persist a chunk bitmap or verified byte ranges and request only those not already stored. Use a stable content identifier and validators when resuming, so bytes from two different builds do not become one corrupted installer. A final strong hash check verifies the assembled artifact before installation. The CDN's exact range caching behavior depends on vendor support and response headers. Treat it as a capability to test, not an assumption that every edge caches every possible range.

Security is the other half of the problem. HTTPS protects a transport connection, but delivery infrastructure should not alone decide what code clients install. A mirror can be misconfigured or compromised. Sign release metadata with appropriately protected keys and let clients verify the expected artifact digest, target identity and approved version. The Update Framework, or TUF, separates root trust, target descriptions, snapshots and freshness metadata to help defend against rollback, freeze and mix-and-match attacks. Adopting that design is more precise than telling a client to trust whatever JSON arrives from the closest CDN.

An apparently easy rollback also needs care. If version 42 is faulty, product operators may want devices back on version 41. But a client that blindly accepts any lower version is vulnerable to a network attacker repeatedly serving an old insecure release. Define a signed emergency rollback authorization or recovery policy that reconciles operational rollback with anti-rollback protections. Treat artifact authenticity and suitability for this device as separate checks. A correctly signed binary for another operating system is not necessarily a valid update here.

Rollout policy should avoid synchronized demand. Start with a small canary population, evaluate download failures, install success, launch crashes and user-visible regressions, then expand by geography and device class. Add randomized polling intervals and backoff for metadata checks. Limit bandwidth per origin and per edge path where the infrastructure supports it. Release waves reduce blast radius as well as peak bandwidth, which is useful when the failure is a broken installer rather than a capacity shortage.

Consider the difference between downloaded and installed. A device may finish transferring bytes and still fail integrity verification, run out of disk during patching, or fail to restart the application. Track the release state machine: offered, downloaded, verified, staged, installed, booted and healthy. It is tempting to stop measuring at HTTP 200, but that metric does not establish successful delivery of a working product.

The final architecture is deliberately boring in one respect. The large files are immutable, caches can reuse them, and clients verify exact bytes. The sophisticated work is in scheduling, metadata trust, safe recovery and staged rollout. A Staff or Principal candidate should be able to defend the capacity math, recovery behavior, security contract and rollout controls together, because an update distributed perfectly to a hundred million broken devices is still a failed system.

Go deeper

CloudFront documents HTTP range GET support and caching of parts of large objects, with response constraints and provider-specific cache-size limits. The Update Framework specifies signed metadata roles and expiration/version checks to mitigate stale or malicious repository responses. Binary delta systems must validate the expected base hash before applying a patch and the resulting artifact afterward. A hash alone supplies byte integrity relative to an expected digest. Authentication requires the expected digest to come from trusted signed metadata or an equivalent validated authority. For package downloads, distinguish delivery errors, verification failures, disk-space failures and install-time rollback signals.

Check your reasoning

  • Compute payload and peak transfer requirements with explicit units.
  • Design immutable CDN-distributed artifacts and checked range resume.
  • Use signed trust metadata and client-side integrity and version checks.
  • Consider deltas, full-artifact fallback and storage or patching costs.
  • Stage rollout and measure verified installation and boot health.

Common mistake: An HTTPS download from a trusted CDN needs no further artifact authorization.

Which distribution plan protects both capacity and client trust during a worldwide software rollout?

Read the answer explanation

Sign metadata, verify bytes and platform, and stage rollout with health gates — Transport identity does not substitute for signed release authorization and independently verified artifacts.

Apply it elsewhere

Deliver a 12 GB AI model update to a worldwide inference fleet where older workers must continue serving while new weights are fetched and verified.

Compare your approach

Quantify transfer demand, use immutable artifacts and range-resumable delivery, verify signed metadata and hashes, stage rollout with guardrails, and track installation health separately from download success.

One more constraint

If a client downloaded a file over HTTPS from the official CDN, can it safely skip independent artifact verification?

Quick revision

A global software update is a distribution and trust problem. Two gigabytes across 100 million devices is 200 PB before retries. Keep release metadata small and signed, serve immutable artifacts through CDNs, support verified range resume and deltas with full-package fallback, and make clients check hashes and authorized versions before installation. Roll out in controlled waves and observe installation health, not just transfer success. A compromised edge or mirror must not be able to choose which code a device runs.

References