Where the checksum is created matters
On upload, the client can compute a checksum over the expected content and send it with the request. The storage service independently computes its own checksum and rejects a mismatch. This catches failures on the upload path, subject to using a suitable algorithm and trusting the client's original bytes. For a security-sensitive untrusted-content identity or deduplication design, a cryptographic digest such as SHA-256 may be necessary because checksums intended for random errors are not collision resistant against malicious inputs.
The service stores a durable object-version identity and its authoritative checksum alongside the appropriate metadata. If an implementation recomputes the stored checksum from a damaged replica and overwrites the original expected checksum, it has normalized the corruption instead of detecting it. Keep the trust boundary clear between what the object was at commit time and what any single storage node claims today.
Multiple checksum granularities are useful. A full-object checksum validates a complete download or restore. Chunk checksums locate damage without reading or retransferring an entire multi-GB object. In an erasure-coded system, the parity and fragment layout also need validation, because one wrong fragment can affect reconstruction. A read of only one byte range may not be verifiable using a provider's full-object hash unless the service exposes checksums for the relevant chunks.
Read-time verification cannot protect cold data
Consider an object uploaded in 2021 that nobody reads until 2026. If corruption appears in 2023 and stays undetected, normal request monitoring sees nothing. A periodic scrub must proactively visit stored data, verify it against independent metadata, and detect unreadable or mismatched sectors long before the next restore.
A light scrub can examine placement and metadata consistency. A deep scrub actually reads data and validates content. Systems such as Ceph distinguish these forms. The deep operation costs bandwidth, IOPS, CPU, and cache pollution, so it must run with a schedule and resource budget. Scrubbing the most critical copies during a repair emergency can also compete with the very rebuild work that restores redundancy.
Put a number on a 100 PB scrub
Suppose we want to read all 100 PB once every 30 days. Using decimal units, 100 PB is 100,000,000,000 MB. Spread evenly over 2,592,000 seconds, that requires about 38.6 GB per second of sustained aggregate read throughput. That is an illustrative lower bound for one full logical pass, not a guarantee about the real physical bytes. Three replicas, erasure-coded fragments, retries, index reads, and geographic placement change the actual work. A seven-day full pass would require roughly 165 GB per second, before overhead. Those are cluster-wide numbers, which still have to be budgeted per rack and failure domain.
I would schedule scrubs with bounded tokens per device, rack, and region, then prioritize objects whose redundancy is low, whose last successful check is old, or whose storage devices show rising error signals. At the same time, I would prevent cold objects from being ignored forever just because hot objects continually generate higher-priority work. Track the age distribution since last validated read, not simply the aggregate count of successful scans.
Repair from evidence, not from a guess
When a fragment fails verification, quarantine it. Do not let a normal anti-entropy process silently spread its bytes to otherwise healthy copies. Identify a trusted full replica or enough independently validated erasure-coded fragments to reconstruct the expected content. Repair the damaged placement, verify the new copy against the authoritative checksum, and only then mark redundancy restored. If no valid reconstruction exists, the system must report irrecoverable corruption rather than return guessed bytes.
This interacts with normal lifecycle work. A scrub may be checking an old version that the deletion system is about to remove. It should either hold an appropriate lease or use a version and generation protocol so it does not accidentally recreate deleted data as if it were current. Conversely, a restore workflow should be able to pin the exact object version and its checksum, even if newer versions have replaced the visible key.
I would test by injecting corrupted bytes below the application layer, not merely by making a GET return an error. Flip one bit on one replica, corrupt parity while data fragments remain valid, corrupt metadata, and crash halfway through repair. Check that corruption is detected, the original expected checksum is not overwritten, readers never receive unverified bytes, and recovery survives a second crash. Measure mean detection delay and time spent with inadequate valid redundancy.
The strongest interview answer isn't 'use SHA-256' or 'run a cron job every week'. It establishes independent evidence of what the object should be, a bounded process that eventually checks cold data, and a repair protocol that restores valid copies without turning one silent error into permanent widespread corruption.
