The database and the object store both restored. Why are files missing?

Interview question. Take a recoverable backup of a database, a search index, and an object store while the application continues accepting writes. Explain what it means for the three restored components to be consistent with one another.

Think about a customer attaching an invoice PDF to an order. The service writes the bytes to object storage, inserts the invoice row in PostgreSQL, and sends an indexing event so search can find it. All three systems have their own durability and backup machinery. At midnight their respective dashboards turn green. The next morning you restore the three midnight backups and discover an invoice row pointing at a blob that is not there.

Nothing required any individual backup to fail. The database snapshot might include a committed reference at 00:00:05 while the object snapshot reflects 23:59:55. The index could have documents from 00:00:20. Each copy is internally consistent, but the combination represents no single valid application state. Taking independent snapshots at approximately the same wall-clock time does not establish an atomic snapshot across systems.

First decide which component owns truth. In this design, the database owns invoice metadata and visibility, the object store owns immutable document bytes, and the search index is derived. That choice simplifies recovery. Search does not need a perfectly synchronized disk snapshot if we can reconstruct it from authoritative records plus an ordered stream of changes. But the database still must never expose an invoice referencing an unavailable blob. The write protocol matters as much as the backup tool.

I would enforce a publication order: upload to an immutable object key, verify the object is durable and readable, then commit the database reference and a durable index-outbox record together. An orphaned object can be cleaned later. A committed reference to missing bytes is much harder to defend. Make the object key immutable or retain its exact version so a later overwrite cannot change the meaning of the reference. A backup that restores old database references must retain the associated old object versions for at least the same recovery window.

A practical snapshot protocol starts by writing a known checkpoint marker in the authoritative database log. Record a database-consistent backup point and a change-stream offset, including the relevant database recovery timeline. Ensure every object reachable from the database at that checkpoint is preserved in the object retention set. If object storage uses versioned snapshots or inventory manifests, reconcile the set of referenced keys and versions against the protected inventory. Then take a search snapshot for speed if desired, but label it as an optimization. After recovery, rebuild or replay search changes from the same protected checkpoint until the index reflects the restored authoritative state.

What if the platform has already allowed delete operations? An object that is unreferenced now may have been referenced by yesterday's database snapshot. A cleanup job cannot treat current reachability as permission to remove every older object generation. Align object retention and GC grace periods with the oldest still-restorable database checkpoint. Otherwise a backup can pass all its tests and become unrestorable after a routine storage cleanup.

The interview follow-up is usually, 'We cannot pause writes.' That's fine. You can build a logical recovery barrier using an authoritative event position rather than freezing every system simultaneously. Capture database consistency at log position L. Subsequent writes continue, but restorations choose whether to stop at L or replay to a later known position. Independently archived objects must remain available for references visible at that chosen position. A change-data-capture cursor that starts too late leaves holes in the derived index. A cursor too early may cause duplicates, so derived updates need stable document identities and idempotent replay.

There are limits to the approach. If two independent authoritative databases commit parts of one business operation with no common transaction log or protocol, simply recording two offsets does not manufacture a globally consistent cut. You need a coordinated checkpoint, domain-level reconciliation, or a redesign with a clear source of truth. A consistent storage snapshot at a block level does not magically resolve an incomplete cross-service business operation either.

How would I verify the result? Restore into an isolated environment. Walk database references and check object existence plus the relevant version or checksum, not only aggregate object counts. Build the search index, then compare counts by tenant and collection, sample representative queries, and audit the indexing high-water mark. Exercise delete and overwrite cases as well as create. The exact recovered state need not include every event before the outage. It must instead satisfy its declared recovery point and all application invariants.

A strong answer separates three issues: backups that work individually, coordinated recoverability across dependencies, and the right to publish a recovered combination. The last requires verification. It is entirely possible to have three healthy backup systems and a broken application snapshot.