Every backup job is green. What evidence says the service can return?

Interview question. A SaaS product has completed nightly backups for eighteen months but has never tested a full restoration. Design a recurring disaster-recovery program that proves restored data is correct, dependencies are usable, and customer-facing recovery meets the advertised time objective.

I'd start by asking what the green check mark means. Usually it says the backup software completed an operation and stored something. That's useful, but narrower than most people imagine. It does not prove that you still have the decryption key, that the right database version can read the data, that a missing WAL segment will not stop replay, or that your service's authorization and payment paths work in the recovery environment. If nobody has restored the system, the claimed RTO is a hypothesis.

I would create a sterile recovery environment that cannot write into live production or reach customer-facing payment processors by accident. It should be built with documented infrastructure automation and emergency identities, not secret dependencies on the original region. Pick a recovery point according to a policy: sometimes the latest, sometimes a randomly selected older point inside retention, sometimes a point just before a deliberately injected bad write. Testing only yesterday's most convenient snapshot can hide old-key loss or expired archive segments.

Then perform a complete recovery, including the backup catalog, encryption and access keys, base data, the required logs, application configuration, identity integration, service discovery, and the minimum dependent services. If the identity service is itself down, 'restore the database, then log in to the identity service to get the recovery credentials' is a cycle. A recovery dependency graph should expose this before an actual disaster.

Data verification should be specific to the domain. For an order system, check that all committed orders within the declared recovery point have corresponding order lines and consistent totals. Compare transaction counts by time bucket and tenant, examine foreign-key integrity where enforced, verify a sampled set of blob references, and ensure tombstoned or privacy-deleted records are not unintentionally made visible by an old restore. A restored database passing SELECT 1 tells you almost nothing about financial correctness.

Now test actual workflows. Create a synthetic order, read it from the supported API, issue a controlled refund in a sandbox integration, and confirm the appropriate audit event and state transition. For chat, replay and fetch messages across devices. For the analytics tier, run a representative query and verify its declared data freshness. Use seeded synthetic accounts, never unauthorized use of customer data, and mock or sandbox irreversible external side effects. These tests answer whether customers can accomplish important tasks, not merely whether processes are alive.

The RTO clock needs a defined start. If an outage begins at 09:00 but a human declares disaster at 09:35, excluding those thirty-five minutes makes the metric misleading. Record discovery, decision, authorization, provisioning, data restoration, log replay, application startup, validation, and route switch. Stop the clock only when the agreed minimum service functions under representative load. If a test recovers in two hours with one engineer sitting next to the console but takes six hours at night, the practiced RTO should reflect the realistic staffing model.

Repeat under unfavorable conditions. One drill should assume the primary region is completely unavailable. Another should assume that the backup account is reachable but normal production credentials have been revoked. One should restore a recovery point from the middle of the retention period, and another should discover a broken dependency, such as a missing object generation or stale search index. Make these exercises safe and scheduled, with guardrails against touching real production resources. The objective is not heroic chaos. It is evidence that the documented failure envelope has been tested.

Automate what is mechanically repeatable. Backup providers, including AWS Backup, support recurring restore tests and record restore-job outcomes and duration. That is a strong starting signal. It is not a substitute for testing application invariants, external integrations, access controls and actual user transactions. Combine provider restore-status evidence with your own application-level acceptance suite and signed drill report. Keep test artifacts such as the restore point ID, source region, manifest hash, software versions, elapsed stages and exception list.

Some defects should block acceptance immediately: unrecoverable encryption keys, broken WAL continuity, insufficient backup retention, inability to authenticate a recovery operator, or restored committed orders referring to missing data. Others can be explicitly accepted for degraded mode, such as recommendation ranking unavailable while checkout works. Assign each finding an owner and a retest deadline. An unresolved critical defect should change the stated disaster-recovery readiness, not disappear in a meeting note.

As the product evolves, keep the drills current. A new payment gateway, encryption provider, search tier or cross-region dependency changes the restore graph. Make disaster recovery review part of introducing critical infrastructure, and maintain two views of readiness: engineering evidence that the backup can be restored, and business evidence that the restored service meets agreed recovery promises. Both matter, and neither can be inferred merely from a green nightly job.

I would close the interview by asking the team to demonstrate its most recent successful customer transaction inside the restored environment, along with the elapsed time from simulated failure and the exact data cutoff. That is the point where a backup stops being an intention and starts becoming a tested recovery capability.