Learning series · 6 episodes · Staff / Senior / Principal
The Backup Succeeded. Can We Recover?
Six connected investigations into recovery targets, cross-system consistency, point-in-time recovery, compromised backups, restore bottlenecks and DR proof.
Start series →Follow the decisions in order
- Part 1 of 6
Staff
Orders, messages, and dashboards went down. Which one comes back first?Turn RPO and RTO into explicit business promises and architecturally enforceable acknowledgment, failover and restore behavior.
- Part 2 of 6
Staff
The database and the object store both restored. Why are files missing?Restore an authoritative database, versioned blob store and derived index to one valid application cut using a checkpoint and protected references.
- Part 3 of 6
Senior
Someone dropped the table at 15:07. Can we recover 15:06?Reconstruct PostgreSQL to a chosen point while preserving the original WAL history and legitimate post-incident commits.
- Part 4 of 6
Principal
The attacker had administrator access. Why would the backups survive?Design backup retention and clean-room restore so an administrator-level production compromise cannot erase the last recoverable copy.
- Part 5 of 6
Staff
The restore plan said three hours. Why did it take nine?Diagnose a nine-hour restore by measuring transfer ceilings, replay, dependencies and the customer-ready critical path.
- Part 6 of 6
Principal
Every backup job is green. What evidence says the service can return?Define realistic restore drills with varied recovery points, domain invariants, customer transactions and measured RTO.