A regional outage affects an ordering platform with 30,000 orders per minute. Product demands five-minute RTO and zero lost acknowledged orders. Present two architectures and explain the exact moment an order can be acknowledged.
Reliability & Failure Recovery · Staff
Orders, messages, and dashboards went down. Which one comes back first?
The Backup Succeeded. Can We Recover? · Part 1 of 6 · 5 min read
Read the worked explanation ↓Think first
A regional outage affects an ordering platform with 30,000 orders per minute. Product demands five-minute RTO and zero lost acknowledged orders. Present two architectures and explain the exact moment an order can be acknowledged.
Take a few minutes to form an approach, then compare it with the worked explanation.
Reveal the worked explanation
Interview question. A company operates an order service, a chat application, and an analytics pipeline. A regional outage takes out all three. Choose a recovery point objective and a recovery time objective for each, then explain what architecture those promises require and what they cost.
An interviewer asks for disaster recovery and someone immediately draws two regions with arrows between them. I'd ask about the arrows later. First I want to know which loss the business can actually survive. If a customer paid for an order that vanishes, somebody will spend the next week reconciling bank statements. If yesterday's revenue dashboard is late by an hour, the sales team may be annoyed but the underlying orders still exist. Those are not the same incident.
RPO is the maximum acceptable age of recoverable data. RTO is the maximum acceptable time before an acceptable service is available again. They measure different things. Suppose region A fails at 14:00, region B becomes usable at 14:12, and its latest complete recovery point is 13:58. You delivered a twelve-minute recovery time but lost two minutes of recent changes. A warm service that boots instantly with data from yesterday has an impressive RTO and a terrible RPO.
For this exercise, I would propose the following business assumptions, not universal industry standards. Orders should survive a regional outage without acknowledged financial commitments disappearing, and checkout should return within 15 minutes. Chat may tolerate roughly a minute of missing recent messages and 30 minutes of interruption, provided the product clearly communicates the limitation. Analytics may accept an hour-old recovery point and a four-hour recovery. Before committing to any of those, I'd confirm them with the business and legal owners.
Now make the architecture earn those numbers. For orders, a zero-loss regional RPO for acknowledged writes demands that acknowledgment be withheld until the required remote durability condition has been met, or a stronger distributed commit architecture that makes the same promise. That introduces cross-region network latency and unavailable-write behavior during a partition. Asynchronous replicas and frequent snapshots can offer excellent recovery, but they cannot truthfully promise zero loss of acknowledged commits if region A disappears before the remote copy receives them. Exactly what is included in the guarantee matters: database rows alone, or payment-intent events, inventory reservations, and the external payment processor's state too?
For chat, async replicated message logs and a backup region might be a reasonable trade. Keep message IDs stable so reconnecting devices can reconcile missing or duplicate deliveries. Some users may tolerate temporary read-only history while sending is disabled. Others may prefer brief loss of availability over messages silently vanishing. Spell that out in the user-visible contract. If the business changes the RPO requirement to ten seconds, ordinary one-minute backups no longer fit and you need to examine replication lag and remote persistence more closely.
For analytics, durable raw events plus replayable processing jobs may be more valuable than keeping a hot copy of every derived table. Restore the input data first, then rebuild dashboards. A pipeline can accept a four-hour RTO only if its dependencies, compute quota, catalogs, transformation code, and credentials can also be restored in that time. A warehouse snapshot alone does not deliver a functioning analytics service.
There is a second trap. RTO is not the time to start an instance. Include outage detection, incident declaration, deciding the failover is safe, provisioning any missing capacity, restoring dependencies, validating correctness, shifting traffic, and confirming that customers can actually complete their work. If your runbook says ten minutes but escalation alone takes twenty, the architectural diagram is irrelevant.
Cost follows the requirement. Backup and restore is relatively inexpensive but often slower. A pilot light preserves critical state and requires substantial startup work. Warm standby keeps a smaller functioning copy running and scales it during an incident. Active-active can reduce switching time but introduces more complicated coordination, correctness, and operational costs. None of these labels automatically establishes an RPO or RTO. A system with warm compute and a lagging database may meet one target but not the other.
At Staff level, I would ask who can authorize a cutover that risks losing acknowledged orders. Failing over while the old region is merely unreachable, rather than proven dead or safely fenced, can create two writers. That is a different failure from replaying an old backup, and recovery procedures must account for both. Keep an external transaction reconciliation path and a documented point at which it is safe to accept new orders.
I would finish the interview with a scorecard per workload: specified failure scope, target RPO and RTO, actual replica or archive lag, measured drill recovery time, unprotected dependencies, and the last successful full rehearsal. After one real exercise we may find that the order target requires more than the budget allows. That's when we renegotiate the business promise or the architecture, not when we quietly relabel a nightly backup as zero-RPO disaster recovery.
Go deeper
AWS distinguishes backup/restore, pilot light, warm standby and multi-site approaches, and describes warm standby as already capable of handling reduced traffic. The exact behavior depends on which state is replicated synchronously, which resources are actually ready, and whether the workload remains write-safe when communication breaks. A useful refinement is to separately measure detection time, decision time and execution time inside RTO. For regulated payment workflows, include processor settlement and reconciled financial state in the recovery definition rather than promising that one database replica is the complete truth.
Check your reasoning
- Separate RPO data loss from RTO service interruption with explicit failure assumptions.
- Tie zero-loss acknowledged order writes to sufficient remote durability or equivalent coordination.
- Choose distinct chat and analytics recovery policies rather than one global number.
- Include detection, declaration, dependencies, safe writer ownership and customer readiness.
- Defend the performance and cost of backup, pilot light, warm standby or synchronous operation.
Common mistake: Starting the replica in thirty seconds proves no orders were lost.
Read the answer explanation
Define service-specific RPO/RTO and verify acknowledgment, replication and failover paths — Fast service restoration and data freshness are separate guarantees. The acknowledgment path determines whether an acknowledged write can be lost.
Apply it elsewhere
A multiplayer game can recover chat history asynchronously but must not lose confirmed purchases. Choose different protection boundaries for chat messages, match scores, and purchases.
Compare your approach
Separate recovery time from recovery point, label business assumptions, link durability acknowledgment to regional loss, include dependencies and safe failover, and present cost and availability tradeoffs.
One more constraint
If the recovery region starts in thirty seconds but contains a five-minute-old database snapshot, did the system meet a zero-data-loss RPO?
Quick revision
RPO limits acceptable loss of recent data. RTO limits acceptable service interruption. They have separate implementations and failure assumptions. Critical acknowledged orders may need remote synchronous durability or equivalent coordination, at a latency and availability cost. Chat can often trade a small loss window for responsiveness, while analytics can replay durable source events. Count detection, decision, validation and traffic cutover inside RTO. Prove both objectives in drills, not by naming a replication topology.