Distributed Systems & System Design
Find your next engineering question
Start with a subject, or narrow the library to the level and problem you want to practise.
18 questions
Clear filtersReliability & Failure Recovery · Senior
The customer clicked once. Why are there two orders?An HTTP timeout leaves the outcome unknown. Stable operation identity and a database constraint turn retries into one business decision.
Reliability & Failure Recovery · Staff
Orders, messages, and dashboards went down. Which one comes back first?Turn RPO and RTO into explicit business promises and architecturally enforceable acknowledgment, failover and restore behavior.
Reliability & Failure Recovery · Staff
The database and the object store both restored. Why are files missing?Restore an authoritative database, versioned blob store and derived index to one valid application cut using a checkpoint and protected references.
Reliability & Failure Recovery · Senior
Someone dropped the table at 15:07. Can we recover 15:06?Reconstruct PostgreSQL to a chosen point while preserving the original WAL history and legitimate post-incident commits.
Reliability & Failure Recovery · Principal
The attacker had administrator access. Why would the backups survive?Design backup retention and clean-room restore so an administrator-level production compromise cannot erase the last recoverable copy.
Reliability & Failure Recovery · Staff
The restore plan said three hours. Why did it take nine?Diagnose a nine-hour restore by measuring transfer ceilings, replay, dependencies and the customer-ready critical path.
Reliability & Failure Recovery · Principal
Every backup job is green. What evidence says the service can return?Define realistic restore drills with varied recovery points, domain invariants, customer transactions and measured RTO.
Reliability & Failure Recovery · Staff
The order completed in 200 ms. Why did the customer wait 40 seconds?Find a customer-visible 40-second gap that is absent from a fast synchronous HTTP trace, using messaging context, event timestamps and population SLIs.
Reliability & Failure Recovery · Staff
p99 doubled after deployment. Why do the sampled traces look fine?Diagnose a release-specific tail regression with comparable cohorts, unbiased histograms and tracing aware of head and tail sampling.
Reliability & Failure Recovery · Staff
Every dashboard is green. Where did 12,000 account updates go?Detect silently dropped source updates with freshness, completeness and reconciled source-to-projection version checks.
Reliability & Failure Recovery · Senior
Checkout is 99.9% reliable. Why did one bad hour consume so much budget?Derive actionable multi-window burn-rate alerts from a service-specific request SLI rather than thresholds on average success rate.
Reliability & Failure Recovery · Principal
The outage began. Why did every graph stop updating at the same time?Design independent service checks, bounded collector queues and degraded diagnostic paths without making customer requests depend on exporters.
Reliability & Failure Recovery · Principal
Five dashboards disagree. What do you do in the first ten minutes?Lead the first ten minutes of an ambiguous production incident with customer evidence, controlled changes, safe mitigation and documented hypotheses.
Reliability & Failure Recovery · Senior
The recommendation service is taking five seconds. Why are we still calling it?Use failure and slow-call signals, timeouts and bounded recovery probes to protect an optional dependency without inventing mandatory business outcomes.
Reliability & Failure Recovery · Staff
One service went down. Why did the other eleven follow?Follow cascading failure propagation through nested attempts, deadlines, abandoned work and redistributing failed backends, then cut the causal path.
Reliability & Failure Recovery · Principal
One tenant's import crashed a shard. Why did half the company go offline?Define a self-contained application and state cell with safe local routing, bounded cell capacity, controlled deployments and ownership-aware migration.
Reliability & Failure Recovery · Staff
Every customer has three workers. Why do the same noisy tenants keep colliding?Derive shuffle-shard full-collision and partial-overlap probabilities, and use those to evaluate shared-pool noisy-neighbor protection.
Reliability & Failure Recovery · Principal
The database recovered an hour ago. Why is the platform still down?Diagnose self-sustaining failure after a trigger ends and stage recovery using goodput, reduced admission, retry control and safe backlog draining.