Distributed Systems & System Design
Find your next engineering question
Start with a subject, or narrow the library to the level and problem you want to practise.
135 questions
Database Internals & Transactions
Database Internals & Transactions · Senior
The writes are fast. Why does one lookup scan a terabyte?Locating the newest value in an append-only store through an in-memory key directory, preserving tombstones, reclaiming segments, and rebuilding the index on restart.
Database Internals & Transactions · Staff
COMMIT succeeded. What exactly survived the crash?Defining a local durable-commit boundary and the role of WAL ordering, fsync, checkpoints, torn pages, and recovery under process versus system failure.
Database Internals & Transactions · Senior
One insert filled a B-tree page. What has to move?How B-tree leaf and parent page splits preserve ordered lookup under concurrent writers, including latches, fillfactor, key distribution, and secondary-row fetches.
Database Internals & Transactions · Senior
We stopped updating disk pages on every write. Where did the data go?Tracing LSM updates through WAL, memtables, immutable SSTs, overlapping L0 files, point reads, versioned deletes, and snapshots.
Database Internals & Transactions · Staff
Writes were fast for an hour. Why did the engine suddenly stop?Identifying an unsustainable background work rate and the distinct RocksDB stall triggers, then validating steady state without defeating durability or read health.
Database Internals & Transactions · Senior
We added an index. Why is the full table scan still faster?Read-path economics with correct selectivity: random heap fetches versus sequential scan; why index presence does not establish a faster access route.
Database Internals & Transactions · Senior
The index has all three columns. Why is there still a Sort?Composite B-tree key ordering and ORDER BY satisfaction across multiple equality groups; indexes do not promise one global ordering across grouped ranges.
Database Internals & Transactions · Staff
The SQL is identical. Did PostgreSQL really use the same plan?Planner statistics and prepared-statement generic-vs-custom plan choice under extreme tenant skew, distinguished from correct high-volume work.
Database Internals & Transactions · Staff
A hash join spilled gigabytes. Should we just add memory?Memory and I/O behavior of a hash join crossing its working-set threshold, including batch spill, concurrent memory multiplication, and remediation.
Database Internals & Transactions · Staff
Index Only Scan still hits the heap. What is it checking?Why index-only scans still consult heap MVCC visibility via the visibility map, and how volatile rows, vacuum, and HOT index-maintenance constraints change the tradeoff.
Database Internals & Transactions · Senior
Both updates succeeded. Why is the counter missing one?Lost update from stale application-calculated assignments under Read Committed, distinguished from atomic SQL arithmetic, compare-and-swap, and short row locking.
Database Internals & Transactions · Staff
The reader never blocked the writer. Where did the old row go?Multi-version visibility and snapshot lifetime, distinguishing Read Committed statement snapshots from Repeatable Read transaction snapshots and the vacuum retention cost.
Database Internals & Transactions · Staff
Both doctors followed the rule. Why was nobody left on call?Write skew under snapshot isolation, demonstrating a cross-row predicate invariant violated without a write/write conflict and evaluated through guard-row locking or serializable retry.
Database Internals & Transactions · Staff
Each transfer holds one row. Why is nothing moving?Real row-lock deadlocks versus ordinary contention, deterministic acquisition ordering, database victim selection and bounded whole-transaction recovery.
Database Internals & Transactions · Staff
Serializable fixed the bug. Why did checkout get slower?Serializable Snapshot Isolation conflicts, SQLSTATE 40001, bounded full retries, read-footprint granularity and goodput versus wasted retries under contention.
Database Internals & Transactions · Senior
The profile lookup takes 3 ms. Why does a new filter scan every customer?Decide when keyed lookup is sufficient, when a scan is unacceptable and what freshness contract a separate read model needs.
Database Internals & Transactions · Staff
The phone lookup found a user. Why did it return the wrong account?Distinguish secondary lookup from uniqueness, then protect reassignment of an alternate key across separate records.
Database Internals & Transactions · Senior
The post loads in one read. Why is adding one comment becoming expensive?Choose embedding versus referencing from boundedness, lifecycle, query shape and single-document atomicity.
Database Internals & Transactions · Senior
The inbox query is simple. Why does Cassandra need another table?Derive Cassandra partitions, clustering order, bucketing and denormalized tables from concrete user queries.
Database Internals & Transactions · Staff
Both writes were in one batch. Why did a reader see only one?Distinguish Cassandra logged-batch atomic application from isolation, and assign authority and repair for duplicated query tables.
Database Internals & Transactions · Senior
The title contains the word. Why is the database reading every row?Distinguish leading-wildcard scans, token retrieval and arbitrary substring matching, then select suitable indexes and semantics.
Database Internals & Transactions · Staff
Search takes 40 milliseconds. Why are the first ten results wrong?Separate lexical BM25 score, candidate recall, graded ranking quality and user success using judged queries and safe experiments.
Database Internals & Transactions · Staff
Elasticsearch acknowledged the document. Why can't search find it?Differentiate index durability, search refresh, Lucene commit, and immutable-segment merging without false ordering assumptions.
Database Internals & Transactions · Staff
Each shard returned its best ten. Why is the global top ten wrong?Prove when local top-K is globally sufficient, identify shard-local scoring divergence, and design consistent pagination and fanout.
Database Internals & Transactions · Staff
The filter says 1,200 jackets. Why can search find only 1,130?Define exact facet query populations, distinguish post filters, and reason about approximate shard-local terms buckets.
Database Internals & Transactions · Principal
The new index has two billion documents. Can we switch search to it now?Migrate search mappings through consistent snapshots, change history, version-safe writes, cutover and reversible rollback.
Replication, Consistency & Consensus
Replication, Consistency & Consensus · Principal
The region failed over. Which promises survived?Local transaction atomicity does not imply regional survival. Acknowledgment, promotion authority, and consequential publication must protect the same business decision.
Replication, Consistency & Consensus · Staff
The follower has the commit record. Why is the read still old?Distinguish WAL receive, durable flush and replay visibility, and gate reads on a reliable apply boundary.
Replication, Consistency & Consensus · Senior
The profile saved. Why does your laptop still show the old name?Define scope of read-your-writes and design client or account-level consistency tokens with fallback behavior.
Replication, Consistency & Consensus · Staff
Two of three replicas agreed. Why did a later read return older data?Distinguish quorum intersection from monotonic reads and linearizability under partially applied or concurrent writes.
Replication, Consistency & Consensus · Staff
We added twenty read replicas. Why did the primary get slower?Diagnose write-history fanout, WAL-retention pressure, standby feedback and alternative read architectures.
Replication, Consistency & Consensus · Staff
The old leader is alive. Why won't the cluster accept its writes?Explain minority behavior, term-based elections, candidate freshness and the scope of reads under a network partition.
Replication, Consistency & Consensus · Staff
Three servers have the log entry. Why won't Raft commit it?Distinguish log replication from commitment of prior-term entries, and connect Raft terms, election restriction and current-term commitment.
Replication, Consistency & Consensus · Staff
The lease expired. Why can its previous owner still write?Distinguish lease expiration from the physical ability to write, and enforce non-regression with destination-side fencing.
Replication, Consistency & Consensus · Staff
The leader has applied every entry. Why might its read still be stale?Establish linearizable read authority using a quorum-confirmed read barrier and state-machine application, separate from local replay lag.
Replication, Consistency & Consensus · Principal
We added two voters. How could a routine change split the cluster?Prove overlapping authority across reconfiguration and plan learner promotion, joint consensus or sequential membership operations with failure recovery.
Networking & Protocols
Networking & Protocols · Staff
The server finished in 30 ms. Where did the other 370 ms go?Separating end-to-end client latency from instrumented server work, including requests that never produce an application span.
Networking & Protocols · Staff
We added six servers. Why did none get traffic?The difference between assigning transport connections and routing requests when HTTP/2 channels are long-lived.
Networking & Protocols · Staff
The link is 10 Gbps. Why is one file moving at 200 Mbps?Calculate BDP and distinguish receiver-window, congestion-window and application pacing limits on a long-RTT TCP flow.
Networking & Protocols · Staff
Only one percent of packets are lost. Why did API p99 explode?Diagnose how a small segment-loss rate can block ordered bytes, trigger retransmission timers and alter tail latency and congestion state.
Networking & Protocols · Staff
The server is idle. Why are new connections timing out?Differentiate incomplete SYN requests from completed connections awaiting accept, measure burst capacity and repair slow accept loops.
Networking & Protocols · Staff
All services are healthy. Who used the last outbound port?Calculate destination-specific NAT mapping occupancy and distinguish gateway port exhaustion from local ephemeral ports and bandwidth limits.
Networking & Protocols · Staff
Tiny requests work. Why do large responses hang over the VPN?Diagnose size-dependent VPN loss through tunnel overhead, MTU and MSS arithmetic, ICMP feedback and black-hole-safe probing.
Networking & Protocols · Principal
The load balancer says healthy. Why do 30 percent of requests fail?Detect partial backend failures invisible to liveness probes using client-outcome dimensions and safely bounded outlier ejection.
Networking & Protocols · Senior
The API is healthy. Why does the first request wait on DNS?Trace stub, recursive and authoritative lookups, including TTLs, CNAME chains and negative caching.
Networking & Protocols · Staff
We changed the DNS record. Why are 20% of users still on the old IP?Design a safe DNS-based migration despite existing resolver caches, HTTP connections, stale answers and rollback limitations.
Networking & Protocols · Senior
The server has a valid certificate. Why does the browser refuse HTTPS?Explain SAN hostname validation, intermediate and root trust chains, SNI and client-specific certificate policy.
Networking & Protocols · Staff
TLS 1.3 is faster. Should a payment API accept 0-RTT?Compare full TLS 1.2 and TLS 1.3 setup, PSK resumption, early-data replay and safe application policy.
Networking & Protocols · Staff
HTTP/2 multiplexes requests. Why can one lost packet stall them all?Distinguish HTTP application serialization, TCP stream-level head-of-line blocking and independent QUIC stream delivery.
Networking & Protocols · Staff
The phone switched from Wi-Fi to 5G. Why did the request survive?Explain address-independent QUIC connection IDs, path validation, anti-amplification and backend affinity across network changes.
Networking & Protocols · Senior
The database is healthy. Why can't the pod reach it?Trace DNS, pod source address, private routes, security policy and the return path without assuming NAT.
Networking & Protocols · Principal
We connected four regions. Why did one routing change affect all of them?Choose peering, transit gateways and service endpoints from a traffic matrix while limiting failures and privileges.
Networking & Protocols · Staff
Every server is healthy. Why did the internet lose the route to them?Separate BGP control-plane route withdrawals from healthy application servers, and diagnose internet reachability from outside.
Networking & Protocols · Staff
The IP address stayed the same. Why did the connection move to another city?Explain shared anycast IPs, routing catchments, and why stateful TCP connections can fail when the chosen site changes.
Networking & Protocols · Staff
We enabled IPv6. Why did only some customers lose the API?Diagnose failures after adding IPv6 AAAA records, including client fallback variation, NAT64, MTU and policy parity.
Networking & Protocols · Staff
We upgraded the inter-region link. Why is checkout still taking a second?Quantify the critical path from dependent WAN calls and move work to the data without weakening consistency.
Networking & Protocols · Senior
The browser uploaded the photo. Why is it not yet part of the account?Move upload bandwidth off the API without delegating final publication or authorization to the browser.
Networking & Protocols · Staff
A photo was uploaded once. Why did the image service generate 40,000 files?Bound transformation cardinality and secure decoding while using source and recipe identity in derivative cache keys.
Networking & Protocols · Staff
We put a CDN in front of storage. Why is the origin still receiving thousands of requests?Explain independent edge misses and how hierarchical caching reduces eligible repeated origin fetches.
Networking & Protocols · Staff
The origin is down. Should the CDN show yesterday's page or return an error?Derive cache freshness and extension windows and choose a business-safe permitted staleness contract.
Networking & Protocols · Principal
The edge lost its control plane. Why did it send users to a retired origin?Define policy-specific offline behavior using versioned, validated configuration snapshots and revocation limits.
Networking & Protocols · Staff
One software release is 2 GB. How do you deliver it to 100 million devices?Design signed content delivery, partial resume, rollout waves and post-install health for massive downloads.
Partitioning & Distributed Data
Partitioning & Distributed Data · Senior
We added one cache server. Why did nearly every key move?Compute actual key movement under modulo 4 to 5, compare stable rings and logical slots, and separate owner mapping from physical migration.
Partitioning & Distributed Data · Staff
We doubled the cache shards. Why did one shard get no relief?Diagnose one hot key versus a hot range or placement imbalance, then distinguish bounded-staleness read distribution from inventory-write authority.
Partitioning & Distributed Data · Staff
The writes are spread evenly. Why does one customer's history need 80 queries?Choose physical placement from point-read and per-tenant ordered scan patterns, then manage whale tenants and the ordering cost of sub-bucketing.
Partitioning & Distributed Data · Principal
The new shard is caught up. Can we send it writes now?Specify online shard migration with snapshot and CDC boundaries, tenant-scoped write fencing, stale-router rejection, and rollback after target writes.
Partitioning & Distributed Data · Staff
One storage node died. Why did the rest of the cluster follow it?Diagnose feedback between lost capacity, replica repair I/O, health checks and rebalancing, then size and prioritize recovery under a durability deadline.
Partitioning & Distributed Data · Senior
The bytes are durable. Why can't the database use this object bucket as its disk?Classify storage by operations, mutation granularity, namespaces and durability contracts.
Partitioning & Distributed Data · Staff
The 200 GB upload finished. Why shouldn't we show the file yet?Separate part acceptance, checksums, completion, conditional catalog publication and cleanup.
Partitioning & Distributed Data · Staff
Three copies cost too much. Will six data shards and three parity shards be safer?Derive redundancy overhead, recovery thresholds, placement risks and repair IO.
Partitioning & Distributed Data · Staff
The backup was written five years ago. How do we know its bytes are still correct?Detect latent corruption using expected checksums, scheduled scrub work and verified reconstruction.
Partitioning & Distributed Data · Principal
The cleanup job deleted the chunks. Why were active files still using them?Protect committed manifests from stale mark-and-sweep decisions, delayed DELETE and ABA.
Caching & Performance
Caching & Performance · Senior
The price changed. Why did the old price come back?Recognize a read-fill/invalidation interleaving and select a freshness contract with correctly preserved ordering evidence.
Caching & Performance · Staff
One key expired. Why did database reads surge?Turn an expired hot key from a database stampede into bounded refill work, with explicit failure and staleness policies.
Caching & Performance · Senior
Hit rate fell from 95% to 80%. Why did the database suddenly struggle?Translate a percentage-point hit-rate regression into origin demand, then diagnose workload and key-distribution causes.
Caching & Performance · Staff
The IDs do not exist. Why are they melting the database?Protect misses for absent keys using validation, scoped negative caching, optional membership filters and a safe creation freshness contract.
Caching & Performance · Staff
The cache was fast. Why did one customer see another customer's orders?Treat cached identity as part of the authorization boundary, applying HTTP shared-cache rules to personalized responses.
Caching & Performance · Principal
Every cache node restarted. Why did adding capacity make recovery worse?Classify cache as a capacity dependency and design bounded recovery when the entire working set is cold.
Queues, Streams & Event Processing
Queues, Streams & Event Processing · Staff
The order exists. Why has the warehouse never heard about it?Atomic order creation cannot make broker publication atomic. A durable publication obligation, relay recovery, and source ordering solve a new boundary.
Queues, Streams & Event Processing · Staff
The warehouse processed the event. Why did it arrive again?Consumer offsets are progress markers, not proof of completed business effects. Atomic receipts and business identities defend separate races.
Queues, Streams & Event Processing · Staff
The replay worked. Why did it reserve stock again?Replay is a separate capability with permission boundaries, history completeness, and cutover conditions. It is not just a consumer group reset.
Queues, Streams & Event Processing · Senior
Publishing got faster. Why are urgent events arriving late?Choose producer batching policies by tracing buffering and end-to-end event freshness, including quiet-partition behavior.
Queues, Streams & Event Processing · Staff
We started 80 consumers. Why is one partition still two hours behind?Diagnose partition-level limits and per-key skew before claiming more consumer instances add throughput.
Queues, Streams & Event Processing · Staff
We quarantined the bad event. Which Kafka offset can we safely commit?Choose quarantine, repair and progress policy from per-key dependencies rather than applying a generic DLQ rule.
Queues, Streams & Event Processing · Staff
The consumer was down eight days. Where did day one go?Treat retention as a replay and recovery contract, distinguishing retained offsets from recoverable historical events.
Queues, Streams & Event Processing · Principal
One tenant published three times more. Why did every team slow down?Design end-to-end per-tenant quotas, admission and physical isolation while preserving other workloads under shared broker saturation.
Queues, Streams & Event Processing · Senior
Ten terabytes went into the job. Why is the shuffle taking longer than the scan?Explain the all-to-all exchange and distinguish input volume from shuffled bytes, with safe combinable pre-aggregation.
Queues, Streams & Event Processing · Staff
Nine hundred ninety-nine tasks finished. Why is the last one still running?Diagnose a slow executor versus one oversized key and select speculation or workload restructuring accordingly.
Queues, Streams & Event Processing · Staff
The lookup table was tiny. Why did the join move four terabytes?Choose distributed broadcast versus shuffle join from post-filter size, per-worker memory and source cardinality.
Queues, Streams & Event Processing · Staff
The dataset is only 200 GB. Why does opening it take longer than reading it?Explain why high file counts hurt planning and reads, and how snapshot-safe compaction and writer layout repair it.
Queues, Streams & Event Processing · Staff
Five hundred files were uploaded. Why can readers still see yesterday's table?Describe multi-file atomic visibility through immutable data, manifests and catalog compare-and-swap, including unknown commits.
Queues, Streams & Event Processing · Principal
Every pipeline task succeeded. Why is a week of revenue missing?Separate job success from semantic and business data correctness, gate publication, and recover through versioned lineage.
Reliability & Failure Recovery
Reliability & Failure Recovery · Senior
The customer clicked once. Why are there two orders?An HTTP timeout leaves the outcome unknown. Stable operation identity and a database constraint turn retries into one business decision.
Reliability & Failure Recovery · Staff
Orders, messages, and dashboards went down. Which one comes back first?Turn RPO and RTO into explicit business promises and architecturally enforceable acknowledgment, failover and restore behavior.
Reliability & Failure Recovery · Staff
The database and the object store both restored. Why are files missing?Restore an authoritative database, versioned blob store and derived index to one valid application cut using a checkpoint and protected references.
Reliability & Failure Recovery · Senior
Someone dropped the table at 15:07. Can we recover 15:06?Reconstruct PostgreSQL to a chosen point while preserving the original WAL history and legitimate post-incident commits.
Reliability & Failure Recovery · Principal
The attacker had administrator access. Why would the backups survive?Design backup retention and clean-room restore so an administrator-level production compromise cannot erase the last recoverable copy.
Reliability & Failure Recovery · Staff
The restore plan said three hours. Why did it take nine?Diagnose a nine-hour restore by measuring transfer ceilings, replay, dependencies and the customer-ready critical path.
Reliability & Failure Recovery · Principal
Every backup job is green. What evidence says the service can return?Define realistic restore drills with varied recovery points, domain invariants, customer transactions and measured RTO.
Reliability & Failure Recovery · Staff
The order completed in 200 ms. Why did the customer wait 40 seconds?Find a customer-visible 40-second gap that is absent from a fast synchronous HTTP trace, using messaging context, event timestamps and population SLIs.
Reliability & Failure Recovery · Staff
p99 doubled after deployment. Why do the sampled traces look fine?Diagnose a release-specific tail regression with comparable cohorts, unbiased histograms and tracing aware of head and tail sampling.
Reliability & Failure Recovery · Staff
Every dashboard is green. Where did 12,000 account updates go?Detect silently dropped source updates with freshness, completeness and reconciled source-to-projection version checks.
Reliability & Failure Recovery · Senior
Checkout is 99.9% reliable. Why did one bad hour consume so much budget?Derive actionable multi-window burn-rate alerts from a service-specific request SLI rather than thresholds on average success rate.
Reliability & Failure Recovery · Principal
The outage began. Why did every graph stop updating at the same time?Design independent service checks, bounded collector queues and degraded diagnostic paths without making customer requests depend on exporters.
Reliability & Failure Recovery · Principal
Five dashboards disagree. What do you do in the first ten minutes?Lead the first ten minutes of an ambiguous production incident with customer evidence, controlled changes, safe mitigation and documented hypotheses.
Reliability & Failure Recovery · Senior
The recommendation service is taking five seconds. Why are we still calling it?Use failure and slow-call signals, timeouts and bounded recovery probes to protect an optional dependency without inventing mandatory business outcomes.
Reliability & Failure Recovery · Staff
One service went down. Why did the other eleven follow?Follow cascading failure propagation through nested attempts, deadlines, abandoned work and redistributing failed backends, then cut the causal path.
Reliability & Failure Recovery · Principal
One tenant's import crashed a shard. Why did half the company go offline?Define a self-contained application and state cell with safe local routing, bounded cell capacity, controlled deployments and ownership-aware migration.
Reliability & Failure Recovery · Staff
Every customer has three workers. Why do the same noisy tenants keep colliding?Derive shuffle-shard full-collision and partial-overlap probabilities, and use those to evaluate shared-pool noisy-neighbor protection.
Reliability & Failure Recovery · Principal
The database recovered an hour ago. Why is the platform still down?Diagnose self-sustaining failure after a trigger ends and stage recovery using goodput, reduced admission, retry control and safe backlog draining.
Capacity, Isolation & Operations
Capacity, Isolation & Operations · Senior
CPU is at 20 percent. Why are 12,000 requests waiting?Using Little’s Law and pool budgets to explain why low CPU does not mean spare request capacity.
Capacity, Isolation & Operations · Staff
Traffic rose 15 percent. Why did latency jump sixfold?Understanding nonlinear queueing delay near saturation without turning a simplified model into a universal rule.
Capacity, Isolation & Operations · Staff
One refund failed. Why can't the metrics dashboard find that request?Choose which telemetry signal answers fleet health, individual decisions and distributed request paths without exploding metric cardinality.
Capacity, Isolation & Operations · Staff
Samples per second stayed flat. Why did the metrics server run out of memory?Diagnose series-creation and label-churn pressure independently of sample volume and design observable safety limits.
Capacity, Isolation & Operations · Staff
The server received 160 MB of samples. Why are the compressed chunks so small?Explain why delta-of-delta timestamps and XOR floats compress predictable series and when that advantage disappears.
Capacity, Isolation & Operations · Staff
We kept hourly metrics for two years. Why can't we reconstruct last Tuesday's spike?Design retention, rollup dependencies, valid mergeable statistics and safe query precision across raw and coarse intervals.
Capacity, Isolation & Operations · Staff
Region A reports 100 ms p99. Region B reports 10 seconds. What's the global p99?Demonstrate the non-mergeability of local quantiles and design correct aggregation of histogram populations.
Capacity, Isolation & Operations · Staff
One dashboard opened a 30-day graph. Why did every metrics query slow down?Protect the shared metrics store using read-work cost budgets, fairness, workload isolation and appropriate query resolution.
Capacity, Isolation & Operations · Staff
CPU is at 40%. Why is the pod being throttled?Explain bursty cgroup CPU quota enforcement and validate scheduling-related tail latency.
Capacity, Isolation & Operations · Senior
Every few minutes, p99 jumps by 300 ms. Is it garbage collection?Diagnose Java GC-related tail latency using JFR events, allocation rate and live data size.
Capacity, Isolation & Operations · Staff
We upgraded from 16 cores to 64. Why did throughput fall?Measure serialization caused by global locks and redesign for per-key independent ownership.
Capacity, Isolation & Operations · Principal
The database moved to a bigger server. Why did memory get slower?Measure remote memory access and decide CPU/memory placement for database working sets.
Capacity, Isolation & Operations · Staff
The database never crashed. Why did swapping make every query slow?Distinguish swap occupancy, active memory faults and reclaim stalls while designing memory budgets.
Capacity, Isolation & Operations · Senior
The disk is fast and the network is fast. Why does copying data use all the CPU?Explain why kernel-assisted file-to-socket transfer removes redundant user-space copies and its limits.
Capacity, Isolation & Operations · Senior
Fifty services sit behind one gateway. Why is every launch waiting on its team?Separate cross-cutting edge responsibilities from domain authorization and client-specific BFF composition without creating a shared business monolith.
Capacity, Isolation & Operations · Staff
The contract says 100 requests per second. Why did 200 requests pass at midnight?Explain fixed window burst boundaries and choose among exact or approximate sliding counts, token buckets and paced queues based on workload semantics.
Capacity, Isolation & Operations · Staff
We admit only 500 requests per second. Why does latency still collapse?Differentiate request arrival rate from concurrent scarce work using Little’s Law and build a feedback limiter with bounded queueing and oscillation controls.
Capacity, Isolation & Operations · Principal
Checkout is overloaded. Should premium customers get every available slot?Allocate scarce API work by criticality and work cost, preserve tenant fairness, and shed early rather than treat paid-user priority as unlimited access.
Capacity, Isolation & Operations · Staff
The policy service is down. Should the gateway let everyone through?Specify different safe degraded behaviors for authentication, rate limiting, fraud scoring and feature flags with bounded freshness and auditable recovery.
Capacity, Isolation & Operations · Senior
ZooKeeper is healthy again. Why is every API request still waiting for it?Move ordinary configuration reads into validated atomic local snapshots, with safe bootstrap, watch recovery and differentiated staleness policies.
Capacity, Isolation & Operations · Staff
The dead instance was removed. Why do clients keep calling it?Distinguish updated service registry membership from per-client caches, existing connection pools and graceful versus unexpected backend loss.
Capacity, Isolation & Operations · Principal
One configuration field changed. How do you roll it out to 1,000 services?Deliver versioned, semantically validated and compatible configuration to large fleets through measurable staged rollout and tested rollback.
Capacity, Isolation & Operations · Staff
The feature flag says off. Why are some customers still seeing the feature?Design deterministic local feature-flag evaluation, context propagation, version observability and critical kill-switch limits.
Capacity, Isolation & Operations · Principal
The routing config passed every check. Why did one cell receive 70% of traffic?Reject routing configurations that pass parsing but violate weighted capacity, state ownership, residency or safe rollback conditions.
Capacity, Isolation & Operations · Principal
One resource changed. Why did thousands of controllers start hammering the API?Use list/watch, coalesced rate-limited reconcile queues, controlled retries and idempotent external operations to recover from watch and API-server interruptions.