One storage node died. Why did the rest of the cluster follow it?

The repair process competes with the customers

Losing a node does two things at once. It removes serving capacity and reduces redundancy for some partitions. The remaining machines take a larger share of live traffic while also becoming sources and destinations for data repair.

A replica copy uses network bandwidth, source disk reads, target disk writes, checksums, and perhaps compaction. Those resources are shared with foreground requests. On a busy cluster, repair can cause more latency than the missing node.

Then the feedback starts. Slower disks increase request latency. Request queues grow. Health checks time out and mark another server unavailable. Traffic concentrates on fewer survivors. The membership controller interprets this as more lost capacity and starts moving more data. Every sensible-looking local recovery action increases pressure on the cluster as a whole.

Google's SRE guidance describes this class of cascading overload and warns that recovery automation, health checks, and rebalancing reactions can amplify the failure instead of restoring service. Addressing cascading failures

Calculate the recovery work before choosing a throttle

Suppose each node stores 6 TB of replica data and losing one node requires restoring 6 TB elsewhere. This is an illustrative repair-size assumption, not a claim that all replication layouts produce exactly this amount.

To restore 6 TB in one hour, the system must transfer an average of about 1.67 GB per second of payload. That is roughly 13.3 Gbit per second of aggregate payload traffic, before protocol overhead, uneven placement, retries, additional reads, and disk amplification. The bottleneck may be one source link or destination disk, not total network capacity.

Now look at foreground headroom. Imagine the cluster was using the equivalent of 7.2 full node capacities before failure, or 60 percent of its 12-node fleet. If the workload is evenly redistributable, the remaining 11 nodes carry about 65 percent of their combined rated serving capacity before any repair cost. But averages hide hot partitions and unequal disks.

For the one-hour objective, measure how much repair work can fit into the residual per-resource capacity while preserving latency and useful throughput. “Copy at full network speed” is not a recovery policy.

Protect minimum redundancy before optimizing balance

I would separate two controller goals. Repair restores the required number and placement of durable replicas for affected partitions. Rebalancing tries to make ordinary load or storage utilization more even. During a real failure, those jobs should not automatically compete as equal priorities.

First prevent the failed node from repeatedly entering and leaving placement decisions during a short network flap. Use deliberate failure detection and a bounded stability interval where appropriate. Do not immediately destroy potentially recoverable copies merely because a heartbeat was missed. But do not wait indefinitely if the cluster has lost the only remaining safe replica for some data.

Next prioritize missing redundancy by risk. A partition with one surviving copy in a fragile failure domain deserves earlier attention than a partition that still meets its intended durable replica count but happens to be unevenly placed. Use admission or token budgets for per-node streaming bandwidth, disk IO, and concurrent transfers. Reserve serving headroom for foreground requests. Spread repair sources and destinations.

As load rises, reduce optional rebalancing first. Keep recovery controllers from issuing another wave of moves simply because a temporarily overloaded repair target misses a shallow health check. Distinguish failed storage from temporary repair-induced slowness without leaving clients waiting indefinitely.

An urgent one-hour deadline changes the decision

A fixed repair throttle that misses the deadline is not acceptable when another failure could cause permanent loss. The choice is not to disable throttling. Confirm the actual remaining failure tolerance, the likelihood and consequence of a second loss, and the capacity available for the required 6 TB transfer.

If the fleet cannot meet the target alongside every foreground SLO, shed low-priority analytics, bulk scans, and other deferrable work. Protect essential requests and tighten admission instead of allowing unbounded backlogs. Reserve recovery headroom to make the promise realistic.

Do not defer compaction or integrity checking blindly. Some storage engines need them to keep accepting writes or to validate repaired files. Set priorities from actual device pressure, replication health, and the nature of the affected dataset.

Google's guidance favors load tests that reach and pass the failure point, controlled rejection, and explicit degraded modes. It also warns that sudden load shifts can keep a recovering cluster in a crash loop after the original trigger is gone. Google SRE

Prove recovery, not merely quiet dashboards

The finish line is not “all nodes report green.” Verify that the intended replica count and failure-domain placement have been restored, repair checksums or state comparisons are complete, and client error rates, latency, and oldest pending repair age have returned to acceptable bounds. A cancelled repair job must be resumable without accepting stale or corrupt replicas as current.

Test node loss under realistic utilization. Record useful throughput, hottest disk, repair completion time, and whether the controller starts extra moves.

The key interview judgment is that repair traffic has a resource budget and a correctness deadline. The right controller protects both, instead of assuming that making every replacement as fast as possible is the safest response.