Why an aggregate dashboard hides the problem
Imagine eight backends receive roughly equal traffic. Seven succeed, while the eighth fails 30 percent of its requests. Assuming even distribution, the region's overall failure rate is 1/8 × 30% = 3.75%. A 3.75% fleet failure rate is a serious availability incident, not a harmless change. The aggregate error rate should trigger investigation, but it dilutes the faulty node’s 30% failure rate and cannot identify the affected customer segment. Fleet CPU may remain normal, while sticky sessions can expose individual customers to much worse failure rates.
The arithmetic assumes uniform routing. Load balancers may route by weights, reused connections or affinity. Long-lived connections can keep requests pinned to one backend and make the distribution uneven. Here the new problem is failure detection when the probe path and customer path disagree.
Microsoft Research calls this differential observability: components disagree about health because they observe different code paths or failure domains.
Finding the failing slice
I would examine successful and failed requests with dimensions such as target backend, route, source region or ASN where appropriate, protocol version, new versus reused connection, response size and downstream dependency. Trace sampling must include errors, not only successful requests that reached the service. Client-side error metrics matter because a request that never reached the application leaves no server span.
Next I would create a targeted synthetic probe that exercises the suspected user path. If image downloads fail, an HTTP probe returning 20 bytes of static text is the wrong test. If IPv6 requests fail while IPv4 works, probing a loopback endpoint over IPv4 will miss it. Synthetic probes are only useful when they overlap the failures we are trying to detect, and their cost should be controlled.
Possible causes include a faulty NIC queue, packet-size failure, stale dependency route or corrupted local cache. The symptom alone cannot choose among them. Compare the failing slices until captures or dependency traces isolate the path.
A backend showing normal CPU tells us that expensive work was not happening at scale on the node. It does not tell us whether requests successfully arrived, were admitted, or completed. Several earlier networking failures would evade that check.
What should the load balancer do?
Marking nodes unhealthy after any error is dangerous. If a shared database fails, ejecting all backends reporting errors can remove the entire fleet. If one customer submits malformed requests, treating those 4xx responses as backend failures would also produce false ejection. Define meaningful backend errors and compare enough samples against peers.
I would use per-backend success rates for eligible requests, connect errors and timeout rates, plus outlier detection with minimum volume thresholds. Once a node is a statistically meaningful outlier, reduce its weight or eject it temporarily with hysteresis and a maximum fraction of capacity removed. Observe whether customer errors improve. Keep a controlled way to probe and reintroduce the node, because permanently blacklisting a temporarily faulty host can waste capacity and mask the underlying defect.
When possible, an independent control plane should decide health using several signals rather than asking a potentially faulty node to certify itself. This does not mean running the full checkout workflow every second. Deep dependency checks are expensive and can worsen incidents. The useful combination is cheap liveness, meaningful readiness, client outcome telemetry and selective end-to-end probes.
Separate availability by customer journey too. A failing checkout matters even if thousands of image requests succeed.
The interview decision
Would I keep routing to the node? Not at full weight while its customer failure rate is a supported outlier, assuming enough healthy capacity is available. I would reduce or eject it in a controlled way and gather evidence, while guarding against mass ejection if the problem is shared. A temporary traffic shift is incident mitigation, not proof that the node's hardware is at fault.
I would set acceptance criteria around per-backend error distribution, client success by route, health probe coverage, ejection safety and recovery behavior under injected partial failures. Kill-one-node chaos tests are insufficient. We need tests where a node accepts TCP but delays half the responses, only one packet size fails, or one dependency route becomes unreachable.
In the interview I would keep returning to the same evidence: can this backend complete real user work on the affected path, and can routing around it avoid harming the remaining fleet?
