The dead instance was removed. Why do clients keep calling it?

Interview question. A Java service discovers payment endpoints from a registry and caches them. A payment instance is drained and removed, but hundreds of clients continue sending it traffic for minutes. Design discovery, connection management, and failover so clients tolerate stale membership without requiring a registry lookup on every request.

Start with a misleading dashboard. The registry shows six healthy payment endpoints. A deployment deleted one of the old pods three minutes ago, yet request logs still show traffic to its former IP. The registry is not necessarily wrong. It may know the newest membership while a client still holds an older address list, a pooled TCP connection, or a load-balancer assignment built from an earlier list. Service discovery is not one lookup. It is an agreement about which endpoint set a caller is willing to use, and how quickly that willingness changes.

A useful mental model has three clocks. First, the control plane observes that an instance is terminating or unhealthy. Second, clients receive or refresh a new endpoint set. Third, existing connections finish or are closed. A ten-second discovery TTL controls neither the time required to classify an unhealthy instance nor the lifetime of a pooled connection already established to it. This explains why changing one TTL often fails to fix the incident.

For Kubernetes-backed discovery, be precise about the API in use. EndpointSlices expose ready, serving, and terminating conditions. Normally a terminating endpoint stops receiving new Service traffic, but serving can remain true during draining. Implementations may have a defined fallback when all endpoints are terminating, so describing ready=false as a universal packet-level kill switch would be incorrect. A client that consumes EndpointSlices directly must handle multiple slices for one Service and decide how the endpoint conditions affect new connections. A client that resolves a stable ClusterIP delegates much of the membership mechanics to the cluster service proxy instead.

There are two main consumption patterns. In server-side load balancing the client addresses a stable virtual endpoint and the balancer maintains backend membership. The client can still have long-lived connections to the balancer, but it need not track every backend. In client-side discovery, the client maintains a local endpoint set and chooses a target for each new connection or RPC. Client-side selection can support locality and per-endpoint health, at the cost of more update and failure-handling logic in every language SDK. Neither pattern eliminates draining or already-started requests.

Now work through a rollout. At 12:00:00 instance A receives a termination signal. It marks itself unavailable for new work and begins draining. At 12:00:02 the registry or EndpointSlice indicates termination. At 12:00:05 one client learns the new membership but continues to reuse a payment HTTP/2 connection pinned to A. At 12:00:30 A exits. Requests multiplexed onto that connection may be reset even though discovery looked correct. The client needs a connection-drain policy and the server needs a shutdown grace period. When possible, protocols such as HTTP/2 can use graceful connection retirement so clients stop creating new streams on a draining connection, but the specific proxy and application behavior must be tested.

A second failure occurs when A dies unexpectedly rather than draining. You cannot wait forever for a registry TTL. Per-endpoint transport failures, failed connection establishment, and carefully classified timeout signals should eject the suspect endpoint locally for a bounded period. This is a circuit for one target, not proof that the entire payment service is down. Maintain failure budgets and periodically probe recovery. Avoid ejecting every endpoint when a common downstream database is slow. If all targets fail together, the correct response may be bounded admission and a clear dependency error rather than retrying each endpoint repeatedly.

How stale may the membership cache become? A 20-second refresh cycle plus a 10-second watch-disconnection interval can produce longer staleness than the nominal TTL alone. Expose last successful refresh time, membership version, endpoint counts and per-endpoint request outcomes. A refresh must swap the complete endpoint list atomically. Never update a shared list in place while request threads iterate over it. And never replace a healthy old snapshot with an empty list just because the registry request timed out. An authoritative empty result has different meaning from a failed fetch.

For strongly controlled deployments, publish endpoint changes in stages. Announce draining, stop admitting new requests, allow ongoing requests to finish within a maximum drain deadline, and only then remove the instance. Consider the client's clock and caches: no deploy controller can guarantee a remote client will update within exactly five seconds under partitions. Therefore keep drain windows conservative and require request semantics that tolerate resets and retries when appropriate. A payment charge must retain its idempotency identity across retries. The discovery library should not invent a fresh charge ID whenever it switches endpoints.

A small numerical example helps. Suppose 120 clients each maintain eight connections and half their pools still include A. Up to 480 connection slots may continue targeting A even after the six-endpoint membership list updates, depending on how the pool distributes connections. That is not the number of ongoing requests. HTTP/2 streams can multiply work per connection, so connection count alone is a poor measure of exposure. Track new streams sent to draining endpoints, inflight streams, resets, and successful drains.

To validate the design, kill a pod without warning and then repeat with a graceful rollout. Interrupt the discovery subscription, delay updates, and test clients that keep long-lived connections. Confirm that clients prefer valid local state over a registry timeout, stop selecting an endpoint once it is known to be draining, and bound the time to abandon a demonstrably dead endpoint. Record what behavior is not guaranteed during partitions. The interview answer is not "set DNS TTL to five seconds." It is a complete contract linking endpoint membership, actual connections, health evidence, and operation-level correctness.