Networking & Protocols · Staff
Where the time goes · Part 2 of 5 · 6 min read
Read the worked explanation ↓Interview question. A service has two busy backend servers behind a layer 4 load balancer. You add six healthy servers. For the next twenty minutes, the old pair stays busy and the new six receive almost no requests. Clients use a handful of long-lived HTTP/2 connections. Explain what is happening and rebalance traffic without abruptly terminating active streams.
You may change the routing layer and client connection policy. You may not assume every request establishes a new TCP connection.
A gRPC client fleet sends millions of calls through four existing long-lived channels. A layer 4 load balancer learns about six new backends, but all calls remain on the older hosts. Diagnose the placement behavior and design a safe transition.
Take a few minutes to form an approach, then compare it with the worked explanation.
The deployment succeeded. The new servers pass health checks. The load balancer knows about all eight destinations. Yet their request counters barely move.
That is plausible when the balancing decision is made per connection rather than per application request. The original connections were assigned when only two backends existed. HTTP/2 allows many independent request streams to share a connection, so those connections can carry most of the application's traffic for a long time. Adding servers changes the placement options for new connections. It does not normally teleport existing TCP connections to another backend.
This is different from a layer 7 proxy that terminates HTTP/2, observes individual streams, and can distribute upstream requests across a pool of backend connections. The protocol alone does not decide the behavior. The placement of protocol termination and the proxy's routing policy do.
Before prescribing anything, check the topology. Is HTTP/2 terminated at the edge, at the intermediate load balancer, or at each backend? Does the balancer select a destination using a connection's address tuple? Does the client maintain a fixed channel for every call? Is there a second proxy doing request-level balancing later?
The answers tell us which connections matter and where new routing decisions can be made.
I would compare three things on the same time axis: active connections per backend, active HTTP/2 streams per backend, and completed requests per backend. A host with few connections but many streams might be busier than a host with many mostly idle connections. Counting connections alone could therefore send us toward the wrong fix.
Suppose the two old backends each have two upstream TCP connections carrying thousands of streams per minute. The six new backends each have no established upstream connection. Adding more CPU cannot change the owner of those four existing flows. If new connections do appear but still choose old servers, inspect weights, consistent-hash policies, health eligibility, and endpoint discovery. Long-lived connections are the likely explanation in this scenario, not the only possible explanation of idle hosts.
For a service where every HTTP request can safely be routed independently, I would prefer a proxy that performs appropriate request-level load balancing. It can keep client channels stable while using independent upstream pools and selecting a suitable backend for new requests. Envoy's documented upstream HTTP/2 pooling is one example. Its behavior depends on stream limits, worker pools, protocol settings, and circuit-breaking limits. [1]
If the topology must remain layer 4, rebalance by controlled connection lifecycle management. Give upstream connections a bounded age or request count, gradually drain old ones, and let replacements connect using the expanded backend set. The rollout should avoid synchronizing every client's connection expiry at the same moment.
For HTTP/2, a graceful GOAWAY tells a peer that no more new streams should be initiated on the draining connection, while allowing eligible in-flight streams to finish. The RFC describes a staged graceful shutdown because a single abrupt close can strand work whose outcome is uncertain. [2]
A proxy closing a TCP connection does not automatically make an application operation safe to retry. A streaming upload, a long-running RPC, and an ordinary read may have different replay and cancellation contracts. Gracefully rotate when you can, and bound the drain interval. For connections whose streams never end, you need a documented maximum drain policy and client recovery behavior.
Rotating all channels at once can create a wave of DNS lookups, TCP and TLS handshakes, authentication, and connection-pool warmup. The new backends may have cold caches and JIT warmup. A large synchronized drain can briefly make latency worse than the original imbalance.
I would introduce a low rate of new connections first, observe actual request redistribution, and increase gradually while respecting in-flight work. Alert on active streams on draining hosts, new connection error rate, and the proportion of traffic still landing on the old pair.
One subtle choice belongs to the workload. If sessions require affinity, per-request balancing might move related requests somewhere that lacks their state. Either move the state out of a specific host, or preserve the affinity explicitly and understand its skew cost. Changing proxy type is not a free correctness improvement.
The engineering distinction is adding destinations is useful only when the system makes new placement decisions at a granularity that can use them. A long-lived connection is not a long-lived request, and neither is the same as a business session.
After this change, distribution looks healthy. But a new dashboard shows a different puzzle: application CPU is only twenty percent, yet thousands of requests are waiting to acquire a connection to the database.
How can a mostly idle application already be out of capacity?
HTTP/2 permits multiple concurrent streams on one connection, but every implementation has limits. Some proxies create additional upstream HTTP/2 connections once a connection reaches its allowed concurrent-stream capacity. Envoy documents that behavior, together with limits on requests per connection and per-worker connection pools. A configuration does not imply exactly one connection for an entire cluster.
Layer 4 flow balancing cannot ordinarily move the bytes of an established TCP connection to a different server without preserving its transport state. A layer 7 proxy may route new upstream requests separately, but it still needs healthy destinations, resource limits, and adequate connection pools. This design changes where traffic balancing occurs, not the statefulness of the business application.
An HTTP/2 GOAWAY contains a last stream identifier. The RFC uses it to communicate which streams could have been processed. A graceful drain avoids unnecessary disruption, but if a connection dies while a non-idempotent stream has an unknown outcome, the application still needs its own recovery contract.
For streams that legitimately last hours, first decide if reconnecting is supported. A bounded drain window might require checkpoint or resumption tokens. Abruptly moving streams simply to improve load distribution can trade a capacity problem for lost user work.
Common mistake: Assuming HTTP/2 automatically gives per-request balancing through a layer 4 load balancer.
No, new streams still travel on the old pinned TCP flows. — A layer 4 routing decision applies to the transport flow. HTTP/2 multiplexing does not automatically create new transport placement decisions.
During a WebSocket chat deployment, old pods keep almost every active session. How would you replace them without dropping messages?
Ask for session duration, affinity and resumability. Gracefully drain and reconnect with recovery semantics rather than assuming each message can be routed independently.
The proposed connection rotation doubles TLS handshakes for five minutes. What measurements and rollout controls would you use?
Long-lived HTTP/2 connections can carry many requests. With layer 4 balancing, the backend is chosen when the TCP connection is established. Adding servers affects new connection placement, not existing flows, so old servers can remain busy while new ones are idle. I would confirm where HTTP/2 terminates and compare requests and active streams per backend, not just connection count. For request-independent traffic, an appropriate layer 7 proxy can route new upstream requests across backends. Otherwise I would rotate connections gradually and drain them using HTTP/2 GOAWAY when applicable, letting active work finish where feasible. I would account for reconnect spikes, warmup, long-lived streams and session affinity. A different proxy does not automatically make business operations safe to retry.