Skip to content
ArchCrux

Networking & Protocols · Staff

The server is idle. Why are new connections timing out?

The Network Was Up. The Requests Were Not. · Part 3 of 6 · 6 min read

Read the worked explanation ↓

Interview question. During a sale, a checkout service starts timing out for new users. The dashboard shows 18 percent CPU utilization, normal database latency and plenty of memory. Requests on connections established five minutes earlier mostly work. New TCP connections sometimes take seconds or never complete. The team wants to double the application replicas. Explain what you would inspect first.

I would ask one question before looking at the application thread pool: where do new connections stop? A request can fail before an HTTP handler ever runs. A listening TCP socket has kernel-maintained queues, and those queues can fill even while most processors are idle.

On Linux there are two useful stages to distinguish. During the handshake, the kernel tracks requests that have not completed the SYN, SYN-ACK, ACK exchange. After the handshake, established sockets can wait for the application to call accept(). The second is commonly called the accept queue. The backlog argument to listen() on modern Linux limits that completed-connection queue, subject to a somaxconn cap. The incomplete-request queue has a separate tcp_max_syn_backlog setting, with SYN cookies changing how overflow is handled.

The queues are not the same and neither is the HTTP request queue.

SYN requests and completed connections occupy different Linux listen queues before an application accept loop takes ownership.
SYN requests and completed connections occupy different Linux listen queues before an application accept loop takes ownership.
Think first

A Java gateway receives a six-second new-connection burst at 4,000 attempts per second, while a single acceptor drains only 2,500 per second during a GC-heavy window. Existing HTTP/2 connections work. Explain the expected backlog pressure, metrics to collect and a fix that doesn't just make clients wait longer.

Take a few minutes to form an approach, then compare it with the worked explanation.

Check the exercise numbers

If all 4,000 connection attempts per second complete their handshakes and reach an initially empty accept queue while it drains at 2,500 per second, the offered excess over six seconds is (4,000 − 2,500) × 6 = 9,000 connections. A smaller effective queue can overflow earlier; this is not a claim that 9,000 sockets will actually be queued. Separate attempted, established and accepted connections when testing the assumption.

Reveal the worked explanation

How low CPU can accompany a full queue

Imagine the load balancer opens 10,000 fresh connections in a short burst. Suppose this server's application loop has paused for a runtime stop-the-world event or is accepting sockets serially while processing expensive TLS or initialization work in the same thread. The kernel may complete TCP handshakes faster than the application drains accepted connections. Its accept queue fills. Existing sockets that have already been accepted can continue serving through other workers, so the service looks partially healthy.

The effective arrival rate temporarily exceeds the accept() drain rate. You do not need sustained high CPU to make that happen. One blocked acceptor, bad event-loop scheduling, a lock, or a process briefly descheduled at the wrong moment can cause a burst of connection-establishment failures. The kernel's response when the completed queue is full depends on OS version and settings. Clients might observe delayed connections, retransmitted handshake packets, connection resets or timeouts. Don't teach one universal symptom.

A different pattern occurs when the queue of incomplete handshakes is exhausted. Perhaps a SYN flood or a sudden arrival spike consumes half-open request state. SYN cookies may help under appropriate conditions, but they are not a generic speed upgrade. They change how the system survives an overflowing SYN queue and can have compatibility or diagnostic implications. If the accept queue is the actual bottleneck, enabling SYN cookies doesn't make the application call accept() faster.

A useful diagnosis sequence

I would capture client connection timing separately from request processing and report SYN-to-SYN-ACK and handshake-to-application-accept latency. On the server, inspect listening sockets with ss -lnt, the effective backlog, socket state distributions and OS counters such as TCP listen overflows and listen drops. Interpret queue columns using the installed ss and kernel version, rather than assuming every platform reports them identically.

Then sample the accept loop. Is one listener responsible for thousands of connections? Is TLS termination inside the process or in the load balancer? Are there reactor stalls, GC pauses, a file-descriptor ceiling, a low per-process descriptor limit, or a mutex shared across accepted sockets? Do scheduled load-balancer health checks use persistent connections that avoid the failing path? If so, green health is misleading.

An especially revealing test is to compare a fresh TCP connection against a reused connection to the same target during the incident. If reuses work while fresh connects fail, kernel queues, connection establishment and admission become much more likely suspects than ordinary handler performance. A packet capture at the load balancer and server can separate dropped SYNs from handshakes that completed but waited for application acceptance.

Capacity math for an acceptor

Suppose burst traffic delivers 2,500 completed handshakes per second while the application can accept only 1,500 sockets per second for four seconds. Ignoring other effects, the queue adds (2,500 - 1,500) × 4 = 4,000 waiting connections. An effective accept backlog below that may overflow during the burst even though normal traffic is only 900 connections per second.

The numbers are illustrative, not a promise that precisely 4,000 entries appear in one Linux queue. Real handshakes are staggered and OS behavior can retransmit or shed traffic. The calculation gives a capacity question to measure: how much of a transient arrival spike can the queue absorb, and for how long can the acceptor lag?

If the loop really is single-threaded and slow, I would make it cheap: accept promptly, hand sockets to bounded workers, and keep expensive initialization off the accept path. Depending on the server architecture, multiple acceptors and SO_REUSEPORT can distribute new connections, but I would verify their behavior and failure modes before making that change. A reasonable backlog increase absorbs a short burst while code is repaired. It cannot fix an acceptor that is consistently slower than arrivals. Eventually every finite queue fills.

Connection reuse and load-balancer connection pooling can reduce arrival bursts, but new clients still need a functioning path. Adding application replicas helps only if new connections actually distribute to the new acceptors and the shared load balancer itself is not saturated. I would cap connection admission where appropriate rather than converting the problem into millions of queued sockets and long client hangs.

The interview conclusion is to identify which boundary received the connection and which component failed to hand it forward. Application CPU utilization is not enough to answer that. The next episode follows outbound connections and shows how a different finite network resource can fail even when every service instance is apparently healthy.

Go deeper

On modern Linux, listen(backlog) governs established sockets waiting to be accepted, capped by net.core.somaxconn, while net.ipv4.tcp_max_syn_backlog relates to incomplete requests. SYN cookies affect the effective half-open queue limit. Some operating systems and older Linux versions have different interpretations, so do not generalize these parameter names across stacks. Listen overflow counters, drops, retransmissions and accept queue occupancy need to be considered together. TLS handshakes may happen after a TCP socket is accepted, depending on the stack, and can indirectly starve acceptor resources. Keep the distinction between kernel connection acceptance and application readiness explicit.

Check your reasoning

  • Separate incomplete SYN queue from established accept queue.
  • Explain backlog and somaxconn roles and SYN-cookie limits.
  • Compute queue accumulation under a short arrival burst.
  • Inspect listen queue occupancy, overflows, and accept-loop delays.
  • Repair admission/accept throughput instead of relying solely on larger backlogs.

Common mistake: Low CPU means the Linux kernel cannot be dropping or delaying new TCP connections.

What is the appropriate response when new connections fail but existing connections work?

Read the answer explanation

Separate SYN and accept queue evidence, then fix the slow accept path — New connections may fail before reaching a handler because the application is slow to accept sockets.

Apply it elsewhere

A database proxy shows normal statement latency but connection acquisitions stall after a deployment. Distinguish the proxy's TCP listen queues, TLS admission, application connection pool and the database's own max-connection limit. Design tests for each boundary.

Compare your approach

Separate SYN and accept queues, account for OS-dependent overflow behavior, diagnose fresh versus reused connections, calculate a burst backlog and fix acceptor throughput rather than only increasing kernel limits.

One more constraint

If SYN cookies eliminate half-open handshake queue overflow, does that prove the established-connection accept queue cannot fill?

Quick revision

Low CPU does not rule out connection failure before an HTTP handler runs. On Linux, incomplete TCP handshakes and completed connections waiting for accept() are distinct queueing stages with different limits. A slow accept loop can fill the second queue during bursts while established connections keep working. Inspect ss, effective backlog and listen-overflow counters, compare fresh versus reused connections and identify where the handshake stops. Tune short-burst buffering carefully, but repair an acceptor that cannot keep pace.

References