Work out the connection population
Suppose the fleet establishes 1,200 outbound connections every second to that destination, and each translation remains occupied for an average of 60 seconds under the observed traffic and gateway behavior. A simple steady-state estimate gives 1,200 × 60 = 72,000 simultaneously occupied mappings. That exceeds the 55,000-per-destination capacity of one NAT IPv4 address. The assumed 60-second occupancy must be measured, not treated as an AWS default.
This is a version of Little's Law applied to active translated connections. The important quantities are new connections per second and how long translations consume scarce state. API requests per second is not the same as new TCP connections per second. One persistent connection can carry many sequential requests and, with HTTP/2 where supported, multiple concurrent streams. Conversely, 200 workers opening fresh connections for each small request can burn thousands of translations for very little payload.
This is why blindly increasing application replicas may amplify the incident. More replicas can create more separate pools and more simultaneous dial attempts. If they all exit through the same NAT public IPv4 address toward one upstream destination, the scarce pool is still shared. Horizontal scaling adds new consumers of a limit instead of increasing that limit.
Prove where ports disappear
For an AWS NAT gateway I would chart ErrorPortAllocation and PacketsDropCount, correlate them with new outbound dial errors, and inspect destination concentration using flow logs and application metrics. AWS also exposes other NAT metrics that help separate idle timeouts and connection attempts. I'd identify the precise destination IPs and ports, remembering that a provider hostname can resolve to multiple addresses or change over time.
I would separately rule out source-host ephemeral-port exhaustion, socket-descriptor limits and a local connection-pool cap. They produce similar client symptoms but require different fixes. When local ephemeral ports run out, the operating system may fail connect() before traffic reaches the NAT gateway. When the gateway runs out of translated ports, the host may initiate connections that cannot receive a working mapping. Client exception wording alone is not a reliable location detector.
Another clue is the difference between reused and new connections. If established connections carry requests successfully but fresh connection attempts fail at the same time gateway port-allocation errors rise, that is strong evidence for NAT exhaustion. Still check the upstream's own connection admission and firewalls, because overload can coexist at multiple stages.
Fix it in the right order
The first application change is usually to stop treating every API call as a new TCP handshake. Reuse connections through bounded, correctly configured clients. Cap pool expansion and the number of concurrent dials. Spread reconnects with jitter so a rolling restart or upstream idle timeout doesn't trigger synchronized storms. Account for safe eviction of stale connections, because keeping connections forever can also be wrong.
If service behavior allows it, HTTP/2 multiplexing reduces the number of TCP connections needed for simultaneous requests. But a provider may not support it, and concentrating everything on one connection introduces its own flow-control or failure coupling. Tune by observed concurrency, response latency and connection health rather than making protocol assumptions.
At the network layer, AWS documents adding secondary IPv4 addresses to expand NAT connection capacity. Another design might distribute traffic through multiple NAT gateways and appropriate routes, or use private endpoints for supported services so it does not traverse public NAT. These changes have cost, route, security and failure-domain implications. They should be justified with destination-specific measurements, not a generic instruction to add gateways.
I would be cautious about simply lowering timeouts. Shorter client timeouts can increase connection churn, which consumes even more NAT state, and can leave payment operations ambiguous. A bounded request budget and upstream rate limit reduce demand more predictably. If a provider publishes capacity constraints, align outbound admission with them rather than retrying aggressively.
The steady-state target is not zero connection errors at arbitrary offered load. It is a predictable maximum number of live translations, a controlled dial rate during failure or restart, and clear backpressure to callers when capacity is unavailable. Measure successful completions, fresh dials per second, occupancy proxies, NAT allocation failures and upstream p99 together.
The useful distinction is between bandwidth capacity and connection-identity capacity. Both live in the network, but changing one does not necessarily improve the other. The next incident will have neither high bandwidth use nor a port shortage. Small packets get through, and large ones disappear.
