Skip to content

The link is 10 Gbps. Why is one file moving at 200 Mbps?

The Network Was Up. The Requests Were Not. · Part 1 of 6 · 6 min read

Read the worked explanation ↓

Interview question. A team is moving multi-gigabyte objects from Mumbai to Virginia. Network monitoring shows a 10 Gbps circuit with plenty of spare capacity, but a single TCP transfer averages only 200 Mbps. Round-trip time is 180 milliseconds. The storage disks are fast enough and neither endpoint is CPU-bound. What would you check before paying for more bandwidth?

I would begin by refusing to equate link capacity with the rate available to one connection. Ten gigabits per second describes the pipe. TCP must also have enough data in flight to fill that pipe while acknowledgments travel back. The useful starting number is the bandwidth-delay product, or BDP.

At 10 Gbps and 180 ms, the path can hold 10,000,000,000 × 0.180 / 8 = 225,000,000 bytes, approximately 225 MB of unacknowledged data at line rate. That's a large flight of packets. If the sender can keep only 4.5 MB outstanding, its idealized steady-state upper bound is 4.5 MB / 0.18 s = 25 MB/s, or 200 Mbps. That matches the observed transfer without needing a congested link.

The numbers do not tell us which limit is binding. They tell us which class of limit to investigate.

A 10 Gbps path with 180 ms round-trip time needs roughly 225 MB in flight for line-rate use, while a 4.5 MB effective flight cap limits throughput near 200 Mbps.
A 10 Gbps path with 180 ms round-trip time needs roughly 225 MB in flight for line-rate use, while a 4.5 MB effective flight cap limits throughput near 200 Mbps.
Think first

A backup service sends 2 TB nightly across a 140 ms link. A single stream sustains 480 Mbps while the WAN has 8 Gbps spare. Design a diagnosis and a safe throughput improvement plan. Include the approximate BDP, TCP state you need, and whether you would introduce parallel byte-range copies.

Take a few minutes to form an approach, then compare it with the worked explanation.

Check the exercise numbers

Using 8 Gbps as the target copy budget at 140 ms RTT, BDP = 8,000,000,000 × 0.140 / 8 = 140 MB. If 480 Mbps is limited only by bytes in flight, it corresponds to about 8.4 MB outstanding. A decimal 2 TB copy at a sustained 480 Mbps takes about 9.26 hours before additional overhead. These estimates do not prove which TCP or endpoint limit is binding.

Reveal the worked explanation

Three limits, not one

A TCP sender is constrained by what the receiver says it can accept, what congestion control believes the network can sustain, and what the application actually supplies. An approximate model for a sustained flow is throughput ≤ min(rwnd, cwnd) / RTT, assuming the application keeps the socket fed and the path and endpoints impose no lower ceiling. Here rwnd is the advertised receive window and cwnd is the congestion window. They are different.

The receiving machine might have small receive buffers, slow application reads, or autotuning limits. In that case the sender sees a receiver-window ceiling even when the network is clean. TCP window scaling allows receive windows beyond the old 16-bit header limit, but support is negotiated during the SYN handshake. Enabling a larger socket buffer afterward does not repair a connection whose window scaling was never negotiated.

Or the receiver advertises 64 MB while the sender's congestion window is 4.5 MB because the connection suffered loss, is still ramping up, or follows a congestion controller that cannot maintain a large flight. Turning up receive buffers would not solve that case. A third possibility is an application that reads from object storage in tiny chunks or deliberately serializes network writes. The TCP stack cannot transmit bytes it has not been given.

I would check ss -ti or equivalent TCP diagnostics on both endpoints, packet traces when permitted, send and receive buffer measurements, retransmissions, RTT estimates, flow-control stalls and CPU or disk activity. Tools such as iperf3 can isolate path behavior from the actual application, but I would keep the test topology and security conditions representative.

The arithmetic changes the design

Suppose the team can raise the effective in-flight limit from 4.5 MB to 45 MB. The same idealized calculation gives around 2 Gbps, not 10 Gbps. To approach the full circuit rate at the stated RTT, we need roughly 225 MB in flight, subject to congestion control, receiver resources and all other endpoints sharing that path. More bandwidth alone helps little when this connection remains window-limited.

Now the interviewer offers 16 parallel transfers. Sixteen connections, each sustaining 200 Mbps, might deliver 3.2 Gbps in a favorable test. The calculation is straightforward. The guarantee is not. Parallel flows compete for buffers and link capacity. They may consume many more file descriptors, object-store requests and memory buffers, and they can be unfair to unrelated traffic. If the limitation is a shared route or receiver aggregate ceiling, adding connections just moves the bottleneck.

Parallel byte ranges can be reasonable for a bulk object copy, especially if the transfer tool can retry failed ranges and verify final checksums. They are a less comfortable default for a transactional API, where splitting work can alter ordering and error handling. The point is to choose concurrency because of a measured per-flow bottleneck, not because sixteen sounds faster than one.

What I would ship

First run a controlled path test with one sustained TCP stream, then several, recording sender and receiver TCP state. Compare the observed flight size to the 225 MB BDP. Check whether the limiting condition is receiver-advertised space, congestion behavior, application pacing, traffic shaping, or storage. Confirm that the measured RTT is representative in both directions and that the VPN or proxy does not introduce another limit.

If receive buffering is the issue, tune bounded buffer and autotuning settings with a rollback plan. If congestion control is limiting the flow, investigate retransmissions, ECN signals, pacing and the deployed algorithm instead of blindly enlarging queues. If the application starves the socket, change its I/O pipeline. If multiple connections are justified, cap concurrency and verify it does not degrade neighboring customers.

I would evaluate success using end-to-end transfer duration, retransmission behavior, memory use, fairness and delivered bytes, not merely the circuit's advertised capacity. A Principal-level answer should also consider whether transferring every object across an ocean is necessary. Regional placement or moving compute to data could eliminate a large recurring network bill.

The part worth remembering for the next network incident is that spare bandwidth tells you almost nothing about a single TCP flow until you know how much data it can keep outstanding. In the next question, the bottleneck changes: the flow has enough window space, but a small amount of packet loss keeps interrupting its progress.

Go deeper

The receive window is flow control. The congestion window is a sender-side estimate shaped by congestion control, recovery and pacing. RFC 7323 describes window scaling negotiated in SYN packets. The BDP calculation is a capacity planning estimate rather than a promise of observed throughput: MSS, ACK behavior, losses, slow start, delayed acknowledgment, buffer accounting and implementation constraints matter. BBR, CUBIC and Reno-style behavior differ, so diagnosing a modern host should not assume one universal multiplicative decrease rule. A 10 Gbps optical link can also be limited by host NIC offload settings, CPU scheduling, encryption and application storage, none of which the BDP equation proves away.

Check your reasoning

  • Compute the 10 Gbps by 180 ms BDP in bytes and compare actual in-flight cap.
  • Distinguish `rwnd` from `cwnd` and application data availability.
  • Identify TCP window scaling negotiation and endpoint measurements.
  • Evaluate bounded parallel flows and fairness rather than assuming a speedup.
  • Choose a root-cause-specific intervention and realistic acceptance metrics.

Common mistake: A link with spare physical bandwidth cannot be limiting a single TCP connection.

What should you do first when a single TCP flow is slow despite spare link capacity?

Read the answer explanation

Measure effective in-flight limits and sender pacing, then test bounded fixes — Link rate alone does not identify the single-flow bound. Determine the effective flight window and other limits before changing capacity.

Apply it elsewhere

A cross-region database snapshot is copied over encrypted tunnels. A bulk tool with eight concurrent streams speeds up, but query p99 on the same network worsens. Specify fair admission, pacing and transfer scheduling controls rather than maximizing snapshot bandwidth.

Compare your approach

Compute and interpret BDP with units, distinguish receiver and congestion windows, separate application starvation from network limits, diagnose before tuning, and discuss parallel-stream tradeoffs and fair use.

One more constraint

If RTT doubles while the effective in-flight limit stays fixed, can throughput remain unchanged merely because physical link capacity is unchanged?

Quick revision

Link bandwidth is not single-flow throughput. A sender needs roughly bandwidth times RTT worth of outstanding bytes to fill a path. On a 10 Gbps, 180 ms link that is about 225 MB. An effective 4.5 MB in-flight limit yields a best-case rate near 200 Mbps. Determine whether rwnd, cwnd, the application, or another endpoint is limiting before adding capacity. Parallel connections can increase aggregate flight size but introduce fairness and resource costs. Measure the transfer, not the circuit label.

References