Skip to content

The disk is fast and the network is fast. Why does copying data use all the CPU?

The Machine Had More Resources. Why Was It Slower? · Part 6 of 6 · 6 min read

Read the worked explanation ↓

Interview question. A broker serves large immutable log segments from local storage to many consumers. Disk throughput and network bandwidth both have headroom, but CPU is spent copying buffers, and adding consumer connections makes throughput fall. Explain the ordinary file-to-socket path, what sendfile changes, and when zero-copy cannot be used.

Think about the bytes a broker is supposed to deliver. A consumer asks for a range of records already stored in a log file. The broker does not need to inspect, decompress or alter every byte to satisfy that request. It mainly needs to determine which file offset to send, enforce access controls, and move the selected bytes onto the network.

An ordinary implementation might read data into a user-space buffer and then write that buffer to a socket. That can cause avoidable copies and syscall overhead. At high throughput, moving the same data through extra buffers consumes CPU cycles and memory bandwidth that could have served more clients.

The conventional read and write path includes a user-space data copy, while sendfile can transfer cached file data to a socket without that user-space bounce buffer.
The conventional read and write path includes a user-space data copy, while sendfile can transfer cached file data to a socket without that user-space bounce buffer.
Think first

A file-serving gateway saturates four CPU cores at 4 Gbps but can send 8 Gbps of cached static content in a benchmark that bypasses application transformations. Design a fair benchmark for buffered I/O versus sendfile, identify correctness concerns, and explain how per-customer encryption affects the answer.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Start with the ordinary buffered path

On a typical Linux buffered read, the kernel finds file data in its page cache or brings it there from storage. A read() call then copies requested bytes into a buffer in the application's address space. The application calls write() or send() on a socket, and the kernel obtains those bytes again for the networking path.

A simplified diagram often shows disk-to-kernel, kernel-to-user, user-to-kernel, and kernel-to-network. That teaches the main idea, but modern devices use DMA and the details of packet assembly, scatter-gather lists, and copies vary. Don't count every arrow in a diagram as one obligatory CPU copy. The important avoidable part is bouncing an unchanged payload through a user-space buffer solely to hand it straight back to the kernel.

A broker can instead ask Linux to transfer data from a suitable file descriptor to a socket using sendfile(). That lets the kernel move the data without making the application own a duplicate payload buffer. When supported by the kernel, socket, filesystem and network path, the transfer can reuse page-cache-backed data efficiently. Zero-copy does not mean zero CPU, zero I/O, or that no copy is ever made anywhere in the stack.

The workload decides whether the optimization is useful

This helps most when the content is already in the correct wire representation. An append-only log segment containing records that consumers want unchanged is a good candidate. The producer writes a durable representation, and many consumers read the same range. Page cache may serve those repeated reads without disk activity, and the network stack can send the bytes without an extra application-level copy.

But suppose the broker must compress each batch differently for each consumer, transform a schema field, redact sensitive content, or encrypt at the application layer. Now the application needs to inspect or modify bytes. A straightforward sendfile() from the original file cannot provide that transformation. The system might preprocess immutable variants or accept a copy on the transformation path. Which choice wins depends on the frequency and cost of the transformation.

Transport encryption is another important boundary. Plain sendfile() does not magically turn file bytes into encrypted TLS records in user space. Some platforms support kernel TLS and other optimized paths that can preserve parts of the benefit, but they depend on configured support. For an arbitrary Java TLS socket, you must verify the actual stack before claiming a zero-copy path.

Measure where the cycles went

Before adopting zero-copy, I would profile CPU time, memory bandwidth, syscalls, socket queueing, page-cache hit rate and disk read activity. If the process spends most of its CPU in payload serialization rather than copying, sendfile() may not address the bottleneck. If the network interface is already saturated, eliminating copies may lower cost per byte without increasing total bytes per second.

A basic controlled test would hold log size, message size distribution, file-cache warmth, TLS mode and consumer concurrency constant. Compare a buffered read/write implementation with the optimized file-to-socket path. Measure throughput per core, user and system CPU, context-switch and syscall counts, tail latency, and the effect on other workloads sharing memory bandwidth.

There is another trap: sendfile() does not promise that one call transfers the entire requested byte range. Calls can transfer fewer bytes than requested, be interrupted, or report an error. Correct code must account for partial progress and retry from the right offset without duplicating or skipping bytes. Certain input/output descriptor combinations and filesystems are unsupported, so the application needs a safe fallback. When the socket path retains references to file-backed pages, those bytes must remain unchanged until the receiver has consumed them; immutable segments make that contract much easier to meet. That correctness handling matters more than a headline throughput number.

Why log brokers care about representation

The file format is part of the architectural choice. If a broker persists record batches in a representation consumers can read directly, it can reuse those bytes efficiently. If every read requires reconstructing an object graph, reserializing it and recompressing it, the storage format imposes recurring CPU cost even when the data never changes. Zero-copy is therefore tied to the wire/storage boundary, not just a clever system call.

This also affects cache behavior. Keeping useful log segments in the operating system page cache can be an effective shared caching strategy. A broker may avoid maintaining a separate large user-space cache for unchanged file bytes. But page cache competes with other memory demands, and a cold read still requires storage I/O. The lesson is not that page cache guarantees memory residency.

For the interview, I would make the boundaries explicit. Show which copies the data path actually eliminates, prove that the payload is unchanged, handle partial sends correctly, and test whether CPU copying is the active bottleneck. If those conditions don't hold, zero-copy can add implementation complexity without solving the user's problem.

Go deeper

Linux sendfile(2) supports moving data between suitable file descriptors, commonly from a regular file to a socket. Semantics include short transfers, offsets, restrictions on descriptors and errors requiring fallback. The kernel page cache can avoid repeated storage reads for hot segments, but cache residency is not guaranteed. DMA, scatter-gather I/O, NIC offloads and kernel TLS produce multiple meanings of the phrase zero-copy. Evaluate actual copied bytes and CPU work rather than inferring them from a syscall name alone. Kafka's efficient transfer path depends on protocol and message representation as well as host facilities.

Check your reasoning

  • Explain the buffered page-cache to user-buffer to socket path.
  • Show how sendfile avoids a redundant user-space bounce copy.
  • Qualify filesystem, descriptor, TLS and transformation restrictions.
  • Handle partial transfers and correct file offsets explicitly.
  • Benchmark throughput per core while controlling cache warmth, message format and TLS.

Common mistake: Using sendfile guarantees that data is sent without any CPU work, storage reads or partial transfers.

Which use of sendfile preserves correctness while reducing an avoidable user-space payload copy?

Read the answer explanation

Use sendfile for supported unchanged file-to-socket transfers and retain short-send and fallback handling — Avoiding an application bounce buffer does not remove all I/O, CPU work or syscall correctness concerns.

Apply it elsewhere

A video-on-demand service must add a personalized watermark to every segment. Decide whether to transform content ahead of time, keep distinct immutable variants, or accept per-request user-space work. Explain how the choice changes cache behavior and network efficiency.

Compare your approach

Describe page cache, user-space bounce buffers and socket writes, distinguish DMA from CPU copies, explain immutable wire representation, qualify TLS optimizations, handle short writes, and measure throughput per core at equivalent load.

One more constraint

Does calling `sendfile()` guarantee that no bytes are ever copied, no storage I/O occurs, and the full requested range is sent in one call?

Quick revision

The ordinary buffered file-to-socket path may copy unchanged payload bytes into user space and back into the kernel. Linux sendfile() can avoid that bounce buffer for supported file/socket paths, often improving throughput per CPU core for immutable log or static-file data. It does not eliminate all CPU work, all hardware copies or storage reads, and partial transfers must be handled correctly. Transformations, encryption and unsupported descriptor combinations can require different paths. Verify the actual bottleneck with controlled profiling.

References