What happens after one gap
If enough later packets arrive, the receiver can signal a hole through duplicate acknowledgments and selective ACK information. In classic fast retransmit behavior, the sender can resend a segment before its retransmission timer fires. Modern stacks have more sophisticated loss detection, but the useful idea is the same: the sender needs evidence that something did not arrive.
A small request or short response may not provide that evidence quickly. Suppose the final segment of a response disappears. There may be no later data segments to generate duplicate acknowledgments. The connection might sit waiting for a timer-based recovery. RFC 6298 describes an initial retransmission timeout of one second before RTT measurements under its standard algorithm, while real implementations and established connections can have different adaptive timers. I would not claim every lost packet adds exactly one second.
Congestion control matters too. TCP generally treats inferred loss as a possible congestion signal. Depending on the implementation and controller, the sender can reduce its permitted in-flight data or adjust its pacing, which hurts throughput beyond the retransmitted packet itself. A long-lived cross-region connection can recover and then spend time rebuilding its sending rate. That's why the effect of loss depends on RTT, congestion-control algorithm, burstiness and connection lifetime.
Do not confuse that with the head-of-line blocking discussion from HTTP/2 versus HTTP/3. Here the first-order issue is TCP's ordered-stream recovery within a connection. If multiple application streams share that TCP connection, missing transport bytes can affect unrelated streams too. QUIC can avoid cross-stream head-of-line blocking, but its lost packets still need recovery and the network path can still be congested.
Why the backend trace looks normal
A server span often begins when the application receives a complete request. Missing packets during TLS setup, request delivery or a stalled response might not be included in that span. The server can finish its computation in 30 ms while the client waits through retransmissions or a connection timeout. Looking only at downstream service duration would falsely clear the network.
I would compare client-observed durations with server spans and correlate failures by source region, egress gateway, address family, connection reuse and route. On controlled hosts, I would inspect TCP retransmission counters and per-socket RTT/RTO state, selective-ACK behavior, connection resets and packet traces. In cloud networks, traces on both sides of a path can help locate where sequence numbers disappear, but capture itself must respect traffic and privacy policies.
The symptom may also come from packet reordering, policers, congestion, MTU problems or a flaky middlebox. Retransmission counters alone do not prove physical packet loss. Capture on both ends and inspect what was sent, what arrived, and what the sender inferred. An average interface drop percentage cannot answer which client requests suffered.
Repair the path, then reconsider the timeout budget
For this scenario I would first isolate the affected route, compare it with a healthy path under similar load, and work with the network owner to investigate queue drops, interface errors, routing asymmetry, traffic shaping and link congestion. If one regional path is unhealthy, route around it deliberately with capacity and retry limits. Do not let an emergency failover stampede the supposedly healthy path.
Client changes can help, but they need judgment. Persistent connections avoid repeated handshakes, and request concurrency can help hide delay for independent operations. Neither repairs data loss. Making every request timeout at 100 ms might reduce waiting but increase retries and payment ambiguity. Retrying a payment on a new connection may duplicate a side effect unless the application has an idempotency contract. Here the design decision is to distinguish transport recovery time from the business action's completion semantics.
I would monitor connection-establishment latency, retransmissions per path, TCP RTT distribution, client completion time and the ratio between client and server latency. The acceptance test would include small versus large payloads and warmed versus new connections, because their loss-recovery patterns differ.
One percent packet loss is not harmless or universally catastrophic. It can be almost invisible for some requests and dominate others. What matters is which packets are missing, how quickly the sender recognizes the gap, how congestion state reacts, and how long the application can wait.
