We admit only 500 requests per second. Why does latency still collapse?

Interview question. A payments API is limited to 500 arrivals per second, yet its p99 latency rises from 80 ms to four seconds during database slowdowns. CPU utilization is modest and autoscaling has not helped. Design an adaptive concurrency limiter that protects useful throughput, and explain how to avoid reacting badly to noisy latency signals.

Picture two mornings with exactly 500 requests per second. On Monday the downstream read takes 40 milliseconds and checkout works. On Tuesday one database index disappears and the same read takes 800 milliseconds. The input rate is identical, but the number of requests occupying resources can be very different. A requests-per-second limit constrains arrivals. It does not directly constrain how many operations are alive inside a service, holding connections, memory and work slots while they wait.

Little's Law gives the relationship for a stable system: average concurrency L = λ × W, where λ is throughput and W is mean time in the system. At 500 completed requests per second and 40 milliseconds of average service time, the mean number in flight is about 20. At 800 milliseconds, it is about 400 if 500 requests per second continue completing. That last condition is crucial. If the service cannot complete 500 per second, a stable-state calculation no longer applies and the backlog grows. Little's Law is not a formula that guarantees the system can safely operate at the calculated concurrency.

Start with one practical defense. Put a semaphore or equivalent in-flight admission limit directly before the scarce work, perhaps before a database connection pool. A request must acquire capacity before starting that work. If capacity is unavailable, a very short, bounded wait may be acceptable for an interactive API. Otherwise reject promptly while the request can still be retried or shown as unavailable within the user's remaining deadline. A long queue is not a solution. It turns an overload into older work that is less likely to matter by the time it executes.

A fixed limit is hard to choose. A healthy node might manage hundreds of lightweight cached lookups, then become overwhelmed by forty expensive transactional writes. Hardware generations, traffic mixtures, replica lag and database load change the safe value. Adaptive concurrency limiting tries to discover a moving limit from measured response time and overload signals rather than setting one universal constant forever.

One family of algorithms compares short-term observed latency with a baseline near-uncongested latency. The intuition is that growing latency can represent queueing in the system. Netflix's concurrency-limits library describes a Vegas-style estimate of queue occupancy, approximately limit × (1 - minRTT / sampleRTT). Its controllers increase concurrency gently while apparent queueing is low and decrease it when queueing grows too high. That is a heuristic, not a direct count of real queued requests. The baseline can drift, and measurements include service work as well as waiting.

Let's use a deliberately simple numerical example. With an in-flight limit of 100, a baseline of 40 milliseconds and observed latency of 50 milliseconds, the estimated queue signal is 100 × (1 - 40 / 50) = 20. If observed latency jumps to 100 milliseconds, that signal becomes 60. Both are too high for a controller targeting only a few queued operations. The precise adjustment comes from the chosen algorithm and sampling policy, not from those numbers alone. The point is that the ratio of observed latency to healthy latency carries information an incoming RPS counter misses.

Why not immediately shrink the limit whenever latency rises? Because latency can rise for other reasons. A request mix may shift toward expensive reports, a network hop can become slower, a downstream might perform a legitimate maintenance operation, or instrumentation can add overhead. If we interpret all latency growth as internal queueing, the controller may aggressively reduce admission, starve throughput and never gather enough representative samples to recover. Separate request classes or cost weights where necessary, and use multiple signals such as rejection rates, timeouts, actual utilization and time spent waiting for the protected resource.

There is also a feedback problem. Rejections reduce the observed workload. Latency falls, so a naive controller increases the limit again. It can oscillate between overload and underutilization. Smooth measurements over appropriate windows, increase cautiously, decrease faster on confirmed overload, cap both minimum and maximum limits, and use a deliberate probe policy during recovery. Netflix's Gradient2 algorithm uses divergence between short and long latency averages to detect sustained queueing trends. That is one implementation option, not the only correct answer.

I would place the limiter where it protects the actual bottleneck. An edge limiter can protect an API fleet, but it may not know that one reporting endpoint consumes twenty times the database work of a profile lookup. A per-dependency, per-operation-class bulkhead can prevent one expensive route from exhausting the entire connection pool. Multiple instances must also consider shared downstream capacity. If every pod independently discovers that it can safely run 100 operations, adding ten pods can overwhelm a database that only supports 300 in flight across the whole fleet. A coordinated budget or downstream-side admission may be necessary.

Finally, compare accepted requests, successful completions, rejected admissions, p50 and p99, measured in-flight occupancy, queue wait and downstream utilization. Run a controlled test that slows a dependency while preserving the same arrival rate. Does the controller maintain an acceptable number of successful payments, or simply produce a beautiful latency chart by rejecting everybody? Useful work and customer outcomes determine whether the limiter is successful. Rate limits still belong at the edge for abuse and entitlement. Concurrency admission belongs near work that must not overload.