Interview question. Your API receives 2,000 requests per second, and each request makes one database call taking 250 ms on average. Twenty application instances each allow ten concurrent database calls. Application CPU is mostly idle. After ten seconds of sustained traffic, a queue has grown by about 12,000 requests. Explain the number, decide what to measure, and propose a safe way to increase useful throughput.
Assume these are steady-state averages for a simplified sizing exercise, that every request needs one database call, and that waiting requests are not yet timing out. The database's own sustainable throughput has not been measured.
Think first
Thirty API replicas receive 3,500 requests per second. Each request occupies one database connection for 100 ms. They share a global budget of 100 active calls. What throughput can they support under these assumptions, how quickly does the queue grow, and how would you change the system?
Take a few minutes to form an approach, then compare it with the worked explanation.
Check the exercise numbers
At 100 ms occupancy, 100 slots support about 100 / 0.1 = 1,000 calls/s. With 3,500 arrivals/s, the queue grows by about 2,500/s before cancellations. Sustaining 3,500/s would require an average 350 active slots if service time stayed unchanged; this does not prove the database can support them. At the follow-up’s 500 ms occupancy, the same 3,500/s would require 1,750 active slots, a warning to measure contention rather than keep raising limits.
Reveal the worked explanation
The server is waiting, not working
When you hear that CPU is low, it is tempting to increase the number of request-processing threads. Yet most of those threads could be blocked, waiting for a database connection, a lock, or a reply. CPU is one resource limit among many.
Here the connection pool gives us a measurable bound. Twenty instances, each with ten active connections, allow at most 200 database calls in flight at once. If each call occupies one connection for an average of 0.25 seconds, the pool can complete about 800 calls per second under the assumed service time and sustained occupancy.
Pool capacity = 20 instances × 10 slots = 200 calls
Mean holding time = 0.25 seconds
Approximate maximum = 200 / 0.25 = 800 calls per second
Arrivals = 2,000 calls per second
Backlog growth = 2,000 - 800 = 1,200 per second
After ten seconds = 12,000 queued requests
This is not a mysterious memory leak. It is accumulated work that the chosen concurrency limit cannot retire quickly enough. In practice timeouts, cancellation, and finite queue limits should prevent unbounded accumulation, so the number is an intentionally simplified illustration.
Flow of incoming requests through a bounded connection pool and growing waiting queue
Little's Law helps before you open a dashboard
For a stable system, Little's Law relates average number of operations in the system, arrival or completion rate, and average time spent there: L = λ × W. [1] If we want to sustain 2,000 completed calls per second at an average connection occupancy of 250 ms, the required average concurrency is 500 calls.
That calculation establishes a lower-order capacity requirement under the assumptions. It does not mean setting every pool to 500, and it does not guarantee the database can execute that much concurrent work. A pool slot is permission to impose work on a dependency, not proof that the dependency has spare capacity.
Our twenty instances might each need about twenty-five active slots to support 500 concurrent calls in aggregate, assuming the distribution is balanced. But before increasing ten to twenty-five, I would find out what the database does when concurrency rises. If calls queue behind row locks or saturate storage, their holding time might grow from 250 ms to 750 ms. Little's Law would then imply 1,500 in-flight calls to maintain the same arrival rate, while the real system may become slower rather than faster.
The quantity on the right side is not fixed independently of our decision. That is the trap in sizing pools entirely from one load-test number.
Instrument the waiting room
I want a separate metric for time spent waiting to acquire a connection, alongside time occupying it. A database query that executes in 5 ms may still produce a client wait of 300 ms when the caller waited 295 ms for a slot.
Measure active, idle, and pending connections per instance. Record acquisition latency by percentile, database query duration, lock waits, transactions in progress, and the actual throughput of successful business requests. Look for instances that hold connections while doing application work unrelated to the database. A connection kept open through a remote API call can waste scarce capacity even when the database itself is idle.
There is also an aggregate limit to check. A database with a practical concurrency budget of 400 calls cannot safely receive 1,000 active queries simply because ten application teams each increased their pools. The sum of pool maxima across replicas and services matters. Pool sizing is part of dependency admission control, not just application configuration.
What would I change first?
If the database can sustain the required throughput at moderate concurrency, increase pool capacity gradually, distribute it within a global database budget, and verify that both application waiting time and database execution time improve. Keep client timeouts and queues bounded while observing goodput, not just accepted requests.
If the database is already saturated, increasing the pool is counterproductive. Reduce how long calls occupy slots, remove unnecessary queries, improve indexes, batch where it is appropriate, or limit incoming work. A read replica helps only when the workload and consistency requirements support moving those reads. Offloading expensive queries might be better than giving every caller more connections.
I would also be wary of a large queue in front of this pool. Even if the database recovers, an old request may have already missed the user's deadline. Continuing to execute it consumes the capacity needed for fresh work. Each pending request needs a deadline and a clearly defined rejection or cancellation path.
The arithmetic is only the start. I would challenge its assumption: does increasing concurrent calls increase the number of successful results, or only the number of calls waiting inside the database? That is the measurement that decides the remedy.
After increasing the pool, the application is no longer the obvious constraint. The database itself approaches its sustainable service rate. Traffic climbs a little, and end-to-end latency changes dramatically.
Why can a modest increase in arrivals cause a very large increase in waiting time?
Go deeper
What Little's Law does and does not guarantee
L = λW is a long-run relationship for a stable system. Apply it to a clearly defined boundary: calls holding a database connection, all calls including waiting, or complete API requests. Those are three different values of L and W. It is easy to use request latency, including pool waiting, to size only the active connection pool. That mixes boundaries and overstates the number of useful slots.
For this example, 500 is the concurrency required when a completed call holds a connection for 250 ms and completions reach 2,000 per second. It is not the same as 500 threads, since asynchronous calls or virtual threads can allow different scheduling models. Nor does it prove that the dependency can sustain 500 simultaneous queries.
If each API request makes two sequential database calls, and each holds a connection 250 ms, sustaining 2,000 API requests per second requires 4,000 completed database calls per second and roughly 1,000 average active connection-slots worth of work, provided the workload and times remain unchanged.
We also need a pool policy for deadlines and cancellation. A waiting request whose deadline has expired should not sit in a FIFO queue indefinitely. The application must release checked-out connections reliably when requests fail. Otherwise the measured cap may be dominated by leaked slots rather than intentional capacity settings.
Check your reasoning
Compute approximate active concurrency needed for 3,500/s: 350 slots at 100 ms.
Derive pool-limited throughput at 100 slots: about 1,000/s under stated assumptions.
Compute backlog growth: approximately 2,500/s before cancellations.
Check DB execution time and safe concurrency before increasing global pools.
Bound pending work and show connection acquisition versus execution metrics.
Common mistake: Raising all per-instance pool limits without considering the aggregate database connection budget or impact on query latency.
Read the answer explanation
No, the database may slow when concurrency increases. — The equation describes average work in progress for a stable system, not downstream capacity under contention. Measure database goodput and latency.
Apply it elsewhere
An inference gateway has low CPU and a huge wait queue while GPU execution slots are full. What transfers from this lesson?
Compare your approach
Separate waiting from active work, size occupancy by throughput and duration, confirm accelerator service capacity, and bound the waiting queue.
One more constraint
Average database occupancy rises from 100 ms to 500 ms after you increase pool sizes. Recalculate necessary concurrency and explain why the change may be harmful.
Quick revision
The application can be idle while callers queue on a database connection pool. With 20 instances and 10 slots each, there are 200 active calls. At 250 ms occupancy, that yields at most about 800 calls per second under the assumed steady service time. If 2,000 arrive, the backlog grows by roughly 1,200 per second. Sustaining 2,000 calls per second would require about 500 concurrent slots if occupancy stayed at 250 ms, using Little's Law. But increasing pool sizes is safe only if the database can handle the added concurrency. I would measure connection acquisition wait, execution duration, lock and I/O pressure, and successful completions. Then I would increase concurrency carefully or reduce demand and keep the waiting queue bounded.
