Distributed Reliability · Principal
One agent run opens a hundred tool calls at once
The question
Interview question
A research agent creates 100 parallel tool calls. User QPS is flat, but the search service times out and its connection pool fills. The application already has a per-user request limit. Locate the amplification and keep useful parallelism. Another team also calls the same service.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The rate limiter counts runs. The dependency feels calls and concurrent work. Ten simultaneous runs at 100 calls each can create a thousand requests before any single user crosses a modest run-rate threshold. A model may add retries or spawn children, so the transitive fan-out matters as much as the visible plan. Google's SRE discussion of cascading failures explicitly treats transitive fan-out, cancellation propagation, and early overload rejection as service design problems. Here the failure is a capacity contract between an agent runtime and a shared tool service.
Put a budget on each layer that can multiply work: maximum outstanding calls per run, per tenant, per tool and route, and at the dependency itself. A run can have 100 planned searches but launch a bounded window, consume results, and choose the next window only if the evidence gap remains. Charge a permit before dispatch. Release it on a verified terminal result, not simply because the model got bored.
Use a global or shared limiter at the dependency for aggregate safety, with fair allocation so one application's burst does not starve the other team. The dependency should reject early when overloaded and say whether a retry is sensible. The runtime must respect its end-to-end deadline and avoid turning a rejected batch into 100 immediate retries.
Which calls should be parallel? Independent sources with distinct likely evidence can improve latency. One hundred near-identical queries against the same shard mostly increase contention and duplicate context. A planner can propose calls, but a deterministic executor can deduplicate identical scoped requests and enforce a concurrency ceiling. Treat high-value required calls before optional exploration, and cancel remaining work when the answer has enough evidence or its deadline is gone. Cancellation has to reach the search service, otherwise the frontend looks idle while backend work continues.
If the other team owns the dependency, do not assume our gateway's semaphore protects its whole capacity. Agree on a per-client budget and a service-wide admission signal, observe actual queue time and in-flight work there, and rehearse bursts. Track user runs, tool calls per run, retries, rejected calls, duplicate queries, and useful evidence found per marginal call. If ten calls find the same policy paragraph, more fan-out is not a quality gain. The answer is not “serialize all tools.” It is to make parallelism proportional to independent useful work and to the capacity that exists.
Continue reading
Related questions
Read beyond the question
Explore more distributed reliability
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →