Retrieval and RAG · Principal
The search returned results, but the policy source timed out
The question
Interview question
A support assistant searches a product manual, a policy store, and incident notes in parallel. The policy store times out. The other two return plausible passages within the two-second deadline. The assistant answers, “There is no exception,” because none of those passages mention one. Design the response behavior and the search contract. The product team says returning some answer is better than failing.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The key missing bit is coverage. “No exception found in two responding sources” cannot become “no exception exists in the policy.” If the policy store is the source that could establish the exception, the result set is incomplete for this question. A federated retriever should return not only hits but a status for each required source: searched successfully at which revision, timed out, unavailable, skipped by policy, or returned zero after a complete search. Those statuses have different meanings. Elasticsearch's search response includes shard failure counts for a similar reason inside one engine. At the application layer, we need the same honesty across source systems.
The query planner should say which sources could answer which part of the question. For a policy exception, mark the policy store required. For a question about an incident log line, it may be optional. Give each source a deadline within the overall request deadline and record its response status, source revision, and authorization outcome. Partial hits can still be shown if they are useful, but the answerer gets the missing-source status as structured state, not an empty list that looks like “nothing matched.”
Under this scenario I would answer the general rule only with a clear limitation and avoid deciding eligibility. Something like: “I found the general rule in the manual, but the policy source did not respond, so I cannot check exceptions right now.” If the product requires a yes/no decision, return an unresolved state or route to a human. A confident negative claim is the most dangerous result of a partial search. The exact threshold depends on the question and source authority, not a blanket policy that any timeout makes every answer impossible.
The product team pushes on latency. I would use a bounded retry or an authoritative direct lookup if it can finish inside the remaining deadline. A recently verified snapshot may be usable for low-risk explanation if its age, permission scope, and policy effective time still meet the product contract. If the rule changes frequently or the consequence is material, an old cache is a poor substitute. Do not extend all requests by two seconds to wait for an optional wiki, and do not use an optional wiki to excuse missing required policy. Return the coverage decision in telemetry so we can measure how often users see incomplete answers.
Now switch the failure: incident notes time out, while the policy store and product manual both respond at coherent versions. The assistant can usually answer the policy question and omit incident context. That different outcome is why source dependency belongs in the query plan rather than a global “partial results allowed” switch.
There is still a security edge. A source marked unavailable because the principal cannot be authorized is not a reason to probe it through a broad service account and leak a title. The answer may say it lacks sufficient authorized evidence without confirming that a restricted exception exists. Test source outage, zero hits, authorization failure, and a stale policy snapshot separately. They may all look like an empty array to careless code, but they justify different user-facing claims.
Continue reading
Related questions
Read beyond the question
Explore more retrieval and rag
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →