Retrieval and RAG · Staff
RAG found five rows. Can it answer how many accounts breached a limit?
The question
Interview question
An enterprise assistant is asked, “How many enterprise accounts exceeded their contracted API limit at least once last quarter?” Search retrieves five very relevant incident notes and a policy page. The model answers “five” and cites the notes. The underlying events contain hundreds of millions of rows, account plans changed during the quarter, and each employee may see only some accounts. Design the path to a defensible answer. What changes if the user asks *why* one particular account exceeded its limit?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Five retrieved notes establish that at least five incidents were documented. They do not count all qualifying accounts. Retrieval samples likely evidence. An exact count asks a different question: define a population, evaluate a predicate over that population, and deduplicate at the right unit. A citation can support a definition or one example, but it cannot turn a top-k sample into a complete aggregate.
I would make the metric explicit before writing any query. “Last quarter” needs a calendar and time zone. Is “exceeded” measured against the contracted limit effective at the moment of each request, a daily quota, or a monthly allowance? Does one account with ten breaches count once? What qualifies as an enterprise account if its plan changed on the last day? Are trial, test, and deleted accounts included? These are product definitions, not things the model should infer from an incident note. A semantic metric definition owned by the billing or usage team gives the assistant a versioned contract to call.
The execution plan is a bounded read-only aggregate over authoritative usage and contract history. Apply the authorized account set at query time. Group usage by the quota window and account, compare it with the limit effective for that window, then count distinct accounts with at least one qualifying window in the quarter. A per-request limit is a different predicate from a monthly allowance. If a contract changes mid-window, the billing owner has to define whether the window splits or the limit is prorated. Do not let SQL pick that rule by accident. If the history cannot reconstruct the effective limit, we cannot answer precisely. Say which piece is missing. A precomputed breach table can make this affordable at scale, but its definition, update watermark, and backfill history must be visible. A fast stale aggregate is still a stale aggregate.
The model may translate the question into a proposed metric and parameters. It should not get arbitrary SQL access to a warehouse with broad credentials. A query service validates the allowed metric, tenant, date range, resource budget, and caller's account visibility. The warehouse or service enforces row and column permissions, not a model instruction that says “only use my accounts.” Watch the identity of a service account that can bypass row policies. PostgreSQL's row security documentation explicitly describes owner and bypass behavior. BigQuery documents row access policies and separate restrictions on historical access. The exact controls depend on the data system, so test them with a caller who must not see a neighboring account.
I would return more than a naked 5: metric definition and version, interval, visibility scope, data freshness watermark, snapshot or query identifier, distinct count, and whether any source partitions were incomplete. The natural-language answer can say, for example, “17 of the accounts you can access breached their effective limit at least once in Q2, as of the usage feed at 09:00 UTC.” That number is illustrative, not a claim about any company. It should not say “17 enterprise accounts in the company” if the caller can see only a subset. For small groups, even an aggregate can leak information through repeated queries, so apply the organization's minimum group size or query policy where needed.
Suppose that count also decides billing adjustments. I would pin a reproducible data snapshot and metric version, reconcile late events, and have the billing owner approve the definition. PostgreSQL's repeatable read behavior is an example of a consistent snapshot for repeated reads within one transaction. It does not make separately loaded warehouse tables coherent unless the system establishes a common cut. A dashboard number that changes after late ingestion should display a revision, not quietly overwrite the basis of an already approved adjustment.
“Why did account A exceed its limit on Tuesday?” changes the method. Start with the exact account and breach window from the structured query, then retrieve the contract clause, change log, and relevant incident or rollout notes. Give an explanation only when those records actually connect the events to a cause. If usage proves a breach but no evidence proves why, say so. Here retrieval is useful for explanation, while the database remains responsible for the count. The two paths meet at a shared account, time window, and source revision.
Continue reading
Related questions
Read beyond the question
Explore more retrieval and rag
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →