Retrieval and RAG · Staff
Two retrieved facts are true. Why is the answer built from them false?
The question
Interview question
A user asks whether a customer was eligible for a refund on the day of a flight. Search returns a policy stating that Flex fares are refundable and a booking record showing the customer bought Flex. Both passages are accurate and cited. The policy came into force the next day, and the booking was changed to Flex after the flight. The assistant says yes. How would you design retrieval and answer construction so two true facts do not produce a false claim?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The missing operation is a join with conditions. The facts need to refer to the same booking, the same decision time, and versions that were effective then. “Flex is refundable” is a rule with an effective interval and exclusions. “The booking is Flex” is a state with its own history. A current snapshot of each does not establish the intersection on the flight date. A relevant citation to each page only proves that the text exists.
I would make the question planner extract the claim it needs to establish: booking identity, fare class at flight time, governing policy at that time, geography and exception conditions. Retrieval brings candidates, but evidence assembly checks these typed fields and their provenance before generation. A booking change event after the flight cannot satisfy the historical fare-class predicate. A policy effective tomorrow cannot govern yesterday, even if it ranks highest today. If timestamps are ambiguous or one source has only a publication date rather than an effective date, the assistant needs to say what is missing or ask for the right record.
This does not require inventing a universal knowledge graph. For high-stakes claims, a small claim-specific join can combine an authoritative booking history with versioned policy text. Search may locate a policy clause, then a structured lookup resolves its release and effective interval. Store citations to exact spans and source revisions. The model can explain the result after the deterministic conditions are checked. HotpotQA illustrates why multiple supporting facts and explainable reasoning matter, but a production eligibility decision also needs the business timestamps and authority that a benchmark may not represent.
There is a tempting shortcut: put both passages into a long prompt and tell the model to consider dates. That may help on easy cases, but it does not give us a reliable invariant if a source uses “updated on” where another uses “effective from.” The ingestion schema must preserve those meanings. Test the swapped-date counterexample, a booking modification just before cutoff, a policy exception, and two customers with the same fare class. Evaluate the joined claim, not recall of each fact separately.
The pushback is when both sources have incomplete history. Maybe the airline only stores the current booking state. Then we cannot infer past eligibility from today's Flex field. Find a change log, ticket receipt or transaction record, or give a bounded answer saying the required historical state is unavailable. How do you retrieve a contract clause and the exception that changes it? asks for a clause and its exception in one legal context. This question is about the compatibility of otherwise correct facts across time and entity state.
Continue reading
Related questions
Read beyond the question
Explore more retrieval and rag
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →