Retrieval and RAG · Staff
The reranker never sees the policy exception
The question
Interview question
First-stage search retrieves a long policy document containing the right exception near its end. A cross-encoder reranker scores that document below a short overview page. The team blames the embedding model because recall@50 was fine. Trace the actual evidence seen by the reranker. The exception may instead be a table footnote omitted during parsing.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
I would inspect the exact query and document tokens passed to the reranker, not the full source page in a search UI. A cross-encoder scores a query-document pair with a finite input length. If the application passes 5,000 tokens and its configured maximum keeps the beginning, the decisive exception may never be in the model's input. The Sentence Transformers CrossEncoder documentation says longer sequences are truncated beyond the configured maximum. That is a specific implementation behavior to verify in this stack. The ranking score cannot judge text it did not receive.
Trace one answer-bearing span through source extraction, chunk construction, first-stage candidate, reranker input after tokenization, final rank, context slot and cited answer. Count whether the exception was present at each stage. If it exists in the full retrieved document but not the reranker input, changing the embedding model is aimed at the wrong boundary. If it never made it through parsing, the reranker is also innocent.
I would rerank evidence-sized passages with their section heading, policy version and needed qualifiers. A later exception can be a candidate in its own right instead of the last 200 tokens of a giant pair. Group related passages by source and revision afterward so the answer can see the base rule and its exception together. Do not just split every 512 tokens without thinking. A condition in a heading or a plan type in a table header may determine what a short exception sentence means.
Would a longer-context reranker fix it? It may, if the model really uses the distant span and the added cost fits the latency budget. Test that against passage-level reranking and a small targeted expansion of promising sections. Measure answer-bearing recall after reranking, claim support, and tail latency, with cases whose exception moves from the start to the end. If the footnote disappeared during PDF parsing, repair the extraction and the cell-to-footnote link first, as in the retrieved-table question about a wrong number. A good reranker cannot recover evidence that never entered its candidates.
Continue reading
Related questions
Read beyond the question
Explore more retrieval and rag
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →