Data and Knowledge Systems · Staff
A retrieved table gives the right row and the wrong number
The question
Interview question
In a hypothetical benefits scenario, an assistant retrieves the correct PDF page. It says the employee reimbursement cap is $5,000, but the page's table has $5,000 under the family column and $2,000 under individual. A footnote further limits a specific plan to $1,500. The search hit, citation page, and document title are all correct. Where did the system fail, and what would you change?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
The page match is not the answer. The answer depends on a relationship among a row, a column, a plan, and a footnote. A text extractor that emits Individual Family $2,000 $5,000 as one line may give the model all the tokens and still lose which number belongs to which header. A chunker may detach the footnote or carry a merged “annual cap” heading to only the first row. This is a parsing and evidence-assembly problem before it is a model reasoning problem.
I would inspect the original page next to the extracted representation and the exact model context. Did the parser preserve cells, row and column headers, merged-cell spans, units, footnote markers, and page coordinates? Does the plan qualifier sit in a caption outside the table? Did OCR read $1,500 as $7,500? It is easy to blame generation when the model was never given a faithful table. Google's Document AI layout parser is one example of layout-aware processing. Its output still needs validation on the actual PDFs and table types.
For indexing, I would keep a structured table object alongside the original page. A cell has its value, row key, column key, parent headers, units, footnote references, page and bounding box, source revision, and extraction confidence when available. Search can use lexical terms for exact plan names and numbers and a table-level representation for semantic discovery. Once a table is retrieved, assemble the relevant row and its headers plus the footnote into the context. Do not ask an embedding to encode the entire two-dimensional relationship in one flat sentence.
The assistant's claim should resolve to a particular cell and its qualifier. For “individual, plan X,” find the individual column's $2,000 cell, then apply the plan X footnote that lowers it to $1,500. The answer is $1,500 for that plan, with a citation that can highlight the cell and the footnote, assuming no other governing rule. For an unspecified plan, asking a clarifying question or stating the conditional values may be better than choosing $1,500 for everyone. The arithmetic here is a minimum cap, not a sum of two dollar amounts. If the footnote describes an entirely different condition, the answer changes. The parser must preserve the link so a reviewer can tell.
At the generation boundary I would require the proposed number, currency, population, time period, and supporting cell/qualifier IDs as structured evidence. A deterministic check can verify that the cited cell contains the number or that an explicit calculation links cited inputs to the output. It cannot decide every policy exception, so sampled human review should inspect difficult tables and disagreement cases. Evaluate cell selection, qualifier application, and final claim support separately. A page-level citation correctness metric would mark this wrong answer as right.
Remove the table header from an OCR scan and the choice gets harder. I would not infer “individual” solely from left-to-right position if the scan can have rotated pages or split columns. Reprocess the page with a better extractor, use a source-native spreadsheet when available, or abstain from the exact amount and route the document for repair. The system should make incomplete extraction visible rather than convert missing layout into confident policy.
Then the footnote changes but the table image does not. Treat the page or table plus footnote as one versioned evidence dependency. Reindex affected table representations and invalidate answers that depended on the old qualifier. A parser confidence score can prioritize review, but the visible failure is a wrong claim with a correct page citation, so the release test has to check the claim against the right cell and condition.
Continue practicing
Related questions
Read beyond the question
Explore more data and knowledge systems
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →