I would ask to see four strings before touching the vector database: the original question, the rewrite, the actual search queries, and the passage that reached the model.

If the original says AtlasPay SDK 4.2 E731, those tokens are carrying different jobs. The product identifies a namespace. The version limits which behavior applies. The error code may be the strongest exact lookup key in the whole question. A fluent rewrite can preserve the topic while throwing away the keys needed to find the right evidence.

Dense retrieval is good at finding similar meaning. An arbitrary error code or a newly introduced version is not guaranteed to have useful semantic neighbors. Google's hybrid search documentation makes this point for product numbers and new names. A lexical path is valuable for precisely those strings.

I would change the query contract. The rewrite may expand natural language intent, but it does not get to erase protected entities. A small parser identifies likely version strings, error codes, quoted phrases, symbols, and product names in the original. Preserve the original query as one retrieval branch. Search exact and lexical fields with those terms, and search a semantic branch with the expanded question. Fuse or rerank candidates only after checking the constraints. If source metadata reliably records product version, version eligibility can be a hard filter for a question about one release. If metadata is incomplete, I would keep the exact string in lexical retrieval and validate the source version before answering rather than filter away all results.

The query itself can be ambiguous. “What changed since 4.2?” requires material from more than one version. “Does 4.2 support this API?” usually needs 4.2 evidence. The parser should represent the user's intent and constraints, not impose one version filter on every sentence containing a number. When uncertain, search both paths and let the answer say what evidence supports which version.

Why did aggregate recall look healthy? The golden set may contain mostly questions without exact tokens. Recall at twenty can also count the right manual at rank nineteen while the context builder only passes six chunks to the model. The old manual may be semantically similar and easier for a reranker to score. If the citation merely points to a relevant looking page, a final answer evaluator may accept it without checking that the page supports behavior in 4.2.

I would make a small regression set from real versioned queries. Record protected tokens before and after rewrite, lexical and dense candidates separately, final selected context, source version, answer, and supporting citation span. Evaluate the exact failure: did the right source appear, did it reach the model, and did the answer stay within it? Put unseen errors, renamed products, and questions comparing versions in the set. A synthetic set made only by the same rewrite system will likely miss the awkward phrasing that caused the problem.

The interviewer may say, “But rewriting improved recall by ten points.” I would keep it for the traffic where it helps. I would not average away a version regression. Segment quality by questions with protected entities and by consequence of a wrong answer. It may be enough to change one branch of retrieval, not remove rewriting entirely.

Then they remove the version metadata from half the corpus. A hard version filter now has a serious false negative problem. I would fix ingestion to extract version and effective date with provenance, but the immediate fallback is exact lexical matching plus a source page check. Unknown metadata is unknown. It is not permission to label a page as current.

The useful lesson for an interview is that a rewrite is a transformation of user intent. It needs an invariant. Here the invariant is that identifying tokens and version constraints either survive or are represented explicitly in the query plan. “Use hybrid search” by itself does not guarantee that.

See Google's hybrid search explanation for the limitation of semantic matching on arbitrary identifiers. The protected token rule and evaluation slices are design choices for this scenario.