Retrieval recall improved, so why did the answers get worse?

I would stop expanding the exposure while I check whether those quality signals are reliable and whether the embedding rollout caused the harm. If the unsupported-claim increase is confirmed, I would pause or roll back the affected path. A before-and-after dashboard alone does not prove causality, and I would not revert solely on a noisy satisfaction movement.

Recall@20 answers a narrow question: did at least one labeled relevant item appear somewhere in twenty results? It does not say whether the right item was first, whether the source was current, whether the context builder included it, or whether the answer stayed within that evidence. It also does not say whether the label set reflects real user questions.

Here are four ways the reported numbers can coexist:

Those are hypotheses. I would test them on paired requests, not defend one in a meeting because it sounds plausible.

I want the old and new systems' traces for the same queries, with user permission, corpus snapshot, model and prompt version, reranker, and context budget held fixed where possible. Log candidate IDs and ranks, the passage text that actually entered the model, source and effective versions, citation spans, answerability decision, and the final claims. Protect sensitive content in the telemetry. For each losing case, find the first stage where the new path differs in a way that matters.

I would separate the analysis into a few measurements. Retrieval: recall at the number of chunks the model actually receives, rank of the authoritative source, version correctness, and precision among selected passages. Context: whether the sufficient evidence survives deduplication, compression, ordering, and token trimming. Answer: claim-level support, refusal when evidence is missing, and citation validity. Product: user correction rate and satisfaction on comparable traffic. A single overall score can hide a permission-sensitive or high-impact slice.

The embedding rollout also needs a mechanical check. Are query and document vectors from compatible model versions? Was the whole corpus reembedded, or did an index migration leave mixed spaces? Did dimension or normalization change? Did ANN parameters, chunking, or metadata filters move at the same time? Did the new model find more general-topic passages and lose exact error codes or product versions? These questions are about the system around the embedding, not just the embedding model.

For a controlled test, I would run old and new retrieval on the same query and authorized corpus snapshot, feed each selected context to the same generation setup, and repeat enough to see model variation. Human reviewers should label a small, stratified set of disagreements without being told which system produced them. An LLM judge can help sort cases, but I would calibrate it against those human labels, especially for unsupported claims and abstentions. OpenAI's evaluation guidance makes the same broad point about task-specific evals, production-shaped distributions, and human feedback. The release decision should be tied to the application behavior we care about.

If the experiment is randomized and the harm is real, I would roll back the new embedding path or keep it only for slices where the evidence supports it. If the metric is confounded by a simultaneous traffic change, I would hold the rollout and fix the measurement. Either way, “recall went up” is not enough to ship.

The interesting Principal-level part is ownership. Retrieval, context assembly, and generation may have separate teams. Someone still has to own the final answer and the rollout gate across those boundaries. I would make the versioned trace and the end-to-end quality slices part of that contract, so the next team cannot declare success while the user-facing system gets worse.

If the harm appears only in billing policy questions, I would make that a protected release slice and inspect how old and new policy versions rank. The overall recall number does not make a high-impact regression acceptable.

The retrieval team may answer that its golden set has 50,000 examples. I would ask how those examples were sampled and what the labels mean. Fifty thousand clean questions about a policy that never changes can give us a precise estimate of the wrong thing. I want ambiguous, stale, denied, and unanswerable cases represented too.

Primary-source note: OpenAI evaluation best practices. The scenario's metric changes are hypothetical. The diagnosis above is an inference from how a multi-stage RAG system works.

Related ArchCrux reading: Your RAG Evals Are Measuring the Wrong Thing.