That set has become a development set. The team used its errors as feedback for design choices. It may still be useful for regression checks, but the 14 point gain is a biased estimate of performance on new questions. We do not need proof that someone put labels into a gradient update. Repeated human choices can fit a system to a finite test set too.

I would reconstruct what changed. Which versions were evaluated, how many times did the team look at each case, which changes were motivated by a particular failure, and were prompts or examples copied into demos or tuning data? Freeze the current system before measuring it on a truly new set. The new sample should be drawn from the intended traffic, with its time, customer mix, task types, permissions, and answerability preserved. If one customer sample is small or unusually hard, report uncertainty and slices rather than calling the old result fake. A gain on the old cases can be real for those cases and still not transfer.

The evaluation unit needs to be more than “did a relevant chunk appear.” For each question, pin source revisions and authorized principal, decide whether an answer is possible, identify the evidence that supports the needed claims, and judge the final answer and citations. Compare retrieval candidate recall, evidence surviving the context builder, supported claims, false answers, abstentions, latency and cost. A query rewrite that improves a score by dropping hard version constraints should fail an exact-version slice even if the aggregate climbs. The team's former holdout becomes a named development corpus, with its exposure history recorded.

For the fresh test, choose cases after the current system freeze where feasible and keep their labels away from the people and automated optimizers making changes. Limit the number of final-set peeks, predeclare the primary metrics and release threshold, and rotate or retire cases once they have influenced decisions. OpenAI's guidance on maintaining a holdout describes its role in detecting overfitting. The exact access and rotation process here is a system governance decision. An encrypted file that the whole team queries after every PR is not held out in the practical sense.

The cost objection is fair. Do not label 20,000 questions just to recreate an impressive number. Sample a modest fresh representative set for the release decision, then add targeted cases for rare high-impact failures. Use active sampling to find disagreements, but do not estimate population accuracy from only the hard cases without accounting for selection. Label a subset twice and adjudicate disagreements so the team knows whether a two point movement is larger than label noise. Automatic graders can screen or scale review after calibration, but a grader tuned on the same repeated cases can overfit alongside the retrieval system.

Suppose the fresh sample shows a gain for common documentation questions and a regression for permission-scoped incident questions. The release decision follows the product and risk slices. Ship the improvement behind a route that excludes the harmed slice if that isolation is real, or fix it before global release. Publish the old-set result as a regression result and the fresh-set result as the prospective estimate. A coding benchmark score jumps, but new private tasks do not already covers a public coding benchmark versus private tasks. This case is about our own quiet leakage through months of iteration, even when every engineer acted honestly.