Not necessarily fake. It measures performance on those benchmark tasks with that harness, budget, and grader. It may not measure improvement on new tasks. There are at least three explanations to separate: the public issues or patches appeared in training or tuning material, the agent learned benchmark-specific patterns without memorizing exact solutions, or the private tasks have a different difficulty and repository mix. The oracle itself may also be wrong. OpenAI's recent analysis of SWE-bench Verified reports both contamination concerns and tests that reject valid solutions in the audited set. Those findings are about that benchmark and analysis. They do not license dismissing every public score or assuming a particular vendor trained on a particular issue.

I would reproduce the 12-point gain with a pinned agent build, model route, tool budget, repository snapshot, task instruction, test harness, and scoring rule. Keep failures and timeouts in the denominator. Audit a stratified sample of wins and losses: did the patch actually satisfy the issue, did it rely on unavailable future files, and did tests accept a shortcut? Search for exact or near-exact issue text and gold patches in any training or prompt-tuning data we control. For an opaque external model, we can test exposure patterns and ask the vendor, but cannot prove clean training lineage from output behavior alone.

Then create a rolling set of fresh, access-controlled issues with creation time after the evaluated model or agent freeze where feasible. Keep setup and hidden tests reproducible, but have human engineers verify that the tests distinguish a valid fix from a test-specific hack. Record task categories, repository complexity, required exploration, and time or cost limits. Never let the evaluation team repeatedly tune against the same “private” holdout until it has become a development set. Rotate new tasks in, retain a locked comparison set for a limited number of decisions, and retire compromised cases explicitly.

What if the public and private sets disagree because your issues are much harder? That is still useful: slice by difficulty and task type, with confidence intervals. A zero average gain might hide a strong improvement on migrations and a regression on bug fixes. If public performance matters as a general capability signal, report it with its scope. For buying or deploying this agent, the relevant result is success on your task distribution at your actual attempt budget, verified by an oracle that engineers trust. A benchmark name is not a substitute for that contract.