Pass@8 asks whether at least one of eight attempts solves a task under the benchmark's success test. The product asks what one user gets from its actual selection process within a deadline. Those are different questions. If attempts were independent with identical success probability p, the chance of at least one success would be 1 - (1-p)^8. In a real agent, attempts share model weaknesses, task ambiguity, and tools, so that simple formula is only an illustration. More importantly, the product does not get an oracle that points to the winning attempt.

I would first run the current product path: one attempt, the same repository checkout and tool permissions, twenty minute budget, and the same patch validation a user sees. Score whether the issue is fixed, existing behavior survives, the diff is safe, and the patch can be reviewed. Measure time, tool calls, cost, and rate of abandoned or partial runs. A benchmark pass is one useful signal, not a complete release gate. SWE-bench Verified's original description explains its test based issue resolution setup. Anthropic's infrastructure noise analysis shows why harness configuration can move coding eval scores. Neither says our users' repositories behave like that benchmark.

The eight patch proposal changes the product, so evaluate that product. Can we run the attempts independently without sharing a dirty workspace or exhausting one tool quota? Which tests are available before selection? Does a passing test distinguish two patches that both satisfy the visible cases but one breaks a hidden contract? The selector must run under the same twenty minutes and count all eight attempts, test executions, and merge work. If it picks by a judge, evaluate that judge on patches it did not help generate. Report success of the selected patch, not the best patch found in hindsight.

Maybe eight attempts are parallel and finish in time. That can be worth buying on a hard slice. It can also create eight nearly identical patches, saturate CI, and make one unsafe patch look attractive because it changes a test. Try a small fixed retry budget triggered by a meaningful failure signal, or a different tool strategy for the cases where pass@1 is low. A good selection system needs diversity and a credible verifier, not just more samples.

The interviewer then points out that the benchmark tests may have leaked into model training or be too narrow. That is another reason to treat the headline score cautiously. OpenAI's 2026 analysis of SWE-bench Verified reports contamination and test validity concerns for frontier progress measurement. I would use private, fresh tasks from our workflows with hidden independent checks and human review of a subset. Do not publicly claim a benchmark is invalid for every use or that any one replacement is perfect.

If pass@1 is flat, selected-patch quality is flat, and cost grows, I would not ship this change for the one attempt product. If the eight attempt system wins on a hard, valuable slice within the user deadline and passes safety gates, I might route only that slice to it. The unit of value is the patch the user receives, with the time and effort it cost to get there.