The score can become very precise about the wrong distribution. If I take 1,000 seed questions and generate ninety-nine paraphrases of each, I have 100,000 rows, but perhaps only 1,000 independent situations. A narrow confidence interval over those rows mostly tells me the system is consistent on that family's wording. It does not tell me how it handles an unseen user need, a different tenant, or a rare failure with an external effect.

I would look at how the new cases were made. Did generation start from real failures and preserve the fact that made each case difficult? Were answers and labels checked against a fixed corpus snapshot and permissions? Did the generator invent a contract exception, a nonexistent API, or a tool result that violates the real schema? A model can generate plausible but unanswerable questions and then give them confident reference answers. The eval rewards the same unsupported story the product should avoid.

The same model generating the cases is an extra circularity risk. It may favor its own phrasing, solution patterns, or preferred tool path. That is not proof it trained on the test set, and I would not casually call it training-data contamination. We can have evaluation bias without any training leak. A study of synthetic test collections for retrieval found bias in evaluation results from generated queries or labels. The size of that bias in our system has to be measured on our tasks.

I would group the 100,000 rows by underlying source case, template, generator prompt, and failure mode. Report scores at the family level and by slice, not only per row. Keep a held-out set of real, permissioned production-shaped tasks with human labels. Compare whether a change that wins synthetic cases also wins the held-out set and online outcomes. If not, inspect which generation pattern is overrepresented. Synthetic data is still useful for testing controlled variations such as a missing version or an injected instruction. It just needs a known relationship to the task distribution.

Would a different generator model fix it? It may remove one source of stylistic alignment. It does not validate the case or make the labels independent of errors. I would use several creation routes: real anonymized task seeds where permitted, expert-written edge cases, programmatically transformed cases with known invariants, and model-generated variations that a reviewer or an executable oracle can verify. Separate the set used for prompt tuning from a frozen release set. Otherwise a team can improve the score by learning the test set's quirks.

For a concrete adversarial check, take a versioned documentation question and create four controlled changes: swap only the version, remove the governing passage, restrict that passage to another principal, and add a newer exception. The expected decision changes in known ways. We can verify those transformations against the corpus and permissions. That is more informative than asking a model for 10,000 “hard RAG questions” and accepting its answers as gold.

I would track label disagreement and case provenance. How many generated cases were rejected as impossible, mislabeled, redundant, or outside the product contract? If rejection is high, the stable score is especially misleading. A hundred thousand examples can improve statistical power when they represent genuinely varied valid cases. Volume alone cannot buy independence, coverage, or truth.