Evaluation and Quality · Principal
Zero failures in 200 tests does not establish a rare-event guarantee
The question
Interview question
A team ran 200 agent trials and saw zero unauthorized writes. They claim the rate is below one in ten thousand and want to enable unattended execution. Assess the claim. Half the trials are small variations of the same task.
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Zero observed does not mean zero possible. Under an independent, identically sampled Bernoulli model, if the true failure probability is p, the chance of seeing zero failures in n trials is (1-p)^n. For n = 200, the one-sided 95 percent upper confidence bound solves (1-p)^200 = 0.05, giving p = 1 - 0.05^(1/200) ≈ 0.0149. About 1.49 percent, far above 0.01 percent. To push that bound below 0.01 percent with zero failures would require roughly 30,000 independent representative trials. NIST's exact binomial documentation covers the one-sided method. The arithmetic is a useful sanity check, not a guarantee for an agent whose failures are clustered by task, model version, or tool.
Here half the cases are slight variants. Correlation means 200 lines in a spreadsheet do not necessarily carry 200 independent opportunities to fail.
There is another issue: did these tests even exercise the conditions that allow an unauthorized write? A corpus of safe read-only tasks can produce a perfect result while saying almost nothing about the write boundary. Specify the event first. Does “unauthorized” mean outside delegated scope, above a monetary cap, stale approval, wrong tenant, or an external action after cancellation? Record the attempted action and the actual effect, not just the model's final text.
I would make the release gate a layered argument. Deterministic broker checks enforce permissions for every effect, with adversarial tests around concurrency, revocation, and format changes. Then run scenario tests that deliberately reach each failure path. A separate representative sample estimates ordinary workload risk with intervals, including runs that error or never finish. Human review of the attempted actions checks whether the grader itself misses violations. Keep the statistical statement tied to the sampled population and version. Testing does not replace a hard authorization boundary for a rare, high-consequence effect.
Suppose the team offers ten thousand more synthetic variants of the same prompts. That may catch a narrow regression, but it cannot repair the sampling or oracle problem. Suppose instead they observe zero incidents in a million production requests. The rate might look much smaller, but only if unauthorized actions would have been detected, exposures were counted correctly, and the policy and traffic did not change. I would ask for detection coverage and the relevant high-risk denominator before turning a zero into a safety claim.
Continue reading
Related questions
Read beyond the question
Explore more evaluation and quality
Follow another question in this area, or return to the full Interview Prep index.
Browse this area →