Keep the old set. It tells us whether known failures returned. But once an attacker can observe responses and change the next prompt, the thing we are measuring is a process, not a fixed file of prompts. A memorized pass on yesterday's examples does not bound what the next attacker can find. OpenAI's safety work describes turning weaknesses in earlier checkpoints into evaluations, and its deployment simulation work discusses how test distributions can differ from deployment. Those motivate a changing test program, not a universal safety score.

I would run a separate adaptive exercise with a stated threat model. Who is the attacker, what can they see, which tools or documents can they control, what is the allowed number of attempts and turns, and what counts as a successful harmful outcome? Keep full attack trajectories, including failures and abandoned attempts, with model, policy and detector versions. “Five successes out of ten submitted winning prompts” is not a success rate. Neither is five successes divided by millions of unrelated normal requests. Use an attempt-level denominator for a defined attacker budget, and also report per-attacker or per-campaign success where repeated trials are correlated. Be explicit that this estimates risk under the tested attacker setup, not real-world prevalence.

Keep the discovery pool apart from a held-out evaluation pool. Red teamers can test live and report new mechanisms. A different team can build variants and fresh tasks to see whether the mitigation generalizes beyond the exact string. Some tests should exercise the complete product, including tools, retrieval and output controls, because a model-only refusal is not the whole system. Analyze legitimate requests in nearby topics too. A broad block on multi-turn questions might reduce attacks while making an ordinary user unable to get help. The safety update refuses more users but blocks more harmful requests. Do you roll it back? is the separate decision about that safety and utility trade-off during a rollout. Here the attacker changes the distribution after the rollout.

For a severe live exploit, containment does not wait for a perfect benchmark. Narrow a vulnerable tool capability, add a temporary control, rate limit an abuse path or roll back, then investigate with preserved evidence and a privacy-aware incident process. After a fix, measure known regression, fresh adaptive attack results and legitimate-task quality before a guarded ramp. Monitor for new clusters, not just exact prompt matches. Feed newly verified mechanisms into the regression set with dates and provenance.

Suppose an interviewer asks for one number to gate release. I would give a small set of numbers with definitions: known regression failures, adaptive success at a fixed budget, severe product-level outcomes, and false refusals in adjacent legitimate tasks. No finite red-team sample proves an attacker will fail tomorrow. The useful claim is narrower: what was tested, under which attacker access and budget, what the mitigation changed, and how fast the system can detect and respond to the next route.