The overall refusal rate does not tell me whether the model refused the right things. More refusals could mean better handling of harmful requests, worse handling of legitimate work, or a traffic mix shift. The red-team win also does not tell me whether real user tasks are still served safely. I want a table of intent and outcome, with severity, rather than one number on each side.

Build a reviewed sample of actual requests where privacy rules permit. Label the task intent and the response separately: disallowed request refused, legitimate request answered, legitimate request unnecessarily refused, ambiguous request safely completed, harmful detail leaked. Dual-use security questions need a careful rubric. A defensive request may contain dangerous terms, and an answer can be useful without giving operational misuse instructions. OpenAI's safe-completions work treats safety and helpfulness as simultaneous output properties, and its rule-based rewards work explicitly examines over-refusal. These sources support the need for both axes, not a particular release threshold for this product.

I would split by task class, customer cohort, language, prompt length, tool use and severity of harm. Count rates within each class with uncertainty intervals, then estimate impact under production traffic. If three large enterprise workflows account for most new false refusals, we can quantify the lost work. If the safety gain comes from a rare but severe misuse class, average utility must not wash it out. A single weighted score can hide a catastrophic category, so set explicit ceilings for severe harm and floors for legitimate task completion.

During investigation, I can hold the rollout at a small exposure, stop it in a proven damaged workflow, or route an affected legitimate class through a reviewed policy path. I would not create a keyword exception for “security” without testing misuse variants. Keep a versioned policy and a rollback route, monitor appeals and human escalations, and evaluate any targeted change on both sides again. If there is a serious newly exposed harm, containment takes priority while measurement continues.

Suppose the enterprise complaints are only anecdotes. I would still sample them, reproduce them with context and compare against a matched control. Complaints are a signal for where to look, not a denominator. Suppose the red-team set was used repeatedly while tuning. Then an improvement on it is weaker evidence. Use fresh adversarial cases and a held-out legitimate set, including near-boundary examples, before claiming a win. You tuned against the holdout for six months. Is it still a test set? covers the repeated holdout problem. Here the decision combines real product benefit with harm severity.

The interview answer is conditional. Roll back if the update breaks a critical legitimate task beyond its agreed floor, or if we cannot explain a large safety regression elsewhere. Continue or narrow the rollout if severe harm is reduced and false refusals are within a measured, remediable bound. Report the two outcomes plainly. “Safer” and “more helpful” are not proven by counting refusals alone.