How would you evaluate a docs assistant that should sometimes refuse to answer?

I would label the situation before I label the model's sentence. The same question can be answerable for one product version and not another. It can be answerable for an administrator and unavailable to a regular user. It can be answerable on Tuesday and stale after Wednesday's release.

An eval case therefore needs the question, tenant and principal, corpus snapshot, source versions, time of the question, and any missing user details. The label says which action is appropriate: answer from permitted current evidence, ask a clarifying question, or decline to make the claim. For answerable cases, record the necessary evidence spans and the allowed claim. For an unanswerable case, record why: absent source, only an old version, conflicting authoritative records, missing product version, or relevant content restricted to this principal. Those are different system failures and different possible repairs.

I would not use the model's confidence as the gold answerability label. Confidence is a model output. The label comes from a human looking at the corpus and permissions defined for that case, ideally with a second reviewer for disputed or high-impact examples. If annotators disagree, keep the disagreement, write an adjudication rule, and do not quietly force a single label from a guess. Some questions are genuinely ambiguous. The system should be allowed to ask for the version instead of being penalized for not inventing one.

A useful set needs common cases and deliberately hard slices. Include exact error codes, renamed APIs, two versions with different behavior, deleted pages, inaccessible current pages, partial evidence, misleading but relevant excerpts, and questions that require two documents. Use real user wording where it can be handled safely, then add expert-authored cases for failures that production logs underrepresent. Keep a held-out slice for decisions about a new model or retrieval change. Synthetic paraphrases are useful, but a thousand paraphrases of one known answer do not give a thousand independent failure modes.

I would score the path, not only the final prose:

For the decision, report unsafe answer rate on unanswerable cases and false abstention on answerable ones. Also report coverage: the fraction of cases the system chooses to answer. A system that refuses everything can achieve a low unsupported-answer rate and still be a bad assistant. Compare these together, by consequence and slice. I would not let strong performance on easy public documentation compensate for a permission leak.

The judge's preference for fluent unsupported answers is a calibration failure. Give it source text, answer, claim boundaries, and an explicit rubric that marks unsupported claims wrong even when plausible. On a human-reviewed set, measure false acceptance of unsupported answers and false rejection of careful, concise ones. Use an independent check for permission and exact version fields. A judge should not be asked to infer from its own parametric knowledge whether an absent document supports a claim.

Here is the interviewer's harder push: the assistant gives the correct API behavior from memory, but the allowed corpus contains only an obsolete manual. If this product promises answers grounded in the available docs, the answer is unsupported for that case even if it happens to be factually correct. We can record factual correctness separately. Lucky correctness is not evidence that the retrieval and grounding path is safe. If the product allows outside knowledge, define that contract and evaluate it separately.

Release gates depend on the use case. I would set a strict block on restricted-content exposure and high-consequence unsupported answers, then compare answerability decisions, citation support, and user utility on the remaining slices. Review false abstentions because over-conservatism pushes work back to the user. OpenAI's evaluation best practices recommend production-shaped cases and human calibration of automated scoring. The case schema, labels, and gates above are the proposed eval for this assistant.