I would start with the user-visible promise. Is the assistant supposed to give a supported answer from permitted documents, finish a support task, or make a good creative suggestion? A judge score over a nightly test set is useful, but it is not automatically a measure of the service users received today. An SLO needs a defined population, good event, time window, and measurement path that the team can act on. The Google SRE workbook frames an SLI as good events over total events and ties it to a user journey.

For a documentation assistant, I might have an operational SLO for answers delivered within the deadline and a quality objective for claims supported by current permitted evidence. Permission leaks and forbidden tool calls should have direct policy checks and incident gates. A judge can review sampled support quality, but it must see the cited source, the question, the principal's allowed scope, and the rubric. A score for “helpfulness” from an ungrounded answer is the wrong good event.

There are at least three reasons I would hesitate to make raw judge score the sole SLO. The judge can drift when its model or prompt changes. It can reward length and confidence instead of the relevant condition. And the mix of questions changes, so a global average can improve while the high-impact slice gets worse. Research on LLM evaluator position bias is one concrete warning that even pairwise judging can depend on presentation. We need to measure those errors on our own tasks rather than assume the paper's effect size applies here.

I would calibrate the judge against a human-reviewed set. Blind the candidate identity, randomize answer order in pairwise comparisons, compare the judge's false acceptance of unsupported claims and false rejection of careful answers, and repeat after each judge update. Keep a sample for adjudication and a queue of disagreements. If the judge can only separate good from bad on the easy public-docs slice, do not use its overall score to release changes to a policy assistant.

Human agreement varies by topic. That should change the rubric, not make the problem disappear. For an exact API version or a numeric claim, reviewers can be given a source and a precise support criterion. For subjective tone or usefulness, disagreement may be genuine. Report those slices separately, retain multiple reviewer judgments where useful, and avoid a target such as “99 percent correct” when nobody can consistently label the denominator. Low agreement does not mean every answer is acceptable. It means we should be honest about what the metric can resolve and where expert adjudication is needed.

Could a judge-derived quality measure ever be an SLO? Yes, if the task is narrow enough, the sample represents production, the judge is calibrated, the version is pinned, and the error bounds are acceptable for the decision it drives. I would write down its denominator, sampling method, exclusions, and fallback when the judge is unavailable. Even then, pair it with deterministic security and operational measures. A weekly sampled quality objective may guide release and review. It should not pretend to be a real-time guarantee for every individual response.

If that score drops during an incident, I would inspect the sampled cases and upstream changes before automatically disabling the product. Was the source corpus updated? Did the judge version change? Did the workload mix move? A validated quality alarm can pause a rollout, while a single noisy score should trigger investigation. The stronger the action attached to the SLO, the stronger the evidence that its measurement means what we think it means.