How do you evaluate an agent when several trajectories are correct?

Say the task is to fix a parser that mistakes an escaped delimiter for a real one. One run reads the failing test, patches the parser, and adds a regression test. Another starts with the parser code, reproduces the failure, and reaches the same patch. A third copies a patch from an unrelated branch, passes the visible test, but quietly breaks quoted fields. Matching the first trace would reject the second and might reward the third if it happened to take the same steps.

So I would define the task by its starting state and acceptable ending states. The starting state includes the repository revision, test fixture, instructions, allowed tools, credentials, and any external services. The ending state includes the code diff and the behavior of the changed system. Then I would add constraints on how an ending state may be reached. For example, no secrets in an external tool call, no edits outside the allowed workspace, no destructive operation without approval, and no unsupported claim that tests passed.

That gives three different kinds of signal:

The first two have hard failures. I would not trade a leaked credential for a faster patch by hiding both in one weighted score. Among runs that satisfy the hard conditions, we can compare quality, latency, and cost. That comparison depends on the product. A background maintenance task may tolerate extra minutes. An interactive coding session may not.

The trace is still useful. I want tool inputs and outputs, model turns, guardrail decisions, handoffs, the final diff, and state changes, with sensitive content handled according to retention rules. OpenAI's agent evaluation guidance describes traces of model calls, tools, guardrails, and handoffs for finding workflow failures. I use that record to explain why a case passed or failed, not as a string to compare with one golden trace. A test can specify a necessary event when it really is necessary, such as obtaining approval before a production write. It should not prescribe the order of two harmless reads.

For a fair comparison, run candidate and baseline from the same clean snapshot. Isolate external tools or give them deterministic fixtures where possible. A mocked tool can verify the agent's decision, but it cannot tell us whether the real integration handles a timeout, pagination, or a permission change. Keep both: controlled evals for attribution and a smaller set against realistic integrations. Repeat cases where model or tool behavior varies. Report the distribution and failure cases, not just one run's score.

I would build cases from actual task patterns, including ambiguous requirements, a misleading test name, an unavailable dependency, a tool that returns partial data, and a user who cancels halfway through. The set needs fresh held out work as well as old regressions. If every case is a minor edit to a known example, the agent can look excellent at the eval and still fail on new work. For a release gate, I would show per-slice outcomes and severity, especially unauthorized effects and regressions on high-impact tasks. Human reviewers should adjudicate a sample of disagreements and the cases the automated grader is least certain about. The grader's agreement with itself is not validation.

Now take the interviewer's slower trajectory. “It avoids the risky tool, but takes twice as long. Which agent wins?” First I need to know what risky means. If the tool is allowed only under conditions the fast run did not satisfy, the fast run fails a boundary and the latency comparison is over. If both paths are permitted but one has a measured, low-probability disclosure risk, compare that risk explicitly with the latency budget. A product owner may set a higher bar for interactive latency, but they cannot make an unknown data exposure disappear inside an average.

Suppose the interviewer then says the tool never leaked data in our test set. I would inspect the permission and payload at the boundary, not infer safety from zero observed incidents. A thousand trials with no leak do not prove a sensitive write is impossible. Construct adversarial cases where the tool result contains secrets and where the agent is prompted to transmit them. The policy should be enforced by the tool layer, and the eval should verify that enforcement even if the model asks for the call.

There is one more trap. An LLM judge might prefer the agent that writes the most detailed explanation of its steps. That preference can punish a concise correct patch and reward a convincing story about tests that never ran. Calibrate the judge against blinded human review of the artifact and evidence. Keep independently checked facts, like test execution and file changes, out of the judge's imagination. Where the label is disputed, keep the disagreement visible.

I would ship a version only if it clears hard boundaries, improves or holds the task slices we care about, and stays within an agreed latency and cost band. A single “agent score” is too lossy for that decision. Several trajectories can be right. The eval's job is to say what right means for this task, and catch the ways a seemingly successful trajectory got there badly.

Source for the trace mechanism: OpenAI's agent evaluation guide. The invariant and release gate design is the proposed evaluation for this scenario.