The unit of an agent eval is a trial in an environment, not just a prompt and a final sentence. If trial A creates a task and trial B sees it, the measured difference may come from contamination between trials rather than the agent version. Before comparing models, define the initial world state, permitted actions, time budget, and the final state that counts as success. “The agent said it created a task” is not the same as a task in the system of record.

I would give each trial a separate account, ticket namespace, storage snapshot, and tool ledger, or reset a private fixture to an exact version. Tool calls read and write that fixture through the same contract as production where possible. Record source revisions, tool schemas, policy revision, clock, model route, and accepted tool results. An operation that times out must enter the same unknown state the production runtime would use, not a harness shortcut that returns success. Anthropic's agent eval guidance makes the environment's final state and full trajectory part of the eval, which is the right framing for this scenario.

For a message sending tool, I would use a controlled inbox sink that records delivery attempts and recipients, not send real customer messages. The grader checks exactly one appropriate task, the correct recipient and content, no disallowed tool effect, and the answer's claims. It need not require one fixed sequence of read calls. If the agent takes a legal alternative route, outcome and policy invariants decide, while latency and cost are separate measurements. A fixture that accepts any syntactically valid message misses the product risk.

The live API probe is useful. A deterministic fixture lets us attribute a regression to a changed agent, but it may hide real 429s, stale reads, price changes, and unavailable tools. Use two layers. A controlled regression suite pins cases and injects named fault schedules, such as a rate limit on the second call or a price update between read and write. A smaller integration suite runs against an isolated live service, records observed versions and fault conditions, and checks invariants without pretending bit for bit replay is possible. Do not call a single live success rate a model score when the environment changed at the same time.

Multiple trials per case help estimate variability. Keep random seeds where an API supports them, but do not claim a seed guarantees the same model output across a provider update or parallel tool timing. Compare versions on matched fixtures and report uncertainty, failure categories, and the exact environment generation. If one version succeeds only when the price never changes, that is a conditional result, not an overall win.

The hardest edge is a real write that cannot be simulated faithfully. Then define what the fixture proves and run a narrowly scoped canary with human oversight before broader release. A harness can verify policy checks and operation state under injected timeouts. It cannot prove a third party's eventual delivery semantics merely by returning a canned JSON response. The eval report should make that boundary visible to the person deciding to ship.