The first mistake is treating a transcript excerpt as the tool result. The process outcome is a separate fact. A green line in stdout can mean one test passed, not that the command passed. The runner must keep the command identity, attempt, start and end time, exit status or timeout, output byte counts, truncation flag, and locations of full artifacts outside the model context. These fields are set by the runtime, not inferred from the text that happened to fit. OpenAI's shell tool documentation is a concrete example of exposing stdout, stderr, exit code, and timeout separately. Our application still has to decide what to preserve when outputs are shortened for a model.

I would present the agent with the structured status first: failed, exit code 1, output truncated, test summary if a reliable parser found one, and pointers to the complete logs. Then choose excerpts based on the task. For a test run, preserve the failure summary and relevant tail or targeted spans, with a little surrounding context. Keep stdout and stderr distinct. If the runner cannot parse a summary, it can show head and tail plus a clear omission marker, then offer bounded search or file-read tools over the stored artifact. Do not pretend a 20 KB snippet is the complete command output.

There is a practical token question. Sending two megabytes into the context is expensive and may bury the one useful line. A small envelope plus a focused excerpt usually works better. Store the full output under a run-scoped artifact ID with access controls and retention, so another step can inspect it without asking the model to remember a path from a prior prompt. Record the excerpt's byte ranges and a hash or revision of the log. If the agent later cites a failure line, a reviewer can verify that it came from this run, not a previous attempt.

What if the failing test appears in the middle and the tail is only a huge cleanup trace? Search the full artifact for structured test failures, exit markers, and error patterns, but verify parser results against raw spans. The model can request a range around a candidate error. Head-and-tail truncation is a fallback, not a correctness guarantee. For an unfamiliar test framework, mark the summary unverified instead of manufacturing a count from green lines.

The timeout probe changes the state again. A timeout with no observed exit code is timed out, not passed and not necessarily failed tests. The process might still be alive unless the runner confirmed termination. Cancel or fence the process, record whether termination succeeded, and only then choose whether a fresh isolated retry is safe. A test command can execute repository code and may write files or call services, so “just rerun” needs the same sandbox and effect awareness as the first attempt.

I would test the contract with failures at the beginning, middle, and end of a long log, output exceeding the storage limit, process termination races, and two concurrent attempts. The acceptance condition is simple: no context compaction or truncation can turn an unverified or failed tool execution into a confirmed success claim.