Every span in this trace is green. The agent run still failed.

An engineering agent gets a request to investigate a rise in payment API failures, decide whether a code change is needed, and create a pull request if the change is safe. It searches the repository, reads a recent incident, queries production logs, changes PaymentClient, runs the tests, and opens a pull request. The model calls succeed, the tools return successfully, tests pass, and the run finishes with COMPLETED.

The change is wrong because the production logs used during the investigation were forty seven minutes old while that source was expected to be refreshed within five minutes. Nothing in a normal service error rate catches this. To investigate the run we need the execution path, the context available at each decision, the external effects created by tools, the state changes made by the runtime, and an independent view of the final outcome.

This article stays on operational observability. Semantic evaluation, such as whether a model interpreted evidence correctly or generated good tool arguments, connects to the same run data but needs its own evaluation system.

Give the logical agent run its own identity#

Define the execution model inside the agent runtime before mapping it to a tracing backend.

AgentRun
    |
    +── Step
    |     |
    |     +── ContextAssembly
    |     +── ModelInvocation
    |     +── Retrieval
    |     +── ToolInvocation
    |     +── PolicyCheck
    |     +── StateTransition
    |
    +── Outcome

The runtime owns identifiers such as:

run_id
step_id
attempt_id
agent_id
agent_version
context_manifest_id
operation_id
tool_call_id
delegation_id

run_id identifies one logical piece of work requested from the agent. It should not be replaced by trace_id because one run can cross worker restarts, approval waits, queues, deployments, and asynchronous verification.

run_id = R9831

trace_id = T1    initial execution
trace_id = T2    resumed worker
trace_id = T3    outcome verification

Each trace still follows normal distributed tracing rules. run_id joins those execution segments into one logical run.

The same applies when a coordinator delegates work.

run_id = R9831

delegation_id = D1    investigate logs
delegation_id = D2    inspect repository
delegation_id = D3    verify patch

delegation_id tells us which logical assignment produced an execution branch. Trace parent relationships tell us how processes called each other. Those are not always the same thing, especially when work goes through queues, is retried on another worker, or several agents execute in parallel.

Diagram showing one logical AI agent run identified by a stable run ID spanning multiple distributed traces, worker resumptions, and delegated agents. Separate delegation IDs identify work assigned to research, code analysis, and verification agents, while outcome verification runs asynchronously in another trace.

A logical agent run can span several traces, workers, and delegated agents. The run ID remains stable across the complete task, while trace IDs represent execution segments and delegation IDs identify logical assignments given to other agents.

Open diagram

Record the context manifest used for every model decision#

If we want to know why an agent made a decision, a trace pointing to context_manifest_id needs to resolve to the actual evidence shown to the model.

A compact manifest can look like this:

ContextManifest {
    manifest_id
    run_id
    step_id
    assembled_at
    rendered_context_hash

    items[] {
        source_ref
        source_version
        rendered_content_hash

        observed_at
        freshness_policy
        max_age_seconds
        freshness_status

        selection_reason
    }
}

Suppose the agent queries production logs at 10:48 using a snapshot observed at 10:01.

source_ref = production_logs/payment_api
observed_at = 10:01
assembled_at = 10:48

max_age_seconds = 300
freshness_status = EXPIRED

observation_age is derived from assembled_at and observed_at. The freshness expectation comes from the policy captured with the manifest entry. This gives the observability system enough evidence to say that stale context entered a specific model call instead of guessing later from unrelated timestamps.

The manifest also needs the rendered content hash because the model may receive an excerpt or transformed representation rather than the entire source document. Source identity tells us where the information came from. The rendered hash tells us what the model actually received.

The context layer can keep richer metadata than this. Observability only needs enough of it to reconstruct the evidence boundary of the decision.

Keep the agent telemetry contract above OpenTelemetry#

OpenTelemetry provides the transport and interoperability layer for traces, metrics, logs, propagation, collection, and export. Its GenAI conventions cover model operations, agent invocation, retrieval, tool execution, token usage, and related attributes. Those conventions are still developing, so application correctness should not depend on a particular experimental attribute name.

Keep a small runtime contract:

AgentTelemetry {
    run_id
    step_id
    attempt_id
    operation_id
    context_manifest_id
    policy_decision
    effect_type
    outcome_status
}

Then map it into OpenTelemetry.

Agent Runtime
      |
      v
Agent Telemetry API
      |
      v
OpenTelemetry Adapter
      |
      v
OTLP
      |
      v
Collector

This model also fits how current agent runtimes are being instrumented. OpenAI's Agents SDK traces agent runs, turns, model generations, function calls, guardrails, handoffs, and custom spans, while still allowing custom processors and external trace destinations.

Trace execution boundaries#

Create spans around operations where latency, dependency behavior, retries, or external effects matter.

invoke_agent
    |
    +── context_assembly
    |
    +── model_call
    |
    +── retrieval
    |
    +── execute_tool search_logs
    |
    +── model_call
    |
    +── execute_tool edit_file
    |
    +── execute_tool run_tests
    |
    +── execute_tool create_pull_request

For model calls, useful attributes include model identity, prompt version, context_manifest_id, input tokens, output tokens, cached tokens, latency, time to first token when streaming, finish reason, and attempt.

For retrieval, record source type, retrieval strategy, query hash, candidate count, selected count, and latency. The content selected for the model remains available through the context manifest rather than being copied into every span.

Keep high cardinality values such as run_id, document IDs, user IDs, and operation IDs on traces and structured events. Metric dimensions should use bounded values such as agent version, model, environment, tool name, and outcome class.

Tool telemetry should describe the external effect#

A span saying create_pull_request completed in 430 milliseconds describes an API call. It does not tell us what happened to the external system.

For tool execution, record:

tool_name
tool_version
tool_call_id
operation_id

effect_type
resource_scope

arguments_hash
attempt
latency

result_status
external_reference

A small effect classification is enough for many agents.

READ
WRITE
IRREVERSIBLE

For writes, include the workflow view of the operation state:

PREPARED
COMMITTED
FAILED
STATUS_UNKNOWN

This lets us search for runs that completed while a write remained unknown, repeated one logical operation across attempts, or changed external state after a policy intervention.

Do not put full tool arguments into every span. Tool payloads can contain code, SQL, credentials, customer information, or other content with different access and retention requirements. A hash and an artifact reference are usually enough for the normal debugging path.

Record execution decisions as structured events#

Several useful things happen between normal dependency spans. The runtime can refresh context, retry a tool, pause for approval, delegate work, or stop an operation because policy rejected it.

Record those events explicitly.

TOOL_RETRY
CONTEXT_REFRESH
ACTION_BLOCKED
APPROVAL_REQUESTED
HANDOFF
RUN_PAUSED
RUN_RESUMED

For example:

event = CONTEXT_REFRESH
reason = OBSERVATION_EXPIRED

run_id = R9831
step_id = S14
context_manifest_id = CM481

These events describe what changed in execution without trying to capture private model reasoning. For new OpenTelemetry instrumentation, structured log events correlated with the active trace are a good fit for this kind of data.

Separate traces, metrics, logs, and large content#

Putting every agent artifact into a tracing backend becomes expensive and creates unnecessary access problems.

Traces reconstruct execution and dependency timing. Metrics support aggregation and alerts. Structured logs carry decisions, state transitions, policy results, and detailed failures. Prompts, responses, screenshots, retrieved documents, context artifacts, and large tool payloads should use separate storage when they need to be retained.

Google's current agent observability guidance similarly treats agent traces, operational metrics, prompt and response information, tool behavior, and evaluation as different observability concerns rather than one trace payload.

                         Agent Runtime
                              |
                              v
                       Telemetry SDK
                              |
                              v
                  OpenTelemetry Collector
                              |
              +---------------+---------------+
              |               |               |
              v               v               v
            Traces          Metrics          Logs
              |                               |
              +---------------+---------------+
                              |
                        Correlation IDs


Large or sensitive content
              |
              v
       Encrypted Artifact Store
              |
              v
       artifact_reference

The artifact store needs its own authorization and retention policy. Otherwise the observability platform becomes another copy of the source code, customer data, internal documents, and prompts touched by agents.

Design the telemetry write path for failure and volume#

Telemetry export should stay outside the main execution path as much as possible.

Buffers need explicit limits. The local collector can batch telemetry and keep processing outside the application process. Gateway collectors handle routing, sampling, and backend export.

Architecture diagram showing telemetry from a production AI agent flowing through an agent telemetry API, OpenTelemetry adapter, bounded buffer, local collector, and collector gateway before being routed to trace, metric, log, and archive systems. Context manifests and large sensitive artifacts are stored separately and correlated through stable run and operation identifiers.

Agent telemetry moves through a bounded collection pipeline before being routed to trace, metric, event, manifest, and artifact stores. The agent runtime owns the logical run identities, while the observability data plane remains outside the agent correctness path.

Open diagram

If the trace backend goes down, the agent should continue running. Exporters can queue and retry telemetry, and selected collectors can use persistent storage when losing buffered records during a collector restart is unacceptable. OpenTelemetry documents in memory sending queues, write ahead storage, retries, and message queues for these failure cases. It also documents the cases where telemetry can still be lost, including queue overflow and prolonged backend failure.

When capacity is exhausted, shedding should follow the value of the record.

external write event        preserve
security event              preserve
failed run                  preserve
unknown external effect     preserve

ordinary successful span    sample or drop
verbose model content       drop first

Agent traffic can produce much more telemetry per user request than an ordinary API. A run with several model calls, retrieval operations, tools, retries, and state transitions can easily produce 30 to 60 spans plus structured events.

Assume an average of 40 spans per run at 50 runs per second.

50 runs / second
x 40 spans / run

= 2,000 spans / second

At an illustrative 2 KB per serialized span, that is roughly 4 MB per second or about 345 GB per day before compression, structured logs, or retained artifacts. The numbers are only an example, but doing this calculation early tells us whether keeping every successful trace for months is realistic.

Store telemetry according to how it is read#

There is no reason to force all signals into one database.

DataMain access patternStorage type
MetricsTime range aggregationTime series store
TracesTrace lookup and attribute searchTrace or columnar event store
Agent eventsRun and failure investigationLog or event store
Context manifestsRun and step lookupMetadata or document store
Large artifactsObject retrievalObject storage
Long term telemetryRare forensic accessCheap object archive

Retention can differ by value and risk. Detailed successful traces may only need short hot retention. Failed runs, security events, runs that changed external state, or traces linked to customer incidents can be retained longer. Aggregate metrics usually live much longer because their cost is lower.

Raw prompts and retrieved content should follow the retention rules of the underlying data. Trace retention should not quietly override a customer's document deletion policy.

Make the read path start from run_id#

An engineer investigating an agent should not have to correlate five systems manually using timestamps.

Engineer searches run_id R9831
          |
          v
       Run Index
          |
          +------ traces
          |
          +------ structured events
          |
          +------ context manifests
          |
          +------ tool effects
          |
          +------ artifacts
          |
          +------ outcome evaluation
          |
          v
    Reconstructed Run

The run index can stay small. It needs enough metadata to locate records in the other stores and support common searches such as agent version, time, outcome, failure classification, tenant, and whether external writes occurred.

A run viewer can then show the execution in order while allowing the engineer to open the exact context manifest or artifact associated with a step.

Workflow state remains authoritative when telemetry is incomplete#

Observability is a derived record of execution. It should not become the source of truth for execution correctness.

Suppose telemetry contains:

operation = OP82
state = PREPARED

and there is no later COMMITTED event.

That does not prove the operation failed. The external write may have committed while the telemetry event was dropped during a collector outage.

Recovery code should use durable workflow state and external reconciliation to determine what happened. If workflow state later says:

operation = OP82
state = COMMITTED
external_reference = PR_1842

the observability UI should show that the trace is incomplete rather than claiming the write never happened.

The same issue occurs with out of order records. An asynchronous evaluation, resumed worker, or queue consumer can report after later events have already arrived. Reconstruct using run_id, step_id, attempt identity, delegation identity, operation identity, parent relationships, and state versions where needed. Timestamp ordering alone is not enough.

This boundary also keeps observability outages from changing agent correctness. Losing a trace is an operational problem. It must not cause the workflow engine to repeat a tool call or change durable run state.

Apply data policy before telemetry leaves the trust boundary#

A small field policy keeps handling explicit:

DROP
HASH
REDACT
STORE_REFERENCE
STORE_ENCRYPTED

Credentials should be dropped. User identifiers can be hashed or tokenized where appropriate. Retrieved documents can normally be referenced. Tool payloads may be redacted or stored in a separately protected artifact store. Prompt and response capture should follow tenant and product policy instead of being switched on globally.

The observability backend needs its own authorization. Permission to inspect agent latency should not automatically expose customer conversations, repository contents, or restricted documents encountered during execution.

OpenAI's tracing documentation makes this concern visible as well. Generation and function spans can contain model and tool input or output, and the SDK provides controls for disabling capture of sensitive trace data.

Sample based on run risk and keep a short promotion window#

Random sampling works well for large volumes of similar requests. Agent runs vary much more in cost and external impact.

Keep detailed telemetry for runs where:

external write occurred
run failed
policy blocked an action
tool retries exceeded threshold
cost exceeded threshold
step count became abnormal
human correction occurred
external effect remained unknown
outcome verification failed

Successful read only runs can be sampled more aggressively.

There is one timing problem. A run may look successful when its execution ends and fail later when an evaluator, external verification, or human reviewer checks the result.

Keep a lightweight envelope for every run:

RunEnvelope {
    run_id
    agent_version
    started_at
    completed_at
    execution_status
    write_count
    risk_flags
    trace_segment_refs
    retention_class
}

Detailed trace segments can remain in a short retention buffer even when the run has not yet qualified for long retention.

For example, R9831 completes at 10:50 and initially looks successful. Its detailed segments have a 24 hour temporary retention. At 11:20 an outcome verifier finds that the production failure still exists. The promotion controller updates the envelope to VERIFIED_FAILURE and moves the related trace segments, manifest, and artifacts into the longer failed run retention class.

Once telemetry has actually been deleted it cannot be promoted later. The short buffer is what makes late classification useful.

Keep execution completion and verified outcome separate#

The engineering agent can reach:

execution_status = COMPLETED

while still producing the wrong result.

Outcome verification should independently check the state that matters.

pull request exists
expected repository changed
tests ran against expected commit
required checks passed
production issue addressed

Then record:

outcome_status =
    VERIFIED_SUCCESS
    VERIFIED_FAILURE
    UNKNOWN

Whenever possible, verification should read external state rather than trust the model's final message. If the agent says it created a pull request, query the repository. If it says a deployment completed, query the deployment system.

Evaluation can attach semantic quality later using the same run_id.

run_id
evaluation_name
evaluation_version
score
classification

The detailed grader design is outside this article. The observability requirement is simpler. Evaluation results must be joinable back to the exact run, context, tools, and outcome they judged.

Measure the system at task level#

Normal infrastructure metrics still matter, including model latency, tool latency, collector saturation, backend errors, and queue depth.

Agent level metrics should describe complete work:

verified outcome rate
time to verified completion
steps per verified success
tokens per verified success
cost per verified success
tool retries per run
human intervention rate
policy intervention rate
unknown external effect rate

cost_per_verified_success is more useful than cost_per_model_call when comparing two agent versions. A cheaper invocation does not help if the agent now needs more model calls, tool calls, and retries to complete the same task.

The observability pipeline also needs completeness metrics. Missing context manifest identifiers, broken propagation, dropped tool records, queue saturation, or outcome checks that never execute can make a clean looking dashboard unreliable.

Reconstruct the failed payment run#

Now the opening failure can be reconstructed without guessing.

run_id = R9831

step 3
context_manifest = CM481

step 4
search_logs

manifest item:
source = production_logs/payment_api
observed_at = 10:01
assembled_at = 10:48
max_age = 5 minutes
freshness_status = EXPIRED

step 5
model selects PaymentClient

step 6
edit completed

step 7
tests passed

step 8
pull request created

execution_status = COMPLETED

outcome
production failure still present

outcome_status = VERIFIED_FAILURE
Execution timeline of a failed AI agent run where context assembly, model calls, log search, code editing, tests, and pull request creation all complete successfully. The context manifest shows that production log data was forty seven minutes old against a five minute freshness limit, identifying the point where the run first became invalid before outcome verification later reports failure.

Every execution step completed successfully, but the context manifest shows that the production observation was already expired when the model used it. Outcome verification later exposes the failure and allows the run to be classified as a context freshness problem.

Open diagram

The incorrect code decision appears at step 5. The evidence showing where the run first became invalid is attached to step 4 through CM481. The production observation was forty seven minutes old and the policy allowed five minutes.

The model API worked. The log query worked. The edit tool worked. Tests passed. The repository accepted the pull request.

The failure classification can now be specific:

CONTEXT_FRESHNESS_FAILURE

A useful observability system should let an engineer reach that conclusion from one run_id, even when the run crossed workers, traces, agents, storage systems, and asynchronous verification. It should also keep telemetry loss separate from workflow truth, because the system recording what happened will eventually fail too.