I would push back on “all.” A retrieval trace is useful because it tells us what the model saw and why it answered. It can also become a second, poorly governed copy of every confidential document in the company. A debug pipeline often has wider access and longer retention than the source. If we dump every snippet there, source ACLs and deletion rules no longer describe who can read the content.

Start from the incident questions we actually need to answer. For ordinary requests I want the run ID, tenant, region, principal reference, source and revision IDs, authorization decision and policy revision, candidate ranks, reranker scores, selected passage IDs, prompt and model versions, cited span IDs, timing by stage, and typed tool outcomes. Those fields let me see whether the right document was found, filtered, discarded, or cited. They do not require the whole passage text in a central dashboard. Even identifiers and query strings can be sensitive, so access and retention still apply.

Sometimes a bad citation cannot be diagnosed without the exact text the model received. For that case, use a separately protected evidence snapshot, captured under an explicit purpose and retention rule, in the allowed region and tenant boundary. It can be encrypted, access logged, and short lived. A trace can point to that snapshot through an opaque ID. Whether to retain any raw snapshot by default is a product and data governance decision, not an observability engineer's convenience switch. Where full input retention is prohibited, make the system reproducible from immutable source revisions and the context assembly manifest if allowed. If the source itself has been deleted, reproduction may correctly be impossible.

The probe crosses both region and tenant. A global control plane can aggregate counts, latencies, model versions, and failure categories without importing all content. Local collectors redact or drop sensitive fields before export. The OpenTelemetry sensitive data guide describes attribute filtering, redaction, and transformation processors. Those processors need tests against real payload shapes. A regex that removes one account number format while preserving a free text tool result is not a data boundary.

I would partition raw traces and protected snapshots by the required region and tenant scope. An incident investigator gets time limited access to a specific trace or case, with an approval trail. An on call engineer can see that authorization removed 19 candidates without reading the 19 documents. If a request crosses services in two regions, propagate an opaque correlation ID and non-sensitive state. Do not assume that propagating the full prompt or document excerpt is necessary for correlation. A cross-region export of raw text must be an explicit permitted path, not a consequence of using one default telemetry backend.

What happens on deletion or revocation? Index copies, prompt captures, caches, and evidence snapshots need lineage and their own removal or access stop procedures. An audit record that a request happened may have a distinct retention basis from the document body. State the contract honestly. “We delete the source but keep every prompt forever for debugging” is a contradictory product promise unless it is clearly scoped and authorized.

My evaluation of this design would include a real debugging exercise: take a wrong numeric citation, a late ACL change, and a cross-region trace. Can an authorized investigator find the failed stage without routinely seeing private source text? Can an unauthorized engineer avoid seeing it even with a trace URL? The aim is enough evidence to fix failures while keeping the logging system from becoming the largest unreviewed knowledge base in the stack.