I would keep the cache on during diagnosis only where the current quality and capacity guardrails allow it. A hit rate says that some prefix was reused. It does not say how many tokens were read at the cache price, what new tokens were written, how many output tokens were generated, or whether the whole task succeeded. A single cached token and a 50,000-token prefix both count as a request with a hit if the metric is defined that way.

The first move is to reconcile three ledgers for the same time window: provider invoice, per-call usage, and product task outcomes. Give each task and attempt an ID and attach every model call, tool call, cache event, retry, and cancellation to it. Price each provider usage category with the contract effective at the time, then sum by task, tenant, model, and task slice. For example, Anthropic's prompt caching documentation reports uncached input, cache creation, and cache read tokens separately, and it warns that its input_tokens field by itself is not total input. OpenAI's reasoning guide explains that non-visible reasoning tokens appear within output usage and are billed. Those are provider-specific examples, not a universal accounting schema. The platform must normalize each contract without throwing away its distinct categories.

Possible causal paths are quite different. The team may have cached an enormous prefix that is rewritten often, paying for repeated creation and pushing out a cheaper prefix. It may now include longer tool descriptions or conversation history in every call. The agent may generate more output or reasoning tokens, call more tools, or retry after a malformed tool response. A router may have shifted hard tasks to a more expensive model while the cache project shipped. Capacity contention can lengthen queues and cause deadline retries, lowering completed tasks while increasing billed attempts. Look for the first divergence by stage, not one global “cost” graph.

Suppose a task previously cost $0.20 and succeeded 80 percent of the time, and now costs $0.18 per attempt but succeeds 40 percent within its deadline. On a simple one-attempt basis, spend per timely success went from $0.25 to $0.45. Those numbers are hypothetical and the calculation assumes comparable tasks. With retries, use the sum of all attempt costs divided by successful tasks in an assigned cohort, while also reporting cost and success separately. Do not discard failed tasks from the numerator or claim a cost win from a route that silently refuses hard work. Compare matched task distributions and latency budgets, then randomize an eligible slice if a rollout effect is still in doubt.

The dashboard uses request hit rate, but the agent now makes twenty calls per task. That changes the denominator. Even if 19 of 20 calls read cached input, one expensive initial write, twenty uncached suffixes, twenty outputs, and tool costs may dominate. Plot cached bytes and billed categories per completed task, together with number of calls and cache lifetime. If most extra calls come from a new planner loop, fix that loop. If the prefix changes on every turn, stabilize only the shareable portion while keeping tenant rules and current authorization outside unsafe reuse. If output length drives the bill, changing the cache will not fix it.

I would keep caching only where it improves cost per successful task or a consciously valued latency target under acceptable quality. The release decision might be “cache the stable prefix for long tasks, leave short tasks alone,” not an all-or-nothing switch. Before calling the invoice wrong, account for lag and credits in the provider bill and verify that our per-call usage sum actually matches its scope. An unaccounted remainder is a reason to investigate measurement, not to invent a cache failure.