A security team has an agent preparing evidence for its quarterly access review. The agent starts with the review policy, a list of control owners, tool definitions, and the evidence required for each control. It then examines identity-provider exports, compares access grants with employee records, opens a few approval tickets, and asks the model which exceptions still need investigation.

By the twentieth model call, it has found several legitimate exceptions and one account that appears to have retained access after a transfer. The investigation is progressing, but the input bill keeps increasing. The dashboard reports more cached tokens than it did at the beginning, so the team first assumes caching must be working well. It is. The problem is that each request also carries the growing history of the investigation.

Prompt caching can make repeated input cheaper without making later agent turns cheap. It discounts eligible matching prefixes. It does not remove them from the context window, and it does not prevent the remaining input, model output, or tool work from growing. If an agent keeps appending history, the cumulative input across a long run can grow quadratically with the number of turns. Caching reduces the price of that growth, but does not eliminate it.

This matters now because OpenAI, Anthropic, and Google offer different ways to retain and reuse context, with different minimum lengths, write charges, expiry rules, and observability. The decision is not simply which provider advertises the largest cached-input discount. It is what your application repeatedly sends, what it actually reuses, and whether retaining that information improves the next decision.

What gets reused when the next model call begins?#

A model request contains more than the latest question. Depending on the API and model, the rendered input can include tool definitions, application instructions, policy excerpts, earlier model responses, tool output, and the current task. Prompt caching looks for an eligible prefix that matches material processed earlier. On a hit, the provider can reuse work associated with that prefix and bill those input tokens at the applicable cache-read rate.

The match is on the prefix and relevant model settings, not on the meaning of the text. Two policy excerpts that express the same rule differently are not the same cache entry. A cache is also not a store of completed answers. The model still processes new input and produces a new answer. Even identical requests can generate different responses.

In our access review, the agent's input might be assembled this way:

Tool schemas and stable reviewer instructions       8,000 tokens
Control definitions and policy reference material  12,000 tokens
Earlier findings, messages, and tool results        grows each turn
New evidence and the next question                  changes each turn

The first 20,000 tokens are meant to remain stable during a review. If the application puts Review generated at 10:32:08 before the tool definitions, it can make the rendered prefix different on every request. Moving the observation after the shared material preserves the earlier matching opportunity without hiding the time from the model. Changing the order of tool definitions can have the same effect even when every tool still works as before.

Stable and changing prompt material placed in different orders, showing why an early timestamp can prevent prefix reuse.
The drawing shows a possible input layout, not a provider cache trace. The cache also depends on eligible breakpoints, model settings, and the lifetime of the stored entry.

OpenAI's current prompt-caching guide describes explicit and implicit prefix boundaries for newer models. Anthropic's cache documentation uses its own block and breakpoint rules. Neither means that anything somewhere in an old conversation will automatically be found and reused.

Five turns explain the cost, but not the whole problem#

Assume the review agent starts every call with the same 20,000-token prefix and retains an additional 3,000 tokens of findings or tool history per turn. All five requests use a compatible model and settings, the relevant cache entries survive, and the earlier serialized content stays unchanged. These are deliberately favorable assumptions for caching.

CallShared prefixAccumulated historyTotal input
120,0003,00023,000
220,0006,00026,000
320,0009,00029,000
420,00012,00032,000
520,00015,00035,000
All five100,00045,000145,000

Without caching, this is 145,000 ordinarily priced input tokens. To compare the alternatives without depending on the current dollar price of one model, let a normal input token cost one base-price unit. Use a 30-minute cache write charged at 1.25 units and a cache read charged at 0.10 units. Those multipliers apply to supported OpenAI models such as GPT-5.6, and are not universal across providers or models.

If only the fixed 20,000-token block is cached, the first request writes that prefix for 25,000 units and processes its 3,000-token suffix at the ordinary rate. The next four calls read the fixed prefix for 2,000 units each while their accumulated suffixes are ordinary input. Over five calls, the input bill is 78,000 units, a 46.2 percent reduction from 145,000.

Now suppose the application preserves each previous request exactly and creates an eligible cache breakpoint after each new 3,000-token increment. Each later call can read the previous input and write only the new increment. That is an idealized growing-history cache, but it helps reveal the actual accounting.

CallPreviously cached inputNew cache writeGrowing-history input costShare billed as cache reads
1023,00028,7500%
223,0003,0006,05088.5%
326,0003,0006,35089.7%
429,0003,0006,65090.6%
532,0003,0006,95091.4%
Total110,00035,00054,750

The useful observation is in calls two through five. The cached-read share improves from 88.5 percent to 91.4 percent, while the price of the individual call increases from 6,050 to 6,950 units. The earlier history is cheap to reread, but the amount of history being reread is larger every time. There is also a newly written 3,000-token increment on every call.

Separate per-turn views of input cost and cached-read share, showing that both increase during the example.
The cost panel and cached-share panel have separate scales. The chart shows calls two through five, using normalized units rather than dollars. Output tokens and tool execution are excluded.

The five-call growing-history total is 54,750 units. If the fifth call really is the end of the review, the application might avoid writing the final 3,000 tokens by leaving them beyond its last explicit cache breakpoint. Billing those as ordinary input instead would bring the total to 54,000 units. The saving is modest here, but the same principle matters for expensive long-lived writes that never receive a follow-up read. It assumes the API and model permit that breakpoint control and that the fifth input is otherwise identical.

None of these figures predicts a vendor invoice or a latency improvement. The model still receives 35,000 tokens on call five. We have measured input billing under a hypothetical reuse pattern, not whether the security review found the right exceptions or completed faster.

What happens after fifty turns?#

The five-call example is manageable because the history is still fairly short. Long-running agents produce a different scale of problem. If the shared prefix contains S tokens, each turn adds d tokens that remain in subsequent calls, and the agent makes N calls, then the total presented input is:

T(N) = N·S + d · N(N + 1) / 2

The first term represents the shared prefix sent on every call. The second accounts for all the accumulated history. As N grows, the second term dominates, so the number of tokens presented across the run grows approximately with N² under these assumptions.

Under ideal incremental caching, with one write for each new increment, a write-price multiplier w, a read multiplier r, no misses, and every call creating the next reusable prefix, the normalized input bill becomes:

C(N) = w(S + N·d) + r((N − 1)S + d · N(N − 1) / 2)

The formula separates tokens written once from earlier tokens read on subsequent calls. It assumes the provider can reuse the exact growing prefix at every turn and that cache entries survive long enough. Real agents change settings, compact history, branch, and lose cache coverage. The equation is a useful lower-bound scenario for a particular billing model, not an SLO or universal pricing rule.

For the same S = 20,000, d = 3,000, w = 1.25, and r = 0.10, fifty calls present 4,825,000 input tokens in total. Ideal growing-history caching reduces their normalized input bill to 678,000 units. That is a large saving, yet the bill remains much larger than it was at five calls. At a hundred calls, the corresponding figures become 17,150,000 presented tokens and 2,083,000 cost units.

The quadratic term survives because cached reads are not free. When r is greater than zero, the bill still contains a term proportional to N², only with a smaller coefficient. Full-history prompting also runs into context-window limits before N can grow without bound. A production harness therefore needs compaction, targeted retrieval, or a different way to hold older tool evidence. A better cache cannot resolve that architectural limit.

The break-even rate is not the hit rate on your dashboard#

Cache writes can cost more than ordinary input. On eligible newer OpenAI models, a 30-minute write is priced at 1.25 times ordinary input and a 12-hour write at 2.5 times. For a model charging 0.10 for cache reads, that changes what a low-reuse workload should do.

Suppose a token in a cacheable prefix is always billed either as a write or as a read. Let h be the fraction of those prefix tokens billed as reads, with write multiplier w and read multiplier r. Its average normalized price is:

E = h·r + (1 − h)w

If ordinary input costs 1, caching is cheaper in this simplified model only when:

h > (w − 1) / (w − r)

With w = 1.25 and r = 0.10, the threshold is about 21.7 percent. With w = 2.5, it is 62.5 percent. These numbers apply to the specified eligible prefix reads and writes, not automatically to the cache-hit percentage in a provider dashboard.

A dashboard often reports total cached-read tokens divided by all input tokens. That denominator includes ordinary suffix tokens that were never eligible for caching. The break-even calculation instead uses cached reads divided by cached reads plus cache writes within the eligible population. Some requests have several breakpoints or different write prices, so even that simplified denominator can become inadequate. Compare the actual cost categories, not just percentages.

For the review agent, I would monitor four different facts: how much input was billed as cache reads, how many newly written prefix tokens were never reused, how much ordinary input was sent beyond the cache boundary, and total cost per accepted review. A good cache-hit ratio can coexist with a bad last metric. The accounting export should make that visible without requiring somebody to infer it from a graph.

There is a useful one-entry calculation too. A 12-hour write costs 2.5 units per token, and each later read costs 0.10. One write plus one read costs 2.6 units, worse than two ordinary uses costing 2. One write plus two reads costs 2.7 units, better than three ordinary uses costing 3. It is a break-even illustration against no caching. Frequent requests might be better served by a cheaper short-lived cache, so a 12-hour write should also be compared with the actual sequence of shorter writes and reads.

Short shared prompts can fail the eligibility test#

Before changing retention, check whether the prompt qualifies for caching at all. OpenAI documents a minimum of 1,024 visible input tokens for GPT-5.6 and later. Earlier models have different eligibility rules. Gemini documents model-dependent minimums such as 2,048 or 4,096 tokens. Anthropic also has model-specific minimum cacheable lengths. An identical prefix shorter than the relevant minimum may not earn the advertised cached-input price. OpenAI's minimum-length guidance and Gemini's caching guide explain those thresholds.

Consider an agent with only 700 tokens of stable instructions, followed immediately by changing user input. Across ten calls, those repeated instructions cost 7,000 ordinary input units at our normalized rate because the reusable portion is below the assumed 1,024-token minimum.

Suppose there are 324 additional tokens of genuinely useful, stable rubric examples that the application otherwise would not include. If the resulting 1,024-token prefix is eligible and reused perfectly, one 1.25× write and nine 0.10× reads cost 2,201.6 units for that prefix across ten calls. That can be cheaper than the original 700-token prefix processed ten times, despite sending more text on each call.

That is not permission to pad prompts with filler. The added examples can change model behavior, increase context use, and make the prompt harder to maintain. It also assumes perfect cache reuse and excludes the changing suffix. The correct experiment compares output quality and actual billed usage with and without the extra useful material. Often the better answer for a tiny agent is to leave the prompt alone, particularly if it makes only one or two requests.

The same caching plan does not work unchanged across providers#

OpenAI's current Responses API on GPT-5.6 and later supports implicit caching and explicit breakpoints. A 30-minute entry is the normal choice for frequent agent turns. Twelve-hour coverage is available for stable material reused after longer waits, at a higher write price. Most of these models charge 0.10 times ordinary input for reads, while GPT-6.1 Sol and Terra currently charge 0.05 times. A cost spreadsheet built for one model's read rate should not silently become the price estimate for another. Reads can refresh the shorter coverage, while the twelve-hour lifetime has different refresh behavior. These are minimum eligibility periods, not deletion deadlines.

Anthropic's Messages API has a five-minute default and a one-hour option. The shorter lifetime refreshes when reused, but its clock starts when the request begins, which matters if a response streams for several minutes. Cache writes typically cost 1.25 times ordinary input for five minutes or twice ordinary input for one hour, with model-specific read discounts. You can place explicit cache_control markers at stable blocks or use automatic caching for a growing conversation. Anthropic also documents that a new cache entry only becomes available after the first response starts. Launching many identical requests simultaneously before the first one reaches that point should not be assumed to produce immediate cache hits. Anthropic's caching guide explains the lifetime and concurrency rules.

Gemini's Interactions API uses implicit caching, whereas explicit cache objects belong to supported GenerateContent models. Gemini 4 Argon and newer do not support that explicit caching path. Google documents a special multi-turn rule on Gemini 4: a new request must add at least 2,050 tokens to receive full previous-request caching, otherwise some recent content is reprocessed at the ordinary input rate. A small-agent workload with short turns may therefore have different economics from the access-review example even if the prefix is stable. Gemini's caching documentation provides the model-specific limits.

The common architecture principle is stable reusable material followed by changing task data. The implementation, minimum length, billing categories, and observed hit rate still need to be verified on the exact provider, model, and API in use.

A tool-schema rollout can create a fleet-wide cold start#

The previous examples follow one review. Now imagine 2,000 active review sessions across the service. A deployment changes a commonly used tool's schema, and the tool definitions are serialized near the beginning of every request. Each affected session now has a different prefix from the one it used earlier.

If all 2,000 sessions independently miss a 20,000-token cache prefix and write it again, that represents 40 million written tokens, or 50 million base-price units at a 1.25× write rate. If those same requests had instead been billed as 0.10× cached reads, that portion would cost 4 million units. The potential difference is 46 million normalized units. This is a deliberately severe illustrative scenario, not a prediction that OpenAI or another provider will bill every session for an independent write. Shared identical prefixes, routing, retained entries, and provider-side reuse can change the outcome substantially.

The more important failure mode is correlated work. A deployment can move many sessions from a warmed prompt version to a cold version at once. That may increase cache-write charges, input processing, and time to first token just when all your customers are active. It can also increase request load against token-per-minute limits, which OpenAI counts even for cached input. The total number of active sessions matters more here than the average hit rate from a quiet test run.

I would stage nonurgent tool-schema and prompt-template rollouts, track cost and time to first token by prompt version, and compare the first requests after rollout with the steady-state requests that follow. Supported OpenAI models offer a cache-prewarm option, but prewarming is itself a billable write. It makes sense only if a sufficient number of subsequent requests reuse the prepared prefix or if the latency benefit justifies that cost. OpenAI's prewarming guidance describes that trade-off.

There is an obvious exception. If the tool schema or instruction contains a security defect, fix it immediately. Protecting a cache-hit rate is not a reason to keep exposing an unsafe tool. The point of rollout controls is to make ordinary compatibility changes predictable and to help the team understand the expected cold-prefix cost when an urgent change is necessary.

Cache discounts and latency savings are different measurements#

Caching can reduce input processing time, but the amount that matters to a user depends on where the request spends its time. Imagine an agent model call that takes two seconds from request submission to the last output token. Suppose only 200 milliseconds of that time comes from the input-processing work that caching might accelerate, with the rest spent in routing, output generation, or other overhead. Even a 50 percent reduction in that 200-millisecond portion saves only 100 milliseconds, about 5 percent of the full call.

That is a hypothetical latency decomposition, not a measured provider result. A very large input with short output can have a much larger input-processing share, so caching may be more visible in time to first token. A task that spends most of its time waiting on identity-provider exports or approval tickets may barely move end-to-end latency even when model input becomes much cheaper.

For a real deployment, compare p50 and p95 time to first token, time to the last output token, tool latency, queue time, and total time to an accepted review. A model API may not expose its internal prefill duration directly, so don't pretend a time-to-first-token chart precisely measures that stage. Treat it as a useful end-user proxy, paired with usage and provider diagnostics.

This distinction also prevents a poor optimization. If longer cache retention reduces the model bill but causes a review to use outdated control evidence, the application has not improved. Policy and IAM state should be refreshed from their owners when needed, and any consequential action or final compliance assertion must use current authorized evidence, regardless of whether an older document prefix is still cheap to reuse.

Retention is part of the security decision#

A cached prefix can contain internal policies, private documents, or customer-specific context. Keeping that prefix eligible for twelve hours may save money, but it changes the data-handling discussion. OpenAI currently documents that prompt cache entries contain key-value representations rather than the raw prompt text, that caching is eligible for Zero Data Retention, and that caches are not shared across organizations or regional processing boundaries. The documentation also says manual cache clearing is not available. Entries expire under the applicable retention behavior, with a maximum documented retention period of 24 hours. OpenAI's data-retention FAQ makes these distinctions.

Removing a document from your retrieval index or deleting a conversation from the application's database does not cause a provider-side cache entry to be cleared immediately. You can stop including that material in subsequent requests, invalidate the application's references, enforce current permissions at the tool or action boundary, and follow the provider's retention contract. You should not promise a synchronous purge of a cache the provider does not let you purge.

Cache keys are not an authorization system either. OpenAI's prompt_cache_key can help separate accounting and reduce certain cross-user probing opportunities, but the application must still build its prompt from appropriately scoped data. The cache should never decide whether one tenant can read another tenant's evidence. A security review has to cover the ordinary request and retrieval paths before it considers retention or reuse settings.

Other providers have different caching and data-retention rules. Anthropic supports five-minute and one-hour cache entries with its own model-dependent billing and retention terms. Google's explicit caches, where supported, can incur storage-duration charges. Do not substitute one provider's TTL, deletion behavior, or privacy controls for another's without checking the specific model and endpoint.

Turn the provider usage into a cost report you can reproduce#

For supported newer OpenAI Responses models, usage distinguishes total input, cached reads, and cache writes. The write details further separate 30-minute and 12-hour writes. Those last two numbers add up to total writes and must not be charged again on top of the write total.

Here is the core of a small Python accounting function. It prices input only, with the rates passed in as dollar amounts per million input tokens for the model being measured:

def input_cost_usd(usage, prices):
    detail = usage["input_tokens_details"]
    writes = detail["cache_write_tokens_details"]
    read = detail["cached_tokens"]
    written = detail["cache_write_tokens"]
    write_short = writes["ttl_30m_tokens"]
    write_long = writes["ttl_12h_tokens"]
    ordinary = usage["input_tokens"] - read - written

    if ordinary < 0 or written != write_short + write_long:
        raise ValueError("Inconsistent token usage")

    return (
        ordinary * prices["ordinary"]
        + read * prices["cache_read"]
        + write_short * prices["cache_write_30m"]
        + write_long * prices["cache_write_12h"]
    ) / 1_000_000

The five-turn figures above come from a slightly longer version of this function that reads a small CSV of illustrative usage records, not raw responses from a live API. That version checks the accounting invariants before calculating totals, so changing the cache-read rate or adding an unexpected write makes the impact visible. The published rate card and actual response schema must be used before applying it to production billing.

Other providers require separate adapters. Anthropic's input_tokens field excludes separately reported cache creation and cache-read tokens, while OpenAI's input_tokens includes its categories. Treating these two fields as interchangeable leads to false savings estimates. Gemini has its own cached-usage reporting. A platform-level cost dashboard needs provider-specific normalization instead of one generic field called input_tokens.

Record the model, prompt-template version, tool-schema version, cache policy, ordinary input, cached reads, writes, output usage, total model cost, request latency, and logical task ID. Avoid copying prompt text or customer identifiers into high-cardinality monitoring labels. A versioned fingerprint of the intended shared prefix can help the team diagnose changes, but it does not prove which prefix the provider cached. Retain source evidence under your normal access controls if deeper debugging is required.

The measurement that ultimately matters is cost per accepted security review, not input cost per model call. Failed analyses, repeated tool calls, human corrections, and infrastructure usage belong in that number. The cache cost report explains where money went. The acceptance evaluation tells you whether that spending produced the correct evidence and findings.

Where I would optimize first#

For the access-review agent, the first experiment would be to keep the policy and tool prefix deterministic, then run a representative batch of reviews at realistic concurrency. Compare the intended prefix version with observed cache reads, writes, and diagnostics. Include the first request after a new tool version is deployed, not only warmed steady-state calls.

Next, measure how much history the agent actually needs after each investigation stage. Evidence that can be reopened from a durable artifact should not necessarily remain as thousands of lines in every model request. A compact record of the unresolved exceptions, evidence references, and verified decisions can retain the useful state while leaving the original exports and logs outside the context window. A compacted session may have a lower hit rate on its next call, yet still save cost and improve accuracy over the rest of the review. The fuller context-layer discussion on ArchCrux covers that separate design problem.

Only then would I evaluate longer cache retention. Segment reviews by reuse intervals and pause behavior. A shared rubric consulted continually may justify a different lifetime from case-specific evidence that is unlikely to be used again after an approval. Compare the 30-minute default, a 12-hour breakpoint where eligible, and the cost of rewriting after idle gaps. Include the cost of prewarming and the fraction of writes never read again. Model-specific minimum lengths and provider settings must be checked before porting the design to Anthropic or Gemini.

I would accept the change only if it lowers total cost per correct review without increasing stale-evidence errors, unauthorized access, or unacceptable p95 latency. A higher cache-hit rate by itself is not an acceptance criterion. The five-call example tells you why. The longer-run equation tells you when the problem becomes structural. At that point, the most useful question is not how much more context the provider can cache, but which evidence the next decision actually needs to see.

Primary references#