Two caches, never summed

When people say “the cache saved us money,” they are usually talking about one of two completely different things — and TokenTriage refuses to let those two things blur together. One is a discount the provider gives you for resending a stable prompt prefix. The other is a call TokenTriage never made at all because it had your answer already. Both are real savings. They are not the same saving, they are not measured the same way, and the load-bearing rule is that they are never summed.

This page explains the distinction so that when you read a dollar figure, you know which cache it came from and why the two are kept apart.

The two caches

The upstream provider’s prompt cache. When you resend the same long system prefix on every request, providers like Anthropic, OpenAI, and DeepSeek can reuse the computed attention state for that prefix instead of re-reading it at full price. You turn this on yourself, in the request body — Anthropic uses cache_control breakpoints, and other dialects have their own markers. The provider then reports, on its own usage, how many tokens were served from its cache. This is a discount on tokens the provider did process for you, on a call that did happen.

TokenTriage’s own response cache. Separately, TokenTriage can keep its own store of provider responses and serve a repeat request straight from that store — with no upstream call at all. This is not a token discount; it is an avoided call. The saving is a counterfactual: “had this not been cached, you would have paid to generate it again.” TokenTriage’s response cache is disclosed on every hit, byte-faithful, and its savings are licensed by evidence rather than asserted. It is documented in full in the caching guide; this page is only about how its accounting stays separate from the provider’s.

How each one shows up

The two vocabularies never overlap — that separation is a deliberate design decision, not an accident of naming.

Provider prompt cache. It appears as token classes on the provider’s reported usage: cache_read (tokens the provider served from its cache) and cache_write (tokens it charged you to fill the cache entry). Filling an entry is not free — the provider charges a write premium the first time, and only later reads collect the discount. TokenTriage prices these classes through the same cost engine as everything else and surfaces them in your ledger and cost views as provider-reported, priced spend.

There is a further wrinkle TokenTriage accounts for honestly: the cache-write premium is not a single rate. Anthropic (and Bedrock) split it by how long you asked the entry to live — a 5-minute write and a 1-hour write are billed differently, with the 1-hour entry carrying the higher premium. TokenTriage reads that split from the provider’s usage and prices each bucket at its own rate rather than folding them into one number.

TokenTriage response cache. It appears in a completely separate place: a response_cache block on the ledger row and a set of x-tt-cache-* response headers. A served hit makes no provider call, so the row carries zero fresh token buckets; the saving is stated as a counterfactual repriced from the original request’s stored usage, always labeled as an estimate and UNPRICED-safe. The licensed savings total sums hit rows only — never provider cache discounts, and never shadow “would-hit” projections.

The rule: never summed

Here is the trap. Both caches produce a “savings” number. It is tempting to add them and report one big figure. That figure would be wrong — it double-counts.

The two numbers answer different questions. “Your provider’s prompt cache saved you X” is a statement about a discount the provider applied to a real, billed call. “Your response cache saved you Y” is a statement about a call that never occurred. Adding X and Y would imply a single pool of avoided spend that does not exist. Worse, on a request that hit TokenTriage’s response cache, no provider call was made — so there is no provider usage to have earned a prompt-cache discount in the first place. The overlap is not just conceptually wrong; on a hit it is arithmetically nonsensical.

So TokenTriage keeps the two epistemic classes strictly separate, never summed. The dashboard shows them side by side, each labeled with what it is:

• provider-reported, priced — the provider’s own prompt cache, from cache_read / cache_write usage
• counterfactual, licensed — TokenTriage’s response cache, from the response_cache ledger block

Why this matters when you optimize

Because the two caches are different levers, you reach for them in different situations. The provider’s prompt cache is worth enabling when you resend a large, stable prefix often enough that repeated warm reads repay the write premium before the entry expires — a break-even that depends on how tightly your requests are spaced. TokenTriage’s response cache is worth enabling when identical or near- identical whole requests repeat, so the entire answer can be replayed. You can absolutely use both at once; you just account for them separately.

The cost optimization guide walks through when each lever pays off — including why TokenTriage will sometimes refuse to recommend prompt caching, quoting the write premium it would cost you rather than the discount it would never collect.