What a cache saving means

A saving is a claim about money you did not spend, and money you did not spend leaves no receipt. Every such figure therefore rests on something, and the useful question about any of them is what.

For the response cache the answer is unusually good. A hit is a call that did not go upstream, and the call it replaced actually happened once, with real token counts recorded. So the claim is: this request would have cost what the identical earlier request cost.

That is still an estimate, and this page is about which parts of it are solid and which are not.

How the figure is computed

On a hit, the stored entry carries the source request’s usage. That usage is repriced under the current pricing snapshot for the stored provider and model, through the same path every other dollar in the product travels.

"source_usage": { "input_tokens": 1200, "output_tokens": 350, "total_tokens": 1550 },
"saved": { "usd": 0.00039, "estimated": true, "unpriced": false },
"saved_basis": "source_usage repriced @ litellm@2026-07-14#179affb, fixed usage"

That row came from a sandbox against a mock upstream, so the tokens are the mock’s and the dollar is arithmetic you can check: 1,200 × $0.15/M + 350 × $0.60/M.

Three things about it:

It is always an estimate. estimated: true is not a hedge about the arithmetic. It says the token counts are the earlier request’s, held fixed. A fresh generation would have produced a different number of output tokens; nobody knows how many.

It uses today’s prices, not the prices at store time. The basis string names the snapshot, so a figure is re-derivable.

It is unpriced-safe. If the stored provider and model do not resolve to a price, the field records unpriced: true and makes no dollar claim. It does not report $0. The count of such hits is disclosed separately as unpriced_hits.

The hit row’s own cost is zero, and that is a different statement: usage.source is replay with zero token buckets, so replayed tokens are never counted as fresh spend anywhere.

Which rows may contribute

Three populations, kept apart.

Population	Contributes to the licensed figure	Why
hit rows	yes	the call did not happen
would_hit rows	no	the request forwarded and was paid for
semantic hit rows without a calibrated policy	no, counted separately	nothing has measured the wrong-hit rate

The stats surface reports the first two under separate basis strings and does not add them:

"savings": {
  "usd": 0,
  "estimated": true,
  "basis": "response_cache hit rows, saved repriced from source usage (hit-only)",
  "verdict_ref": "/tt/api/v1/cache/evidence"
},
"shadow_savings": {
  "usd": 0.00078,
  "estimated": true,
  "basis": "shadow-mode would-save, nothing avoided"
}

The $0 licensed figure above is from a rule still in shadow. It is a not-yet-measured zero, and estimated: true beside it says so. A shadow would-save is what the cache would have avoided; the money was spent.

Both figures carry a verdict_ref pointing at the evidence surface, and a savings value that lacks its basis or that reference cannot be serialised at all — the API refuses to emit an unlabelled cache dollar.

The semantic exclusion

A semantic hit’s saving appears on its row and does not appear in the licensed total.

"savings": { "usd": 0, "excluded_semantic": 1, "basis": "…(hit-only)" }

The excluded_semantic count is the disclosure: one hit occurred, its dollar was withheld. The evidence surface says why — either no calibration policy is configured, or one is and has not accumulated enough labels to bound the error rate. Either way it reads cannot_measure with a reason rather than producing a smaller confident number.

Once a calibrated policy licenses the kind, the evidence row switches to licensed and reports the effective threshold, the lower bound on the correct-serve rate, and the sample count behind it, so the decision can be re-derived. When a semantic hit is wrong covers the calibration itself.

The two caches are not summed

The diagram separates them: on the left the provider’s prompt cache discounting a call that happened, on the right this cache removing a call that would have happened.

Both produce a dollar and the dollars mean different things. A prompt-cache discount is provider-reported, priced from tokens on your bill. A response-cache saving is a counterfactual over a request that produced no bill at all. Adding them charges one request to two ledgers.

Cache accounting is the canonical statement of that rule, with the arithmetic worked through. Nothing on this page or anywhere in this section adds the two.

What this costs to produce

Semantic caching’s embeddings are spend, and a metered embedder’s charge lands on the row as embed_cost and joins the metacost surface. On a rule whose semantic tier is not yet licensed to report savings, that is spend against a saving of zero.

The measurement’s own cost is Cost & attribution’s subject; this section owns where the number is produced.

Where this figure appears elsewhere

The cache stats and evidence surfaces are the authority. Downstream, the same population feeds the savings verdict vocabulary as cache-licensed-replay-v1, which is the method name a cost surface prints beside a claimed cache dollar. Verdicts, methods and baselines is where that vocabulary is defined; the reason a cache figure earns its strongest class is the replay evidence described on this page.

What can go wrong

A finance-facing total silently excludes rows. Quote the exclusions with the number: unpriced hits and excluded semantic hits are both reported and both mean the total is smaller than the row count implies.

A big shadow figure gets forwarded as a saving. It is labelled, and labels get lost in screenshots. Say “would have saved” out loud when you quote it.

The figure changes without traffic changing. The reprice runs under the current snapshot, so a pricing update moves it. That is intended; the basis string names the snapshot each figure used.