# Response caching

Answer a repeated request from a stored response instead of paying for it twice, and know whether the saving you report is measurable.

Source: https://tokentriage.com/docs/build/caching

---
import Callout from '../../../components/Callout.astro';
import CacheFlowDiagram from '../../../components/CacheFlowDiagram.astro';
import NextSteps from '../../../components/NextSteps.astro';

Some of your traffic is the same question asked again. A support assistant answering "how do I
reset my password" a thousand times a day sends a thousand identical requests upstream, and you
pay for a thousand generations of one answer.

A response cache stores the answer the first time and returns the stored copy afterwards. The
upstream call does not happen, so the money is not spent. That makes caching the one cost lever
whose effect is directly measurable: a routing decision saves money you would have spent on a
model you did not use, which is a counterfactual, but a call that never left the building either
happened or it didn't.

The hard part is not storing the answer. It's knowing when the stored answer is still the right
one, and being able to say how much you actually saved.

## What you get, and what it costs you to find out

TokenTriage's response cache is off out of the box. An absent `cache:` block, or one with
`enabled: false`, changes nothing about how a request is handled: no lookup runs, no header is
set, and the decision ledger row carries no cache fields at all. You turn it on by adding a
block, and the shipped starting point puts every rule in `mode: shadow`, which measures what the
cache *would* have done without serving anything.

Three tiers of matching are available, and they are not equally safe. Two of them are
deterministic: a request either produces the same cache key as an earlier one or it doesn't. The
third matches on similarity, and a similar question is not the same question. That distinction
runs through this whole section.

<CacheFlowDiagram />

Read the diagram left to right. Every cacheable request runs a scoped lookup; the top branch is
the served replay, the bottom is the forward that may write the store. In shadow the lookup
still runs and the row records what it found, but the request forwards anyway.

## What a hit is

A hit is a replay of the bytes a client received earlier, plus a disclosure of where those bytes
came from. Three properties hold on every one.

**It is disclosed.** A served hit carries `x-tt-cache: hit` and four more `x-tt-cache-*` headers
naming the match kind, the entry's age, the entry id, and the request whose response this is. A
semantic hit carries three more: the similarity score, the threshold it cleared, and the
verifier's verdict. The same facts land on the ledger row as `decision.response_cache`. There is
no configuration that turns the headers off.

**It is byte-faithful.** The stored bytes are the ones the client saw, written back verbatim.
A streamed response replays frame by frame, with a flush after each frame and no invented
pacing. The replayed body still contains the provider's original `usage` object, because nothing
is edited into it. Disclosure rides the headers and the ledger instead.

**It is not free money by default.** A hit row records `usage.source: "replay"` with zero token
buckets, so replayed tokens can't be double-counted as fresh spend. The *saving* is a separate,
typed figure: the source row's recorded usage repriced under the current pricing snapshot, always
labelled an estimate, and reported with the basis string that says how it was computed. Semantic
hits are counted but their dollars withheld until evidence licenses them.

Here is one of those rows, from a request served out of the durable tier:

```json
"response_cache": {
  "outcome": "hit",
  "kind": "exact",
  "rule": "chat-cache",
  "strategy": "exact",
  "entry_sha7": "3a2e388",
  "source_req_id": "tt_3fg5aqubovpekklypzj5fxq3de",
  "age_ms": 102885,
  "scope_sha7": "799cda7",
  "stored_provider": "openai",
  "stored_model": "gpt-4o-mini",
  "source_usage": { "input_tokens": 1200, "output_tokens": 350, "total_tokens": 1550 },
  "saved": { "usd": 0.00039, "estimated": true, "unpriced": false },
  "saved_basis": "source_usage repriced @ litellm@2026-07-14#179affb, fixed usage",
  "stored": false,
  "lookup_us": 518,
  "serving_tier": "l2"
}
```

That block was produced by a local run against a mock upstream, so the token counts are the
mock's and `$0.00039` is arithmetic you can check: 1,200 input tokens at $0.15 per million plus
350 output tokens at $0.60 per million. It is nobody's bill. The `usage` block on the same row
reads all zeros with `source: "replay"`.

## The two caches, kept apart

Your provider has a cache too. Anthropic, OpenAI and Google all discount the input tokens of a
prompt they have seen recently, and report the discount back as `cache_read_tokens`. TokenTriage
prices those tokens and shows them, but it does not own that cache.

The response cache on this page is a different thing: TokenTriage's own store, serving an answer
with no upstream call at all. The provider's cache makes a call cheaper. This one removes the
call.

<Callout type="warning">
The two figures are computed over different populations and must not be added together. A
provider prompt-cache discount is money the provider says you didn't pay on a call that
happened; a response-cache saving is a call that didn't happen. Adding them counts one request
twice. [Cache accounting](/docs/concepts/cache-accounting) is where that rule is stated and
worked through.
</Callout>

## What this section does not cover

It does not cover routing's `cache_affinity`, which pins a trace to the upstream that already
holds its *prompt* cache. Similar name, different mechanism; see
[routing limits](/docs/build/routing-limits). It does not define the savings verdict vocabulary
or the `UNPRICED` rule, which belong to [the honesty model](/docs/concepts/honesty-model), and it
does not own the cost reporting surfaces, which are
[Cost & attribution](/docs/build/cost-analytics)'s.

## Where to go from here

If you want the cache on, start with the rollout page and come back. If you are debugging a cache
that isn't hitting, the key is almost always the answer. If you are deciding whether to enable
the semantic tier, read the correctness page before the config.

<NextSteps items={[
  {title: "Roll out a cache rule safely", href: "/docs/build/caching-rollout", description: "Measure the would-hit rate in shadow, then promote one rule to serving."},
  {title: "How a request becomes a cache key", href: "/docs/build/caching-keys", description: "The seven fingerprint segments and the scope prefix that decide whether two requests share an entry."},
  {title: "When a semantic hit is wrong", href: "/docs/build/caching-semantic", description: "What similarity matching can get wrong, and the four bounds that keep it in range."},
  {title: "What a cache saving means", href: "/docs/build/caching-savings", description: "Why hit rows license a dollar, would-hits don't, and semantic hits wait for evidence."}
]} />