Response caching

Some of your traffic is the same question asked again. A support assistant answering “how do I reset my password” a thousand times a day sends a thousand identical requests upstream, and you pay for a thousand generations of one answer.

A response cache stores the answer the first time and returns the stored copy afterwards. The upstream call does not happen, so the money is not spent. That makes caching the one cost lever whose effect is directly measurable: a routing decision saves money you would have spent on a model you did not use, which is a counterfactual, but a call that never left the building either happened or it didn’t.

The hard part is not storing the answer. It’s knowing when the stored answer is still the right one, and being able to say how much you actually saved.

What you get, and what it costs you to find out

TokenTriage’s response cache is off out of the box. An absent cache: block, or one with enabled: false, changes nothing about how a request is handled: no lookup runs, no header is set, and the decision ledger row carries no cache fields at all. You turn it on by adding a block, and the shipped starting point puts every rule in mode: shadow, which measures what the cache would have done without serving anything.

Three tiers of matching are available, and they are not equally safe. Two of them are deterministic: a request either produces the same cache key as an earlier one or it doesn’t. The third matches on similarity, and a similar question is not the same question. That distinction runs through this whole section.

Read the diagram left to right. Every cacheable request runs a scoped lookup; the top branch is the served replay, the bottom is the forward that may write the store. In shadow the lookup still runs and the row records what it found, but the request forwards anyway.

What a hit is

A hit is a replay of the bytes a client received earlier, plus a disclosure of where those bytes came from. Three properties hold on every one.

It is disclosed. A served hit carries x-tt-cache: hit and four more x-tt-cache-* headers naming the match kind, the entry’s age, the entry id, and the request whose response this is. A semantic hit carries three more: the similarity score, the threshold it cleared, and the verifier’s verdict. The same facts land on the ledger row as decision.response_cache. There is no configuration that turns the headers off.

It is byte-faithful. The stored bytes are the ones the client saw, written back verbatim. A streamed response replays frame by frame, with a flush after each frame and no invented pacing. The replayed body still contains the provider’s original usage object, because nothing is edited into it. Disclosure rides the headers and the ledger instead.

It is not free money by default. A hit row records usage.source: "replay" with zero token buckets, so replayed tokens can’t be double-counted as fresh spend. The saving is a separate, typed figure: the source row’s recorded usage repriced under the current pricing snapshot, always labelled an estimate, and reported with the basis string that says how it was computed. Semantic hits are counted but their dollars withheld until evidence licenses them.

Here is one of those rows, from a request served out of the durable tier:

"response_cache": {
  "outcome": "hit",
  "kind": "exact",
  "rule": "chat-cache",
  "strategy": "exact",
  "entry_sha7": "3a2e388",
  "source_req_id": "tt_3fg5aqubovpekklypzj5fxq3de",
  "age_ms": 102885,
  "scope_sha7": "799cda7",
  "stored_provider": "openai",
  "stored_model": "gpt-4o-mini",
  "source_usage": { "input_tokens": 1200, "output_tokens": 350, "total_tokens": 1550 },
  "saved": { "usd": 0.00039, "estimated": true, "unpriced": false },
  "saved_basis": "source_usage repriced @ litellm@2026-07-14#179affb, fixed usage",
  "stored": false,
  "lookup_us": 518,
  "serving_tier": "l2"
}

That block was produced by a local run against a mock upstream, so the token counts are the mock’s and $0.00039 is arithmetic you can check: 1,200 input tokens at $0.15 per million plus 350 output tokens at $0.60 per million. It is nobody’s bill. The usage block on the same row reads all zeros with source: "replay".

The two caches, kept apart

Your provider has a cache too. Anthropic, OpenAI and Google all discount the input tokens of a prompt they have seen recently, and report the discount back as cache_read_tokens. TokenTriage prices those tokens and shows them, but it does not own that cache.

The response cache on this page is a different thing: TokenTriage’s own store, serving an answer with no upstream call at all. The provider’s cache makes a call cheaper. This one removes the call.

What this section does not cover

It does not cover routing’s cache_affinity, which pins a trace to the upstream that already holds its prompt cache. Similar name, different mechanism; see routing limits. It does not define the savings verdict vocabulary or the UNPRICED rule, which belong to the honesty model, and it does not own the cost reporting surfaces, which are Cost & attribution’s.

Where to go from here

If you want the cache on, start with the rollout page and come back. If you are debugging a cache that isn’t hitting, the key is almost always the answer. If you are deciding whether to enable the semantic tier, read the correctness page before the config.