# Roll out a cache rule safely

Turn the response cache on without risking a wrong answer, by measuring the would-hit rate on live traffic before you promote a rule.

Source: https://tokentriage.com/docs/build/caching-rollout

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

A cache changes what your users see. Before you serve a single stored answer you want two numbers:
how often a repeat would have been found, and what it would have been worth. Both are available
without serving anything.

Cache rules move through three settings, in order: `off`, `shadow`, `on`. In `shadow` the lookup
runs in full and the ledger row records what it found, including the would-be saving, and then the
request forwards upstream byte for byte. Nothing is served from the store. That is the whole
on-ramp: run in shadow until the evidence justifies the change, then flip one rule.

This mirrors the engine's own [trust ladder](/docs/concepts/core-concepts), the `ledger → shadow →
route` progression the router uses, at rule scope. The two settings are separate, and a hit is
served only when both are at their last rung.

## Before you start

| You need | Why |
|---|---|
| a deployment whose config you can edit and restart | `cache.enabled` and `cache.store.*` are restart-required |
| `ledger.dimensions.enabled: true` | without it every caller shares one cache partition |
| admin access to `/tt/api/v1/cache/stats` and `/cache/evidence` | role floor `viewer`, scope `cache:read` |
| repeated traffic | a cache with no repeats has nothing to measure |

You do not need an embedder, a vector database, Redis, or any external service. The `exact` and
`normalized` tiers are pure Go and call nothing.

## 1. Add a cache block with the rule in shadow

The cache is off until you add the block. Start from the offline tiers, which need no embedder:

```yaml
cache:
  enabled: true                     # master gate; absent or false is a no-op
  store:
    l1: { max_entries: 4096 }       # bounded in-memory tier
    l2:
      path: /var/lib/tokentriage/respcache.db   # required for a durable tier
      max_total_mb: 512
      max_entry_kb: 1024            # must be < server.max_body_bytes
  scope: [tenant]                   # tenant|user|feature|trace|upstream-key
  budgets:
    lookup_exact_us: 500
    lookup_semantic_ms: 5
    judge_daily_usd: 1.0
  rules:
    - id: chat-cache
      match: { endpoint: chat }     # chat and responses are the cacheable endpoints
      mode: shadow                  # off | shadow | on
      strategy: [exact, normalized] # a chain, first match wins
      ttl: 1h
```

<Callout type="warning">
Leaving `l2.path` empty gives you an in-memory-only cache that is lost on restart, and no file is
created anywhere. The comment in the shipped examples describes a default path that the binary
does not apply. Set the path yourself. [The store](/docs/build/caching-store#the-l2-path-trap)
has the evidence.
</Callout>

Validate before you restart. Cache errors are path-named, so the message tells you which key to
edit:

```console
$ tokentriage validate --config tokentriage.yaml
mode:      route
listen:    127.0.0.1:8791   admin: 127.0.0.1:9191
upstreams: 1
  - mock         openai    http://127.0.0.1:18911
tiers:     1 (cheapest -> most capable)
  0. standard   -> mock/gpt-4o-mini
rules:     0
OK: configuration is valid
```

That capture is from a local sandbox pointed at a mock upstream, so the addresses and the tier
name are the sandbox's; yours will show your own. A `rules: 0` line counts *routing* rules, not
cache rules.

## 2. Drive traffic and read what the lookup found

With the rule in shadow, send your normal traffic. Every cacheable response carries
`x-tt-cache: miss`, because that is what the client received — a live passthrough. The ledger row
is where the interesting outcome lands:

```json
"response_cache": {
  "outcome": "would_hit",
  "kind": "exact",
  "rule": "chat-cache",
  "strategy": "exact",
  "entry_sha7": "3a2e388",
  "source_req_id": "tt_3fg5aqubovpekklypzj5fxq3de",
  "age_ms": 32,
  "scope_sha7": "799cda7",
  "saved": { "usd": 0.00039, "estimated": true, "unpriced": false },
  "saved_basis": "source_usage repriced @ litellm@2026-07-14#179affb, fixed usage",
  "stored": false,
  "serving_tier": "l1"
}
```

The header says `miss` and the row says `would_hit` on the same request. Both are correct: the
header describes what the client got, the row describes what the lookup found.

Then ask the admin plane for the totals:

```console
$ curl -sS "$TT_ADMIN/tt/api/v1/cache/stats"
{
  "outcomes": { "hit": 0, "would_hit": 2, "miss": 1, "bypass": 0, "error": 0 },
  "hits_by_kind": {},
  "savings": {
    "usd": 0,
    "estimated": true,
    "unpriced": false,
    "basis": "response_cache hit rows, saved repriced from source usage (hit-only)",
    "verdict_ref": "/tt/api/v1/cache/evidence"
  },
  "shadow_savings": {
    "usd": 0.00078,
    "estimated": true,
    "unpriced": false,
    "basis": "shadow-mode would-save, nothing avoided"
  }
}
```

Two figures, two basis strings, and they are kept apart. `savings` counts hit rows only,
so while the rule is in shadow it is `$0`: nothing was served, so nothing was avoided.
`shadow_savings` is the counterfactual. The provenance envelope on the real response is trimmed
here for length.

**What to look for.** A would-hit rate that justifies the change, and an age distribution on
`/cache/evidence` showing that the entries being matched are fresh enough for your content. If
your would-hit rate is near zero and you expected repeats,
[why didn't this request hit?](/docs/build/caching-misses) is the next stop, not a bigger TTL.

<Details summary="Estimating from history instead, before you can run shadow">

`tokentriage cache preview` reads your existing decision log and estimates what an exact-tier cache
would have caught. It carries three approximation labels it cannot drop, and the most important is
`fingerprint_approximated`: the ledger stores a messages-only prompt hash, not the full cache key,
so requests differing only in system prompt, tools or sampling parameters collide in the estimate
and would not share an entry in production.

```console
$ tokentriage cache preview --log ./decisions.jsonl
tokentriage cache preview — response-cache repeat-rate + would-save estimate

  measurement:    SECONDARY (labeled replay estimate over logged rows)
  primary:        shadow mode is the primary cache measurement (real lookups on live traffic, §2.7); this is the labeled replay estimate (DC15)
  kind:           response_cache_preview_estimate
  approximations: fixed_usage, recorded_runtime_state, fingerprint_approximated
  pricing:        litellm@2026-07-14#179affb
  window:         2026-08-10T03:22:17Z .. 2026-08-10T03:24:00Z
  rows scanned:   10   (keyable: 10, unkeyable/no-prompt-hash: 0)

EXACT tier (canonical-identical repeats)
  collision key:  prompt_sha256 + model + endpoint + param-class(max_tokens, reasoning, stream)
  distinct keys:  2   (the would-be cache entries)
  repeat rows:    8   <- would-be exact hits (UPPER BOUND, fingerprint_approximated)
  repeat rate:    80.0%   (repeat rows / keyable rows)
  would-save:     $0.002340   (repriced from recorded usage; fixed_usage)
                  unpriced repeats: 0 (counted, never $0); estimated: 2
```

Read it as a ceiling. On the sandbox that produced this transcript the same ten requests yielded
two served hits and three would-hits, not eight: the preview counted a request carrying tools, a
request that opted out, a request from a different tenant, and a request whose `temperature`
differed, none of which can share an entry. Shadow measures; this estimates.

</Details>

## 3. Promote one rule

Change the rule and reload:

```yaml
    - id: chat-cache
      mode: on                      # was: shadow
```

A hit needs both settings at their last rung: the rule at `mode: on` **and** the engine at
`mode: route`. Under engine `ledger` or `shadow` the rule still only records would-hits.

Rule changes are classified as hot: `cache.rules`, `cache.budgets` and `cache.embedders` reprogram
the live cache on reload. `cache.enabled`, `cache.scope` and everything under `cache.store` change
the store's identity and need a restart.

## 4. Confirm a hit, and that no upstream call happened

Send a request you know is a repeat and read the headers:

```console
$ curl -sS -D - -o /dev/null "$TT/v1/chat/completions" \
    -H 'content-type: application/json' -H 'x-tt-tenant: acme' \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is the capital of France?"}]}'
HTTP/1.1 200 OK
Content-Type: application/json
X-Tt-Cache: hit
X-Tt-Cache-Age: 102
X-Tt-Cache-Entry: 3a2e3889d0a87a520dce119a216f57b64631b2514a094c71d65f078a67452f23
X-Tt-Cache-Kind: exact
X-Tt-Cache-Source: tt_3fg5aqubovpekklypzj5fxq3de
X-Tt-Mode: route
X-Tt-Request-Id: tt_n67vihukwdsdhxma57ugkvzm74
```

`x-tt-cache-source` names the request whose response you are looking at. `x-tt-decision` is absent
because no upstream was chosen for this request. On the sandbox that produced the capture, the
mock upstream's own request counter did not move across two served hits.

Then check that the licensed figure has started moving, and that `/cache/evidence` reports the
kinds you expect as `licensed`.

## What can go wrong

**The would-hit rate is high but the served hit rate is lower.** Entries expire. The lookup treats
an entry past its TTL as a miss, so a shadow window measured against warm entries can overstate a
steady state with a short `ttl`.

**Nothing hits after a restart and you expected it to.** If `l2.path` is empty there is no durable
tier; the store is L1 only and starts cold every time.

**A rule you added does not seem to apply.** Rules are ordered and first-match, and the only
`match` key that narrows a rule is `endpoint`. A second rule sitting behind a rule with no
`match.endpoint` is unreachable. See [cache rules](/docs/build/caching-rules#what-match-can-and-cannot-narrow).

<NextSteps items={[
  {title: "Why didn't this request hit?", href: "/docs/build/caching-misses", description: "Work back from a miss you did not expect to the key segment that changed."},
  {title: "Operating the cache", href: "/docs/build/caching-operations", description: "The stats, evidence, preview and purge surfaces, and what purge can and cannot reach."},
  {title: "When a semantic hit is wrong", href: "/docs/build/caching-semantic", description: "Read this before adding the semantic tier to a rule you plan to promote."}
]} />