Roll out a cache rule safely

A cache changes what your users see. Before you serve a single stored answer you want two numbers: how often a repeat would have been found, and what it would have been worth. Both are available without serving anything.

Cache rules move through three settings, in order: off, shadow, on. In shadow the lookup runs in full and the ledger row records what it found, including the would-be saving, and then the request forwards upstream byte for byte. Nothing is served from the store. That is the whole on-ramp: run in shadow until the evidence justifies the change, then flip one rule.

This mirrors the engine’s own trust ladder, the ledger → shadow → route progression the router uses, at rule scope. The two settings are separate, and a hit is served only when both are at their last rung.

Before you start

You do not need an embedder, a vector database, Redis, or any external service. The exact and normalized tiers are pure Go and call nothing.

1. Add a cache block with the rule in shadow

The cache is off until you add the block. Start from the offline tiers, which need no embedder:

cache:
  enabled: true                     # master gate; absent or false is a no-op
  store:
    l1: { max_entries: 4096 }       # bounded in-memory tier
    l2:
      path: /var/lib/tokentriage/respcache.db   # required for a durable tier
      max_total_mb: 512
      max_entry_kb: 1024            # must be < server.max_body_bytes
  scope: [tenant]                   # tenant|user|feature|trace|upstream-key
  budgets:
    lookup_exact_us: 500
    lookup_semantic_ms: 5
    judge_daily_usd: 1.0
  rules:
    - id: chat-cache
      match: { endpoint: chat }     # chat and responses are the cacheable endpoints
      mode: shadow                  # off | shadow | on
      strategy: [exact, normalized] # a chain, first match wins
      ttl: 1h

Validate before you restart. Cache errors are path-named, so the message tells you which key to edit:

$ tokentriage validate --config tokentriage.yaml
mode:      route
listen:    127.0.0.1:8791   admin: 127.0.0.1:9191
upstreams: 1
  - mock         openai    http://127.0.0.1:18911
tiers:     1 (cheapest -> most capable)
  0. standard   -> mock/gpt-4o-mini
rules:     0
OK: configuration is valid

That capture is from a local sandbox pointed at a mock upstream, so the addresses and the tier name are the sandbox’s; yours will show your own. A rules: 0 line counts routing rules, not cache rules.

2. Drive traffic and read what the lookup found

With the rule in shadow, send your normal traffic. Every cacheable response carries x-tt-cache: miss, because that is what the client received — a live passthrough. The ledger row is where the interesting outcome lands:

"response_cache": {
  "outcome": "would_hit",
  "kind": "exact",
  "rule": "chat-cache",
  "strategy": "exact",
  "entry_sha7": "3a2e388",
  "source_req_id": "tt_3fg5aqubovpekklypzj5fxq3de",
  "age_ms": 32,
  "scope_sha7": "799cda7",
  "saved": { "usd": 0.00039, "estimated": true, "unpriced": false },
  "saved_basis": "source_usage repriced @ litellm@2026-07-14#179affb, fixed usage",
  "stored": false,
  "serving_tier": "l1"
}

The header says miss and the row says would_hit on the same request. Both are correct: the header describes what the client got, the row describes what the lookup found.

Then ask the admin plane for the totals:

$ curl -sS "$TT_ADMIN/tt/api/v1/cache/stats"
{
  "outcomes": { "hit": 0, "would_hit": 2, "miss": 1, "bypass": 0, "error": 0 },
  "hits_by_kind": {},
  "savings": {
    "usd": 0,
    "estimated": true,
    "unpriced": false,
    "basis": "response_cache hit rows, saved repriced from source usage (hit-only)",
    "verdict_ref": "/tt/api/v1/cache/evidence"
  },
  "shadow_savings": {
    "usd": 0.00078,
    "estimated": true,
    "unpriced": false,
    "basis": "shadow-mode would-save, nothing avoided"
  }
}

Two figures, two basis strings, and they are kept apart. savings counts hit rows only, so while the rule is in shadow it is $0: nothing was served, so nothing was avoided. shadow_savings is the counterfactual. The provenance envelope on the real response is trimmed here for length.

What to look for. A would-hit rate that justifies the change, and an age distribution on /cache/evidence showing that the entries being matched are fresh enough for your content. If your would-hit rate is near zero and you expected repeats, why didn’t this request hit? is the next stop, not a bigger TTL.

$ tokentriage cache preview --log ./decisions.jsonl
tokentriage cache preview — response-cache repeat-rate + would-save estimate

  measurement:    SECONDARY (labeled replay estimate over logged rows)
  primary:        shadow mode is the primary cache measurement (real lookups on live traffic, §2.7); this is the labeled replay estimate (DC15)
  kind:           response_cache_preview_estimate
  approximations: fixed_usage, recorded_runtime_state, fingerprint_approximated
  pricing:        litellm@2026-07-14#179affb
  window:         2026-08-10T03:22:17Z .. 2026-08-10T03:24:00Z
  rows scanned:   10   (keyable: 10, unkeyable/no-prompt-hash: 0)

EXACT tier (canonical-identical repeats)
  collision key:  prompt_sha256 + model + endpoint + param-class(max_tokens, reasoning, stream)
  distinct keys:  2   (the would-be cache entries)
  repeat rows:    8   <- would-be exact hits (UPPER BOUND, fingerprint_approximated)
  repeat rate:    80.0%   (repeat rows / keyable rows)
  would-save:     $0.002340   (repriced from recorded usage; fixed_usage)
                  unpriced repeats: 0 (counted, never $0); estimated: 2

3. Promote one rule

Change the rule and reload:

    - id: chat-cache
      mode: on                      # was: shadow

A hit needs both settings at their last rung: the rule at mode: on and the engine at mode: route. Under engine ledger or shadow the rule still only records would-hits.

Rule changes are classified as hot: cache.rules, cache.budgets and cache.embedders reprogram the live cache on reload. cache.enabled, cache.scope and everything under cache.store change the store’s identity and need a restart.

4. Confirm a hit, and that no upstream call happened

Send a request you know is a repeat and read the headers:

$ curl -sS -D - -o /dev/null "$TT/v1/chat/completions" \
    -H 'content-type: application/json' -H 'x-tt-tenant: acme' \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is the capital of France?"}]}'
HTTP/1.1 200 OK
Content-Type: application/json
X-Tt-Cache: hit
X-Tt-Cache-Age: 102
X-Tt-Cache-Entry: 3a2e3889d0a87a520dce119a216f57b64631b2514a094c71d65f078a67452f23
X-Tt-Cache-Kind: exact
X-Tt-Cache-Source: tt_3fg5aqubovpekklypzj5fxq3de
X-Tt-Mode: route
X-Tt-Request-Id: tt_n67vihukwdsdhxma57ugkvzm74

x-tt-cache-source names the request whose response you are looking at. x-tt-decision is absent because no upstream was chosen for this request. On the sandbox that produced the capture, the mock upstream’s own request counter did not move across two served hits.

Then check that the licensed figure has started moving, and that /cache/evidence reports the kinds you expect as licensed.

What can go wrong

The would-hit rate is high but the served hit rate is lower. Entries expire. The lookup treats an entry past its TTL as a miss, so a shadow window measured against warm entries can overstate a steady state with a short ttl.

Nothing hits after a restart and you expected it to. If l2.path is empty there is no durable tier; the store is L1 only and starts cold every time.

A rule you added does not seem to apply. Rules are ordered and first-match, and the only match key that narrows a rule is endpoint. A second rule sitting behind a rule with no match.endpoint is unreachable. See cache rules.