Turn on response caching safely

Enable TokenTriage’s response cache shadow-first — measure the would-hit rate and would-save on live traffic while serving nothing, confirm the evidence, then promote the rule to mode: on so a repeat drives a disclosed, byte-faithful replay and never a fabricated fresh generation.

Prerequisites

• A running TokenTriage deployment you can edit config for and reload. See Response caching.
• Business dimensions enabled (ledger.dimensions.enabled: true). Without them, scope: [tenant] collapses to one global partition and validate warns — the two shipped example configs enable dimensions for this reason.
• Admin-plane access to GET /tt/api/v1/cache/stats and GET /tt/api/v1/cache/evidence (role floor viewer, scope cache:read).

Steps

1. Confirm the cache is off, then add an opt-in block in shadow

The cache is OFF by default: an absent (or enabled: false) cache: block is byte-identical no-op behavior — nothing is stored, looked up, or dialed. You turn it on by adding a cache: section, and you keep it safe by shipping the rule in mode: shadow.

Add the offline, air-gapped default tier (exact + normalized matching, no embedder):

cache:
  enabled: true                    # master gate; default OFF (opt-in posture)

  store:
    l1: { max_entries: 4096 }      # bounded in-memory hot tier (RSS is capped)
    l2:                            # dedicated own-file SQLite tier (pure-Go driver)
      path: ""                     # "" ⇒ <data-root>/respcache.db — never the read-model DB
      max_total_mb: 512
      max_entry_kb: 1024           # MUST be < server.max_body_bytes (10 MiB default)

  scope: [tenant]                  # closed vocab: tenant|user|feature|trace|upstream-key

  budgets:
    lookup_exact_us: 500           # hot-path budgets; over budget ⇒ FAIL-OPEN to a live
    lookup_semantic_ms: 5          # forward (a cache slower than the LLM is pointless)
    judge_daily_usd: 1.0           # off-path explore-verification spend cap

  rules:                           # ordered, first-match
    - id: chat-cache
      match: { endpoint: chat }    # chat | responses only are cacheable in v1
      mode: shadow                 # off | shadow | on  — MEASURE before you serve
      strategy: [exact, normalized]
      ttl: 1h

mode is a closed enum — off | shadow | on — validated at load; anything else is a path-named error (cache.rules[0].mode: … is not one of off|shadow|on). This mirrors the committed examples/cache/tokentriage.cache.yaml.

Validate before you reload:

$ tokentriage validate --config tokentriage.yaml
OK: configuration is valid

2. Drive real traffic and read the evidence

With the rule in shadow, send your normal traffic. Every cache-eligible response carries x-tt-cache: miss (the client received a live passthrough), while the ledger row records the would_hit outcome and the shadow would-save under its own basis string. Then ask the two admin surfaces whether it is worth turning on.

GET /tt/api/v1/cache/stats — outcome counts, per-kind hits, and the hit-only licensed savings beside the separate shadow figure:

$ curl -sS -H "Authorization: Bearer $TOKENTRIAGE_ADMIN_TOKEN" \
    "$TOKENTRIAGE_ADMIN_URL/tt/api/v1/cache/stats"

You should see (illustrative):

{
  "outcomes": { "hit": 0, "would_hit": 3184, "miss": 9127, "bypass": 41, "error": 0 },
  "hits_by_kind": {},
  "savings": {
    "usd": 0,
    "estimated": true,
    "unpriced": false,
    "basis": "response_cache hit rows, saved repriced from source usage (hit-only)",
    "verdict_ref": "/tt/api/v1/cache/evidence"
  },
  "shadow_savings": {
    "usd": 12.4718,
    "estimated": true,
    "unpriced": false,
    "basis": "shadow-mode would-save, nothing avoided",
    "verdict_ref": "/tt/api/v1/cache/evidence"
  }
}

Note the honesty seam: while the rule is in shadow, outcomes.hit is 0, so the licensed savings is $0 flagged estimated: true — an honest not-yet-measured zero, never a confident priced $0. Nothing was served, so nothing was saved. The would_hit count and shadow_savings are the counterfactual, and they carry a distinct basis string precisely so they can never be added into the licensed figure. (The real response also carries a top-level provenance envelope describing scan coverage; abbreviated here.)

GET /tt/api/v1/cache/evidence — the authority a savings verdict_ref points at: per-kind hit counts, age distribution, and (for semantic rules) the measured posture, which can honestly say cannot_measure rather than guess.

3. Promote to mode on

Once the would-hit rate and would-save justify it, flip the rule and confirm the engine is in route:

cache:
  rules:
    - id: chat-cache
      match: { endpoint: chat }
      mode: on                     # was: shadow
      strategy: [exact, normalized]
      ttl: 1h

cache.rules.* is a hot-reload change — it reprograms the live cache without rebuilding the store. (cache.enabled, cache.scope, and cache.store.* are restart-required — they change the store’s identity.) Now a repeat drives a disclosed, byte-faithful replay.

You should see, on a served hit:

HTTP/1.1 200 OK
x-tt-cache: hit
x-tt-cache-kind: exact
x-tt-cache-age: 421
x-tt-cache-source: req_9f3c1e0a
x-tt-cache-entry: 3f2a1c9e8b7d6a5c4e3f2a1c9e8b7d6a5c4e3f2a1c9e8b7d6a5c4e3f2a1c9e8b

And the ledger row now carries a decision.response_cache block with outcome: hit, the source request id, and a typed saved figure repriced from the source row’s usage.

Disclosure headers you will see

On every cache-eligible request the proxy sets x-tt-cache; the richer headers appear on hits. They describe what the client received, so a shadow would_hit row pairs with x-tt-cache: miss (the response was a live passthrough).

Header	On	Value
x-tt-cache	always (eligible)	hit | miss | bypass | error
x-tt-cache-kind	hit	exact | normalized | semantic
x-tt-cache-age	hit	age of the stored entry, seconds
x-tt-cache-source	hit	the req_id of the request whose response this is
x-tt-cache-entry	hit	the entry id (the per-entry purge handle)
x-tt-cache-similarity	semantic hit	the match score
x-tt-cache-threshold	semantic hit	the acceptance bar the score cleared
x-tt-cache-verified	semantic hit	true | false | none (no verifier ran)

Clients can also send x-tt-cache: off (bypass read + write), refresh (bypass read, force a fresh write), or no-store (serve hits, skip write). These are TokenTriage control headers and are stripped before the request is forwarded upstream — the provider never sees them. See Response headers for the full reference.

Semantic caching: add the verified-savings guard

A semantic hit is a probabilistic guess — the cached answer to a similar request, not the same one. Read this before you promote a semantic rule. Semantic caching is doubly opt-in: the strategy must include semantic and name a configured embedder. The air-gapped static embedder embeds in-process and dials nothing:

cache:
  embedders:
    potion: { kind: static }       # in-process Model2Vec-method table — AIR-GAPPED

  rules:
    - id: chat-hybrid
      match: { endpoint: chat }
      mode: shadow
      strategy: [exact, normalized, semantic]
      ttl: 15m
      semantic:
        embedder: potion
        threshold: 0.95            # >= ⇒ confident hit
        low_threshold: 0.85        # < ⇒ miss; [0.85,0.95) is the verifier-gated gray band
        verifier: kreciprocal      # model-free mutual-kNN sync verifier (none|kreciprocal|judge)
        history_max_messages: 3    # eligibility gate: skip deeper-history requests
        verified:                  # the verified-savings guard (opt-in)
          delta: 0.05              # operator wrong-hit budget δ ∈ (0,1)
          min_samples: 200         # cold-start label floor
          confidence: 0.95         # Wilson two-sided confidence

The verified: block is the guard that keeps a wrong semantic hit from ever being credited as a saving. Even with a semantic rule in mode: on, semantic savings are excluded-and-counted — the hit is counted but its dollar withheld — until a verified policy has cleared your wrong-hit budget:

• Below min_samples labels, the policy refuses: GET /tt/api/v1/cache/evidence reports posture: cannot_measure, “static threshold stands”, and the semantic saving is excluded (no dollar) — stats.savings.excluded_semantic discloses how many.
• Once the calibrated bound clears 1 − delta, evidence reads posture: licensed with the calibrated threshold T* and the Wilson bound, and the saving counts.