GitHubBook a demoStart routing
Browse docs

Response caching

Serve a stored provider response instead of forwarding — off by default, shadow-first, byte-faithful, and honestly costed. The exact/normalized and semantic tiers, the store tiers, and the embedder ladder.

On this page

TokenTriage can serve a stored provider response instead of forwarding it — and it’s built so that a hit is honest by construction. It is off by default: an absent (or enabled: false) cache: block is byte-identical no-op behavior.

The three properties of a hit

  • Disclosed. Every served hit carries x-tt-cache: hit plus a decision.response_cache block on the ledger row. The headers have no off switch. A cached response is never presented as fresh, and a semantic hit is never presented as exact (x-tt-cache-kind + -similarity + -threshold + -verified).
  • Byte-faithful. A hit is a verbatim replay of the stored client-visible bytes (streaming: frame-by-frame with a flush after each frame, no fabricated pacing). Disclosure rides the headers and ledger, never a body mutation.
  • Honestly costed. The hit row is cost-exempt (usage.source: "replay", no magic $0); the saving is a typed counterfactual repriced from the source row’s recorded usage, always estimated and UNPRICED-safe, summed over hit rows only. The cache’s own embedding spend is counted on the row and a metacost line — and the licensed aggregate never merges with the would-hit (shadow) figure.

Shadow-first

Cache rules climb the same ladder the router uses: mode: off ──▶ shadow ──▶ on. A hit is served only when the rule’s mode: on and the engine’s effective mode is route. Under engine ledger/shadow, or a rule in mode: shadow, the lookup runs fully and the row records a would_hit — but the request forwards byte-identically and nothing is served. You measure the would-hit rate before you ever serve one.

The response-cache request flowA scoped cache lookup (exact, normalized or semantic) runs on every cacheable chat or responses request. On a hit, when the rule is mode on and the engine is in route mode, TokenTriage serves a disclosed byte-faithful replay tagged x-tt-cache hit that is cost-exempt, with a counterfactual estimated saving. On a miss it forwards to the upstream and may write the store. In shadow or off mode the lookup still runs and records a would_hit, but the request forwards anyway — nothing is served.Requestchat · responsesendpoints onlyCache lookupscope: tenant / user / …exact · normalized · semanticHit → byte-faithful replayx-tt-cache: hit · cost-exemptsaving: counterfactual (estimated)Miss → forward upstreamthen maybe write the storehit · servedmissshadow / off modelookup runs · records would_hit · forwards anyway (nothing served)measure first

The strategies

The strategy: chain picks one or more matchers (only chat and responses endpoints are cacheable in v1):

  • exact / normalized — the air-gapped tiers. Deterministic, pure-Go, and they dial nothing; they need no embedder. exact keys on a content fingerprint; normalized canonicalizes first.
  • semantic — a probabilistic tier: the cached answer to a similar request, gated by embedding cosine similarity with dual thresholds and a gray-zone verifier. It requires a configured embedder and is bounded four ways — eligibility gates (no tool calls, no response_format, no binary streams, bounded history), the dual thresholds + verifier, disclosed ANN recall, and evidence-gated savings.

Store tiers and embedders

The store (cache.store) has up to three tiers, disclosed as serving_tier: l1 (bounded in-memory hot tier, max_entries: 4096), l2 (a dedicated own-file SQLite tier via the pure-Go driver — max_total_mb: 512, max_entry_kb must be strictly less than server.max_body_bytes), and l3 (opt-in Redis — an air-gap phone-out, gated). The embedder ladder (semantic only) offers static (recommended) and lexical — both in-process, pure-Go, dial nothing — plus http local-network and external-API tiers that require external: allow and are disclosed and metered (external is real spend). Cache scope is a closed vocabulary: tenant | user | feature | trace | upstream-key (and needs ledger.dimensions.enabled for real isolation).

cache:
  enabled: true
  scope: [tenant]                     # cross-tenant serving is structurally impossible
  rules:
    - id: chat-cache
      match: { endpoint: chat }
      mode: shadow                    # off | shadow | on — measure before you serve
      strategy: [exact, normalized]   # add `semantic` for the opt-in, bounded tier
      ttl: 1h

Related: The honesty model · Cost optimization (the cache-opportunity recommendation).