Cache rules

A cache rule decides three things about a request: whether the cache engages at all, which matchers run and in what order, and how long an entry it writes stays valid. Everything else about the cache is global.

This page is the field list. For the on-ramp see rolling out a rule; for what each matcher matches on, see the three matchers.

The block

cache:
  enabled: true                      # default false — the whole subsystem is opt-in
  store: { }                         # see /docs/build/caching-store
  scope: [tenant]                    # see /docs/build/caching-scope
  budgets: { }                       # see /docs/build/caching-performance
  embedders: { }                     # see /docs/build/caching-embedders
  rules:
    - id: chat-cache                 # required, unique
      match: { endpoint: chat }      # only `endpoint` narrows a rule — see below
      mode: shadow                   # off | shadow | on
      strategy: [exact, normalized]  # ordered chain, first match wins
      ttl: 1h

Key	Default	What it does
cache.enabled	false	master gate; an absent or disabled block runs no lookup and writes no cache fields
rules[].id	—	required, unique across rules; appears on every row and is a purge dimension
rules[].mode	""	off, shadow or on; an unset mode behaves as off
rules[].strategy	—	a matcher name or an ordered list; at least one entry required
rules[].ttl	0	how long an entry this rule writes stays servable; 0 means no expiry
rules[].match.endpoint	""	narrows the rule to one endpoint kind; empty matches any cacheable endpoint

strategy accepts a scalar or a list, so strategy: exact and strategy: [exact] are the same rule.

How a rule is selected

Rules are ordered and first-match-wins, the same idiom routing rules use. For each request:

1. If the deciding policy rule carried a cache: annotation naming a rule id, that rule governs, with any mode or ttl override the annotation set. A directive naming an unknown id falls through to step 2 rather than faulting.
2. Otherwise, walk cache.rules in order and take the first whose match.endpoint is empty or equals this request’s endpoint kind.
3. If no rule matches, the row records outcome: bypass with bypass_reason: no_rule and the request forwards.

Rule selection happens before eligibility. A request that carries tools is bypassed with tool_calls, not no_rule, because a rule was found first.

What match can and cannot narrow

The schema accepts five keys under match: endpoint, model, dialect, tier and dims. Validation checks all of them — an unknown dims key is a path-named error, and a non-cacheable endpoint is rejected.

Only endpoint reaches the engine. The config-to-engine mapping in cmd/tokentriage/cachereg.go carries id, mode, strategy, ttl, the global scope and match.endpoint, and nothing else; the compiled rule has one match field. A rule written as match: { model: "gpt-4o*" } validates cleanly and then matches every cacheable request, because its only effective narrowing key is unset.

Write rules on endpoint only until that changes. If you need per-model or per-tenant behaviour today, the levers that work are the policy-language cache: annotation, which names a rule per routing decision, and cache scope, which partitions entries rather than selecting rules.

Mode, and the setting it is woven with

Rule mode	Lookup runs	Row outcome on a match	Client gets
off (or unset)	no	bypass	a live forward
shadow	yes, in full	would_hit, carrying the would-be saving	a live forward
on, engine in ledger or shadow	yes, in full	would_hit	a live forward
on, engine in route	yes	hit	the stored replay

Shadow cannot serve by construction: the lookup yields only a metadata view with no byte accessors, and the single function that writes cached bytes to a client does not accept it.

Do not confuse the two ladders. Cache rules climb off → shadow → on; the engine climbs the trust ladder ledger → shadow → route. They are separate settings and both must be at the top for a replay to be served.

TTL

ttl is stamped onto each entry a rule writes as an absolute deadline. Expiry is applied when the entry is read: a lookup that finds an entry past its deadline treats it as a miss and moves to the next matcher in the chain, so a stale answer is not served while waiting for a sweep.

$ # ttl: 5s, the same request twice
X-Tt-Cache: hit
X-Tt-Cache-Age: 1
$ # ... six seconds later
X-Tt-Cache: miss

An expired entry still occupies its tier until something displaces it. Nothing physically deletes it on a schedule; see the store.

Which changes need a restart

Change	Applies
cache.rules, cache.budgets, cache.embedders	hot — reprograms the live cache without rebuilding the store
cache.enabled, cache.scope, anything under cache.store	restart — these change the store’s identity or topology

Changing cache.scope changes the scope key, which is a prefix of every storage key, so existing entries become unreachable rather than wrong. The same is true of switching a semantic rule’s embedder.

Validation you will meet

Every cache error names its YAML path and is raised both by tokentriage validate and on reload. Four at once, from one bad file:

$ tokentriage validate --config broken.yaml
Error: configuration is invalid: config broken.yaml: cache.scope[1]: "region" is not one of tenant|user|feature|trace|upstream-key
cache.rules[0].match.endpoint: endpoint kind "embeddings" is not cacheable (chat|responses only in v1)
cache.rules[0].semantic.embedder: a semantic strategy requires a configured embedder
cache.rules[0].semantic.low_threshold: must be strictly less than threshold (low=0.95 >= threshold=0.9)
Error: configuration is invalid: config broken.yaml: cache.scope[1]: "region" is not one of tenant|user|feature|trace|upstream-key
cache.rules[0].match.endpoint: endpoint kind "embeddings" is not cacheable (chat|responses only in v1)
cache.rules[0].semantic.embedder: a semantic strategy requires a configured embedder
cache.rules[0].semantic.low_threshold: must be strictly less than threshold (low=0.95 >= threshold=0.9)

The block prints twice. That is one bug, not two sets of problems: the CLI’s error handler prints the error and the root command prints it again. It affects every failing command, not only validate.

The full cache validation matrix

Structural checks always run. Checks that need the primitive registries (matcher, embedder-kind, verifier and index names) are deferred when the registries are not injected, and run at boot.

Rule	Message names
scope vocabulary	cache.scope[i]
l2.max_entry_kb strictly below server.max_body_bytes	cache.store.l2.max_entry_kb
l3.max_entry_kb strictly below server.max_body_bytes	cache.store.l3.max_entry_kb
L3 present without external: allow	cache.store.l3.external
L3 addr parses as host:port	cache.store.l3.addr
password_env / api_key_env name a variable rather than carrying a value	the key itself
index name is registered	cache.store.index
embedder kind is registered; external class carries external: allow	cache.embedders[name]
rule ids non-empty and unique	cache.rules[i].id
mode in `off	shadow
endpoint known, and cacheable	cache.rules[i].match.endpoint
match.dims keys drawn from the scope vocabulary	cache.rules[i].match.dims[key]
at least one matcher, each registered	cache.rules[i].strategy[j]
semantic strategy has a configured, defined embedder	cache.rules[i].semantic.embedder
threshold in (0,1], low_threshold below it	cache.rules[i].semantic.*
verified.delta and .confidence in (0,1), .min_samples above 0	cache.rules[i].semantic.verified.*

Two warnings are not errors and are worth reading anyway: enabling the cache with ledger.dimensions disabled warns that scope is global, and any enabled semantic rule warns that similarity matching can return a wrong answer and names the embedder’s class.

What can go wrong

A rule validates and then does nothing. Check whether an earlier rule with no match.endpoint is absorbing the traffic, and remember that match.model, match.tier, match.dialect and match.dims do not narrow anything today.

A config that passes validate still aborts at boot. Registry-backed name checks (a matcher, embedder kind, verifier or index that does not exist) are deferred when validation runs without the registries. Start the binary as part of your change, not only the validator.