# Cache rules

The cache block field by field, how a rule is selected, which changes reload hot, and the validation errors you will meet.

Source: https://tokentriage.com/docs/build/caching-rules

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

A cache rule decides three things about a request: whether the cache engages at all, which matchers
run and in what order, and how long an entry it writes stays valid. Everything else about the cache
is global.

This page is the field list. For the on-ramp see [rolling out a rule](/docs/build/caching-rollout);
for what each matcher matches on, see [the three matchers](/docs/build/caching-strategies).

## The block

```yaml
cache:
  enabled: true                      # default false — the whole subsystem is opt-in
  store: { }                         # see /docs/build/caching-store
  scope: [tenant]                    # see /docs/build/caching-scope
  budgets: { }                       # see /docs/build/caching-performance
  embedders: { }                     # see /docs/build/caching-embedders
  rules:
    - id: chat-cache                 # required, unique
      match: { endpoint: chat }      # only `endpoint` narrows a rule — see below
      mode: shadow                   # off | shadow | on
      strategy: [exact, normalized]  # ordered chain, first match wins
      ttl: 1h
```

| Key | Default | What it does |
|---|---|---|
| `cache.enabled` | `false` | master gate; an absent or disabled block runs no lookup and writes no cache fields |
| `rules[].id` | — | required, unique across rules; appears on every row and is a purge dimension |
| `rules[].mode` | `""` | `off`, `shadow` or `on`; an unset mode behaves as off |
| `rules[].strategy` | — | a matcher name or an ordered list; at least one entry required |
| `rules[].ttl` | `0` | how long an entry this rule writes stays servable; `0` means no expiry |
| `rules[].match.endpoint` | `""` | narrows the rule to one endpoint kind; empty matches any cacheable endpoint |

`strategy` accepts a scalar or a list, so `strategy: exact` and `strategy: [exact]` are the same
rule.

## How a rule is selected

Rules are ordered and first-match-wins, the same idiom routing rules use. For each request:

1. If the deciding policy rule carried a `cache:` annotation naming a rule id, that rule governs,
   with any `mode` or `ttl` override the annotation set. A directive naming an unknown id falls
   through to step 2 rather than faulting.
2. Otherwise, walk `cache.rules` in order and take the first whose `match.endpoint` is empty or
   equals this request's endpoint kind.
3. If no rule matches, the row records `outcome: bypass` with `bypass_reason: no_rule` and the
   request forwards.

Rule selection happens *before* eligibility. A request that carries tools is bypassed with
`tool_calls`, not `no_rule`, because a rule was found first.

<Callout type="warning">
A rule with no `match.endpoint` matches every cacheable request, so any rule after it is
unreachable. Put your narrow rules first, exactly as you would with routing rules. Nothing warns
you about a shadowed rule.
</Callout>

## What `match` can and cannot narrow

The schema accepts five keys under `match`: `endpoint`, `model`, `dialect`, `tier` and `dims`.
Validation checks all of them — an unknown `dims` key is a path-named error, and a non-cacheable
`endpoint` is rejected.

**Only `endpoint` reaches the engine.** The config-to-engine mapping in
`cmd/tokentriage/cachereg.go` carries `id`, `mode`, `strategy`, `ttl`, the global scope and
`match.endpoint`, and nothing else; the compiled rule has one match field. A rule written as
`match: { model: "gpt-4o*" }` validates cleanly and then matches every cacheable request, because
its only effective narrowing key is unset.

Write rules on `endpoint` only until that changes. If you need per-model or per-tenant behaviour
today, the levers that work are the policy-language `cache:` annotation, which names a rule per
routing decision, and [cache scope](/docs/build/caching-scope), which partitions entries rather
than selecting rules.

## Mode, and the setting it is woven with

| Rule `mode` | Lookup runs | Row outcome on a match | Client gets |
|---|---|---|---|
| `off` (or unset) | no | `bypass` | a live forward |
| `shadow` | yes, in full | `would_hit`, carrying the would-be saving | a live forward |
| `on`, engine in `ledger` or `shadow` | yes, in full | `would_hit` | a live forward |
| `on`, engine in `route` | yes | `hit` | the stored replay |

Shadow cannot serve by construction: the lookup yields only a metadata view with no byte
accessors, and the single function that writes cached bytes to a client does not accept it.

Do not confuse the two ladders. Cache rules climb `off → shadow → on`; the engine climbs the
[trust ladder](/docs/concepts/core-concepts) `ledger → shadow → route`. They are separate settings
and both must be at the top for a replay to be served.

## TTL

`ttl` is stamped onto each entry a rule writes as an absolute deadline. Expiry is applied when the
entry is read: a lookup that finds an entry past its deadline treats it as a miss and moves to the
next matcher in the chain, so a stale answer is not served while waiting for a sweep.

```console
$ # ttl: 5s, the same request twice
X-Tt-Cache: hit
X-Tt-Cache-Age: 1
$ # ... six seconds later
X-Tt-Cache: miss
```

An expired entry still occupies its tier until something displaces it. Nothing physically deletes
it on a schedule; see [the store](/docs/build/caching-store#what-actually-reclaims-space).

## Which changes need a restart

| Change | Applies |
|---|---|
| `cache.rules`, `cache.budgets`, `cache.embedders` | hot — reprograms the live cache without rebuilding the store |
| `cache.enabled`, `cache.scope`, anything under `cache.store` | restart — these change the store's identity or topology |

Changing `cache.scope` changes the scope key, which is a prefix of every storage key, so existing
entries become unreachable rather than wrong. The same is true of switching a semantic rule's
embedder.

## Validation you will meet

Every cache error names its YAML path and is raised both by `tokentriage validate` and on reload.
Four at once, from one bad file:

```console
$ tokentriage validate --config broken.yaml
Error: configuration is invalid: config broken.yaml: cache.scope[1]: "region" is not one of tenant|user|feature|trace|upstream-key
cache.rules[0].match.endpoint: endpoint kind "embeddings" is not cacheable (chat|responses only in v1)
cache.rules[0].semantic.embedder: a semantic strategy requires a configured embedder
cache.rules[0].semantic.low_threshold: must be strictly less than threshold (low=0.95 >= threshold=0.9)
Error: configuration is invalid: config broken.yaml: cache.scope[1]: "region" is not one of tenant|user|feature|trace|upstream-key
cache.rules[0].match.endpoint: endpoint kind "embeddings" is not cacheable (chat|responses only in v1)
cache.rules[0].semantic.embedder: a semantic strategy requires a configured embedder
cache.rules[0].semantic.low_threshold: must be strictly less than threshold (low=0.95 >= threshold=0.9)
```

The block prints twice. That is one bug, not two sets of problems: the CLI's error handler prints
the error and the root command prints it again. It affects every failing command, not only
`validate`.

<Details summary="The full cache validation matrix">

Structural checks always run. Checks that need the primitive registries (matcher, embedder-kind,
verifier and index *names*) are deferred when the registries are not injected, and run at boot.

| Rule | Message names |
|---|---|
| scope vocabulary | `cache.scope[i]` |
| `l2.max_entry_kb` strictly below `server.max_body_bytes` | `cache.store.l2.max_entry_kb` |
| `l3.max_entry_kb` strictly below `server.max_body_bytes` | `cache.store.l3.max_entry_kb` |
| L3 present without `external: allow` | `cache.store.l3.external` |
| L3 `addr` parses as host:port | `cache.store.l3.addr` |
| `password_env` / `api_key_env` name a variable rather than carrying a value | the key itself |
| index name is registered | `cache.store.index` |
| embedder kind is registered; external class carries `external: allow` | `cache.embedders[name]` |
| rule ids non-empty and unique | `cache.rules[i].id` |
| `mode` in `off|shadow|on` | `cache.rules[i].mode` |
| endpoint known, and cacheable | `cache.rules[i].match.endpoint` |
| `match.dims` keys drawn from the scope vocabulary | `cache.rules[i].match.dims[key]` |
| at least one matcher, each registered | `cache.rules[i].strategy[j]` |
| semantic strategy has a configured, defined embedder | `cache.rules[i].semantic.embedder` |
| `threshold` in (0,1], `low_threshold` below it | `cache.rules[i].semantic.*` |
| `verified.delta` and `.confidence` in (0,1), `.min_samples` above 0 | `cache.rules[i].semantic.verified.*` |

Two warnings are not errors and are worth reading anyway: enabling the cache with
`ledger.dimensions` disabled warns that scope is global, and any enabled semantic rule warns that
similarity matching can return a wrong answer and names the embedder's class.

</Details>

## What can go wrong

**A rule validates and then does nothing.** Check whether an earlier rule with no `match.endpoint`
is absorbing the traffic, and remember that `match.model`, `match.tier`, `match.dialect` and
`match.dims` do not narrow anything today.

**A config that passes `validate` still aborts at boot.** Registry-backed name checks (a matcher,
embedder kind, verifier or index that does not exist) are deferred when validation runs without the
registries. Start the binary as part of your change, not only the validator.

<NextSteps items={[
  {title: "The three matchers", href: "/docs/build/caching-strategies", description: "What exact, normalized and semantic each match on, what they cost, and how each fails."},
  {title: "Cache scope and isolation", href: "/docs/build/caching-scope", description: "The five scope dimensions, and what changing one does to hit rate and to tenant separation."},
  {title: "Roll out a cache rule safely", href: "/docs/build/caching-rollout", description: "The shadow-first path from an empty config to a served replay."}
]} />