The store, and which tiers you need

Most operators need one decision from this page: whether to configure a durable tier, and whether to share entries across replicas. Everything else is defaults.

Pick a tier

You are running	Configure	You get
a single process, and a cold start after a deploy is fine	L1 only	bounded memory, nothing on disk
a single process, and you want the cache to survive restarts	L1 + L2	a durable file, byte-budgeted
several replicas that should share entries	L1 + L2 + L3	shared payloads over Redis, and a network dependency

L1 is always present. L2 exists only if you give it a path. L3 exists only if you add its block and open its gate.

cache:
  store:
    l1: { max_entries: 4096 }
    l2:
      path: /var/lib/tokentriage/respcache.db
      max_total_mb: 512
      max_entry_kb: 1024

How a lookup walks them

Read it top to bottom. L1 answers first; an L2 hit is promoted into L1 before it is served; an L3 hit is promoted the same way. A write goes to L2 first, then L1, then out to L3 as a best-effort copy whose failure is ignored. The served tier is recorded on the row.

Two consecutive hits on the same entry, from a sandbox restarted between the store and the lookup:

{ "outcome": "hit", "kind": "exact", "serving_tier": "l2", "lookup_us": 518 }
{ "outcome": "hit", "kind": "exact", "serving_tier": "l1" }

The first read came off disk because the in-memory tier was cold, and promoted the entry. The second read found it in memory. serving_tier is how you tell, and it is the field to watch if your latency profile changes after a deploy.

L1: bounded memory

An LRU map capped at max_entries (default 4,096) across all scopes, not per scope. A write past the cap evicts the least recently used entry and, if that entry carried a vector, removes it from its recall index so the two cannot drift apart.

Reads hand out a copy of the stored entry, so nothing downstream can mutate the shared value. That copy is the cost you see in the benchmark: an in-memory probe is around 0.25 µs with six allocations.

L1 holds nothing after a restart.

L2: a durable file of its own

A SQLite database with write-ahead logging, one writer, and BLOB columns for the payload and, for semantic entries, the vector. It is not a table in the dashboard’s read-model index; it is its own file and you should treat it as cache, not as a system of record. Deleting it costs you a warm cache and nothing else.

Knob	Default	What it bounds
path	""	the file; empty means no L2 at all
max_total_mb	512	total resident payload bytes
max_entry_kb	1024	one entry’s payload; must be strictly below server.max_body_bytes

On boot the store scans L2’s vector column and rebuilds each scope’s recall index, so a restart loses the index structure but not the vectors behind it. That rebuild is the reason a semantic cache is worth making durable.

The L2 path trap

path: "" does not resolve to a default location. It builds an L1-only store: no file is created, nothing is durable, and no byte budget applies because L1 counts entries rather than bytes.

The comment shipped in both example configs says "" ⇒ <data-root>/respcache.db, and the operator guide repeats it. The binary does not do that. From two sandbox runs of the same config, differing only in this key:

$ # cache.store.l2.path: ""  — restart, then repeat a request that hit before the restart
X-Tt-Cache: miss
$ ls /var/lib/tokentriage
dash  decisions.jsonl  index

$ # cache.store.l2.path: ./respcache.db  — same restart, same request
X-Tt-Cache: hit
X-Tt-Cache-Age: 102

L3: shared entries across replicas

An opt-in Redis tier, and the one part of the cache that leaves the machine. It is refused unless its block carries external: allow, the same gate an HTTP embedder carries, and opening it fires a warning that names the address.

    l3:
      external: allow                  # required; without it, a validation error
      addr: redis.internal:6379
      password_env: TT_CACHE_REDIS_PASSWORD   # names a variable, never the secret
      key_prefix: "tt:respcache:"
      dial_timeout_ms: 500
      op_timeout_ms: 200
      max_entry_kb: 1024               # same rule as L2: below server.max_body_bytes

It rides the shared Redis client, so it has a bounded connection pool, a circuit breaker that fast-fails once a dead server has produced enough consecutive errors, and optional TLS. Each of pool_size, max_conns, failure_threshold, probe_interval_ms, tls and tls_server_name is additive, and a zero leaves the client default in place. The connection is lazy: a mis-wired L3 is a boot error, but an unreachable one is not — it dials on first use and falls back to a live call.

Two things are worth knowing before you reach for it. This block is separate from shared_state.redis, which the fleet and coordination features require and which carries its own external: allow; they are two opt-ins, and they can point at the same Redis server. And the semantic recall index is built per process from local writes and the local L2 file, so an entry another replica wrote is reachable by an exact or normalized key and is not reachable by similarity until this process has seen it.

The recall index

Semantic recall needs a structure to search. One is built per scope.

cache.store.index	What it does
bruteforce (default)	exact k-nearest-neighbour scan; its scores are exact cosines
hnsw	an approximate index for large per-scope working sets, with index_params

The default is exact, so there is no approximation to disclose. When the approximate index is selected, every semantic hit carries recall_exact: false and the index’s configuration on the row, and the approximate score is thrown away and the exact cosine recomputed before any decision is made. An approximate distance never decides anything; it only shortlists.

Choose hnsw when a per-scope working set gets large enough that the exact scan misses its budget. Budgets and performance has the measured crossover.

What actually reclaims space

This is the part most likely to differ from what you expect.

Bound	Enforced	Audited
L1 max_entries	at write time, by the memory tier’s LRU	no
L2 max_total_mb	at write time, by evicting oldest entries until the new one fits	no
L2 max_entry_kb	at write time, by rejecting the entry	no
TTL	at read time, by treating an expired entry as a miss	no
an operator purge	on request, synchronously	yes

There is no scheduled sweep. The evictor implementations for TTL, entry count and byte budget exist in the tree, and the only one wired at composition is the purge evictor, which runs when an operator asks it to. So an expired entry stays resident until a write displaces it, and the “every physical deletion is audited” claim is true of purge deletions and not of budget evictions.

Practically, the bounds still hold: a write that would exceed a budget evicts before it lands. What you do not get is a background process that reclaims an idle cache, and what you do not get in the audit log is a record of budget evictions.

Reading the store's own counters

Each tier keeps counters for puts, hits, misses, evictions, deletes, and — for the durable tier — rejected oversize entries, write errors, and cold starts on a schema version mismatch. They are surfaced through the cache stats path rather than as a separate endpoint. A rising rejected count with a flat hit rate means your responses are larger than max_entry_kb and nothing is being stored.

What can go wrong

Everything misses after a restart. No l2.path. See above.

The durable file grows to its budget and stays there. That is the design: max_total_mb is a budget, not a target, and eviction happens on write.

An L3 outage looks like a latency spike, then stops. The circuit breaker opens after enough consecutive failures and the tier fast-fails to a live call from then on. Reads degrade to L1 and L2.