# The store, and which tiers you need
Pick one tier rather than reading three. What L1, L2 and L3 each hold, what bounds them, and what survives a restart.
Source: https://tokentriage.com/docs/build/caching-store
---
import CacheTierDiagram from '../../../components/CacheTierDiagram.astro';
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
Most operators need one decision from this page: whether to configure a durable tier, and whether to
share entries across replicas. Everything else is defaults.
## Pick a tier
| You are running | Configure | You get |
|---|---|---|
| a single process, and a cold start after a deploy is fine | L1 only | bounded memory, nothing on disk |
| a single process, and you want the cache to survive restarts | L1 + **L2** | a durable file, byte-budgeted |
| several replicas that should share entries | L1 + L2 + **L3** | shared payloads over Redis, and a network dependency |
L1 is always present. L2 exists only if you give it a path. L3 exists only if you add its block and
open its gate.
```yaml
cache:
store:
l1: { max_entries: 4096 }
l2:
path: /var/lib/tokentriage/respcache.db
max_total_mb: 512
max_entry_kb: 1024
```
## How a lookup walks them
Read it top to bottom. L1 answers first; an L2 hit is promoted into L1 before it is served; an L3
hit is promoted the same way. A write goes to L2 first, then L1, then out to L3 as a best-effort
copy whose failure is ignored. The served tier is recorded on the row.
Two consecutive hits on the same entry, from a sandbox restarted between the store and the lookup:
```json
{ "outcome": "hit", "kind": "exact", "serving_tier": "l2", "lookup_us": 518 }
{ "outcome": "hit", "kind": "exact", "serving_tier": "l1" }
```
The first read came off disk because the in-memory tier was cold, and promoted the entry. The second
read found it in memory. `serving_tier` is how you tell, and it is the field to watch if your
latency profile changes after a deploy.
## L1: bounded memory
An LRU map capped at `max_entries` (default 4,096) across all scopes, not per scope. A write past
the cap evicts the least recently used entry and, if that entry carried a vector, removes it from
its recall index so the two cannot drift apart.
Reads hand out a copy of the stored entry, so nothing downstream can mutate the shared value. That
copy is the cost you see in the benchmark: an in-memory probe is around 0.25 µs with six
allocations.
L1 holds nothing after a restart.
## L2: a durable file of its own
A SQLite database with write-ahead logging, one writer, and BLOB columns for the payload and, for
semantic entries, the vector. It is **not** a table in the dashboard's read-model index; it is its
own file and you should treat it as cache, not as a system of record. Deleting it costs you a warm
cache and nothing else.
| Knob | Default | What it bounds |
|---|---|---|
| `path` | `""` | the file; empty means **no L2 at all** |
| `max_total_mb` | `512` | total resident payload bytes |
| `max_entry_kb` | `1024` | one entry's payload; must be strictly below `server.max_body_bytes` |
On boot the store scans L2's vector column and rebuilds each scope's recall index, so a restart
loses the index structure but not the vectors behind it. That rebuild is the reason a semantic
cache is worth making durable.
### The L2 path trap
`path: ""` does not resolve to a default location. It builds an L1-only store: no file is created,
nothing is durable, and no byte budget applies because L1 counts entries rather than bytes.
The comment shipped in both example configs says `"" ⇒ /respcache.db`, and the operator
guide repeats it. The binary does not do that. From two sandbox runs of the same config, differing
only in this key:
```console
$ # cache.store.l2.path: "" — restart, then repeat a request that hit before the restart
X-Tt-Cache: miss
$ ls /var/lib/tokentriage
dash decisions.jsonl index
```
```console
$ # cache.store.l2.path: ./respcache.db — same restart, same request
X-Tt-Cache: hit
X-Tt-Cache-Age: 102
```
Set `cache.store.l2.path` explicitly if you want durability. An empty value is valid, silent, and
gives you an in-memory cache that starts cold on every restart. This is filed against the product;
until it changes, the config comment is wrong and the binary is right.
## L3: shared entries across replicas
An opt-in Redis tier, and the one part of the cache that leaves the machine. It is refused unless
its block carries `external: allow`, the same gate an HTTP embedder carries, and opening it fires a
warning that names the address.
```yaml
l3:
external: allow # required; without it, a validation error
addr: redis.internal:6379
password_env: TT_CACHE_REDIS_PASSWORD # names a variable, never the secret
key_prefix: "tt:respcache:"
dial_timeout_ms: 500
op_timeout_ms: 200
max_entry_kb: 1024 # same rule as L2: below server.max_body_bytes
```
It rides the shared Redis client, so it has a bounded connection pool, a circuit breaker that
fast-fails once a dead server has produced enough consecutive errors, and optional TLS. Each of
`pool_size`, `max_conns`, `failure_threshold`, `probe_interval_ms`, `tls` and `tls_server_name` is
additive, and a zero leaves the client default in place. The connection is lazy: a mis-wired L3 is a
boot error, but an unreachable one is not — it dials on first use and falls back to a live call.
Two things are worth knowing before you reach for it. This block is separate from
`shared_state.redis`, which the fleet and coordination features require and which carries its own
`external: allow`; they are two opt-ins, and they can point at the same Redis server. And the
semantic recall index is built per process from local writes and the local L2 file, so an entry
another replica wrote is reachable by an exact or normalized key and is **not** reachable by
similarity until this process has seen it.
## The recall index
Semantic recall needs a structure to search. One is built per scope.
| `cache.store.index` | What it does |
|---|---|
| `bruteforce` (default) | exact k-nearest-neighbour scan; its scores are exact cosines |
| `hnsw` | an approximate index for large per-scope working sets, with `index_params` |
The default is exact, so there is no approximation to disclose. When the approximate index is
selected, every semantic hit carries `recall_exact: false` and the index's configuration on the
row, and the approximate score is thrown away and the exact cosine recomputed before any decision
is made. An approximate distance never decides anything; it only shortlists.
Choose `hnsw` when a per-scope working set gets large enough that the exact scan misses its budget.
[Budgets and performance](/docs/build/caching-performance) has the measured crossover.
## What actually reclaims space
This is the part most likely to differ from what you expect.
| Bound | Enforced | Audited |
|---|---|---|
| L1 `max_entries` | at write time, by the memory tier's LRU | no |
| L2 `max_total_mb` | at write time, by evicting oldest entries until the new one fits | no |
| L2 `max_entry_kb` | at write time, by rejecting the entry | no |
| TTL | at read time, by treating an expired entry as a miss | no |
| an operator purge | on request, synchronously | yes |
There is no scheduled sweep. The evictor implementations for TTL, entry count and byte budget exist
in the tree, and the only one wired at composition is the purge evictor, which runs when an operator
asks it to. So an expired entry stays resident until a write displaces it, and the "every physical
deletion is audited" claim is true of purge deletions and not of budget evictions.
Practically, the bounds still hold: a write that would exceed a budget evicts before it lands. What
you do not get is a background process that reclaims an idle cache, and what you do not get in the
audit log is a record of budget evictions.
Each tier keeps counters for puts, hits, misses, evictions, deletes, and — for the durable tier —
rejected oversize entries, write errors, and cold starts on a schema version mismatch. They are
surfaced through the cache stats path rather than as a separate endpoint. A rising rejected count
with a flat hit rate means your responses are larger than `max_entry_kb` and nothing is being
stored.
## What can go wrong
**Everything misses after a restart.** No `l2.path`. See above.
**The durable file grows to its budget and stays there.** That is the design: `max_total_mb` is a
budget, not a target, and eviction happens on write.
**An L3 outage looks like a latency spike, then stops.** The circuit breaker opens after enough
consecutive failures and the tier fast-fails to a live call from then on. Reads degrade to L1 and L2.