# Budgets, fail-open, and performance
What a lookup costs on each tier, the deadline that turns a slow cache back into a live call, and how faults degrade.
Source: https://tokentriage.com/docs/build/caching-performance
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
A cache sits on the request path, so it can make things worse. A lookup that takes longer than the
model call it was meant to avoid is not an optimisation, and a store that faults is not a reason to
fail a request.
Both are handled the same way: the lookup is budgeted, and every fault degrades to a normal upstream
call.
## What a lookup costs
Measured on the reference machine documented in [benchmarks](/docs/reference/benchmarks), with the
pure-Go database driver and in-process fixtures.
| Path | Measured | Budget |
|---|---|---|
| key derivation over a four-turn body | ~9.9 µs | ≤ 20 µs |
| in-memory probe | ~0.25 µs | ≤ 5 µs |
| durable-file probe | ~13.8 µs | ≤ 300 µs |
| Redis probe, loopback | ~20 µs | ≤ 5 ms |
| semantic recall, exact scan, 10k entries, 384 dims | ~2.9 ms | 5 ms design |
| semantic recall, exact scan, 10k entries, 768 dims | ~5.4 ms | over the 5 ms design budget |
| semantic recall, approximate index, 10k entries, 768 dims | ~1.8 ms | ≤ 5 ms |
| replay, non-streaming | ~0.2 µs | ≤ 50 µs |
| replay, per stream frame | ~7 ns | ≤ 5 µs |
| added overhead on a cache miss | ~12 µs mean | < 2 ms p99 |
Each is a gate in CI rather than a note: the build fails if a path regresses past its ceiling.
Two rows are worth pausing on.
**Key derivation costs more than the lookup.** Around 9.9 µs against 0.25 µs for the in-memory probe.
That is the price of a key that cannot be fragmented by formatting: decode the body, re-encode it
canonically, hash the chain. It is the right trade, because a cheaper key would miss on whitespace, and it
means the exact tier's cost is dominated by canonicalisation, not by storage.
**The exact scan runs out of room at 768 dimensions.** At a ten-thousand-entry per-scope working set
it measures about 5.4 ms, over the 5 ms design budget for the semantic path, which is the documented
point at which to switch `cache.store.index` to the approximate index and land the same corpus at
about 1.8 ms.
## The budgets
```yaml
cache:
budgets:
lookup_exact_us: 500 # the deterministic path
lookup_semantic_ms: 5 # the semantic path's hard deadline
judge_daily_usd: 1.0 # off-path grading spend, per day
```
`lookup_semantic_ms` is a deadline on the semantic probe. Exceeding it is not an error: the lookup
is abandoned, the row records `outcome: miss` with `bypass_reason: over_budget`, the counter
increments, and the request forwards live. A cache that is slower than the model gets out of the
way.
`judge_daily_usd` caps the off-path grader that produces calibration labels. It is a spend cap on
work that stays off the request path.
If you see `over_budget` on many rows, the per-scope working set has outgrown the exact scan. Switch
the index before you raise the deadline; raising it makes slow lookups slower rather than making
them hits.
## Every fault forwards the request
The lookup does not return an error to the request pipeline. Each of these degrades to a live call
with the fault counted and the row carrying its reason:
| Fault | Result |
|---|---|
| a corrupt or unreadable durable file | `outcome: error`, live forward |
| an unreachable Redis tier | live forward; the breaker opens after repeated failures |
| an embedding call that fails | `outcome: error`, live forward |
| a semantic lookup past its deadline | `outcome: miss` with `bypass_reason: over_budget` |
| a panic anywhere in the lookup | `outcome: error` with a redacted reason, live forward |
The store write is off the request path entirely. It happens after the client has been served, it is
queued, and a full queue drops the write and counts it rather than blocking anything.
A broken cache costs you savings, not availability. There is no cache failure mode that turns into a
5xx: the client is served by the upstream exactly as it would have been with the cache disabled.
## What a hit does to the rest of the system
A replay makes no upstream call, so several subsystems have to be told not to learn from it. Each is
gated on the row being a hit:
- upstream health does not record a result, so a replay cannot make a dead upstream look healthy
- latency signals ignore the replay's timing, so serve time does not enter routing's model of how
fast an upstream is
- prompt-cache affinity is not pinned by a replay, which reports zero cache-read tokens
- shadow duplication does not run against a replayed row
Two things do still run. Governance evaluates before the cache lookup, so a request that would be
turned away is turned away rather than served from the store. And output guardrails re-evaluate on a
hit: a stored entry was checked under whatever policy was in force when it was written, so replaying
it unchecked would serve stale-policy content after a policy tightens.
The soak test runs a filling and evicting cache under streaming load and gates on stable resident
memory with no decision-log drops. The bound that keeps memory flat is the in-memory tier's entry
cap, which evicts on write and removes the evicted entry's vector from its recall index at the same
time, so the two cannot drift.
The added overhead on a miss — the path most requests take on a cold cache — measures at about 12 µs
mean, inside the unchanged sub-2 ms p99 proxy budget.
## What can go wrong
**Latency gets worse after enabling the semantic tier.** Every miss on a semantic rule pays for an
embedding on the request path. Put `exact` and `normalized` in front of it so the common case never
reaches the embedder.
**p99 climbs after the working set grows.** The exact scan is linear in the per-scope entry count.
Watch `over_budget` rows, and switch the index rather than the deadline.
**An external embedder makes the cache as available as the vendor.** It does not fail requests, but
every semantic lookup now depends on a network call inside your latency budget. The in-process
options do not.