Budgets, fail-open, and performance

A cache sits on the request path, so it can make things worse. A lookup that takes longer than the model call it was meant to avoid is not an optimisation, and a store that faults is not a reason to fail a request.

Both are handled the same way: the lookup is budgeted, and every fault degrades to a normal upstream call.

What a lookup costs

Measured on the reference machine documented in benchmarks, with the pure-Go database driver and in-process fixtures.

Path	Measured	Budget
key derivation over a four-turn body	~9.9 µs	≤ 20 µs
in-memory probe	~0.25 µs	≤ 5 µs
durable-file probe	~13.8 µs	≤ 300 µs
Redis probe, loopback	~20 µs	≤ 5 ms
semantic recall, exact scan, 10k entries, 384 dims	~2.9 ms	5 ms design
semantic recall, exact scan, 10k entries, 768 dims	~5.4 ms	over the 5 ms design budget
semantic recall, approximate index, 10k entries, 768 dims	~1.8 ms	≤ 5 ms
replay, non-streaming	~0.2 µs	≤ 50 µs
replay, per stream frame	~7 ns	≤ 5 µs
added overhead on a cache miss	~12 µs mean	< 2 ms p99

Each is a gate in CI rather than a note: the build fails if a path regresses past its ceiling.

Two rows are worth pausing on.

Key derivation costs more than the lookup. Around 9.9 µs against 0.25 µs for the in-memory probe. That is the price of a key that cannot be fragmented by formatting: decode the body, re-encode it canonically, hash the chain. It is the right trade, because a cheaper key would miss on whitespace, and it means the exact tier’s cost is dominated by canonicalisation, not by storage.

The exact scan runs out of room at 768 dimensions. At a ten-thousand-entry per-scope working set it measures about 5.4 ms, over the 5 ms design budget for the semantic path, which is the documented point at which to switch cache.store.index to the approximate index and land the same corpus at about 1.8 ms.

The budgets

cache:
  budgets:
    lookup_exact_us: 500       # the deterministic path
    lookup_semantic_ms: 5      # the semantic path's hard deadline
    judge_daily_usd: 1.0       # off-path grading spend, per day

lookup_semantic_ms is a deadline on the semantic probe. Exceeding it is not an error: the lookup is abandoned, the row records outcome: miss with bypass_reason: over_budget, the counter increments, and the request forwards live. A cache that is slower than the model gets out of the way.

judge_daily_usd caps the off-path grader that produces calibration labels. It is a spend cap on work that stays off the request path.

If you see over_budget on many rows, the per-scope working set has outgrown the exact scan. Switch the index before you raise the deadline; raising it makes slow lookups slower rather than making them hits.

Every fault forwards the request

The lookup does not return an error to the request pipeline. Each of these degrades to a live call with the fault counted and the row carrying its reason:

Fault	Result
a corrupt or unreadable durable file	outcome: error, live forward
an unreachable Redis tier	live forward; the breaker opens after repeated failures
an embedding call that fails	outcome: error, live forward
a semantic lookup past its deadline	outcome: miss with bypass_reason: over_budget
a panic anywhere in the lookup	outcome: error with a redacted reason, live forward

The store write is off the request path entirely. It happens after the client has been served, it is queued, and a full queue drops the write and counts it rather than blocking anything.

What a hit does to the rest of the system

A replay makes no upstream call, so several subsystems have to be told not to learn from it. Each is gated on the row being a hit:

• upstream health does not record a result, so a replay cannot make a dead upstream look healthy
• latency signals ignore the replay’s timing, so serve time does not enter routing’s model of how fast an upstream is
• prompt-cache affinity is not pinned by a replay, which reports zero cache-read tokens
• shadow duplication does not run against a replayed row

Two things do still run. Governance evaluates before the cache lookup, so a request that would be turned away is turned away rather than served from the store. And output guardrails re-evaluate on a hit: a stored entry was checked under whatever policy was in force when it was written, so replaying it unchecked would serve stale-policy content after a policy tightens.

Throughput and memory under sustained load

The soak test runs a filling and evicting cache under streaming load and gates on stable resident memory with no decision-log drops. The bound that keeps memory flat is the in-memory tier’s entry cap, which evicts on write and removes the evicted entry’s vector from its recall index at the same time, so the two cannot drift.

The added overhead on a miss — the path most requests take on a cold cache — measures at about 12 µs mean, inside the unchanged sub-2 ms p99 proxy budget.

What can go wrong

Latency gets worse after enabling the semantic tier. Every miss on a semantic rule pays for an embedding on the request path. Put exact and normalized in front of it so the common case never reaches the embedder.

p99 climbs after the working set grows. The exact scan is linear in the per-scope entry count. Watch over_budget rows, and switch the index rather than the deadline.

An external embedder makes the cache as available as the vendor. It does not fail requests, but every semantic lookup now depends on a network call inside your latency budget. The in-process options do not.