# Operating the cache Read what the cache is doing, estimate before you enable it, invalidate entries, and understand what a purge can and cannot reach. Source: https://tokentriage.com/docs/build/caching-operations --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; Four surfaces answer the four questions an operator has about a running cache: what is it doing, may I trust the saving, what would it have done, and how do I get rid of an entry. | Surface | Answers | Role floor | |---|---|---| | `GET /tt/api/v1/cache/stats` | what is it doing | `viewer`, scope `cache:read` | | `GET /tt/api/v1/cache/evidence` | may I trust the saving | `viewer`, scope `cache:read` | | `GET /tt/api/v1/cache/preview` | what would it have done | `viewer`, scope `cache:read` | | `POST /tt/api/v1/cache/purge` | remove entries | `operator`, scope `cache:write` | All four report the feature as absent, with a hint naming the config that enables it, when the cache is not wired. None of them fabricates a success. ## What the cache is doing ```console $ curl -sS "$TT_ADMIN/tt/api/v1/cache/stats" { "outcomes": { "hit": 2, "would_hit": 3, "miss": 3, "bypass": 2, "error": 0 }, "hits_by_kind": { "exact": 2 }, "savings": { "usd": 0.00078, "estimated": true, "unpriced": false, "basis": "response_cache hit rows, saved repriced from source usage (hit-only)", "verdict_ref": "/tt/api/v1/cache/evidence" }, "shadow_savings": { "usd": 0.00117, "estimated": true, "unpriced": false, "basis": "shadow-mode would-save, nothing avoided" } } ``` Every outcome field is always present, so a zero is a fact rather than a missing key. The response also carries a provenance envelope describing how much of the log was scanned; it is trimmed here. A read that hits its deadline or its concurrency limit returns a valid response with `provenance.truncated: true` rather than an error, so a partial answer says it is partial. ## Whether the saving is trustworthy ```console $ curl -sS "$TT_ADMIN/tt/api/v1/cache/evidence" { "kinds": [ { "kind": "exact", "hits": 2, "posture": "licensed", "licensed": true } ], "age_distribution": [ { "label": "<1m", "count": 0 }, { "label": "<1h", "count": 2 }, { "label": "<1d", "count": 0 }, { "label": ">=1d", "count": 0 } ], "excluded_semantic": 0 } ``` Each hit kind reports one of two states. `licensed` means the savings rest on replay evidence. `cannot_measure` is a refusal with a reason attached, and it is the normal state for semantic hits until a calibration policy has enough labels. All four age buckets render even when empty, so "no entries in this band" is distinguishable from "this band was not reported". The age distribution is the number to watch during a rollout. If your hits are clustered in the oldest bucket and your content changes daily, the TTL is too long regardless of what the hit rate says. ## What it would have done, before you enable it Shadow mode measures. This estimates, over rows already in your log, and it exists for the case where you cannot run shadow first. ```console $ tokentriage cache preview --log ~/.tokentriage/decisions.jsonl --since 30d ``` Three approximation labels travel on every response and cannot be dropped. `fixed_usage`: the would-save reprices each repeat row's recorded tokens, not the tokens a fresh call would produce. `recorded_runtime_state`: it reads logged facts, not live state. `fingerprint_approximated`: the ledger stores a messages-only prompt hash rather than the full cache key. That last label is the one that changes how you read the number. The estimate's grouping key is coarser than the cache's, so its repeat count is an upper bound. On a sandbox where the cache produced two hits and three would-hits over ten requests, the preview reported eight repeat rows and an 80.0% repeat rate, because it also counted a request carrying tools, one that opted out, one from a different tenant, and one whose `temperature` differed. The semantic tier declines rather than guessing: re-embedding needs captured response bodies, which live in the content store rather than the ledger, so a ledger-only preview reports its coverage and says it cannot estimate. `cache preview` is not a dry run of a purge. It cannot see which entries a purge would remove, it takes no `by` or `value`, and nothing links the two operations. There is no preview for purge. ## Removing entries ```console $ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" \ -H 'content-type: application/json' \ -d '{"by":"rule","value":"chat-cache"}' {"by":"rule","value":"chat-cache","deleted":2} ``` `by` is a closed dimension: `entry`, `scope` or `rule`. An unrecognised value is refused rather than treated as a no-op, because a purge that matched nothing would look identical to one that worked: ```console $ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" \ -H 'content-type: application/json' -d '{"by":"nonsense","value":"x"}' {"type":"/tt/api/problems/bad_request","title":"Bad request","status":400,"detail":"cache/purge: by must be one of entry|scope|rule","code":"bad_request","request_id":"req_47ab232b1bdbca71"} ``` `deleted` is the count physically removed. It is often smaller than an operator expects, and seeing `0` is informative. Take the `entry` handle from a hit's `x-tt-cache-entry` header when you need to remove one specific answer someone complained about. ### What a purge can reach A purge only removes entries this process has written since it started. The tracking that maps a scope or a rule to its entries is in-memory, so after a restart a purge by scope or by rule reports `deleted: 0` while the entries are still on disk and still serving. Reproduced on a sandbox with a durable tier. Entries were written, the process was restarted, and then: ```console $ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" -d '{"by":"rule","value":"chat-cache"}' {"by":"rule","value":"chat-cache","deleted":0} ``` One request was then sent so the process wrote entries of its own, and the same purge ran again: ```console $ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" -d '{"by":"rule","value":"chat-cache"}' {"by":"rule","value":"chat-cache","deleted":2} ``` Two entries died: the ones this process wrote. The pre-restart entry still served afterwards, with its original age and source id. Plan around it: to invalidate reliably across a restart, change the key instead — bump `cache.scope`, change the rule's matcher chain, or stop the process and delete the L2 file, which costs you a warm cache and nothing else. ### What is audited Every purge attempt writes one audit row — accepted, rejected or failed — carrying the actor, the dimension, the value and the resulting count, and each physically deleted entry writes a `cache.delete` row. So a purge is attributable after the fact even though it cannot be rehearsed beforehand. It is attributable, not reconstructable. The rows record who purged along which dimension and how many entries died. They do not record which ones, and no read surface can show you the match set of a scope or rule purge before or after. Budget evictions are not audited. Nothing in the audit log records an entry dropped because the L1 entry cap or the L2 byte budget displaced it. See [the store](/docs/build/caching-store#what-actually-reclaims-space).
A build with a semantic rule and no grader wired narrates that at startup rather than leaving it to be discovered: ```text WARN cache explore/label loop is INERT: no observations will be produced reason=no_equivalence_judge consequence=the verified-semantic threshold stays at its static value and no calibration evidence accrues ``` If you configured a `verified` block and expect it to license a semantic dollar, this line is the answer to why it never does.
## What can go wrong **Stats look empty on a busy deployment.** The surfaces read the decision log. If the cache is not enabled, no rows carry cache fields and every count is zero. **A purge reports success and the answer keeps coming back.** Almost always the restart boundary above. Check whether the process has restarted since the entry was written. **The preview number gets quoted as a saving.** It is a ceiling over historical rows, and it says so on every response. Run shadow before anyone forwards a figure.