Operating the cache

Four surfaces answer the four questions an operator has about a running cache: what is it doing, may I trust the saving, what would it have done, and how do I get rid of an entry.

All four report the feature as absent, with a hint naming the config that enables it, when the cache is not wired. None of them fabricates a success.

What the cache is doing

$ curl -sS "$TT_ADMIN/tt/api/v1/cache/stats"
{
  "outcomes": { "hit": 2, "would_hit": 3, "miss": 3, "bypass": 2, "error": 0 },
  "hits_by_kind": { "exact": 2 },
  "savings": {
    "usd": 0.00078,
    "estimated": true,
    "unpriced": false,
    "basis": "response_cache hit rows, saved repriced from source usage (hit-only)",
    "verdict_ref": "/tt/api/v1/cache/evidence"
  },
  "shadow_savings": {
    "usd": 0.00117,
    "estimated": true,
    "unpriced": false,
    "basis": "shadow-mode would-save, nothing avoided"
  }
}

Every outcome field is always present, so a zero is a fact rather than a missing key. The response also carries a provenance envelope describing how much of the log was scanned; it is trimmed here.

A read that hits its deadline or its concurrency limit returns a valid response with provenance.truncated: true rather than an error, so a partial answer says it is partial.

Whether the saving is trustworthy

$ curl -sS "$TT_ADMIN/tt/api/v1/cache/evidence"
{
  "kinds": [
    { "kind": "exact", "hits": 2, "posture": "licensed", "licensed": true }
  ],
  "age_distribution": [
    { "label": "<1m", "count": 0 },
    { "label": "<1h", "count": 2 },
    { "label": "<1d", "count": 0 },
    { "label": ">=1d", "count": 0 }
  ],
  "excluded_semantic": 0
}

Each hit kind reports one of two states. licensed means the savings rest on replay evidence. cannot_measure is a refusal with a reason attached, and it is the normal state for semantic hits until a calibration policy has enough labels. All four age buckets render even when empty, so “no entries in this band” is distinguishable from “this band was not reported”.

The age distribution is the number to watch during a rollout. If your hits are clustered in the oldest bucket and your content changes daily, the TTL is too long regardless of what the hit rate says.

What it would have done, before you enable it

Shadow mode measures. This estimates, over rows already in your log, and it exists for the case where you cannot run shadow first.

$ tokentriage cache preview --log ~/.tokentriage/decisions.jsonl --since 30d

Three approximation labels travel on every response and cannot be dropped. fixed_usage: the would-save reprices each repeat row’s recorded tokens, not the tokens a fresh call would produce. recorded_runtime_state: it reads logged facts, not live state. fingerprint_approximated: the ledger stores a messages-only prompt hash rather than the full cache key.

That last label is the one that changes how you read the number. The estimate’s grouping key is coarser than the cache’s, so its repeat count is an upper bound. On a sandbox where the cache produced two hits and three would-hits over ten requests, the preview reported eight repeat rows and an 80.0% repeat rate, because it also counted a request carrying tools, one that opted out, one from a different tenant, and one whose temperature differed.

The semantic tier declines rather than guessing: re-embedding needs captured response bodies, which live in the content store rather than the ledger, so a ledger-only preview reports its coverage and says it cannot estimate.

Removing entries

$ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" \
    -H 'content-type: application/json' \
    -d '{"by":"rule","value":"chat-cache"}'
{"by":"rule","value":"chat-cache","deleted":2}

by is a closed dimension: entry, scope or rule. An unrecognised value is refused rather than treated as a no-op, because a purge that matched nothing would look identical to one that worked:

$ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" \
    -H 'content-type: application/json' -d '{"by":"nonsense","value":"x"}'
{"type":"/tt/api/problems/bad_request","title":"Bad request","status":400,"detail":"cache/purge: by must be one of entry|scope|rule","code":"bad_request","request_id":"req_47ab232b1bdbca71"}

deleted is the count physically removed. It is often smaller than an operator expects, and seeing 0 is informative.

Take the entry handle from a hit’s x-tt-cache-entry header when you need to remove one specific answer someone complained about.

What a purge can reach

Reproduced on a sandbox with a durable tier. Entries were written, the process was restarted, and then:

$ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" -d '{"by":"rule","value":"chat-cache"}'
{"by":"rule","value":"chat-cache","deleted":0}

One request was then sent so the process wrote entries of its own, and the same purge ran again:

$ curl -sS -X POST "$TT_ADMIN/tt/api/v1/cache/purge" -d '{"by":"rule","value":"chat-cache"}'
{"by":"rule","value":"chat-cache","deleted":2}

Two entries died: the ones this process wrote. The pre-restart entry still served afterwards, with its original age and source id. Plan around it: to invalidate reliably across a restart, change the key instead — bump cache.scope, change the rule’s matcher chain, or stop the process and delete the L2 file, which costs you a warm cache and nothing else.

What is audited

Every purge attempt writes one audit row — accepted, rejected or failed — carrying the actor, the dimension, the value and the resulting count, and each physically deleted entry writes a cache.delete row. So a purge is attributable after the fact even though it cannot be rehearsed beforehand.

It is attributable, not reconstructable. The rows record who purged along which dimension and how many entries died. They do not record which ones, and no read surface can show you the match set of a scope or rule purge before or after.

Budget evictions are not audited. Nothing in the audit log records an entry dropped because the L1 entry cap or the L2 byte budget displaced it. See the store.

WARN cache explore/label loop is INERT: no observations will be produced
     reason=no_equivalence_judge
     consequence=the verified-semantic threshold stays at its static value and no calibration evidence accrues

What can go wrong

Stats look empty on a busy deployment. The surfaces read the decision log. If the cache is not enabled, no rows carry cache fields and every count is zero.

A purge reports success and the answer keeps coming back. Almost always the restart boundary above. Check whether the process has restarted since the entry was written.

The preview number gets quoted as a saving. It is a ceiling over historical rows, and it says so on every response. Run shadow before anyone forwards a figure.