When a semantic hit is wrong

Exact and normalized caching cannot serve the wrong answer. Either a request produces a stored key or it doesn’t, and if it does, the stored answer is an answer to that request.

Semantic caching gives that up. It serves the answer to a similar request, and similarity is a number produced by a model with no idea what your users need. The reason to accept that is that paraphrases are common and expensive. The reason to be careful is that the failures are not random: the pairs of questions that embed most closely include some of the pairs whose answers differ most.

The failure mode, made concrete

Here is a semantic rule with a 0.95 threshold, serving from one stored answer to “How do I reset my password?”. The captures are from a sandbox using the embedder that ships in the binary:

query                                   result   similarity
How can I reset my password?            hit      0.9966
Where do I reset the password?          hit      0.9966
I forgot my password, what now?         hit      0.9931
reset my password please                hit      0.9665
password reset                          hit      0.9391   (gray band, verifier passed)
How do I change my password?            hit      1.0000
How do I reset my email?                miss

The first four are the case for semantic caching. The last is a correct miss. The sixth is the problem: “change my password” is a different request from “reset my password”. One is a routine update, the other is an account-recovery flow, and it scored a perfect 1.0.

The shape of the failure survives the embedder choice, though. Negation, polarity, a changed entity and a changed number all routinely produce very high cosine similarity, because dense embeddings compress precisely the tokens that flip an answer.

The four bounds

TokenTriage does not try to make similarity safe. It narrows where it applies, adds a second opinion in the uncertain band, keeps approximation out of decisions, and withholds the dollar until something has measured the error rate.

1. Most of the request has to match exactly

Only the last history_max_messages user turns are embedded. The system prompt, the tool schema, the sampling parameters, the assistant turns and every earlier user turn are hashed into the partition the search runs inside. Two requests are compared by similarity only when all of that is byte-identical after canonical encoding.

On top of that, three classes are excluded from semantic serving entirely: requests carrying tools or tool results, requests asking for structured output through response_format or a forced tool_choice, and requests with more user turns than the rule’s history bound. These are the cases where a wrong answer is worst, since a mis-shaped JSON payload breaks a caller outright, so they are excluded before any embedding is computed.

2. Two thresholds and a verifier in between

The diagram shows one score line and three regions. At or above threshold a candidate serves without further work. Below low_threshold it is a miss. Between them is the gray band, and a candidate there serves only if a synchronous verifier passes.

The default verifier is model-free and costs one extra index search. Ordinary recall is one-sided: the candidate is near the query because the query found it. Reciprocity asks the other direction — would the query rank inside the candidate’s own ten nearest neighbours? A candidate that only matched because the query drifted into its outer orbit, while the candidate itself sits in a dense cluster of closer paraphrases, fails that test.

Failures are safe. A verifier that returns false does not serve. A verifier that errors does not serve. A gray-band candidate with no synchronous verifier configured does not serve, so setting low_threshold without a verifier gives you the narrow band and none of its benefit.

A gray-band serve is disclosed as one:

X-Tt-Cache: hit
X-Tt-Cache-Kind: semantic
X-Tt-Cache-Similarity: 0.9391311407089233
X-Tt-Cache-Threshold: 0.95
X-Tt-Cache-Verified: true

A score below the threshold with verified: true beside it is the band working. A confident-zone hit carries verified: none, because none ran.

3. An approximate score does not decide anything

The default recall index is an exact scan, so its scores are exact cosines. If you switch to the approximate index for a large working set, its scores are used only to shortlist: the exact cosine is recomputed against each candidate’s stored vector before the threshold test. The row then carries recall_exact: false and the index’s configuration, so a hit recalled approximately is not presented as one recalled exactly.

4. The saving waits for evidence

A semantic hit is counted immediately and its dollar is withheld. From the same sandbox — one semantic hit, and the licensed total:

"savings": {
  "usd": 0,
  "estimated": true,
  "basis": "response_cache hit rows, saved repriced from source usage (hit-only)",
  "verdict_ref": "/tt/api/v1/cache/evidence",
  "excluded_semantic": 1
}

The row for that hit carries "saved": {"usd": 0.00039}. The aggregate reports $0 and discloses that one hit was excluded. Following the verdict reference explains why:

{
  "kind": "semantic",
  "hits": 1,
  "posture": "cannot_measure",
  "licensed": false,
  "reason": "no verifier configured (pre-RC3): semantic savings are excluded-and-counted, never a dollar"
}

What a cache saving means is where that machinery is documented.

Calibrating instead of guessing at a threshold

A cosine threshold is not something an operator can reason about. Nobody knows what 0.95 buys them. The optional verified block replaces it with a quantity you can state: the wrong-hit rate you are willing to accept.

      semantic:
        embedder: potion
        threshold: 0.95
        low_threshold: 0.85
        verifier: kreciprocal
        history_max_messages: 3
        verified:
          delta: 0.05         # the wrong-hit budget you accept
          min_samples: 200    # labels needed before calibration may fire
          confidence: 0.95    # the confidence level for the bound

With it configured, an off-path grader labels sampled traffic for whether a cached answer would have answered as well as the fresh one, and those labels are used to search a threshold grid for the lowest bar whose lower-bounded correct-serve rate still clears 1 − delta. That bar becomes the effective threshold and is what the row reports.

Below min_samples labels it declines rather than guessing:

{
  "kind": "semantic",
  "hits": 1,
  "posture": "cannot_measure",
  "licensed": false,
  "reason": "verified policy active; 0 labeled samples below the sample floor — static threshold stands (excluded-and-counted)"
}

The idea of bounding wrong-hit probability against an operator-set budget calibrated from labels is credited to the vCache paper (arXiv:2502.03771); the procedure here is built from the Wilson score interval over an empirical distribution.

Two things called a verifier

semantic.verifier and semantic.verified are different mechanisms and the words are almost the same.

verifier is the synchronous gray-band gate. It decides whether a candidate between the two thresholds is served. It runs on the request path and it is model-free by default.

verified is the calibration policy. It decides whether a semantic hit’s dollar is licensed, and it depends on off-path labels. It never blocks serving.

The evidence surface’s reason string uses “verifier” for the second sense. A rule with verifier: kreciprocal configured and no verified block reports “no verifier configured”, which is confusing and accurate about the thing it means: no policy licenses the saving.

What to check before you promote a semantic rule

1. Are you using a real embedder table? The bundled one is a fixture.
2. Have you run it in shadow against your own traffic, and looked at the would-hit similarities rather than only the rate?
3. Take twenty of your highest-scoring would-hits and read both questions. If any pair differs in negation, in an entity, in a number, or in a date, raise the threshold or drop the tier.
4. Can you state a wrong-hit rate you would accept? If not, you cannot configure verified, and you are relying on a cosine number nobody can interpret.
5. Is a wrong answer recoverable in your product? A support answer someone can re-ask is a different risk from a number someone pastes into a report.

What can go wrong

Your hit rate is great and your users are complaining. Check the similarity distribution of served hits and the pairs at the top of it. High scores are where near-miss pairs live.

The tier costs money and reports no savings. Expected: a metered embedder charges on every lookup that reaches the semantic link, hit or miss, and the semantic saving is withheld until a calibrated policy licenses it. That spend lands on the row as embed_cost and on the metacost surface.