# When a semantic hit is wrong A similar question is not the same question. What similarity matching gets wrong, the four bounds on it, and what to check first. Source: https://tokentriage.com/docs/build/caching-semantic --- import CacheZoneDiagram from '../../../components/CacheZoneDiagram.astro'; import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; Exact and normalized caching cannot serve the wrong answer. Either a request produces a stored key or it doesn't, and if it does, the stored answer is an answer to that request. Semantic caching gives that up. It serves the answer to a *similar* request, and similarity is a number produced by a model with no idea what your users need. The reason to accept that is that paraphrases are common and expensive. The reason to be careful is that the failures are not random: the pairs of questions that embed most closely include some of the pairs whose answers differ most. ## The failure mode, made concrete Here is a semantic rule with a 0.95 threshold, serving from one stored answer to "How do I reset my password?". The captures are from a sandbox using the embedder that ships in the binary: ```text query result similarity How can I reset my password? hit 0.9966 Where do I reset the password? hit 0.9966 I forgot my password, what now? hit 0.9931 reset my password please hit 0.9665 password reset hit 0.9391 (gray band, verifier passed) How do I change my password? hit 1.0000 How do I reset my email? miss ``` The first four are the case for semantic caching. The last is a correct miss. The sixth is the problem: "change my password" is a different request from "reset my password". One is a routine update, the other is an account-recovery flow, and it scored a perfect 1.0. That 1.0 is a property of the embedder shipped in the binary, which is a 26-token, 16-dimension fixture in which `reset` and `change` are literally the same vector. The project's own benchmarks call its recall unusable and use purpose-built vectors instead. It is the right default for tests and for an air-gapped demo, and it is not a production embedder. Configure a real table before you promote a semantic rule. See [the embedder ladder](/docs/build/caching-embedders). The shape of the failure survives the embedder choice, though. Negation, polarity, a changed entity and a changed number all routinely produce very high cosine similarity, because dense embeddings compress precisely the tokens that flip an answer. ## The four bounds TokenTriage does not try to make similarity safe. It narrows where it applies, adds a second opinion in the uncertain band, keeps approximation out of decisions, and withholds the dollar until something has measured the error rate. ### 1. Most of the request has to match exactly Only the last `history_max_messages` user turns are embedded. The system prompt, the tool schema, the sampling parameters, the assistant turns and every earlier user turn are hashed into the partition the search runs inside. Two requests are compared by similarity only when all of that is byte-identical after canonical encoding. On top of that, three classes are excluded from semantic serving entirely: requests carrying tools or tool results, requests asking for structured output through `response_format` or a forced `tool_choice`, and requests with more user turns than the rule's history bound. These are the cases where a wrong answer is worst, since a mis-shaped JSON payload breaks a caller outright, so they are excluded before any embedding is computed. ### 2. Two thresholds and a verifier in between The diagram shows one score line and three regions. At or above `threshold` a candidate serves without further work. Below `low_threshold` it is a miss. Between them is the gray band, and a candidate there serves only if a synchronous verifier passes. The default verifier is model-free and costs one extra index search. Ordinary recall is one-sided: the candidate is near the query because the query found it. Reciprocity asks the other direction — would the query rank inside the candidate's own ten nearest neighbours? A candidate that only matched because the query drifted into its outer orbit, while the candidate itself sits in a dense cluster of closer paraphrases, fails that test. Failures are safe. A verifier that returns false does not serve. A verifier that errors does not serve. A gray-band candidate with no synchronous verifier configured does not serve, so setting `low_threshold` without a `verifier` gives you the narrow band and none of its benefit. A gray-band serve is disclosed as one: ```console X-Tt-Cache: hit X-Tt-Cache-Kind: semantic X-Tt-Cache-Similarity: 0.9391311407089233 X-Tt-Cache-Threshold: 0.95 X-Tt-Cache-Verified: true ``` A score below the threshold with `verified: true` beside it is the band working. A confident-zone hit carries `verified: none`, because none ran. ### 3. An approximate score does not decide anything The default recall index is an exact scan, so its scores are exact cosines. If you switch to the approximate index for a large working set, its scores are used only to shortlist: the exact cosine is recomputed against each candidate's stored vector before the threshold test. The row then carries `recall_exact: false` and the index's configuration, so a hit recalled approximately is not presented as one recalled exactly. ### 4. The saving waits for evidence A semantic hit is counted immediately and its dollar is withheld. From the same sandbox — one semantic hit, and the licensed total: ```json "savings": { "usd": 0, "estimated": true, "basis": "response_cache hit rows, saved repriced from source usage (hit-only)", "verdict_ref": "/tt/api/v1/cache/evidence", "excluded_semantic": 1 } ``` The row for that hit carries `"saved": {"usd": 0.00039}`. The aggregate reports `$0` and discloses that one hit was excluded. Following the verdict reference explains why: ```json { "kind": "semantic", "hits": 1, "posture": "cannot_measure", "licensed": false, "reason": "no verifier configured (pre-RC3): semantic savings are excluded-and-counted, never a dollar" } ``` [What a cache saving means](/docs/build/caching-savings) is where that machinery is documented. ## Calibrating instead of guessing at a threshold A cosine threshold is not something an operator can reason about. Nobody knows what 0.95 buys them. The optional `verified` block replaces it with a quantity you can state: the wrong-hit rate you are willing to accept. ```yaml semantic: embedder: potion threshold: 0.95 low_threshold: 0.85 verifier: kreciprocal history_max_messages: 3 verified: delta: 0.05 # the wrong-hit budget you accept min_samples: 200 # labels needed before calibration may fire confidence: 0.95 # the confidence level for the bound ``` With it configured, an off-path grader labels sampled traffic for whether a cached answer would have answered as well as the fresh one, and those labels are used to search a threshold grid for the lowest bar whose lower-bounded correct-serve rate still clears `1 − delta`. That bar becomes the effective threshold and is what the row reports. Below `min_samples` labels it declines rather than guessing: ```json { "kind": "semantic", "hits": 1, "posture": "cannot_measure", "licensed": false, "reason": "verified policy active; 0 labeled samples below the sample floor — static threshold stands (excluded-and-counted)" } ``` No grader ships wired. The build injects none, so nothing samples and nothing calls out, and a `verified` block cold-starts at zero labels and stays there until an operator supplies one. Until then the static threshold governs serving and no semantic dollar is licensed. Plan for the semantic tier to cost you money and report no savings. The idea of bounding wrong-hit probability against an operator-set budget calibrated from labels is credited to the vCache paper (arXiv:2502.03771); the procedure here is built from the Wilson score interval over an empirical distribution.
`semantic.verifier` and `semantic.verified` are different mechanisms and the words are almost the same. `verifier` is the synchronous gray-band gate. It decides whether a candidate between the two thresholds is *served*. It runs on the request path and it is model-free by default. `verified` is the calibration policy. It decides whether a semantic hit's dollar is *licensed*, and it depends on off-path labels. It never blocks serving. The evidence surface's `reason` string uses "verifier" for the second sense. A rule with `verifier: kreciprocal` configured and no `verified` block reports "no verifier configured", which is confusing and accurate about the thing it means: no policy licenses the saving.
## What to check before you promote a semantic rule 1. **Are you using a real embedder table?** The bundled one is a fixture. 2. **Have you run it in shadow against your own traffic**, and looked at the would-hit similarities rather than only the rate? 3. **Take twenty of your highest-scoring would-hits and read both questions.** If any pair differs in negation, in an entity, in a number, or in a date, raise the threshold or drop the tier. 4. **Can you state a wrong-hit rate you would accept?** If not, you cannot configure `verified`, and you are relying on a cosine number nobody can interpret. 5. **Is a wrong answer recoverable in your product?** A support answer someone can re-ask is a different risk from a number someone pastes into a report. ## What can go wrong **Your hit rate is great and your users are complaining.** Check the similarity distribution of served hits and the pairs at the top of it. High scores are where near-miss pairs live. **The tier costs money and reports no savings.** Expected: a metered embedder charges on every lookup that reaches the semantic link, hit or miss, and the semantic saving is withheld until a calibrated policy licenses it. That spend lands on the row as `embed_cost` and on [the metacost surface](/docs/build/cost-metacost).