How a request becomes a cache key
The seven fingerprint segments and the scope prefix that decide whether two requests can share a stored answer.
On this page
The most common question anyone asks about a cache is why a request they thought was a repeat didn’t hit. Almost every answer is the same shape: something the key includes was different, and you weren’t thinking of it as part of the request.
So it is worth knowing what the key is made of. There are two parts, and they answer two different questions. The fingerprint answers “would this request produce the same response?” The scope answers “who is allowed to share that response?” An entry’s identity is the hash of both, and a lookup that has one without the other cannot read anything.
Read it left to right: the request splits into the seven fingerprint segments on top and the scope segments below, each side is hashed, and the two hashes are hashed together into the entry id that storage is keyed on.
The fingerprint: would this produce the same answer?
Seven segments, joined with a length prefix on each so no two different splits can produce the same byte stream, then hashed with SHA-256.
| Segment | Why it is in the key |
|---|---|
| dialect id | the same conversation in the OpenAI and Anthropic dialects produces different client bytes |
| endpoint kind | chat and responses return different shapes |
| served provider | the provider that actually answered, not the one the client asked for |
| served model | the model after any routing rewrite, not the requested name |
| profile id | exact or normalized; the same content under two profiles is two entries |
| canonical material | the meaning-bearing content, below |
| stream flag | a streamed entry cannot answer a non-streamed request |
The two bolded segments are the ones that surprise people. The lookup runs after routing, so the key names the upstream and model that would serve this request now. Change a tier’s model, route a request to a different upstream, or add a model rewrite, and every entry stored under the old identity becomes unreachable. That is the correct behaviour, because a stored Haiku answer is not a Sonnet answer, and it means a routing change silently empties the effective cache.
The canonical material
The material is not the raw request body. It is the response-determining content, re-encoded canonically so that formatting cannot fragment the cache:
- the message array, verbatim per message and key-sorted
- the system prompt, when present
- the tool schema, when present
- the sampling-parameter class
Whitespace between JSON tokens, key order, number formatting and Unicode escaping all disappear into the canonical encoding. Two clients that serialize the same conversation differently share an entry.
The parameter class is the part worth reading twice, because each of these makes a new entry:
temperature · top_p · max_tokens · n · frequency_penalty · presence_penalty ·
response_format · tool_choice · seed · stream_options.include_usage · reasoning_effort ·
size · quality
An absent parameter and a parameter set to its default value are not the same thing. A client that
starts sending temperature: 1.0 explicitly stops matching entries written before it did.
What the fingerprint leaves out, and why
The parsed request carries derived fields that are not key material. Counts and sizes
(messages, prompt_bytes, system_bytes, tool_count) are excluded because the content they
summarize is already in the canonical material, and including a count as well would only add a way
to disagree with it. The messages-only prompt hash is excluded for the same reason: the canonical
material supersedes it, and it covers less.
One exclusion is a decision rather than a redundancy: whether the request carries provider
prompt-cache markers. Those affect the provider’s billing for a call, not the content of the
response, so two requests differing only in a cache_control breakpoint share an entry.
The exclusion list is enforced at compile time. Adding a field to the parsed-request type breaks the build in the key derivation until someone decides whether the new field belongs in the key.
The scope: who may share it
The scope key is a mandatory prefix of every storage key and every index partition. An entry written under one scope is not reachable from a probe under another: not filtered out afterwards, but unable to be addressed. It is hashed from four things:
| Segment | What it does |
|---|---|
| cache generation epoch | lets a whole generation of entries be abandoned at once |
| matcher id | exact, normalized and semantic each own their entries |
| embedder fingerprint | empty for non-semantic matchers; changing the embedder repartitions |
| the rule’s ordered scope dims | tenant=acme, user=…, and so on |
The matcher id in the scope is why a three-matcher chain writes three entries per miss, and why
promoting a rule from [exact] to [exact, normalized] starts cold for the new tier.
Which dimensions appear, and what changing them does to isolation and to hit rate, is cache scope.
The semantic key is a different shape
Semantic matching has no fingerprint to compare, because the point is that the text differs. It partitions instead. The scope key is folded together with the context material, meaning everything response-determining that the embedded text leaves out, into a partition. Recall runs only inside it.
Concretely, for a rule with history_max_messages: 3, the embedded text is the last three user
turns joined with newlines. The system prompt, the tool schema, the parameter class, the assistant
turns and any earlier user turns are context material. Two requests can only be compared by
similarity when all of that matches byte for byte after canonical encoding.
Two entry ids, in practice
The row carries the first seven characters of each hash: entry_sha7 and scope_sha7. The
response header carries the full entry id, which is also the handle a per-entry purge takes.
"entry_sha7": "3a2e388",
"scope_sha7": "799cda7",Two rows sharing a scope_sha7 were looked up in the same partition. Two rows sharing an
entry_sha7 matched the same stored answer. A raw tenant id never appears in either.
What can go wrong
A deploy empties the cache without deleting anything. Changing a tier’s model, adding a rewrite,
switching a rule’s matcher chain, changing cache.scope, or pointing a semantic rule at a different
embedder all change the key. The old entries are still on disk and no probe can reach them.
A client library upgrade stops the hits. If it starts sending a parameter it used to omit, every request has a new key. Compare two request bodies before you widen a TTL.