# What the cache will not do

The eligibility gates, the endpoints that are cacheable, and the responses v1 leaves alone rather than gets wrong.

Source: https://tokentriage.com/docs/build/caching-limits

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';

Some requests are not cached, and some responses are not stored. Both lists are short and closed,
and every case lands on the ledger row with a reason rather than passing silently.

## Cacheable endpoints

Two, in v1: `chat` and `responses`. Everything else — embeddings, models, images, audio — is not
cacheable, and naming one in a rule is a validation error:

```text
cache.rules[0].match.endpoint: endpoint kind "embeddings" is not cacheable (chat|responses only in v1)
```

Embeddings responses are deterministic and repeat heavily, which makes them the obvious next
candidate. The schema admits the endpoint and the tier is held back until the chat path has soaked.

## The eligibility gates

Evaluated in order, after a rule has been selected and before any key is derived. Each produces a
`bypass_reason` on the row.

| Gate | Applies to | Reason recorded |
|---|---|---|
| the client sent `x-tt-cache: off` | the whole cache | `client_off` |
| the request could not be parsed | the whole cache | `non_cacheable_endpoint` |
| the endpoint is not cacheable | the whole cache | `non_cacheable_endpoint` |
| the request declares tools, or carries tool results | the whole cache | `tool_calls` |
| `response_format`, or a `tool_choice` that forces a call | the semantic matcher | `structured_output` |
| more user turns than `history_max_messages` | the semantic matcher | `history_too_long` |

The first four turn the whole request away. The last two are semantic-only: in a chain the semantic
link is skipped and the deterministic matchers still get their turn, because an exact match can
serve a structured-output request perfectly well. You see them as a `bypass_reason` only when
`semantic` is the rule's sole matcher.

<Callout type="warning">
Tool traffic is not cached at all. This catches people who expect an agent loop to be the highest
value cache target: a request carrying a tool schema is bypassed before any matcher runs, whichever
strategy the rule names.
</Callout>

## What is not stored

A miss writes the client-visible response to the store only when all of these hold:

- the status is 2xx
- the body is complete: not truncated by the body limit, and for a stream, a terminal finish reason
  with no error and no capture overflow
- the client did not send `x-tt-cache: no-store`
- the response was not served by a cascade
- the payload is within `cache.store.l2.max_entry_kb`

An aborted stream, a client that disconnected mid-response, an oversize body, and an error status
are all left unstored. The row records `"stored": false`.

The size cap deserves a note. It is why `max_entry_kb` must be strictly below
`server.max_body_bytes`: the observing buffer truncates at the body limit, so an entry cap at or
above it would store truncated bodies as though they were complete. Validation refuses that config:

```text
cache.store.l2.max_entry_kb: 10240 KB (10485760 bytes) must be strictly less than server.max_body_bytes (10485760 bytes)
```

## What v1 does not cache

These are scoped out rather than missing, and each leaves a clean edge for later.

**Binary event-stream responses.** Server-sent-events streaming replays frame by frame and is fully
supported. The AWS `application/vnd.amazon.eventstream` binary framing is not decoded for capture,
so those responses are not stored.

**Cascade-served responses.** When a judge-graded cascade produces the answer, the response does not
travel through the observation point the store rides, so nothing is written and the row records
`"stored": false`. Reads are unaffected: a lookup still short-circuits a cascade, so the read side
is complete and only the write side excludes them.

**Endpoints other than chat and responses**, as above.

**Provider prompt-cache directive injection.** TokenTriage forwards `cache_control`, `cachePoint`
and `cachedContent` markers untouched, and prices the tokens they save. It does not place those
markers for you: doing so mutates the client's bytes and creates stateful objects with their own
storage billing. See [cache accounting](/docs/concepts/cache-accounting).

**Gemini explicit-cache storage fees.** A time-by-size charge that appears in no response body, so
it cannot be attributed to a request. It is named rather than approximated.

**Request coalescing.** Two identical requests in flight at once both go upstream. Single-flight
would be the natural companion feature and it is not shipped; it is named here so it is not assumed.

**Shared state across replicas, by default.** L1 and L2 are per-process. Two replicas keep separate
caches and a cold replica misses. The opt-in Redis tier shares payloads across replicas; the
semantic recall index is still built per process. See [the store](/docs/build/caching-store).

<NextSteps items={[
  {title: "Why didn't this request hit?", href: "/docs/build/caching-misses", description: "Which of these gates you are looking at, and how to tell from the row."},
  {title: "The store: three tiers", href: "/docs/build/caching-store", description: "What each tier holds, what bounds it, and what survives a restart."},
  {title: "Cache accounting", href: "/docs/concepts/cache-accounting", description: "The provider's own prompt cache, and why its discount is never added to a response-cache saving."}
]} />