The embedder ladder

Exact and normalized caching need no embedder. Semantic caching needs one, and choosing it is the one place in the cache where you trade an air-gapped default for recall quality, latency or money. Each rung trades a single property, and you pick the rung rather than inheriting its cost.

Embedders are named instances. A rule names one; several rules can share one.

cache:
  embedders:
    potion: { kind: static }
  rules:
    - id: chat-hybrid
      strategy: [exact, normalized, semantic]
      semantic:
        embedder: potion

The ladder

kind	Runs	Cost per lookup	Calls out
lexical	in process, pure Go	CPU, microseconds	no
static	in process, pure Go	CPU, microseconds to milliseconds	no
http at a loopback or local address	your host	a network round trip	yes, gated
http at an external address	a vendor	a round trip and real spend	yes, gated

The two in-process kinds are what the default build ships with, and a deployment using them dials nothing at all.

lexical

Sparse similarity with no model: min-hash and sim-hash signatures over term frequency. High precision, low recall, and surface-sensitive in a way that turns out to be useful — it notices the negation, polarity and date differences that dense embeddings smooth over. When it is wrong it is usually wrong toward a miss.

    lex: { kind: lexical }

Reach for it when you want a paraphrase tier that will not serve a near-miss, and you can accept catching fewer paraphrases.

static

A token-to-vector table looked up per token and mean-pooled with weights. No transformer pass, so it costs a table lookup per token rather than a forward pass, and it runs in process with no network.

    potion:
      kind: static
      table_file: /etc/tokentriage/potion-base-8M.tt

http

An OpenAI-compatible /embeddings client, serving both the local-network rung (a model you host) and the external rung (a vendor API). The distinction is the address, and both are refused unless the instance carries external: allow.

    local:
      kind: http
      base_url: "http://127.0.0.1:11434/v1"    # a model you run
      model: nomic-embed-text
      external: allow
    vendor:
      kind: http
      base_url: "https://api.openai.com/v1"
      model: text-embedding-3-small
      api_key_env: OPENAI_API_KEY              # names a variable, never the secret
      external: allow

The external rung is metered: the provider reports token usage and it is priced through the same path as every other dollar in the product.

The gate, and what it means

Any embedder that leaves the process is refused at load unless it says so:

cache.embedders["vendor"].external: an external-class embedder (kind http / non-loopback base_url) requires external: allow

Opening the gate is one auditable line, and it produces a warning naming the instance and its class:

warning: cache.embedders["vendor"]: external_api embedder — enabling it makes the cache PHONE OUT for embeddings (disclosed on every semantic trace; the air-gap posture is broken for this embedder)

An api_key_env value must name an environment variable. A literal secret in that field is a validation error.

What a metered embedder costs, and where you see it

An embedding runs on every lookup that reaches the semantic link, whether or not it produces a hit, because the vector is what the search needs. On a miss it runs once: the store write reuses the vector the lookup already computed rather than embedding a second time.

A priced call lands on the row as embed_cost, alongside embed_us:

"embed_cost": { "usd": 0.0000021, "estimated": false, "unpriced": false },
"embed_us": 412

It is estimated: false because it is a real charge against reported usage, not a counterfactual. If the price does not resolve, the field records unpriced: true and makes no dollar claim rather than reporting zero.

The same spend joins the metacost surface under a response_cache line, which is where the cost of measurement is collected across the whole product. What the measurement itself costs owns that surface; this page owns where the number comes from.

An in-process embedder produces no embed_cost field at all. It is counted as free because it is: CPU time on a process you are already running.

One embed per miss, and the exception

The vector computed during a lookup is carried to the store write, so a metered miss pays for one embedding rather than two. The exception is a write with no lookup vector — a forced refresh, which bypasses the read. That embedding has no request row in flight to land on, so it is charged to the metacost line and counted per rule on the evidence surface instead of being dropped.

Changing an embedder repartitions the cache

The embedder’s fingerprint is part of the scope key for a semantic matcher. Swapping the instance a rule names, or pointing table_file at a different table, makes every existing semantic entry unreachable. Vectors from two different embedders are not comparable, so this is the outcome you want; plan for a cold semantic tier after the change.

Embedder definitions themselves reload hot, but the effect is the same partition change.

What can go wrong

Recall is poor and every paraphrase misses. Almost always the bundled fixture table. Check whether table_file is set.

The semantic tier is slower than expected on a large working set. Embedding is only part of the cost; the search over a large per-scope set is the rest. See budgets and performance.

A vendor outage turns into misses, not errors. An embedding failure is a fault that forwards the request live, and the row records it. The client is served either way.