# Troubleshooting Symptom-first fixes for the traps operators actually hit — 401 storms, UNPRICED rows, silent routing, cross-replica drift, and a missing dashboard. Source: https://tokentriage.com/docs/reference/troubleshooting --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; Symptom-first. Find your symptom; each entry gives the cause and the fix. For the full machine-readable code list, see [Errors & problem+json](/docs/reference/errors). ## "I set `require_key: required` and everything 401s" **Cause.** `require_key: required` is default-deny: any request that does not present a `ttv_`-prefixed bearer key resolves as anonymous and gets an honest `401 invalid_key`. A presented-but-unresolvable key also 401s. And `require_key` is **restart-required** — it is not hot-reloadable, so you cannot walk it back without a bounce. `require_key: required` is a one-way cutover you cannot un-flip at runtime. Do **not** flip it straight from `off` in production — mint and roll out keys first, or you will 401 all live traffic until you can restart. On a cold-booted replica with no loadable identity snapshot, `required` returns `503` ("cannot verify"), never a fabricated 401. **Fix.** Stage the cutover so no keyed traffic is ever refused: 1. Mint `ttv_` keys and roll them out to every client first (the secret is shown once at mint). 2. Run `require_key: optional`. Keys are honored and attributed, but anonymous traffic still flows — no 401 — so you can confirm every real caller is now presenting a key. 3. Only once traffic is fully keyed, restart into `require_key: required`. ```yaml governance: enabled: true require_key: optional # keys honored + attributed; anonymous still flows (no 401) keys: prefix: "ttv_" # display/namespace only ``` The same observe-before-enforce ladder applies per limit: a budget or quota climbs `posture: { mode: off }` → `posture: { mode: shadow }` (records a `would_refuse`, never refuses) → `mode: on`. See [Governance](/docs/build/governance) for the full trust ladder. ## "Every row shows UNPRICED" **Cause.** No price resolved for that model against the active pricing snapshot. **UNPRICED is not zero** — it is a null cost that is *excluded and counted*, never asserted as `$0`. In `top`, a group with no priced row renders its cost cell as `-` (not the literal string `UNPRICED`), with a footnote below the table: ```text COST excludes 4 unpriced rows of 198 (no price resolved; the $0 is a missing price, not $0 of spend) ``` **Fix.** If the model really costs money, add it to your pricing so a price resolves. If the model is genuinely free (a local endpoint, say), declare it free rather than leaving it unpriced — a *declared* free price is an explicit `$0` with `unpriced: false`, which is kept distinct from "unknown" on purpose. Mark the upstream free: ```yaml upstreams: local: # upstreams is a map keyed by name base_url: http://127.0.0.1:11434 # host only — the inbound /v1 is appended dialect: openai pricing: free # explicit $0 (unpriced: false) — a declared-free price, not a missing one ``` "We spent nothing on this model" and "we don't know what we spent on this model" are different statements, so TokenTriage never collapses one into the other. See the [Honesty model](/docs/concepts/honesty-model) for the UNPRICED-is-not-zero rule. ## "Routing isn't firing / why did this route the way it did?" **Cause.** Two common non-bugs. First, enforcement only happens in `mode: route` — in `ledger` and `shadow` the policy is evaluated and logged but **nothing enforces**, so a "route" you expected simply was not applied. Second, a `{ model, upstream }` target with no `tier` does not enforce a route in `route` mode (enforcement keys off `decision.tier`), so a tier-less target safely drops to passthrough rather than silently mis-routing. **Fix.** Read the decision's own explanation — it is built on the hot path for every request, not optional telemetry. It prints as **text** (not JSON): ```console $ tokentriage policy explain --req-id --log ~/.tokentriage/decisions.jsonl ``` Or read the same block over `GET /tt/api/v1/requests/{id}` or the dashboard record drawer. The explanation shows `req_id`, `policy_rev`, the `decision` line, the node `path`, a `trace[]` of visited nodes in first-read order (each condition's `read x = v`), and per-candidate `scores[]` / `filtered[]` for a `select`. Confirm the top-level `mode` is `route`, and resolve any bare `{ model, upstream }` leaf to a `{ tier }` when you want it enforced. See [Policy language](/docs/build/policy-language) for the full explanation shape. ## "Coordination looks off across replicas" **Cause.** When the shared coordination store is unreachable, coordination is **suspended and fail-open** — traffic still flows; each replica falls back to enforcing its local limit alone, so the fleet-wide admitted rate can widen. This is disclosed, not silent. **Fix.** Read the disclosure, which appears on three surfaces from the same one fact: - the response header `x-tt-coordination: suspended`; - the decision row's `suspended:[]`; - the governance-limits panel, whose bound widens to `membership_stale`, `≤ N × limit`. A healthy fleet carries **no** coordination header — the absence is the default, so if you never see the header, coordination is fine. If you do, restore the shared store (Redis / Postgres) and the fleet re-converges. See [Multi-replica & scale-out](/docs/operate/scale-out) for the convergence bounds and the fail-open ladder. ## "The dashboard won't come up" **Cause.** An admin-plane bind failure is **non-fatal** — the data plane keeps serving traffic, and the dashboard line is simply omitted from the boot banner. The failure is disclosed loudly on stderr: ```text dashboard unavailable: — proxy traffic unaffected (run: tokentriage doctor) ``` **Fix.** The dashboard mounts on the `server.admin` bind, so check that bind: - Confirm `dashboard.enabled: true` (it is opt-in; default off). - Make sure the `server.admin` port is not already in use. - If `dashboard.auth.mode: off`, the `server.admin` bind **must be loopback** (e.g. `127.0.0.1:9090`) — an `off` auth mode on a non-loopback admin bind is rejected at validation. Either bind admin to loopback, or set an auth mode: ```yaml server: admin: "127.0.0.1:9090" # off auth requires a loopback admin bind dashboard: enabled: true auth: mode: off # off | token | users ``` ## "My guard fires on nothing" **Cause.** A missing fire has several explanations; execution receipts distinguish selection, inspection and findings. **Fix.** Work down the list: - **No retained findings are recorded for this ID.** `curl -s …/tt/api/v1/guardrails/evidence | jq '.configured[] | select(.fired == false) | .id'` is the fired-on-nothing roster. Put it on a schedule — see [Guardrails API](/docs/reference/guardrail-api). - **A detector was never armed.** Empty detector bindings are rejected. `format` has no detector; `custom` only contains the `drill.echo` test stub. Check activation and the effective profile. - **The detector is bound to a phase that never carries that content.** An output-phase guard on text your users only ever send inbound will sit at zero forever. - **The text is outside a supported inspection surface, or inspection did not complete.** Read extraction and execution states, including unsupported shapes and exhausted bounds. See [What gets inspected](/docs/guardrails/what-gets-inspected). ## "A secret in a tool definition was not caught" **Cause.** Tool definitions, `tool_choice`, `response_format` and envelope metadata remain outside the mutation surface. They describe the capability the client asked for. Tool-call arguments are different: supported `messages[].tool_calls[].function.arguments` are decoded and inspected. **Fix.** Keep credentials out of tool schemas. Check the supported dialect/path and extraction receipt for tool arguments; malformed or unsupported shapes produce coverage gaps subject to mode and failure posture. See [What gets inspected](/docs/guardrails/what-gets-inspected). ## "The bytes are not changing in `enforce`" **Cause.** Selection, effective mode, a missing finding or a failed transformation may explain it. **Fix.** Check the active profile and binding mode, then read execution stage, action/effect and delivery together. Monitor never mutates. Stock spanless mutation bindings and unavailable actions are rejected during activation. A hit alone is not proof that a transformation committed. ## "`detector_timeout` or `budget_exceeded` is climbing" **Cause.** A detector may exceed its own granted deadline. Cumulative embedded detector work can also exhaust the request's `local_budget_ms`; bounded external call/byte/concurrency work uses `budget_exceeded` too. The execution stage identifies the failed operation. **Fix.** Compare actual supported document sizes and measured work against the applicable bound. Do not treat historical microbenchmarks as a current host latency guarantee. Increasing the local CPU budget does not repair external call admission, malformed content or stream retention loss. Continuation follows mode and failure posture; acknowledged enforcing/closed can withhold output. ## "`caller_canceled` is climbing" **Cause.** The request context was cancelled, possibly before inspection, during a real external call or after partial stream delivery. The reason alone proves neither zero work nor zero delivery. **Fix.** Inspect invocation actual-send state and the delivery receipt, then investigate client or network cancellation. Raising `local_budget_ms` does not repair caller cancellation. ## "My guardrail config will not boot" **Cause.** The layer refuses several configurations at boot rather than arming something that would under-deliver silently. **Fix.** The refusal names itself and the cure; they are enumerated with their verbatim messages in [Errors & problem+json](/docs/reference/errors#guardrail-boot-refusals). `guardrails explain --config ` validates the file, so you can find these without a restart.