Troubleshooting

Symptom-first. Find your symptom; each entry gives the cause and the fix. For the full machine-readable code list, see Errors & problem+json.

“I set require_key: required and everything 401s”

Cause. require_key: required is default-deny: any request that does not present a ttv_-prefixed bearer key resolves as anonymous and gets an honest 401 invalid_key. A presented-but-unresolvable key also 401s. And require_key is restart-required — it is not hot-reloadable, so you cannot walk it back without a bounce.

Fix. Stage the cutover so no keyed traffic is ever refused:

1. Mint ttv_ keys and roll them out to every client first (the secret is shown once at mint).
2. Run require_key: optional. Keys are honored and attributed, but anonymous traffic still flows — no 401 — so you can confirm every real caller is now presenting a key.
3. Only once traffic is fully keyed, restart into require_key: required.

governance:
  enabled: true
  require_key: optional   # keys honored + attributed; anonymous still flows (no 401)
  keys:
    prefix: "ttv_"        # display/namespace only

The same observe-before-enforce ladder applies per limit: a budget or quota climbs posture: { mode: off } → posture: { mode: shadow } (records a would_refuse, never refuses) → mode: on. See Governance for the full trust ladder.

“Every row shows UNPRICED”

Cause. No price resolved for that model against the active pricing snapshot. UNPRICED is not zero — it is a null cost that is excluded and counted, never asserted as $0. In top, a group with no priced row renders its cost cell as - (not the literal string UNPRICED), with a footnote below the table:

COST excludes 4 unpriced rows of 198 (no price resolved; the $0 is a missing price, not $0 of spend)

Fix. If the model really costs money, add it to your pricing so a price resolves. If the model is genuinely free (a local endpoint, say), declare it free rather than leaving it unpriced — a declared free price is an explicit $0 with unpriced: false, which is kept distinct from “unknown” on purpose. Mark the upstream free:

upstreams:
  local:                              # upstreams is a map keyed by name
    base_url: http://127.0.0.1:11434  # host only — the inbound /v1 is appended
    dialect: openai
    pricing: free                     # explicit $0 (unpriced: false) — a declared-free price, not a missing one

“Routing isn’t firing / why did this route the way it did?”

Cause. Two common non-bugs. First, enforcement only happens in mode: route — in ledger and shadow the policy is evaluated and logged but nothing enforces, so a “route” you expected simply was not applied. Second, a { model, upstream } target with no tier does not enforce a route in route mode (enforcement keys off decision.tier), so a tier-less target safely drops to passthrough rather than silently mis-routing.

Fix. Read the decision’s own explanation — it is built on the hot path for every request, not optional telemetry. It prints as text (not JSON):

$ tokentriage policy explain --req-id <id> --log ~/.tokentriage/decisions.jsonl

Or read the same block over GET /tt/api/v1/requests/{id} or the dashboard record drawer. The explanation shows req_id, policy_rev, the decision line, the node path, a trace[] of visited nodes in first-read order (each condition’s read x = v), and per-candidate scores[] / filtered[] for a select. Confirm the top-level mode is route, and resolve any bare { model, upstream } leaf to a { tier } when you want it enforced. See Policy language for the full explanation shape.

“Coordination looks off across replicas”

Cause. When the shared coordination store is unreachable, coordination is suspended and fail-open — traffic still flows; each replica falls back to enforcing its local limit alone, so the fleet-wide admitted rate can widen. This is disclosed, not silent.

Fix. Read the disclosure, which appears on three surfaces from the same one fact:

• the response header x-tt-coordination: suspended;
• the decision row’s suspended:[<limit_id>];
• the governance-limits panel, whose bound widens to membership_stale, ≤ N × limit.

A healthy fleet carries no coordination header — the absence is the default, so if you never see the header, coordination is fine. If you do, restore the shared store (Redis / Postgres) and the fleet re-converges. See Multi-replica & scale-out for the convergence bounds and the fail-open ladder.

“The dashboard won’t come up”

Cause. An admin-plane bind failure is non-fatal — the data plane keeps serving traffic, and the dashboard line is simply omitted from the boot banner. The failure is disclosed loudly on stderr:

dashboard unavailable: <err> — proxy traffic unaffected (run: tokentriage doctor)

Fix. The dashboard mounts on the server.admin bind, so check that bind:

• Confirm dashboard.enabled: true (it is opt-in; default off).
• Make sure the server.admin port is not already in use.
• If dashboard.auth.mode: off, the server.admin bind must be loopback (e.g. 127.0.0.1:9090) — an off auth mode on a non-loopback admin bind is rejected at validation. Either bind admin to loopback, or set an auth mode:

server:
  admin: "127.0.0.1:9090"   # off auth requires a loopback admin bind
dashboard:
  enabled: true
  auth:
    mode: off               # off | token | users

“My guard fires on nothing”

Cause. A missing fire has several explanations; execution receipts distinguish selection, inspection and findings.

Fix. Work down the list:

• No retained findings are recorded for this ID. curl -s …/tt/api/v1/guardrails/evidence | jq '.configured[] | select(.fired == false) | .id' is the fired-on-nothing roster. Put it on a schedule — see Guardrails API.
• A detector was never armed. Empty detector bindings are rejected. format has no detector; custom only contains the drill.echo test stub. Check activation and the effective profile.
• The detector is bound to a phase that never carries that content. An output-phase guard on text your users only ever send inbound will sit at zero forever.
• The text is outside a supported inspection surface, or inspection did not complete. Read extraction and execution states, including unsupported shapes and exhausted bounds. See What gets inspected.

“A secret in a tool definition was not caught”

Cause. Tool definitions, tool_choice, response_format and envelope metadata remain outside the mutation surface. They describe the capability the client asked for. Tool-call arguments are different: supported messages[].tool_calls[].function.arguments are decoded and inspected.

Fix. Keep credentials out of tool schemas. Check the supported dialect/path and extraction receipt for tool arguments; malformed or unsupported shapes produce coverage gaps subject to mode and failure posture. See What gets inspected.

“The bytes are not changing in enforce”

Cause. Selection, effective mode, a missing finding or a failed transformation may explain it.

Fix. Check the active profile and binding mode, then read execution stage, action/effect and delivery together. Monitor never mutates. Stock spanless mutation bindings and unavailable actions are rejected during activation. A hit alone is not proof that a transformation committed.

“detector_timeout or budget_exceeded is climbing”

Cause. A detector may exceed its own granted deadline. Cumulative embedded detector work can also exhaust the request’s local_budget_ms; bounded external call/byte/concurrency work uses budget_exceeded too. The execution stage identifies the failed operation.

Fix. Compare actual supported document sizes and measured work against the applicable bound. Do not treat historical microbenchmarks as a current host latency guarantee. Increasing the local CPU budget does not repair external call admission, malformed content or stream retention loss. Continuation follows mode and failure posture; acknowledged enforcing/closed can withhold output.

“caller_canceled is climbing”

Cause. The request context was cancelled, possibly before inspection, during a real external call or after partial stream delivery. The reason alone proves neither zero work nor zero delivery.

Fix. Inspect invocation actual-send state and the delivery receipt, then investigate client or network cancellation. Raising local_budget_ms does not repair caller cancellation.

“My guardrail config will not boot”

Cause. The layer refuses several configurations at boot rather than arming something that would under-deliver silently.

Fix. The refusal names itself and the cure; they are enumerated with their verbatim messages in Errors & problem+json. guardrails explain --config <file> validates the file, so you can find these without a restart.