# When a provider won't connect Work from the symptom you have to the field that explains it and the config key that fixes it. Source: https://tokentriage.com/docs/build/providers-troubleshooting --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; You have a symptom and not much else: a 401 from somebody, a row with no cost on it, or traffic going somewhere you did not configure. This page starts from those, not from the model. Run the three checks first. Most provider problems are caught by one of them before a request is ever sent, and they take a few seconds together. ```console $ tokentriage validate --config tokentriage.yaml # shape, params, auth pairing $ tokentriage doctor --config tokentriage.yaml # variables actually set, ports, pricing $ curl -s localhost:9090/tt/api/status # a running instance's view ``` ## Start from what you have | You are looking at | Go to | |---|---| | a validation error | [config that will not load](#config-that-will-not-load) | | `validate` turning away two settings that are each legal | [two features that will not load together](#two-features-that-will-not-load-together) | | 401 or 403 from the provider | [credential problems](#credential-problems) | | 404 from the provider | [url problems](#url-problems) | | HTTP 200 and no cost on the row | [a row with no dollar](#a-row-with-no-dollar) | | two surfaces disagreeing about how many rows are unpriced | [a row with no dollar](#a-row-with-no-dollar) | | a stream that stopped early, or one upstream 400ing on every stream | [when a stream misbehaves](#when-a-stream-misbehaves) | | traffic reaching an upstream you did not pick | [routing surprises](#routing-surprises) | | everything intermittently failing | [cooldown and reachability](#cooldown-and-reachability) | ## Config that will not load Validation errors name the YAML path, so read the path before the prose. | Message contains | Cause | Fix | |---|---|---| | `does not name a known provider` | the descriptor id is not in the registry | check the id in [the reference](/docs/build/providers-reference) | | `base_url: must not be empty` | `provider:` set without a host | descriptors carry relative paths only | | `is not a template variable of provider` | a `params:` key the descriptor does not use | the message lists the allowed set | | `requires template variable … (missing)` | a required `params:` key absent | supply it | | `params.model` or `params.region` | putting either in `params:` | `model` is per-tier; `region` has its own field | | `requires a region` | a Bedrock upstream with no `region:` | add it; it signs *and* prices | | `the multi-part auth.* fields are not valid here` | an `auth:` block on bearer or header | use `api_key_env` | | `this looks like a secret value` | a key literal in a `*_env` field | put the variable name there | | `unknown variable {…} at offset` | an inline `paths` template | only six variables are allowed | | `REFUSED while routing.allow_cross_dialect_streaming` | two features that will not run together | [flip one of them](#two-features-that-will-not-load-together) | The binary currently prints each validation error twice: the command layer prints it, then the top-level handler prints it again, because the root command silences usage without silencing errors. Warning lines double for their own reason, a registry pass re-emitting what the load already emitted. Ignore the repeats; every distinct line you see is genuine, and two copies of one message are not two problems. ## Two features that will not load together Stream-phase guardrail inspection and translated streams cannot both be armed in this build, and the config is turned away at validation rather than at the first request. Arm the guardrail layer, give any guard the `stream` phase, set `routing.allow_cross_dialect_streaming: true`, and `validate` exits 1: ```console $ tokentriage validate --config t7-guard-stream-vs-translated-stream.yaml Error: configuration is invalid: config D:/tt-w01-neutral/work/t7-guard-stream-vs-translated-stream.yaml: guardrails.guards[0].phases[1]: guard "pii-stream" inspects at the "stream" phase, which is REFUSED while routing.allow_cross_dialect_streaming is true (BB14). The stream-framing gate supports exactly (openai, chat) and is armed from the ROUTE before the first byte relays (internal/proxy/stream_framing.go:34-97,127-129); it assumes streams never cross dialects, so on a translated stream it would arm an inspector over frames it cannot decode and produce a "guarded, clean" row. Turn off one of the two: drop the "stream" phase from this guard, or set routing.allow_cross_dialect_streaming: false ``` The absolute path is the capture machine's and yours carries your own; the array index and the guard id belong to the config that was written to trip the check; the sentence is the binary's. The second copy the binary prints is cut here, for the reason the note above gives. Grep your terminal for `REFUSED while routing.allow_cross_dialect_streaming` and you have found this. Either single flip loads clean, and the two are alternatives rather than a ranked pair. Each was run through `validate` as a one-line edit of the config that produced the refusal above, and each printed `OK: configuration is valid`. ```yaml # A: keep stream-phase inspection, turn translated streams off. routing: allow_cross_dialect_streaming: false # B: keep translated streams, take the stream phase off the guard. guardrails: guards: - id: pii-stream phases: [input] ``` A guard with no `stream` phase does not trip the check, and neither does a `guardrails.enabled: false` layer. The check reads the config rather than your traffic, so it is wider than the conflict it protects against, which is the safe direction for a gate that fails closed. [Translated streams](/docs/build/providers-streaming-translation) has the reason it is drawn that wide. ## Credential problems Work down this list; the first two are the answer most of the time. **The variable is not set.** `doctor` reports it by name, and the boot banner prints a line at startup: ```console ! keys — 1 upstream(s), 1 *_env reference(s): 1 unset (GROQ_API_KEY) — missing_env set: export GROQ_API_KEY=… (referenced by an upstream but unset) ``` With the variable unset, requests are forwarded carrying whatever credential the client sent. A client that sends nothing gets the provider's own 401, which is why this looks like an authentication bug rather than a configuration one. **The variable is set but empty.** Counted as unset. Reporting a set-but-empty variable as clean would hide the case the check exists to find. **A multi-part credential is unset.** The boot banner covers `api_key_env` only. An upstream whose credential arrives through `auth.credentials_file_env`, the AWS pair or `auth.private_key_file_env` boots silently. `doctor`'s `keys` line counts five of the six credential fields and leaves `auth.private_key_file_env` out, so for a key-file upstream the status payload is the surface that answers: ```json "vertex": { "auth_scheme": "gcp_oauth", "auth_health": { "configured": false, "fresh": false, "state": "missing_env" } } ``` **The credential file is unreadable.** `state: unreadable_credentials` means the path resolved and the file did not parse. Check what the variable resolves to: it holds a path, not a key. A passphrase-encrypted private key lands here too, in both PEM spellings, because neither key-file scheme takes a passphrase. Generate the key with `-nocrypt` and point the variable at that file. **A token expired.** `refresh_failed` means the last good material is still serving and a refresh is failing; `expired` means it lapsed. Both point at provider reachability or clock skew. `auth_health: ok` means a credential resolved, not that the provider accepted it. A revoked key reports `ok` here and 401 at the provider. The sharpest case is `oci_sig`: its authenticator loads the PEM, reports `ok`, and then injects nothing, so every request arrives at the provider unauthenticated and comes back 401. No built-in descriptor declares that scheme, so you meet it only by writing it into a `providers:` block yourself; Oracle Cloud ships on `bearer` with an API key. Nothing here probes your providers, so no state means "this key works". ## URL problems A 404 from the provider is nearly always the path rather than the host. - **Azure**: the deployment in `params.deployment` must exist on that resource. The path carries it; the body carries the model. - **Vertex**: `params.location` must be a location that serves your model, and it appears in both the host and the path. - **OpenAI-compatible vendors**: check whether the vendor's base URL already ends in `/v1`. Doubling it is the most common paste error. - **A vendor whose path is not the driver's default**: declare it under an inline provider's `paths`. See [adding a provider](/docs/build/providers-custom). ## A row with no dollar HTTP 200 with no cost is the symptom most often mistaken for breakage, and there are three distinct causes with three different fields. **`unpriced: true` with a reason.** Read `cost.unpriced_reason`. Six values are defined, in two families that want different things from you. | Reason | Family | First thing to check | |---|---|---| | `model_unknown` | pricing miss | is the upstream naming the descriptor you meant? | | `region_unknown` | pricing miss | is your region outside `us-`, `eu-` and `ap-`? | | `endpoint_unpriced` | pricing miss | is the model priced for the endpoint you called? | | `unit_missing` | pricing miss | did an image call carry a size and quality? | | `settlement_pending` | async settlement | nothing yet; the provider has not reported the money | | `settlement_terminal` | async settlement | the job ended without a priceable result | The four pricing misses mean a lookup ran and found nothing usable, and each is a work item for a catalog or a config. The two settlement reasons mean there is no price to look up: not yet, or not at all. No pricing action clears either, and folding them in with the first four reports a catalog gap that does not exist. [Reading a settled cost](/docs/build/providers-settlement) is the page for a pending row you are trying to add up, and [providers that take a job and answer later](/docs/build/providers-async-jobs) is why the row is written that way in the first place. **`unit_missing` on an error row is a guard, not a catalog gap.** A token-shaped row whose usage did not arrive used to be written as a confident `$0` carrying an `estimated: true` flag. It is now written as `UNPRICED(unit_missing)` and excluded. No total moves, because those rows contributed `$0` before and contribute `$0` now. What moves is the unpriced-excluded counter, and it moves up, most visibly on upstream errors, `no upstream` 502s, policy refusals and cascade exhaustion. If that counter jumped the day you upgraded, re-baseline it rather than alerting on the delta. **No `unpriced` flag, empty `pricing_key`, a `descriptor@` snapshot.** A declared-free upstream. Working as configured. **No cost fields at all.** The request took the passthrough path, because its endpoint kind is not one this descriptor declares. Check `http.endpoint` on the row. **Two unpriced counts that disagree.** The Requests list and the Cost view can report different unpriced totals over the same ledger and the same window. The list counts each row as it was written and does not fold a settlement into it, so a job that has settled still reads `UNPRICED` there while its own record drawer shows the dollar. The folded count is the one to believe. No dashboard surface renders a settlement figure at all, so a pending-settlement diagnosis is a command-line job: `tokentriage top` prints the exclusion and the pending count as two lines under its table. The [pricing-key page](/docs/build/providers-pricing-keys) has the full walk and the diagnosis table. ## When a stream misbehaves **It stopped early and the last thing through was an error.** On a translated stream, a frame the codec cannot encode ends the stream: your client receives an error frame in its own dialect, no terminating `[DONE]` arrives, `finish_reason` is empty, and the row carries the same string in `outcome.error`. That is a named case with a capture behind it, in [translated streams](/docs/build/providers-streaming-translation). **One upstream 400s on every streamed chat request.** On a streaming chat request to an OpenAI-dialect upstream, the gateway adds `stream_options.include_usage: true` to the outbound body and strips the resulting usage chunk from your copy if you did not ask for it. That is how a streamed row carries provider-reported usage instead of an estimate, and it lands on every upstream riding that driver rather than on a chosen list. An upstream that rejects the field therefore fails every streamed chat request, in a way that reads as the gateway's fault. Turn it off with `ledger.usage_injection: false` and note that the switch is global: rows on every OpenAI-dialect upstream then price from an estimated token count, with `usage.source` saying so. Oracle Cloud is the one upstream where this is a watch item rather than a settled question, because its published documentation says nothing about the field. ## Routing surprises **A tier is never chosen.** If its upstream speaks a different dialect than the request and `routing.allow_cross_dialect` is off, the tier is skipped and the request escalates to the same-dialect default. That is the default configuration. **A cross-dialect tier is chosen and the body is not translated.** The target upstream needs both `provider:` and `dialect:` named. With `provider:` alone the request is forwarded untranslated; [the cross-dialect page](/docs/build/providers-cross-dialect) shows both captures, and [the dialect matrix](/docs/build/providers-dialect-matrix) is the table of pairs that can translate at all. **`decision.fail_safe: true`.** Something degraded to a passthrough rather than failing: a model rewrite that could not be applied, a translation that hit an unmodelled parameter, a tier naming an upstream that no longer exists. ## Cooldown and reachability Nothing here probes your providers. Reachability is inferred from requests that actually failed, and it surfaces as one boolean per upstream: ```json "openai": { "base_url": "https://api.openai.com", "dialect": "", "in_cooldown": false, "auth_scheme": "bearer", "auth_health": { "configured": true, "fresh": true, "state": "ok" } } ``` That entry came off `GET /tt/api/status` on the admin port, re-indented from the single line `curl` returns. The upstream name and its `base_url` are yours; the field set is the payload's. `in_cooldown: true` means routing is currently avoiding that upstream after failures. The policy that puts it there and takes it out belongs to [routing limits](/docs/build/routing-limits). Two things this payload does not tell you. It does not say whether the provider is up — only whether requests to it have been failing. And `dialect` reads empty for every upstream configured with `provider:`, because the derived dialect is not written back into the config this payload reads. Use the descriptor id to know which wire you are on.
`/health` on the admin port answers `OK` while the process is running. It performs no provider check, no ledger check and no credential check, so a green `/health` alongside a failing provider is not a contradiction. For provider state, read `/tt/api/status`; for setup state, run `doctor`.
## When it is not you Some catalog prefixes cannot be reached from any configuration: a credential mechanism sitting outside the shipped authentication schemes, a wire with no primary documentation to build a mock against, or a bespoke contract with no verifiable shape. Submit-and-poll is no longer one of those blockers; the class that held it is closed, and [providers that take a job and answer later](/docs/build/providers-async-jobs) is where those rows are explained. [What is not covered](/docs/build/providers-coverage) lists every prefix with its blocker, so you can tell a misconfiguration from an absence in about a minute.