Connect an OpenAI-compatible provider

Any upstream that speaks the OpenAI chat-completions wire connects the same way, whether it is Groq, Together, Fireworks, OpenRouter, DeepSeek, a Cerebras endpoint or a vLLM server on your own hardware. This page walks that path once, with Groq, and ends with a request that comes back carrying the price it was looked up at.

You need the provider’s base URL and an API key. You do not need a database, an account with us, or any change to your application code beyond its base URL.

This is the self-hosted path, where your config names a variable and the process reads its value at startup, so nothing in the file has to be rotated. A hosted gateway has a second mode: the provider key is saved once as a provider connection, stored encrypted at rest under a customer-managed KMS key in TokenTriage’s cloud account, and not readable back through any API, export or log. Request-supplied mode is the third case and it keeps the no-storage guarantee, one request at a time. The mode belongs to the gateway’s configuration and not to a caller.

1. Name the upstream

An upstream is a block under upstreams: with a name you choose. provider: names a built-in descriptor, which already carries the wire dialect, the authentication scheme and the pricing namespace, so you do not repeat any of them.

upstreams:
  groq:
    provider: groq                 # built-in descriptor: openai wire, bearer auth
    base_url: https://api.groq.com/openai   # /openai is Groq's — your client already sends /v1
    api_key_env: GROQ_API_KEY      # an env var NAME, never a secret literal

api_key_env holds the name of an environment variable. Pasting the key itself is the mistake this field exists to prevent, and the credential model explains what the validator does and does not catch.

dialect: is derived from the descriptor, so leave it out. One exception is worth knowing before you meet it: an upstream that a cross-dialect tier will target has to name dialect: alongside provider:, and cross-dialect translation shows both captures. base_url is always required, because a descriptor carries relative paths and no host.

Those five lines under a version: 1 are already a config that loads. Six lines in total, and validate accepts them on their own, in ledger mode, on the default listen and admin addresses. Step 2 adds the tier that gives routing something to choose.

Where do I find my provider's descriptor name?

The provider: value is the descriptor id, which is usually the provider’s own catalog prefix: groq, together_ai, fireworks_ai, deepseek, mistral, openrouter, xai, cerebras. The complete list, with the pricing prefix and endpoints each one serves, is in the provider reference. A name that is not in the registry fails validation and prints what you typed.

2. Point a tier at it

Routing needs at least one tier, and a tier is an upstream plus a model string. This config is otherwise minimal: shadow mode, one tier, no rules.

version: 1

server:
  listen: "127.0.0.1:8787"
  admin:  "127.0.0.1:9090"

mode: shadow

upstreams:
  groq:
    provider: groq
    base_url: https://api.groq.com/openai
    api_key_env: GROQ_API_KEY

tiers:
  - name: cheap
    upstream: groq
    model: llama-3.3-70b-versatile

routing:
  default_tier: cheap

3. Validate it

validate loads the file, compiles the provider registry the way boot does, and prints what it built. It dials nothing and reads no credential.

$ tokentriage validate --config tokentriage.yaml
mode:      shadow
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 1
  - groq                   https://api.groq.com/openai
tiers:     1 (cheapest -> most capable)
  0. cheap      -> groq/llama-3.3-70b-versatile
rules:     0
OK: configuration is valid

The blank column between the upstream name and the URL is where the wire dialect prints. It is empty for every upstream configured with provider:, because the derived dialect is not written back into the config the summary reads. That is a display gap, not a misconfiguration; the descriptor’s dialect is in force either way, and the dialect matrix is where you read which client formats reach which upstreams.

4. Check the credential is actually set

Validation checks that api_key_env is a legal variable name. It cannot check that the variable exists, because that depends on the shell you eventually start the binary in. doctor can.

$ tokentriage doctor --config tokentriage.yaml
✓ binary — tokentriage dev (none), embedded pricing snapshot litellm@2026-07-14#179affb — clean
✓ config — tokentriage.yaml — strict decode clean: mode shadow, 1 upstream(s), 1 tier(s), 0 rule(s)
✓ compile — provider registry + routing policy compiled — 1 upstream(s) wired — clean
✓ ports — listen 127.0.0.1:8787 bindable, admin 127.0.0.1:9090 bindable — clean
✓ pricing — embedded table litellm@2026-07-14#179affb loaded, no overrides configured — clean
! keys — 1 upstream(s), 1 *_env reference(s): 1 unset (GROQ_API_KEY) — missing_env
    set: export GROQ_API_KEY=…   (referenced by an upstream but unset)

ok — no hard failures, 1 soft warning(s) (exit 0)

That capture is from a source build, so it prints dev where a release prints a version, and the config path is this machine’s. Five further check lines are cut for length: ledger, index, dashboard, instance and features. The keys line is the one to read: it reports presence by variable name, and never the value.

Export the key and the line turns green. An unset key is a soft warning rather than a failure because forwarding whatever credential the client sent is a supported way to run.

5. Send one request and read the price

Point a client at the proxy and send a normal OpenAI request.

$ curl -s http://127.0.0.1:8787/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model":"llama-3.3-70b-versatile",
         "messages":[{"role":"user","content":"hello"}]}'

Then read the row it wrote. The field to check is cost.pricing_key: it should start with your provider’s namespace, not the wire dialect’s.

"http":  { "path": "/v1/chat/completions", "dialect": "openai", "status": 200,
           "endpoint": "chat", "provider": "groq" },
"usage": { "input_tokens": 11, "output_tokens": 5, "total_tokens": 16,
           "source": "provider" },
"cost":  { "usd": 1.044e-05, "pricing_key": "groq/groq/llama-3.3-70b-versatile",
           "pricing_snapshot": "litellm@2026-07-14#179affb", "estimated": false }

That row came from a mock upstream shaped like Groq, so the token counts are the mock’s and the dollar is arithmetic over the committed rate table rather than anyone’s bill: 11 input tokens at $0.59 per million plus 5 output tokens at $0.79 per million is $0.00001044. Against the live provider the shape is identical and the counts are theirs.

The doubled groq/ is not a bug. The vendored catalog keys many providers as <prefix>/<the model id the provider publishes>, and that published id already carries the prefix. The pricing-key page shows the resolution order that produces it.

What can go wrong

Symptom	Cause	Fix
`does not name a known provider`	descriptor id misspelled	check the id in the reference
`base_url: must not be empty`	`provider:` set without a host	descriptors carry relative paths only
`this looks like a secret value`	a key literal in `api_key_env`	put the variable name there instead
row has `unpriced: true`	the model has no rate under this prefix	diagnose the key
provider returns 401	the referenced variable is unset	run `doctor`; the `keys` check names it

The 401 case is worth pre-empting. With api_key_env unset, requests are forwarded with whatever credential the client sent, so a client sending nothing gets the provider’s own 401 rather than an error from the gateway. The boot banner says so at startup for any api_key_env it cannot resolve.