# Migrate from LiteLLM

Swap the base URL, then map your LiteLLM model_list, router_settings, virtual keys, and budgets onto TokenTriage equivalents.

Source: https://tokentriage.com/docs/migrate/litellm

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import CodeTabs from '../../../components/CodeTabs.astro';

Migrating from LiteLLM is two moves, in order of effort. First, the **data-plane swap**:
point your OpenAI-style client at TokenTriage instead of the LiteLLM proxy — one line. Then,
if you ran LiteLLM's proxy config (`model_list` / `router_settings` / `general_settings`) or
its virtual-key admin surface, **port that config** onto TokenTriage's equivalents. This guide
covers the drop-in first, then a concept-by-concept mapping table, then a short structural
runbook you finish with `tokentriage validate`.

<Callout type="note">
LiteLLM specifics below are described as of **2026-07** from LiteLLM's current docs
(`docs.litellm.ai`). TokenTriage claims are verified against its own code and config. The
honest framing matters both directions: where the two products differ, this guide says how —
not that one is universally better. TokenTriage does **not** claim a cost win over LiteLLM,
and this guide makes no latency claim.
</Callout>

## 1. The drop-in swap

A LiteLLM proxy listens on `http://0.0.0.0:4000` by default; TokenTriage's data plane listens
on `http://127.0.0.1:8787` and serves the same OpenAI-style routes under `/v1` (e.g.
`/v1/chat/completions`, `/v1/embeddings`). The OpenAI SDK instantiation is otherwise unchanged
— only the base URL moves.

**Before — pointing at LiteLLM:**

<CodeTabs>
<Fragment slot="python">
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://0.0.0.0:4000",   # LiteLLM proxy
    api_key="sk-1234",                 # your LiteLLM master/virtual key
)

resp = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Ping"}],
)
print(resp.choices[0].message.content)
```
</Fragment>
<Fragment slot="typescript">
```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://0.0.0.0:4000",   // LiteLLM proxy
  apiKey: "sk-1234",                 // your LiteLLM master/virtual key
});

const resp = await client.chat.completions.create({
  model: "claude-haiku-4-5",
  messages: [{ role: "user", content: "Ping" }],
});
console.log(resp.choices[0].message.content);
```
</Fragment>
</CodeTabs>

**After — pointing at TokenTriage:**

<CodeTabs>
<Fragment slot="python">
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8787/v1",  # ← the only line you change
    api_key="sk-your-real-provider-key",   # forwarded upstream as-is in the default posture
)

resp = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Ping"}],
)
print(resp.choices[0].message.content)
```
</Fragment>
<Fragment slot="typescript">
```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8787/v1",  // ← the only line you change
  apiKey: "sk-your-real-provider-key",   // forwarded upstream as-is in the default posture
});

const resp = await client.chat.completions.create({
  model: "claude-haiku-4-5",
  messages: [{ role: "user", content: "Ping" }],
});
console.log(resp.choices[0].message.content);
```
</Fragment>
</CodeTabs>

**And the same swap in `curl`:**

```console
# Before (LiteLLM):
curl http://0.0.0.0:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","messages":[{"role":"user","content":"Ping"}]}'

# After (TokenTriage) — only the host/base changes:
curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer sk-your-real-provider-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","messages":[{"role":"user","content":"Ping"}]}'
```

<Callout type="key">
**The one difference to understand is the key.** LiteLLM asks the client to send a LiteLLM key
(`sk-…`) and injects the real provider key server-side. In TokenTriage's default posture — no
`api_key_env` on the upstream and `governance.require_key: off` — the inbound `Authorization`
is **forwarded upstream as-is**, so the client sends the real provider key. To recreate
LiteLLM's "clients never see the provider key" model, set `api_key_env` on the upstream (below)
so TokenTriage injects the credential and the inbound key is replaced — then mint a virtual key
(`ttv_…`) for clients to carry instead.
</Callout>

### What you get the moment traffic flows

With the base-URL swap alone — and TokenTriage in its default `mode: ledger` — the proxied
bytes are identical to going direct (ledger mode is observe-only, zero request mutation), and
every request is now measured:

- **Every request is costed against a typed money envelope**, not a bare float. An unpriced row
  is **excluded from totals and counted in the open**, never silently added as `0` — so an
  unknown cost can't masquerade as free.
- **An append-only decision ledger** records each request's route, model, tokens, and cost —
  the durable record LiteLLM's spend tracking approximates.
- **RFC 9457 `application/problem+json` errors** with a stable `type`, code, and `request_id`.
- **Routing tiers, caching, and per-tenant governance are all opt-in from here** — flip
  `mode` to `shadow` (compute would-route decisions, log them, change nothing) and then `route`
  (enforce) when you trust what you see. You never enforce blind.

## 2. Config mapping — LiteLLM proxy → TokenTriage

If you ran a LiteLLM proxy `config.yaml`, here is where each concept lives in a TokenTriage
config. The shapes differ for a reason: LiteLLM keys a flat `model_list` with routing and
budgets layered on top; TokenTriage separates **where** a model lives (`upstreams`) from
**which tier** serves it (`tiers`, ordered cheapest → most capable), and puts all tenant
governance behind one opt-in `governance:` block.

| LiteLLM concept | LiteLLM key(s) | TokenTriage equivalent |
|---|---|---|
| Model deployment | `model_list[].litellm_params.model`, `api_base`, `api_key` | An `upstreams.<name>` entry (`base_url`, `dialect:`, `api_key_env`) **plus** a `tiers[]` entry (`name`, `upstream`, `model`) that names the served model. The accepted `dialect:` values are derived from the compiled descriptor registry, which names 14 distinct dialect ids; `openai` and `anthropic` are the fallback pair when no registry is injected |
| Provider credential | `api_key: sk-…` (inline value) | `api_key_env: OPENAI_API_KEY` — a reference to an env-var **name**, never the secret in the file. Omit it to forward the inbound key instead |
| User-facing model alias | `model_name` | The requested `model` on the wire; route on it with a `routing.rules[]` `match: { model: "gpt-*" }` |
| Routing strategy | `router_settings.routing_strategy` (`simple-shuffle` / `least-busy` / `usage-based-routing` / `latency-based-routing`) | `routing.rules[]` — ordered, first-match-wins, matching on request signals (`complexity`), `step_class`, `model`, `agent`, `has_tools`, or a header. This is **complexity/attribute routing, not load-based shuffling** — a different model, and the difference is intentional |
| Fallbacks | `router_settings.fallbacks` (ordered model chains) | `routing.default_tier` (the fail-safe target on classifier error — never downgrades on error) and `routing.on_internal_error: passthrough \| default_tier`. TokenTriage does not express per-model ordered fallback chains in this config form |
| Retries / cooldown | `router_settings.num_retries`, `allowed_fails`, `cooldown_time` | `routing.cooldown` (`enabled`, `seconds`, `failure_threshold`, `min_requests`) — cool an upstream when its failure rate crosses a threshold over a minimum request count |
| Per-deployment rate limits | `model_list[].litellm_params.rpm` / `tpm` | `governance.limits[]` of `type: rpm` / `tpm` attached to a tenancy node (a different axis — TokenTriage rate-limits **tenants**, not deployments) |
| Master key | `general_settings.master_key` / `LITELLM_MASTER_KEY` (must start with `sk-`) | No single global master key. Data-plane access is gated by `governance.require_key: off \| optional \| required` and virtual keys (`ttv_`); the management API uses separate `tta_` tokens. Credential classes are split, not merged |
| Backing database | `general_settings` + `DATABASE_URL` (Postgres **required** for keys/budgets) | `governance.store.path` — an embedded SQLite `governance.db`, no external database to stand up. Postgres and Redis are **opt-in** in two independent places: either one for multi-replica coordination (`shared_state`), and Redis alone for the response cache's L3 tier (`cache.store.l3`) |
| Virtual keys | `POST /key/generate`; `duration` for expiry | `tokentriage keys mint --owner <node>` (or `POST /tt/api/v1/governance/keys`). Keys are `ttv_`-prefixed, stored as SHA-256 hashes, and the secret is **shown once**; `--expiry 720h` sets expiration |
| Key spend budget | `max_budget` + `budget_duration` on a key | A `governance.limits[]` of `type: budget_usd` with `amount_usd` and `period: day \| week \| month`, attached to (or templated over) the key/user/team/org node |
| Key token/request quota | `tpm_limit` / `rpm_limit` on a key | `governance.limits[]` of `type: quota_tokens` / `quota_requests` (per-period ledger quotas) or `tpm` / `rpm` (rate windows) |
| Team / org budgets | team & org `max_budget` | Template limits attached by `level: org \| team`, with per-node `override` ceilings — one limit definition materializes per node at that level |

<Callout type="tip">
TokenTriage keys are **grant-frozen at mint**: there is no in-place `key/update`.
To change a key's scope you rotate it (`tokentriage keys rotate <kid> --grace 24h` mints a
successor and keeps the predecessor valid through a grace window) or mint a fresh one. Budgets
and limits, by contrast, are edited in config and hot-reload.
</Callout>

## 3. Structural migration runbook

Work top-down through your LiteLLM `config.yaml`, translate each section, then let
`tokentriage validate` catch anything before traffic hits it.

**1. Map `model_list` → `upstreams` + `tiers`.** For each deployment, create (or reuse) a named
upstream for its host, then add a tier that names the served model. Ordering the tiers cheapest
→ most capable is what lets routing send simple requests down and hard ones up.

```yaml
version: 1
mode: ledger                     # start observe-only; promote to shadow, then route
server:
  listen: "127.0.0.1:8787"       # the data plane your clients now target
  admin:  "127.0.0.1:9090"

upstreams:
  anthropic:
    base_url: https://api.anthropic.com
    dialect: anthropic
    api_key_env: ANTHROPIC_API_KEY   # a reference to an env-var NAME, not the secret
  openai:
    base_url: https://api.openai.com   # host only — your client already sends /v1
    dialect: openai
    api_key_env: OPENAI_API_KEY

tiers:
  - name: cheap
    upstream: anthropic
    model: claude-haiku-4-5
  - name: frontier
    upstream: anthropic
    model: claude-opus-4-8
```

Those upstreams name `dialect:` and no `provider:`. That is the legacy form, it is still valid,
and it is the shape `configs/tokentriage.example.yaml` ships for these same hosts. Keep it, for a
reason that outlives the migration: the cross-dialect branch reads its target from
`Upstream.Dialect`, and in this build nothing writes a descriptor's dialect back into that field
(`internal/api/proxy/route.go:255`). An upstream configured with `provider:` alone reports an empty
dialect on the admin status route, and a cross-dialect tier pointed at it forwards the body
untranslated. When you move to a descriptor-backed upstream,
[Dialects](/docs/build/providers-dialects) says when to name both `provider:` and `dialect:` on
one entry.

**2. Move credentials out of the file.** Any inline `api_key: sk-…` from `litellm_params`
becomes an `api_key_env:` reference plus the value exported in the environment. No TokenTriage
config field holds a provider secret, which is a claim about this self-hosted mode. A hosted
gateway can instead keep the key as a [saved provider
connection](/docs/build/provider-connections), stored encrypted at rest under a customer-managed
KMS key and not readable back; request-supplied mode keeps the no-storage guarantee and is chosen
in the gateway's configuration rather than by a caller.

**3. Translate `router_settings` → `routing`.** Replace `routing_strategy` with ordered
`routing.rules[]` matching on the signals you care about; map `num_retries` / `allowed_fails` /
`cooldown_time` onto `routing.cooldown`; and set `default_tier` as your fail-safe.

```yaml
routing:
  default_tier: frontier
  on_internal_error: passthrough   # never downgrade on an internal error
  rules:
    - id: simple-to-cheap
      match: { signal: complexity, below: 0.35 }
      route: cheap
  cooldown:
    enabled: true
    seconds: 30
    failure_threshold: 0.5         # cool when >50% fail ...
    min_requests: 5                # ... over ≥5 requests in the minute
```

**4. Rebuild keys and budgets under `governance`.** LiteLLM's `master_key` + `DATABASE_URL` +
`/key/generate` collapse into one opt-in block. Declare your tenancy, set `require_key`, and
define limits — starting every limit in `shadow` so you watch would-be refusals before any real
one fires.

```yaml
governance:
  enabled: true
  require_key: optional            # off | optional | required (RESTART-required)
  store: { path: "" }              # "" ⇒ <state>/governance.db (embedded SQLite)
  tenants:
    - org: acme
      teams: [ml, search]
  limits:
    - id: org_monthly
      attach: { level: org }       # a template: one budget per org node
      type: budget_usd
      amount_usd: 1000
      period: month
      posture: { mode: shadow }    # observe would-refusals first; promote to enforce later
```

Then mint the client-facing keys (each secret is shown once):

```console
$ tokentriage keys mint --owner "org:acme/team:ml" --expiry 720h
kid:     vk_7Qm2Xr9dK4pL
expires: 2026-08-28T00:00:00Z
This secret is shown ONCE and cannot be recovered — copy it now:
ttv_…
```

**5. Validate before you cut over.** `tokentriage validate` strictly decodes the config (unknown
keys are errors), re-checks it against the real provider and routing registries exactly as boot
does, and prints a summary plus any warnings.

```console
$ tokentriage validate --config tokentriage.yaml
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 2
  - anthropic    anthropic https://api.anthropic.com
  - openai       openai    https://api.openai.com
tiers:     2 (cheapest -> most capable)
  0. cheap      -> anthropic/claude-haiku-4-5
  1. frontier   -> anthropic/claude-opus-4-8
rules:     1
OK: configuration is valid
```

**What to confirm in that output:**

- The upstream count and dialects match your old `model_list`.
- The tiers are ordered cheapest → most capable.
- Your rule count is what you expect.

If you enabled a `shared_state` coordination tier, `validate` also prints an
"air-gap posture is broken for the coordination tier" warning. That is expected when you opt
in — it is not a failure.

**6. Point the client at `:8787/v1`.** That's the one-line swap from §1. Run in `ledger` first
to confirm cost and routing look right in the decision log, promote `mode` to `shadow` to watch
would-route decisions, and switch to `route` when you trust them.

<Callout type="warning">
Enabling `governance.require_key` and changing `governance.store` are **restart-required** — they
do not hot-reload. Limit *definitions* and routing rules do hot-reload; credential posture and the
store binding are set at boot. Plan the cutover so a client isn't mid-flight when you flip
`require_key` from `off` to `required`.
</Callout>

<NextSteps items={[
  { title: "Governance, virtual keys & budgets", href: "/docs/build/governance", description: "The full model behind ttv_ keys, per-tenant limits, and the shadow-first enforcement ladder your LiteLLM keys and budgets map onto." },
  { title: "Routing", href: "/docs/build/routing", description: "Tiers, rules, signals, and cooldown in depth — how complexity routing replaces routing_strategy and fallbacks." },
  { title: "The release workflow", href: "/product/", description: "See how gateway configuration, candidate evidence, and deliberate deployment fit together." }
]} />