Migrate from LiteLLM

Migrating from LiteLLM is two moves, in order of effort. First, the data-plane swap: point your OpenAI-style client at TokenTriage instead of the LiteLLM proxy — one line. Then, if you ran LiteLLM’s proxy config (model_list / router_settings / general_settings) or its virtual-key admin surface, port that config onto TokenTriage’s equivalents. This guide covers the drop-in first, then a concept-by-concept mapping table, then a short structural runbook you finish with tokentriage validate.

1. The drop-in swap

A LiteLLM proxy listens on http://0.0.0.0:4000 by default; TokenTriage’s data plane listens on http://127.0.0.1:8787 and serves the same OpenAI-style routes under /v1 (e.g. /v1/chat/completions, /v1/embeddings). The OpenAI SDK instantiation is otherwise unchanged — only the base URL moves.

Before — pointing at LiteLLM:

from openai import OpenAI

client = OpenAI(
    base_url="http://0.0.0.0:4000",   # LiteLLM proxy
    api_key="sk-1234",                 # your LiteLLM master/virtual key
)

resp = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Ping"}],
)
print(resp.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://0.0.0.0:4000",   // LiteLLM proxy
  apiKey: "sk-1234",                 // your LiteLLM master/virtual key
});

const resp = await client.chat.completions.create({
  model: "claude-haiku-4-5",
  messages: [{ role: "user", content: "Ping" }],
});
console.log(resp.choices[0].message.content);

After — pointing at TokenTriage:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8787/v1",  # ← the only line you change
    api_key="sk-your-real-provider-key",   # forwarded upstream as-is in the default posture
)

resp = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Ping"}],
)
print(resp.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8787/v1",  // ← the only line you change
  apiKey: "sk-your-real-provider-key",   // forwarded upstream as-is in the default posture
});

const resp = await client.chat.completions.create({
  model: "claude-haiku-4-5",
  messages: [{ role: "user", content: "Ping" }],
});
console.log(resp.choices[0].message.content);

And the same swap in curl:

# Before (LiteLLM):
curl http://0.0.0.0:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","messages":[{"role":"user","content":"Ping"}]}'

# After (TokenTriage) — only the host/base changes:
curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer sk-your-real-provider-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","messages":[{"role":"user","content":"Ping"}]}'

What you get the moment traffic flows

With the base-URL swap alone — and TokenTriage in its default mode: ledger — the proxied bytes are identical to going direct (ledger mode is observe-only, zero request mutation), and every request is now measured honestly:

• Every request is costed against a typed money envelope, not a bare float. An unpriced row is excluded from totals and counted in the open, never silently added as 0 — so an unknown cost can’t masquerade as free.
• An append-only decision ledger records each request’s route, model, tokens, and cost — the durable record LiteLLM’s spend tracking approximates.
• RFC 9457 application/problem+json errors with a stable type, code, and request_id.
• Routing tiers, caching, and per-tenant governance are all opt-in from here — flip mode to shadow (compute would-route decisions, log them, change nothing) and then route (enforce) when you trust what you see. You never enforce blind.

2. Config mapping — LiteLLM proxy → TokenTriage

If you ran a LiteLLM proxy config.yaml, here is where each concept lives in a TokenTriage config. The shapes differ deliberately: LiteLLM keys a flat model_list with routing and budgets layered on top; TokenTriage separates where a model lives (upstreams) from which tier serves it (tiers, ordered cheapest → most capable), and puts all tenant governance behind one opt-in governance: block.

3. Structural migration runbook

Work top-down through your LiteLLM config.yaml, translate each section, then let tokentriage validate catch anything before traffic hits it.

1. Map model_list → upstreams + tiers. For each deployment, create (or reuse) a named upstream for its host, then add a tier that names the served model. Ordering the tiers cheapest → most capable is what lets routing send simple requests down and hard ones up.

version: 1
mode: ledger                     # start observe-only; promote to shadow, then route
server:
  listen: "127.0.0.1:8787"       # the data plane your clients now target
  admin:  "127.0.0.1:9090"

upstreams:
  anthropic:
    base_url: https://api.anthropic.com
    dialect: anthropic
    api_key_env: ANTHROPIC_API_KEY   # a reference to an env-var NAME, not the secret
  openai:
    base_url: https://api.openai.com/v1
    dialect: openai
    api_key_env: OPENAI_API_KEY

tiers:
  - name: cheap
    upstream: anthropic
    model: claude-haiku-4-5
  - name: frontier
    upstream: anthropic
    model: claude-opus-4-8

2. Move credentials out of the file. Any inline api_key: sk-… from litellm_params becomes an api_key_env: reference plus the actual value exported in the environment. TokenTriage never stores a provider secret in config.

3. Translate router_settings → routing. Replace routing_strategy with ordered routing.rules[] matching on the signals you care about; map num_retries / allowed_fails / cooldown_time onto routing.cooldown; and set default_tier as your fail-safe.

routing:
  default_tier: frontier
  on_internal_error: passthrough   # never downgrade on an internal error
  rules:
    - id: simple-to-cheap
      match: { signal: complexity, below: 0.35 }
      route: cheap
  cooldown:
    enabled: true
    seconds: 30
    failure_threshold: 0.5         # cool when >50% fail ...
    min_requests: 5                # ... over ≥5 requests in the minute

4. Rebuild keys and budgets under governance. LiteLLM’s master_key + DATABASE_URL + /key/generate collapse into one opt-in block. Declare your tenancy, set require_key, and define limits — starting every limit in shadow so you watch would-be refusals before any real one fires.

governance:
  enabled: true
  require_key: optional            # off | optional | required (RESTART-required)
  store: { path: "" }              # "" ⇒ <state>/governance.db (embedded SQLite)
  tenants:
    - org: acme
      teams: [ml, search]
  limits:
    - id: org_monthly
      attach: { level: org }       # a template: one budget per org node
      type: budget_usd
      amount_usd: 1000
      period: month
      posture: { mode: shadow }    # observe would-refusals first; promote to enforce later

Then mint the client-facing keys (each secret is shown once):

$ tokentriage keys mint --owner "org:acme/team:ml" --expiry 720h
kid:     vk_7Qm2Xr9dK4pL
expires: 2026-08-28T00:00:00Z
This secret is shown ONCE and cannot be recovered — copy it now:
ttv_…

5. Validate before you cut over. tokentriage validate strictly decodes the config (unknown keys are errors), re-checks it against the real provider and routing registries exactly as boot does, and prints a summary plus any warnings.

$ tokentriage validate --config tokentriage.yaml
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 2
  - anthropic    anthropic https://api.anthropic.com
  - openai       openai    https://api.openai.com/v1
tiers:     2 (cheapest -> most capable)
  0. cheap      -> anthropic/claude-haiku-4-5
  1. frontier   -> anthropic/claude-opus-4-8
rules:     1
OK: configuration is valid

What to confirm in that output: the upstream count and dialects match your old model_list; the tiers are ordered cheapest → most capable; your rule count is what you expect; and if you enabled any shared_state coordination tier, that validate prints its honest “air-gap posture is broken for the coordination tier” warning — that message is expected when you opt in, not a failure.

6. Point the client at :8787/v1. That’s the one-line swap from §1. Run in ledger first to confirm cost and routing look right in the decision log, promote mode to shadow to watch would-route decisions, and switch to route when you trust them.