The honest LLM gateway
Point your apps at one base URL to see where every dollar goes. Test cheaper models in shadow, and route only after quality holds — so every saving is proven, estimated, or honestly labeled can’t-prove-it-yet. Never a number we can’t back up.
git clone https://github.com/inferops/tokentriage && cd tokentriage
CGO_ENABLED=0 go build -o dist/tokentriage ./cmd/tokentriage
./dist/tokentriage run --anthropicBuild from source · data plane on :8787
80+ providers, one gateway — every request priced and attributed from the same catalog.
Head-to-head · measured on bare metal · pinned versions
Same bare-metal run — AWS z1d.metal, an equal 4-core budget, latest stable images, steel-manned configs — and TokenTriage was computing and logging cost for every request while LiteLLM ran its logging-off latency posture. Every number below is reproducible from the open harness.
Offered load swept 100 → 4,000 req/s on the same 4 cores. LiteLLM saturates at ~670 req/s and its p99 collapses past 13 seconds. TokenTriage takes all 4,000 at 1.20 ms.
The current optimized build goes further (TokenTriage-only follow-up run, same hardware class — kept separate; we don't splice runs into the head-to-head)
We pre-committed to publish this axis win or lose: TokenTriage, LiteLLM, and Bifrost all matched an independent provider-doc reference to the nanodollar. The difference is what happens after: TokenTriage stamps every request with its pricing snapshot and decision provenance, so each decimal survives an audit.
Reported cost, all three gateways = ground truth
AWS z1d.metal · 48 vCPU bare metal · equal 4-core gateway budgets · disjoint-core pinned · deterministic mock upstream · 95% bootstrap CIs · LiteLLM v1.94.0 · Bifrost v1.6.6 · pinned image digests
See the full head-to-head & reproduce it yourselfThe problem
If more than one of these is true, there is almost certainly a number worth finding.
Traffic grows 20%; the inference bill grows 90%. Nobody can point at the line that did it, because the invoice is one number per provider.
Which feature, which agent, which route, which token type? The data exists in logs and billing exports — but no one has joined the two.
Someone tried a smaller model for a week, quality complaints arrived, and it was reverted. Now nothing moves, because nothing can be proven safe.
What it does
One gateway your apps point at. Each capability names the outcome first — the mechanism is one layer down for the engineers.
Route
Point each request at the cheapest tier that can still do the job — hard prompts to a frontier model, easy ones to a cheap one. Routing stays off until you turn it on.
11 routing strategies · typed policy language · off by default
See how routing decides
Attribute
Every request is costed and attributed by model, token type, trace, agent, and MCP tool — all traceable to one decision log. Nothing is guessed; unpriced spend is counted, never shown as $0.00.
One JSONL decision log · source@date#sha7 pricing · UNPRICED ≠ $0.00
See where dollars go
Prove
Test a cheaper model in shadow against your real traffic and grade its quality before anything changes. Every saving is labeled PROVEN, ESTIMATED, or honestly can’t-prove-it-yet.
Verdicts: PROVEN / ESTIMATED / UNPROVABLE · shadow-judged, never guessed
Read a real verdict
Govern
Mint virtual keys (hashed, shown once) and enforce per-tenant budgets and rate limits across org, team, user, and key. When the limiter itself dies it fails open and discloses it — never a fake 429.
6 limit types · virtual keys (hashed, shown once) · fail-open with disclosure
Set up a virtual key
Observe
A dashboard that reads the same decision log the ledger writes — spend, savings, and health, live. It sits entirely off the request hot path, so a dead dashboard can never slow your traffic.
OTLP tokentriage.cost.* · dashboard off the hot path
Watch cost live
How it works
You earn enforcement one rung at a time: observe every dollar, prove the cheaper path in shadow, then route. The same decision can only ever be proven, estimated, or honestly refused.
1Ledger
See every dollar
Cost and log every request. Zero request mutation.
2Shadow
Prove it is safe
Compute the cheaper counterfactual and judge its quality — without touching a response.
3Route
Enforce, once earned
Switch to the cheaper tier only after the evidence clears the bar.
One decision, three honest answers
support reply → gpt-5 mini· cheap tierillustrativeThe shadow judge graded the cheaper tier on the same query and it held. The saving is real.
Only the counterfactual cost (would_usd) is known, not a quality judgement yet. The number wears an EST flag.
With served-model labels alone, routing can at best sit on the frontier — never a beat. An honest refusal, never a number.
The trust dashboard
Spend to date, projected period-end, and savings vs. baseline — each stamped with its pricing snapshot, EST flags where estimated, and UNPRICED rows counted, never hidden.
Open at the core
The core is source-available under the Functional Source License and the client SDKs are Apache-2.0. Every honesty invariant on this page is a constraint in the source — so you don’t have to take our word for any of it.
Pricing
Cut your LLM bill only after you prove the cheaper path is just as good.
Building in the open
TokenTriage is being built in the open. Drop your email for occasional, no-spam updates — new releases, fresh benchmarks, and the road to 1.0. Star the repo for the code; join the list for the story.
Double opt-in. Unsubscribe anytime. We never share your address.