Benchmark
Every number survives an audit.
An open, verdict-labeled benchmark of LLM gateways. The flagship result is cost-number accuracy — it is hardware-independent, so it is exact arithmetic anyone can re-run. Every figure on this page traces to RESULTS.md, and where a claim is not yet defensible, this page says so instead of showing a number.
The flagship result · measured
Cost that matches provider-doc ground truth to the nanodollar.
With fixed, known usage and a single cited rate, the correct per-request cost is exact arithmetic. Across a cached-input sweep, TokenTriage's ledger cost equals an independent reference to the ninth decimal — including the cache-read discount.
| Cached | Reported (TT .cost.usd) | Ground truth | Abs error | Rel error | Verdict |
|---|---|---|---|---|---|
| 0% (control) | $0.006400000 | $0.006400000 | $0.000000000 | 0.000% | Correct |
| 25% | $0.006112000 | $0.006112000 | $0.000000000 | 0.000% | Correct |
| 50% | $0.005824000 | $0.005824000 | $0.000000000 | 0.000% | Correct |
Raw evidence — graded JSON, the raw decision-log rows, and per-run provenance — ships underbenchmarks/results/tokentriage-cost-2026-07-28/.
One lever: the cache-read discount.
Cache-read tokens bill 10× cheaper than input — the only real accuracy lever on this axis. TokenTriage bills disjoint buckets (input = prompt − cachedat the input rate, cache_read = cached at the discounted rate), so it removes exactly that error. A naive gateway that bills the cached subset at the full input rate over-charges — and the 0% control proves the 25/50% deltas are only the discount, nothing else.
The bars below are a method reference point, not a competitor measurement. Whether any specific gateway makes the naive error was measured, not assumed — and neither LiteLLM nor Bifrost does: all three are exact, a three-way tie (published in the head-to-head below).
Over-charge if you ignore the cache-read discount
- Fixed usage
prompt=1024·completion=512·reasoning=128- Cached sweep
- 0% (control) / 25% / 50% of prompt tokens
- Priced rate
- published
openai/gpt-5— in $1.25 / out $10.00 / cache-read $0.125 per 1M - Snapshot
litellm@2026-07-14#179affb(committed, dated)- Ground truth
benchmarks/costcheck— imports no TokenTriage core; graded by provider billing docs- Version
- tokentriage @ git
a8ed2ed; sample n=1/bucket (cost is exact arithmetic, not a timing metric)
Performance · bare-metal reference · measured
Fast under load — measured on bare metal, with the caveats shown.
On our virtualized Windows box, sub-millisecond gateway deltas fall below the noise floor, so no latency verdict is banked there. We ran a bare-metal Linux reference(AWS z1d.metal) and report TokenTriage's own numbers from it. The named competitor head-to-head from the same reference is published below.
- 8,000+req/s on 4 coressustained at 100% success; pushes to ~10,205 req/s at 16k offered
- 1.45 msp99 under loadat 8,000 req/s — the tail, not the median
- 0.148 msadded overheadmedian TTFT vs a direct call · 95% CI [0.145, 0.151]
Before → after one commit. The upstream connection-pool optimization (MaxIdleConnsPerHost 2 → 256), same hardware and workload, on a single run:
| Offered load | Before (2 idle conns/host) | After (256 idle conns/host) |
|---|---|---|
| 4,000 req/s | 100% · p99 381 ms | 100% · p99 1.82 ms |
| 8,000 req/s | saturates — 1,704/s (98.9%), p99 5,833 ms | 8,000/s · 100% · p99 1.45 ms |
| 16,000 req/s | collapses — 279/s (45.4%) | 10,205/s · 100% · p99 692 ms |
- Measured against a deterministic mock upstream, so the gateway's own work is the bottleneck — this isolates the overhead TokenTriage adds, not end-to-end latency.
- Single bare-metal run. The direction and magnitude (≈2× ceiling, ≈200× lower p99 at 4k) are large and robust; the exact knee (8k vs 10k) is not multiply-confirmed.
- Single-request TTFT was unchanged — a tie, and one transient p99 blip (13 ms) appeared at 2,000 req/s. Both are reported, not hidden.
- No production/end-to-end latency superiority is claimed. This overhead is a rounding error next to real provider latency; where a thinner router has lower overhead, we'd report the loss.
How it's kept honest
Five hard gates, not polish
A slanted benchmark from the honesty company is worse than publishing nothing. These are constraints the harness holds itself to — when in doubt, it holds the claim and discloses why.
Reproducible by anyone
The whole harness — configs, pinned digests, the mock, the cost reference, and the methodology — is open. One command stands it up. A number you cannot re-run is not a result.
Steel-man every competitor
Each gateway runs on its latest stable version in a sensible, documented config on identical hardware and workload. If one cannot be set up fairly, it is excluded with a stated reason — never handicapped.
Own where we lose
A thinner router may beat us on raw overhead; we measure it and frame the tradeoff. A loss disclosed honestly beats a suspicious win, so the matrix includes the axes where we trail.
Verdict-label everything
Every row carries what was measured, its config, version/digest, sample, variance, and a verdict — or an honest non-verdict (CANNOT-FAIRLY-MEASURE, RUN-PENDING) when no claim is publishable.
No fabrication
Real measured numbers only. Ground truth comes from cited, dated provider docs — not from our own engine. If our engine disagreed with the docs, that would be a finding against us.
The head-to-head · measured · published win, lose & tie
The named numbers — same run, same hardware, pinned versions.
Measured 2026-07-29 on AWS z1d.metal from the latest stable pinned images, steel-manned configs, equal 4-core budgets. Published under the written pre-commitment: the same run's numbers ship together, win or lose — which is why the cost tiebelow is presented as prominently as the latency win.
Added latency (median TTFT overhead @ 0-delay mock)
| Gateway | Overhead | 95% CI | Posture during the run | Verdict |
|---|---|---|---|---|
| TokenTriage | 0.148 ms | [0.146, 0.150] | full ledger: cost computed + logged | Lowest — 62× / 2.6× less |
| Bifrost v1.6.6 | 0.392 ms | [0.384, 0.398] | request-log stores on | measured |
| LiteLLM v1.94.0 | 9.245 ms | [8.91, 9.65] | logging OFF (its latency config) | measured |
Conservative by construction: TokenTriage ran its full ledger (cost computed and logged per request) while LiteLLM ran its logging-off latency posture — we do more per-request work and still measure lowest. "62×" is the gateway's added latency, never an end-to-end speed claim.
Throughput on an equal 4-core budget
| Gateway | Sustained (100% success) | p99 under load |
|---|---|---|
| TokenTriage | ≥ 4,000 req/s — no saturation | 1.20 ms @ 4,000 |
| Bifrost v1.6.6 | ≥ 4,000 req/s — no saturation | 3.54 ms @ 4,000 |
| LiteLLM v1.94.0 | ~670 req/s, then saturates | 13.4–16.8 s past saturation |
The Go gateways (TokenTriage, Bifrost) both clear the 4,000 req/s test ceiling — the 6× vs LiteLLM is a floor, not a max. LiteLLM's ceiling scales with more workers on more cores; this test fixes the core budget equal for everyone. TokenTriage showed one transient p99 excursion (63 ms at 2,000 req/s) that cleared by 4,000 — reported, not hidden.
Cost accuracy — a three-way tie, published as one
- LiteLLM v1.94.0 Measured
Response-cost header; custom pricing under model_info
Cost: exact (tie) · overhead 9.245 ms · saturates ~670 req/s
- Bifrost v1.6.6 Measured
Per-request cost in the request log; pricing overrides
Cost: exact (tie) · overhead 0.392 ms · clears 4,000 req/s
- HeliconeNo local cost path
No local cost path — cost is authored only to its cloud sink (verified empirically, air-gapped)
Architecture note, not a scored result
- Portkey OSSNo local cost path
OSS build emits no per-request cost (hosted/enterprise only)
Architecture note, not a scored result
TokenTriage, LiteLLM, and Bifrost all match provider-doc ground truth to the nanodollar on every cached bucket — no cost-accuracy advantage is claimed over either. TokenTriage's edge on this axis is auditability: a per-request pricing snapshot and decision log, not a better number.
The written pre-commitment, honored: "if we publish our cost or latency number, we publish the competitors' numbers from the same run — win or lose." The founder GO was signed 2026-07-29 with all outcomes known; the tie ships beside the win, the rationale is recorded in the repo, and every number above is reproducible from the open harness with pinned image digests.
Where we lose · owned, not hidden
A matrix of only our wins would read as tilted.
Latency: no end-to-end superiority claim
On our virtualized Windows box, sub-millisecond gateway deltas fall below the platform noise floor — Cannot-fairly-measure, never banked as a favorable "tie." A verdict is publishable only from a bare-metal Linux reference, which we ran (see Performance above) and report our own numbers from. Even so,TokenTriage claims no production / end-to-end latency superiority — the gateway's overhead is a rounding error next to real provider latency — and where a thinner router has lower overhead we will report the loss.
Capabilities we don't lead
- Guardrails / PII — not shipped. Research-stage only; no subsystem in code.
NOT-SUPPORTEDapplies to us too. - Load-balancing is thinner. Weighted pools, least-latency, and passive health cooldown — but no P2C, consistent-hashing, active health probes, or hedging.
- Provider / model breadth trails. LiteLLM's 100+ provider catalog is broader — though TokenTriage now ships 80+ built-in providers, every one real-run-verified.
- Managed ecosystem & community. The hosted competitors lead here.
What we do not claim
The non-claims, stated plainly
TokenTriage is beta software. These are the claims v1 deliberately withholds — dropping them is the point, not an oversight.
No production latency superiority
Overhead is measured against a mock; at 60–600 ms real provider latency a sub-millisecond gateway difference is immaterial, and on this virtualized host it is below the resolution floor. Where a thinner router wins, we report the loss.
No quality or proven-savings number
A quality / proven-savings axis depends on calibration and evals work that does not exist yet. Overclaiming it would be dishonest — it is a documented fast-follow, gated on that work.
Reasoning tokens are not a cost lever
No provider prices reasoning apart from output, so reasoning tokens contribute $0 to any cost gap. They are parsed for completeness only, never used to manufacture an accuracy edge.
No cache-creation or tiered cache pricing
v1 exercises a single flat, cited cache-read rate per bucket. Cache-creation (write) billing and context-tiered or region-differentiated cache pricing are explicit non-claims.
Reproduce it yourself
One clone, one command.
TokenTriage's cost path runs natively — no Docker required for the flagship number. Each run stamps provenance.json (CPU / RAM / OS, container topology, image digests, git SHA, UTC) and writes verdict-labeled JSON per axis.
cd benchmarks && ./run.sh tokentriage --cost-onlyCompetitor runs (./run.sh litellm, ./run.sh bifrost) need Docker — the published head-to-head numbers above came from exactly these commands on pinned images.
Honest status
- MeasuredCost accuracy — exact, verdict-labeled, reproducible. Shippable now.
- MeasuredCompetitor head-to-head — published win, lose, and tie from the same run (founder GO signed 2026-07-29): overhead 62× / 2.6× less than LiteLLM / Bifrost (CIs disjoint), throughput 6×+ LiteLLM on equal cores (a floor), cost a three-way nanodollar tie.
- MeasuredLatency (ours) — TokenTriage's own overhead and throughput are measured on a bare-metal Linux reference (above); the Windows box staysCannot-fairly-measure. No end-to-end superiority is claimed.
TokenTriage is source-available (FSL) and beta software. Everything above traces to the verdict-labeled RESULTS.md in the open harness.