Measure your own overhead

Putting a proxy in front of your model calls raises a fair question: how much does it add? The answer for your deployment, on your hardware, under your traffic, is a scrape away, and it is worth more than any number somebody else published.

By the end of this page you can read both latency histograms, you know precisely what each one measures, and you know the two ways they under-report.

The two histograms

Metric	Measures	Labels
tokentriage_proxy_overhead_seconds	from the request arriving to the moment it is forwarded upstream	dialect
tokentriage_time_to_first_token_seconds	from the request arriving to the first response byte	dialect, upstream

The scopes are different and the difference matters.

Overhead is the inbound pipeline only: reading the body, parsing it, extracting headers, dispatching the endpoint, running the classifier signals and deciding. It stops the instant the request is handed to the upstream. It does not include the provider’s time, and it does not include response-side work, so it is the cost of being in the path on the way in rather than the whole cost of the hop. The same quantity is on every ledger row as latency.overhead_us, which is the field to use when you want one request rather than a distribution.

Time to first token spans the proxy and the provider together. It is dominated by the model, which is what makes it useful for watching a provider and unsuitable for measuring the proxy.

Read them

Overhead buckets run from 100 µs to 50 ms, so the useful quantile is high:

histogram_quantile(0.99, sum by (le, dialect) (
  rate(tokentriage_proxy_overhead_seconds_bucket[5m])
))

The mean is the sum over the count, and it is worth having beside the quantile because the two diverge when a small share of requests does extra work:

rate(tokentriage_proxy_overhead_seconds_sum[5m])
  / rate(tokentriage_proxy_overhead_seconds_count[5m])

First-token latency uses the same shapes against tokentriage_time_to_first_token_seconds, and its upstream label is what makes it worth graphing: one line per provider shows you which one slowed down.

Two ways these histograms under-report

Sub-millisecond first bytes are not recorded. The first-token observation is taken only when the measured time rounds to at least one whole millisecond. Anything faster is skipped, so the histogram’s _count is lower than the request count and the distribution is biased upward. On provider traffic this is invisible, because a model’s first token takes hundreds of milliseconds. It bites where responses are fast: a served cache hit, a local model on the same host, or a test against a mock. In one lab run, 66 requests against a local mock produced 5 first-token observations.

Your clock’s resolution can dominate the fastest bucket. Overhead is a difference between two timestamps, and on a platform whose timer ticks coarsely a sub-tick difference measures as zero while an occasional one lands a whole tick out. The same lab run:

tokentriage_proxy_overhead_seconds_bucket{dialect="openai",le="0.0001"} 58
tokentriage_proxy_overhead_seconds_bucket{dialect="openai",le="0.001"}  65
tokentriage_proxy_overhead_seconds_bucket{dialect="openai",le="0.0025"} 66
tokentriage_proxy_overhead_seconds_count{dialect="openai"}              66

Fifty-eight requests landed in the fastest bucket, seven measured between 500 µs and 1 ms, and one measured 1.5 ms. Reading the rows behind those eight shows they are scattered through the run rather than clustered at the start, and every one of them sits within a tick of 500 µs, which is the signature of timer granularity rather than of work. That capture is from a source build against a mock upstream on one Windows desktop, and it is not a performance figure for anything. It is here to show what a measurement floor looks like: check the shape of your fastest bucket before you quote a mean from it.

What these numbers do and do not support

They support statements about your deployment: what the proxy added at the 99th percentile over a named window, on named hardware, under your traffic. That is a useful thing to put in a design review, and it is checkable by anyone who can reach your Prometheus.

They do not support a comparison with another gateway. A comparison needs both systems on the same hardware, driven by the same workload, in the same run, with the methodology published; a number from your instance and a number from somebody’s blog post are not the same measurement.

Published figures for TokenTriage carry their conditions with them and live in benchmarks. Two rules govern how they are stated, and they are worth knowing if you are quoting one: the quantity is overhead or added latency rather than speed, and figures from different runs are not combined.

What can go wrong

The overhead histogram has fewer observations than tokentriage_requests_total. Requests that were turned away before being forwarded have no forward moment to measure against, so a refusal or a served cache hit contributes to the request counter and not to overhead. That is the metric being accurate rather than losing data.

The 99th percentile is a bucket boundary. Histogram quantiles interpolate within buckets, so a quantile that lands in the top bucket reports its edge. With buckets ending at 50 ms, a p99 of exactly 50 ms means “at least 50 ms” and needs a wider bucket set to resolve, which is a build change rather than a config one.

Overhead looks high and the cause is the classifier. Signal evaluation is measured separately as tokentriage_signal_duration_seconds, labelled by provider. Compare its sum against the overhead sum before assuming the pipeline itself is slow.

Measuring the whole hop instead of the inbound half

tokentriage_request_duration_seconds measures end to end, provider included, labelled by dialect, upstream and stream. Subtracting it from the provider’s own reported latency is not a sound way to isolate proxy cost, because the two are measured at different points and over different clocks.

If you want the round-trip cost of being in the path, the honest experiment is a comparison against yourself: send the same workload directly to the provider and through the proxy, from the same client, in the same window, and compare the two distributions. That is the shape the published benchmark uses, and it is the one whose conditions you can write down.