# Measure your own overhead

Two histograms say what the proxy adds and when the first token arrived. How to read them, and what they will not support.

Source: https://tokentriage.com/docs/build/observability-performance

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

Putting a proxy in front of your model calls raises a fair question: how much does it add? The
answer for your deployment, on your hardware, under your traffic, is a scrape away, and it is
worth more than any number somebody else published.

By the end of this page you can read both latency histograms, you know precisely what each one
measures, and you know the two ways they under-report.

## The two histograms

| Metric | Measures | Labels |
|---|---|---|
| `tokentriage_proxy_overhead_seconds` | from the request arriving to the moment it is forwarded upstream | `dialect` |
| `tokentriage_time_to_first_token_seconds` | from the request arriving to the first response byte | `dialect`, `upstream` |

The scopes are different and the difference matters.

**Overhead** is the inbound pipeline only: reading the body, parsing it, extracting headers,
dispatching the endpoint, running the classifier signals and deciding. It stops the instant the
request is handed to the upstream. It does not include the provider's time, and it does not
include response-side work, so it is the cost of being in the path on the way in rather than the
whole cost of the hop. The same quantity is on every ledger row as `latency.overhead_us`, which
is the field to use when you want one request rather than a distribution.

**Time to first token** spans the proxy and the provider together. It is dominated by the model,
which is what makes it useful for watching a provider and unsuitable for measuring the proxy.

## Read them

Overhead buckets run from 100 µs to 50 ms, so the useful quantile is high:

```text
histogram_quantile(0.99, sum by (le, dialect) (
  rate(tokentriage_proxy_overhead_seconds_bucket[5m])
))
```

The mean is the sum over the count, and it is worth having beside the quantile because the two
diverge when a small share of requests does extra work:

```text
rate(tokentriage_proxy_overhead_seconds_sum[5m])
  / rate(tokentriage_proxy_overhead_seconds_count[5m])
```

First-token latency uses the same shapes against `tokentriage_time_to_first_token_seconds`, and
its `upstream` label is what makes it worth graphing: one line per provider shows you which one
slowed down.

## Two ways these histograms under-report

**Sub-millisecond first bytes are not recorded.** The first-token observation is taken only when
the measured time rounds to at least one whole millisecond. Anything faster is skipped, so the
histogram's `_count` is lower than the request count and the distribution is biased upward. On
provider traffic this is invisible, because a model's first token takes hundreds of milliseconds.
It bites where responses are fast: a served cache hit, a local model on the same host, or a test
against a mock. In one lab run, 66 requests against a local mock produced 5 first-token
observations.

**Your clock's resolution can dominate the fastest bucket.** Overhead is a difference between two
timestamps, and on a platform whose timer ticks coarsely a sub-tick difference measures as zero
while an occasional one lands a whole tick out. The same lab run:

```text
tokentriage_proxy_overhead_seconds_bucket{dialect="openai",le="0.0001"} 58
tokentriage_proxy_overhead_seconds_bucket{dialect="openai",le="0.001"}  65
tokentriage_proxy_overhead_seconds_bucket{dialect="openai",le="0.0025"} 66
tokentriage_proxy_overhead_seconds_count{dialect="openai"}              66
```

Fifty-eight requests landed in the fastest bucket, seven measured between 500 µs and 1 ms, and one
measured 1.5 ms. Reading the rows behind those eight shows they are scattered through the run
rather than clustered at the start, and every one of them sits within a tick of 500 µs, which is
the signature of timer granularity rather than of work. That capture is from a source build
against a mock upstream on one Windows desktop, and it is not a performance figure for anything.
It is here to show what a measurement floor looks like: check the shape of your fastest bucket
before you quote a mean from it.

<Callout type="warning">
Do not derive a headline number from a short local run. Overhead scales with body size, tool
count and which signal providers your classifier runs, and a mock upstream removes the network
entirely. Measure on your own traffic, over a window long enough to include your real request
shapes, and quote the window alongside the number.
</Callout>

## What these numbers do and do not support

They support statements about your deployment: what the proxy added at the 99th percentile over a
named window, on named hardware, under your traffic. That is a useful thing to put in a design
review, and it is checkable by anyone who can reach your Prometheus.

They do not support a comparison with another gateway. A comparison needs both systems on the
same hardware, driven by the same workload, in the same run, with the methodology published; a
number from your instance and a number from somebody's blog post are not the same measurement.

Published figures for TokenTriage carry their conditions with them and live in
[benchmarks](/docs/reference/benchmarks). Two rules govern how they are stated, and they are worth
knowing if you are quoting one: the quantity is *overhead* or *added latency* rather than speed,
and figures from different runs are not combined.

## What can go wrong

**The overhead histogram has fewer observations than `tokentriage_requests_total`.** Requests that
were turned away before being forwarded have no forward moment to measure against, so a refusal or
a served cache hit contributes to the request counter and not to overhead. That is the metric
being accurate rather than losing data.

**The 99th percentile is a bucket boundary.** Histogram quantiles interpolate within buckets, so a
quantile that lands in the top bucket reports its edge. With buckets ending at 50 ms, a p99 of
exactly 50 ms means "at least 50 ms" and needs a wider bucket set to resolve, which is a build
change rather than a config one.

**Overhead looks high and the cause is the classifier.** Signal evaluation is measured separately
as `tokentriage_signal_duration_seconds`, labelled by provider. Compare its sum against the
overhead sum before assuming the pipeline itself is slow.

<Details summary="Measuring the whole hop instead of the inbound half">

`tokentriage_request_duration_seconds` measures end to end, provider included, labelled by
`dialect`, `upstream` and `stream`. Subtracting it from the provider's own reported latency is not
a sound way to isolate proxy cost, because the two are measured at different points and over
different clocks.

If you want the round-trip cost of being in the path, the honest experiment is a comparison
against yourself: send the same workload directly to the provider and through the proxy, from the
same client, in the same window, and compare the two distributions. That is the shape the
published benchmark uses, and it is the one whose conditions you can write down.

</Details>

<NextSteps items={[
  {title: "The metrics catalogue", href: "/docs/build/observability-metrics-reference", description: "Both histograms with their bucket ranges, beside everything else on the endpoint."},
  {title: "Benchmarks", href: "/docs/reference/benchmarks", description: "The published figures, the hardware they were measured on, and the methodology behind them."},
  {title: "Scrape the metrics endpoint", href: "/docs/build/observability-metrics", description: "Getting these histograms into Prometheus in the first place."},
  {title: "Trace one request end to end", href: "/docs/build/observability-request-id", description: "The per-request view: latency.overhead_us on the row, for when a distribution is not the question."},
]} />