# Scrape the metrics endpoint

Point Prometheus at the admin plane, read your first scrape, and load the alert rules the chart already ships.

Source: https://tokentriage.com/docs/build/observability-metrics

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

By the end of this page TokenTriage is a scrape target in the Prometheus you already run, you
know why your first scrape has fewer metrics in it than the catalogue lists, and three alerts
are loaded.

There is nothing to enable. `/metrics` is mounted on the admin listener when the proxy starts.

## 1. Confirm the endpoint

The admin listener is `server.admin`, which defaults to `127.0.0.1:9090` and is separate from
the data plane your traffic uses.

```console
$ curl -s http://127.0.0.1:9090/metrics | grep '^# TYPE'
# TYPE tokentriage_cost_usd_total counter
# TYPE tokentriage_dash_stream_dropped_total counter
# TYPE tokentriage_declog_dropped_total counter
# TYPE tokentriage_dim_requests_total counter
# TYPE tokentriage_dims_dropped_total counter
# TYPE tokentriage_otlp_dropped_total counter
# TYPE tokentriage_proxy_overhead_seconds histogram
# TYPE tokentriage_request_duration_seconds histogram
# TYPE tokentriage_requests_total counter
# TYPE tokentriage_signal_duration_seconds histogram
# TYPE tokentriage_time_to_first_token_seconds histogram
# TYPE tokentriage_tokens_total counter
```

Twelve families, from a lab instance that had served traffic with business dimensions and span
export both switched on. The [metrics reference](/docs/build/observability-metrics-reference)
lists nineteen. That difference is the next section, and it is the thing most worth understanding
before you build a dashboard.

<Callout type="warning">
`/metrics` is served without authentication, and turning on dashboard auth does not change that.
The listener's bind address is the access control: `127.0.0.1` keeps it on the box, and anything
wider exposes model names, upstream names and spend totals to whoever can reach the port. Bind
the admin listener to an interface your monitoring can reach and nothing else can.
</Callout>

## 2. Understand what is missing, and why

A Prometheus metric with labels emits nothing until something has been observed for a label
combination. A counter that has not been incremented for any label set is registered and
absent from the scrape, and a client that has not seen it renders "No data" rather than zero.

So a fresh instance shows a subset that grows as your traffic exercises features, and three
categories are missing for three different reasons.

| Missing because | Example | Appears when |
|---|---|---|
| nothing has happened yet | `tokentriage_pricing_miss_total` | a model with no price is served |
| the feature is off | `tokentriage_dim_requests_total` | dimensions are on and one is promoted |
| no code increments it in this build | `tokentriage_decisions_total` | not yet; see the reference |

The third row is real and there are six of them. The [metrics
reference](/docs/build/observability-metrics-reference#metrics-with-no-emitter-in-this-build)
names each one, so you can leave those panels out rather than deploying a dashboard with six
permanently empty tiles.

A practical consequence for alerting: reach for `absent()` as well as `rate(...)`.
An alert on a metric that has never been emitted does not fire, and does not warn you that it
cannot.

## 3. Add the scrape

For a plain Prometheus, the target is the admin listener:

```yaml
scrape_configs:
  - job_name: tokentriage
    static_configs:
      - targets: ['tokentriage.internal:9090'] # server.admin
    metrics_path: /metrics # the default; shown for clarity
```

On Kubernetes the chart renders a `ServiceMonitor` for the Prometheus Operator. It is off out of
the box, and it renders only when the operator's CRD is present on the cluster, so enabling it on
a cluster without the operator produces no object rather than a broken manifest:

```yaml
metrics:
  serviceMonitor:
    enabled: true # default false
    interval: '' # "" uses the Prometheus default
    labels: {} # selector labels your Prometheus matches on
```

The rendered monitor scrapes the `admin` port at `/metrics` and keeps only targets discovered
through the admin `ClusterIP` service. That last part matters because the headless service used
for peer discovery carries the same labels and publishes not-ready pod addresses; without the
filter each pod would be scraped twice and every counter would read double.

## 4. Load the alert rules

The chart ships three curated alerts, also off out of the box:

```yaml
metrics:
  prometheusRule:
    enabled: true # default false
    additionalRules: [] # appended verbatim after the three below
```

| Alert | Expression | Fires when |
|---|---|---|
| `TokenTriageDeclogDrops` | `rate(tokentriage_declog_dropped_total[5m]) > 0` | the ledger is losing rows to a full writer queue |
| `TokenTriagePricingMisses` | `sum(rate(tokentriage_pricing_miss_total[15m])) by (model) > 0` | a model with no price entry is being served, so its spend is excluded |
| `TokenTriageEnforcementSuspended` | `max by (limit_id) (tokentriage_governance_limit_suspended) > 0` | governance enforcement has fallen open for a limit |

The first two fire on metrics the binary increments today. The third is written against a gauge
that no code path sets in this build, so it cannot fire yet; load it if you want the rule in
place for when the emitter lands, and do not treat its silence as evidence that enforcement is
healthy.

Each alert's description says what the number means rather than only that it moved. A drop alert
that says "the ledger is losing records, each drop is counted rather than replaced by a fabricated
row" tells whoever is paged what they are looking at.

## What can go wrong

**Every counter reads double.** Two scrape targets are hitting the same pod, usually the headless
service alongside the admin service. The chart's relabeling rule handles this; a hand-written
scrape config needs the same filter.

**A histogram's `_count` is lower than `tokentriage_requests_total`.** Expected for
`tokentriage_time_to_first_token_seconds`, which is observed only when a first streamed byte is
timed. Comparing its count against total requests is a category error rather than a lost
observation.

**Totals disagree with the ledger.** They can. Counters reset when the process restarts, and
Prometheus totals are a mirror rather than the record. [What a drop
means](/docs/build/observability-drops) covers the cases where the gap is a drop rather than a
restart.

<Details summary="Scraping several replicas without double counting spend">

Each replica exports its own counters, so `sum(tokentriage_cost_usd_total)` across a fleet is the
fleet's spend and a single instance's value is that instance's share. Keep the instance label
your service discovery adds and aggregate explicitly.

What you should not do is treat a Prometheus sum as the number you bill from. Counters reset on
restart, the mirrors can drop, and the ledger is the record. Use the counters for rate, shape and
alerting; use [the cost surfaces](/docs/build/cost-analytics) for a figure somebody will act on.

</Details>

<NextSteps items={[
  {title: "The metrics catalogue", href: "/docs/build/observability-metrics-reference", description: "All nineteen metric names with their type, labels, when each is emitted, and the six with no emitter."},
  {title: "What a promoted dimension costs", href: "/docs/build/observability-cardinality", description: "Why agent is never a label, how to promote a tenant safely, and the two guards that stop a cardinality explosion."},
  {title: "What a drop means", href: "/docs/build/observability-drops", description: "Four drop counters, what each one loses, and which totals go quiet when they move."},
  {title: "Measure your own overhead", href: "/docs/build/observability-performance", description: "Read the two latency histograms on your own deployment instead of quoting somebody else's figure."},
]} />