Scrape the metrics endpoint

By the end of this page TokenTriage is a scrape target in the Prometheus you already run, you know why your first scrape has fewer metrics in it than the catalogue lists, and three alerts are loaded.

There is nothing to enable. /metrics is mounted on the admin listener when the proxy starts.

1. Confirm the endpoint

The admin listener is server.admin, which defaults to 127.0.0.1:9090 and is separate from the data plane your traffic uses.

$ curl -s http://127.0.0.1:9090/metrics | grep '^# TYPE'
# TYPE tokentriage_cost_usd_total counter
# TYPE tokentriage_dash_stream_dropped_total counter
# TYPE tokentriage_declog_dropped_total counter
# TYPE tokentriage_dim_requests_total counter
# TYPE tokentriage_dims_dropped_total counter
# TYPE tokentriage_otlp_dropped_total counter
# TYPE tokentriage_proxy_overhead_seconds histogram
# TYPE tokentriage_request_duration_seconds histogram
# TYPE tokentriage_requests_total counter
# TYPE tokentriage_signal_duration_seconds histogram
# TYPE tokentriage_time_to_first_token_seconds histogram
# TYPE tokentriage_tokens_total counter

Twelve families, from a lab instance that had served traffic with business dimensions and span export both switched on. The metrics reference lists nineteen. That difference is the next section, and it is the thing most worth understanding before you build a dashboard.

2. Understand what is missing, and why

A Prometheus metric with labels emits nothing until something has been observed for a label combination. A counter that has not been incremented for any label set is registered and absent from the scrape, and a client that has not seen it renders “No data” rather than zero.

So a fresh instance shows a subset that grows as your traffic exercises features, and three categories are missing for three different reasons.

Missing because	Example	Appears when
nothing has happened yet	tokentriage_pricing_miss_total	a model with no price is served
the feature is off	tokentriage_dim_requests_total	dimensions are on and one is promoted
no code increments it in this build	tokentriage_decisions_total	not yet; see the reference

The third row is real and there are six of them. The metrics reference names each one, so you can leave those panels out rather than deploying a dashboard with six permanently empty tiles.

A practical consequence for alerting: reach for absent() as well as rate(...). An alert on a metric that has never been emitted does not fire, and does not warn you that it cannot.

3. Add the scrape

For a plain Prometheus, the target is the admin listener:

scrape_configs:
  - job_name: tokentriage
    static_configs:
      - targets: ['tokentriage.internal:9090'] # server.admin
    metrics_path: /metrics # the default; shown for clarity

On Kubernetes the chart renders a ServiceMonitor for the Prometheus Operator. It is off out of the box, and it renders only when the operator’s CRD is present on the cluster, so enabling it on a cluster without the operator produces no object rather than a broken manifest:

metrics:
  serviceMonitor:
    enabled: true # default false
    interval: '' # "" uses the Prometheus default
    labels: {} # selector labels your Prometheus matches on

The rendered monitor scrapes the admin port at /metrics and keeps only targets discovered through the admin ClusterIP service. That last part matters because the headless service used for peer discovery carries the same labels and publishes not-ready pod addresses; without the filter each pod would be scraped twice and every counter would read double.

4. Load the alert rules

The chart ships three curated alerts, also off out of the box:

metrics:
  prometheusRule:
    enabled: true # default false
    additionalRules: [] # appended verbatim after the three below

Alert	Expression	Fires when
TokenTriageDeclogDrops	rate(tokentriage_declog_dropped_total[5m]) > 0	the ledger is losing rows to a full writer queue
TokenTriagePricingMisses	sum(rate(tokentriage_pricing_miss_total[15m])) by (model) > 0	a model with no price entry is being served, so its spend is excluded
TokenTriageEnforcementSuspended	max by (limit_id) (tokentriage_governance_limit_suspended) > 0	governance enforcement has fallen open for a limit

The first two fire on metrics the binary increments today. The third is written against a gauge that no code path sets in this build, so it cannot fire yet; load it if you want the rule in place for when the emitter lands, and do not treat its silence as evidence that enforcement is healthy.

Each alert’s description says what the number means rather than only that it moved. A drop alert that says “the ledger is losing records, each drop is counted rather than replaced by a fabricated row” tells whoever is paged what they are looking at.

What can go wrong

Every counter reads double. Two scrape targets are hitting the same pod, usually the headless service alongside the admin service. The chart’s relabeling rule handles this; a hand-written scrape config needs the same filter.

A histogram’s _count is lower than tokentriage_requests_total. Expected for tokentriage_time_to_first_token_seconds, which is observed only when a first streamed byte is timed. Comparing its count against total requests is a category error rather than a lost observation.

Totals disagree with the ledger. They can. Counters reset when the process restarts, and Prometheus totals are a mirror rather than the record. What a drop means covers the cases where the gap is a drop rather than a restart.

Scraping several replicas without double counting spend

Each replica exports its own counters, so sum(tokentriage_cost_usd_total) across a fleet is the fleet’s spend and a single instance’s value is that instance’s share. Keep the instance label your service discovery adds and aggregate explicitly.

What you should not do is treat a Prometheus sum as the number you bill from. Counters reset on restart, the mirrors can drop, and the ledger is the record. Use the counters for rate, shape and alerting; use the cost surfaces for a figure somebody will act on.