# Observability and tracing

Get the gateway into the monitoring you already run, and answer what happened to one request when somebody hands you its id.

Source: https://tokentriage.com/docs/build/observability

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import ObservabilitySurfacesDiagram from '../../../components/ObservabilitySurfacesDiagram.astro';

You already run something. Prometheus and Grafana, or Datadog, or an OpenTelemetry collector
in front of a backend you pay for. A new component in the request path is worth having only if
it shows up there beside everything else.

There is a second, sharper moment. Somebody forwards you a request id from a support ticket and
asks what happened to it, and you want the model that served it, the dollar it cost, the rule
that chose it and the agent step it belonged to, inside a minute.

This section is about both. The first is dashboards and alerts, and it is mostly wiring. The
second is a path through the ledger that starts from one response header.

## Three surfaces, three questions

Every request appends one row to the decision ledger. Three surfaces read or mirror that row,
and each answers a different question, which is the fastest way to decide where to look.

<ObservabilitySurfacesDiagram />

Read it left to right. The solid path is the row being written; the two dashed paths are
mirrors of it. Each destination carries its default underneath.

**The ledger** answers *what happened to this one request*. It is a JSON Lines file with one
wide row per request, read by the CLI, the dashboard and the query API. It is the system of
record, and it is on out of the box.

**The Prometheus endpoint** answers *is the fleet healthy*. It is a counter surface on the
admin plane at `/metrics`, aggregated, cheap to scrape, and bounded in cardinality,
which means it cannot answer per-request questions and is not trying to.

**The OTLP exporter** answers *where does this call sit among my other traces*. It posts one
span per ledger row to a collector you run. It is off until you switch it on.

<Callout type="key">
The ledger is the record; the other two are mirrors. Under pressure a mirror drops an entry and
counts the drop rather than slowing the request down, so a Prometheus total and a ledger total
can differ. When they do, the ledger is right. [What a drop
means](/docs/build/observability-drops) works through the arithmetic.
</Callout>

## Identity is on the wire or it is not there

A costed row is only as useful as the names you can group it by, and the names come from
headers your application sets. `x-tt-trace-id` groups calls into one run. `x-tt-step-id` and
`x-tt-parent-id` turn that run into a tree. `x-tt-agent` says which sub-agent made the call.

Nothing is inferred. TokenTriage does not stitch a conversation from timing gaps or from a
shared prompt prefix, and it does not adopt an inbound `traceparent` as identity. A request that
arrives with no identity headers is still priced, still logged, still counted; it lands as a
standalone row rather than being guessed into somebody else's trace.

That trade is worth stating plainly, because it costs something. A gateway that guessed would
show a fuller-looking tree on day one. It would also be wrong in ways you could not see, and a
cost tree you cannot audit is one you cannot use to settle an argument.

## What is on, and what is waiting for you

Three defaults decide what a fresh install can tell you.

| Surface | Default | Turn it on with |
|---|---|---|
| Prometheus `/metrics` | on, on the admin plane | nothing; it is mounted when the proxy starts |
| identity headers | read whenever a client sends them | your application, or `tokentriage wrap` |
| business dimensions | off | `ledger.dimensions.enabled: true` |
| OTLP span export | off | `export.otlp.enabled: true`, plus the dashboard |
| the Live feed | with the dashboard, which is off in a config file | `dashboard.enabled: true` |

The two that surprise people are the last two. Business dimensions being off means a fresh
install groups spend by model, provider and tier but not by tenant, feature or user, because
nothing has told it who those are. And OTLP export needs the dashboard enabled as well as its
own flag, which is not obvious from the config and is
[covered where you configure it](/docs/build/observability-otlp#2-the-flag-that-is-not-enough).

## What this section does not cover

It does not teach the business-dimension headers. `x-tt-tenant`, `x-tt-feature`, `x-tt-user`
and `x-tt-tag-<key>` answer *whose budget this comes out of*, which is a money question, and
[turn on business dimensions](/docs/build/cost-dimensions) is the one page that configures them.
This section owns the identity family and everything downstream of a metric label, including
[what promoting a dimension to Prometheus costs](/docs/build/observability-cardinality).

It does not define the ledger row field by field; that is [decision ledger
fields](/docs/reference/ledger-fields). It does not repeat the wire-header table in [HTTP
headers](/docs/reference/headers). It does not cover the dashboard's other surfaces beyond the
Live feed, which is [the trust dashboard](/docs/build/dashboard)'s job. And it does not publish
comparative latency figures; [benchmarks](/docs/reference/benchmarks) owns those, and
[measuring your own overhead](/docs/build/observability-performance) shows you how to get your
own instead.

<NextSteps items={[
  {title: "Trace one request end to end", href: "/docs/build/observability-request-id", description: "You have a request id and nothing else. Five steps to the row, the trace, the decision and the dollar."},
  {title: "What identity is on the wire", href: "/docs/build/observability-identity", description: "The six identity headers, why none is required, and what happens to them before the request is forwarded."},
  {title: "Scrape the metrics endpoint", href: "/docs/build/observability-metrics", description: "Point Prometheus at the admin plane, read your first scrape, and load the three shipped alert rules."},
  {title: "Export spans to your collector", href: "/docs/build/observability-otlp", description: "Turn on OTLP, wire it to your own collector, and see what actually arrives."},
]} />