Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Observability and tracing

Get the gateway into the monitoring you already run, and answer what happened to one request when somebody hands you its id.

ExplanationLast updated

On this page

You already run something. Prometheus and Grafana, or Datadog, or an OpenTelemetry collector in front of a backend you pay for. A new component in the request path is worth having only if it shows up there beside everything else.

There is a second, sharper moment. Somebody forwards you a request id from a support ticket and asks what happened to it, and you want the model that served it, the dollar it cost, the rule that chose it and the agent step it belonged to, inside a minute.

This section is about both. The first is dashboards and alerts, and it is mostly wiring. The second is a path through the ledger that starts from one response header.

Three surfaces, three questions

Every request appends one row to the decision ledger. Three surfaces read or mirror that row, and each answers a different question, which is the fastest way to decide where to look.

One costed row feeding three observability surfacesA request arrives on the data plane and appends one row to the decision ledger, which is the system of record and is on out of the box. Three surfaces read or mirror that row. The ledger answers what happened to one request. Prometheus counters on the admin plane at slash metrics answer whether the fleet is healthy, and are also on out of the box. An OTLP span posted to your own collector answers where this call sits among your other traces, and is off until you enable it. The two mirrors are drawn with dashed lines because each can drop under pressure without stopping traffic, while the ledger row is written on the solid path.One requestdata plane :8787One ledger rowreq_id · cost · traceon out of the boxThe ledger file and CLIwhat happened to one requesttop --trace · the dashboardPrometheus countersis the fleet healthy:9090/metrics · onOTLP spanswhere this sits in my tracesyour collector · offsystem of recordA dashed mirror may drop under pressure and count the drop; the row stays

Read it left to right. The solid path is the row being written; the two dashed paths are mirrors of it. Each destination carries its default underneath.

The ledger answers what happened to this one request. It is a JSON Lines file with one wide row per request, read by the CLI, the dashboard and the query API. It is the system of record, and it is on out of the box.

The Prometheus endpoint answers is the fleet healthy. It is a counter surface on the admin plane at /metrics, aggregated, cheap to scrape, and bounded in cardinality, which means it cannot answer per-request questions and is not trying to.

The OTLP exporter answers where does this call sit among my other traces. It posts one span per ledger row to a collector you run. It is off until you switch it on.

Identity is on the wire or it is not there

A costed row is only as useful as the names you can group it by, and the names come from headers your application sets. x-tt-trace-id groups calls into one run. x-tt-step-id and x-tt-parent-id turn that run into a tree. x-tt-agent says which sub-agent made the call.

Nothing is inferred. TokenTriage does not stitch a conversation from timing gaps or from a shared prompt prefix, and it does not adopt an inbound traceparent as identity. A request that arrives with no identity headers is still priced, still logged, still counted; it lands as a standalone row rather than being guessed into somebody else’s trace.

That trade is worth stating plainly, because it costs something. A gateway that guessed would show a fuller-looking tree on day one. It would also be wrong in ways you could not see, and a cost tree you cannot audit is one you cannot use to settle an argument.

What is on, and what is waiting for you

Three defaults decide what a fresh install can tell you.

Surface Default Turn it on with
Prometheus /metrics on, on the admin plane nothing; it is mounted when the proxy starts
identity headers read whenever a client sends them your application, or tokentriage wrap
business dimensions off ledger.dimensions.enabled: true
OTLP span export off export.otlp.enabled: true, plus the dashboard
the Live feed with the dashboard, which is off in a config file dashboard.enabled: true

The two that surprise people are the last two. Business dimensions being off means a fresh install groups spend by model, provider and tier but not by tenant, feature or user, because nothing has told it who those are. And OTLP export needs the dashboard enabled as well as its own flag, which is not obvious from the config and is covered where you configure it.

What this section does not cover

It does not teach the business-dimension headers. x-tt-tenant, x-tt-feature, x-tt-user and x-tt-tag-<key> answer whose budget this comes out of, which is a money question, and turn on business dimensions is the one page that configures them. This section owns the identity family and everything downstream of a metric label, including what promoting a dimension to Prometheus costs.

It does not define the ledger row field by field; that is decision ledger fields. It does not repeat the wire-header table in HTTP headers. It does not cover the dashboard’s other surfaces beyond the Live feed, which is the trust dashboard’s job. And it does not publish comparative latency figures; benchmarks owns those, and measuring your own overhead shows you how to get your own instead.