# Trace one request from its ID Start from a support ticket's request ID and read the release that served it, every attempt, its cost state, and what remains unstated. Source: https://tokentriage.com/docs/build/observability-request-id --- import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; A support ticket says "the helpdesk answer was wrong at 11:22", and the only thing attached is a request ID. By the end of this page that ID has told you which gateway, environment and release served the request, which provider attempts it made, what it cost or why there is no figure, and which parts of the answer the service could not state. Everything here uses the hosted customer API through the `tokentriage cloud` CLI or either SDK. It is a read: nothing on this page sends a model request or spends money. If you run the local gateway and read its decision log instead, skip to [Local ledger profile (legacy)](#local-ledger-profile-legacy) at the end. ## What you need | You need | Because | |---|---| | the request ID (`lreq_…`) | every read on this page starts from it | | your organization ID (`org_…`) and workspace ID (`t_…`) | the ID is looked up inside one workspace | | a management key in `TOKENTRIAGE_MANAGEMENT_KEY` | the runtime key your application sends traffic with cannot read management records | | `metadata.read` on that workspace | it gates the whole read | | `cost.read` on that workspace | without it the read still works, and the cost comes back `unavailable` instead of a figure | You do not need the gateway's runtime key, access to the provider, a log file, or `content.read`: no prompt or completion text is returned by any read below. [Who can do what in your company](/docs/build/organizations-and-access) covers how the two grants reach a person. ## 1. Get the request ID A hosted gateway stamps every response with an `x-tt-request-id` header. The ticket should carry it; if you are reproducing the call yourself, [Send your first gateway request](/docs/get-started/first-live-request) prints it from either SDK. The value is the logical request ID: retries and fallbacks inside that one call share it, so one ID is one request however many provider attempts it took. ## In the portal If you would rather click than type, open the gateway in the portal, choose its environment, and open the **Requests** tab. Paste the ID into the form labelled **Find a request by ID** and choose **Find**. The portal uses your signed-in session, with the same two grants as the table above. - **The receipt opens even when the request is not on the list page.** The Requests list is a bounded page of the window; the lookup reads the one request by its ID instead. - **Attempts are read for that request on the server.** Each is labelled "Attempt n of N observed in this window", so N counts what the selected window observed. - **A miss says what it knows.** It reads: "No request with this ID is readable in the selected scope as of this read. It may be mistyped, belong to another application or environment, or not be indexed yet. Refresh to read again." Beneath it are the server's detail, with the range it searched, and the retry interval. **Refresh request evidence** reads again. - **An unavailable reader and a grant you lack are separate states**, each shown differently from a miss, so an outage does not read as "no such request". This view fetches no prompt or completion text. The labels above are quoted from the portal source at this page's date; no capture of this screen is published yet. The rest of this page does the same with the CLI or an SDK. ## 2. Read the receipt `customer.requests.get` returns the request's receipt: where it was served, the release, the final attempt, usage, cost and protection. From the CLI: ```console $ tokentriage cloud request get \ --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --request lreq_2026_09_20_000001 \ --base-url http://127.0.0.1:48241 request lreq_2026_09_20_000001 status succeeded served by gateway gw_ABCDEFGHJKLM, environment env_ABCDEFGHJKLM, release rel_ABCDEFGHJKLM purpose application (chat, streaming) acted as unavailable - attempt att_2026_09_20_000001 primary via openai/gpt-4o-mini usage 812 in, 240 out, 128 cached tokens (provider_reported) cost $0.001845 (1845000 nano_usd) at price-2026-09 guards unavailable (guardrail outcomes are held outside this backend's fence, so it cannot say what the guards did) Not stated: first_output_at (not_projected) Not stated: release.subject_kind (not_projected) Not stated: release.draft_revision (not_projected) Not stated: run (not_projected) ``` One line went to stderr, naming the operation and the URL it called: ```text cloud: customer.requests.get GET http://127.0.0.1:48241/portal/api/v1/customer/v1/organizations/org_ABCDEFGHJKLM/workspaces/t_ABCDEFGHJKLM/requests/lreq_2026_09_20_000001 ``` That capture ran against a loopback server replaying `internal/control/foundation/httpapi/testdata/request-record-projected.golden.json`, the record the service's own projection test writes for the hosted (analytics-backed) reader. The `Not stated:` lines are the answer's coverage, which step 6 explains. The IDs, the port and the figure are that recording's; against your deployment, pass your own `--base-url` and IDs. The `$0.001845` is fixture arithmetic, not anyone's bill. The same read from Python. The answer is an envelope whose `data` member is the receipt: ```python import os from tokentriage.customer import CustomerClient client = CustomerClient(os.environ["TOKENTRIAGE_BASE_URL"], os.environ["TOKENTRIAGE_MANAGEMENT_KEY"]) receipt = client.requests.get("org_ABCDEFGHJKLM", "t_ABCDEFGHJKLM", "lreq_2026_09_20_000001") record = receipt["data"] print(receipt["kind"]) print(record["release"]["release_id"], record["attempt"]["provider_model"]) print(record["cost"]) print([u["member"] for u in record["coverage"]["unstated"]]) ``` ```text request-record rel_ABCDEFGHJKLM gpt-4o-mini {'state': 'priced', 'nano_usd': 1845000, 'nano_usd_decimal': '1845000', 'price_snapshot': 'price-2026-09'} ['first_output_at', 'release.subject_kind', 'release.draft_revision', 'run'] ``` That program ran with the in-repo Python SDK (`sdk/python/src`) under Python 3.14.5, against the same loopback replay. The TypeScript client has the same method: _Not yet executed against a deployment; output omitted._ ```typescript import { CustomerClient } from "tokentriage/customer"; const client = new CustomerClient({ baseUrl: process.env.TOKENTRIAGE_BASE_URL!, managementApiKey: process.env.TOKENTRIAGE_MANAGEMENT_KEY!, }); const { data: record } = await client.requests.get(org, workspace, requestId); console.log(record.release?.release_id, record.cost.state); ``` ## 3. Read which release served it `--json` prints the server's document unchanged, and it is the rendering to keep in a ticket. The serving configuration is the `release` block: ```json "release": { "release_id": "rel_ABCDEFGHJKLM", "effective_digest": "sha256:6f1c0d1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6", "mandatory_generation": 3, "route_version": "route/2026-09-20.1", "policy_version": "guardrail/2026-09-18.2" } ``` That block is cut from the `--json` output of the same replay. Read it this way: - **`release_id`** is the deployed release the routing decision used. Compare it with the release the environment shows as applied now; if they differ, the request ran on an earlier deployment. - **`effective_digest`** identifies the complete effective configuration, so two requests with the same digest ran under the same rules. - **`route_version`** and **`policy_version`** name the routing and guardrail revisions inside that release. - A **`release` of `null`** means the request was turned away before any release was resolved. `status` and `reason_code` then say why. A playground or test run carries a draft revision instead of a `release_id`. The hosted reader does not project that revision yet, which is why this answer lists `release.draft_revision` under `coverage` (step 6). ## 4. List every attempt it made The receipt's `attempt` block is the **final** attempt only. A request that retried, or fell back to a second provider, made more dispatches than that, and the cost on the receipt already covers all of them. To see each dispatch, ask the attempt drill-down for this one request: _Not yet executed against a deployment; output omitted._ ```console $ tokentriage cloud insights drilldown --grain attempt \ --request lreq_2026_09_20_000001 \ --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --from 2026-09-20T11:00:00Z --to 2026-09-20T13:00:00Z ``` `--request` makes the server scope the page to that request's attempts; it is not a filter applied to a page that was already read. In the SDKs the same question is an `insights.query` drill-down with `attempt_filter.logical_request_id`: _Not yet executed against a deployment; output omitted._ ```python answer = client.insights.query(org, workspace, { "schema": 1, "shape": "drilldown", "drilldown": "attempt", "from": "2026-09-20T11:00:00Z", "to": "2026-09-20T13:00:00Z", "purposes": ["application"], "attempt_filter": {"logical_request_id": "lreq_2026_09_20_000001"}, "page_size": 100, })["data"] for row in answer.get("attempts", []): tokens = row["total_tokens"] if row["usage_known"] else "unknown" print(row["ordinal"], row["role"], row["provider_id"], row["provider_model"], row["output_state"], tokens) print("truncated:", answer.get("truncated", False)) ``` Three things decide whether that list is complete: - **The window.** The drill-down answers a half-open UTC window, `--last 24h` by default. Choose `--from` and `--to` around the receipt's `admitted_at`, or an older request's attempts are outside the question. - **`truncated`.** A page cut at its row bound says so. Do not read a truncated page as the whole list. - **`usage_known`.** When it is `false` the provider did not report usage for that dispatch, and the token counts on the row are zeros that mean nothing. Attempt rows carry no money and no content. The spend for the request is the receipt's `cost`, which sums every attempt. ## 5. Read the cost state The receipt's `cost.state` is one of four words, and only two carry a figure. | `cost.state` | carries | read it as | |---|---|---| | `priced` | `nano_usd_decimal`, `price_snapshot` | the settled cost of every attempt, in integer nano-USD (divide by 10⁹ for dollars) | | `adjusted` | `nano_usd_decimal` and `adjusted_nano_usd_decimal` | it was priced, then a refund, credit or re-settlement landed; `nano_usd_decimal` is what the price list produced, `adjusted_nano_usd_decimal` what it became | | `unpriced` | nothing | the request ran and no price has settled yet; it is not $0.00 | | `unavailable` | `reason` | this answer does not contain the money; `scope_not_authorized` means you lack `cost.read` | Read each figure from its `_decimal` member: a decimal string of the exact integer, `null` when the state carries no figure. The numeric `nano_usd` and `adjusted_nano_usd` members carry the same integers as JSON numbers and are deprecated: JavaScript's `JSON.parse` rounds a number above 2⁵³ without an error. A request refused before any provider was called shows how the CLI reports a missing figure. Replaying the golden's `projected_pre_dispatch_refusal` record: ```console $ tokentriage cloud request get \ --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --request lreq_2026_09_20_000001 \ --base-url http://127.0.0.1:48242 request lreq_2026_09_20_000001 status refused (connection_unresolved, configuration) served by gateway gw_ABCDEFGHJKLM, environment env_ABCDEFGHJKLM, no release (refused before one was resolved) purpose application (chat) acted as unavailable - usage none reported (unavailable) cost unpriced (no price has been settled for this attempt yet) guards unavailable (guardrail outcomes are held outside this backend's fence, so it cannot say what the guards did) Not stated: dispatched_at (not_observed) Not stated: first_output_at (not_projected) Not stated: usage (not_observed) Not stated: usage_source (not_observed) Not stated: run (not_projected) ``` and on stderr, after the operation line: ```text note: this request has no cost figure (unpriced (no price has been settled for this attempt yet)). It is NOT $0.00. ``` No attempt line, no release, and no usage: nothing was dispatched. It still counts as one request. An adjusted cost prints both figures on one line. Replaying the golden's `projected_draft_adjusted` record, a playground request whose settled price was later corrected: ```console $ tokentriage cloud request get \ --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --request lreq_2026_09_20_000001 \ --base-url http://127.0.0.1:48243 request lreq_2026_09_20_000001 status succeeded served by gateway gw_ABCDEFGHJKLM, environment env_ABCDEFGHJKLM, an unnamed subject purpose playground (chat, streaming) acted as unavailable - attempt att_2026_09_20_000001 primary via openai/gpt-4o-mini usage 812 in, 240 out, 128 cached tokens (provider_reported) cost $0.001845 (1845000 nano_usd) settled at price-2026-09, adjusted to $0.0012 (1200000 nano_usd) guards unavailable (guardrail outcomes are held outside this backend's fence, so it cannot say what the guards did) Not stated: release.identity (not_projected) Not stated: release.subject_kind (not_projected) Not stated: first_output_at (not_projected) Not stated: run (not_projected) Not stated: attempt.connection_version (not_observed) ``` `$0.001845` is what the price list produced and `$0.0012` is what it became after the correction. No stderr note is printed, because both figures are present. `an unnamed subject` and `release.identity (not_projected)` are the draft case from step 3: the hosted store carries no draft revision to name. [How to read a cost figure](/docs/build/cost-figures) covers where a price comes from and what an unpriced request does to a total. ## 6. Read what the answer cannot state A receipt from the hosted service can include a `coverage` block listing members it could not fill, each with a reason. The CLI prints one `Not stated:` line per member, as in step 2; `--json` and the SDKs return the block itself. The replayed record lists four: ```json "coverage": { "unstated": [ { "member": "first_output_at", "reason": "not_projected" }, { "member": "release.subject_kind", "reason": "not_projected" }, { "member": "release.draft_revision", "reason": "not_projected" }, { "member": "run", "reason": "not_projected" } ] } ``` That block is the replay's `--json` output, reformatted onto fewer lines. | reason | means | |---|---| | `not_projected` | the hosted store has no column for this member, for any request; it is a fact about the service | | `not_observed` | the column exists and holds nothing for this request, such as `dispatched_at` on a request refused before dispatch | | `not_authorized` | the member was withheld for want of a grant | This is the difference between "this did not happen" and "we cannot say". A `run` block that is absent with no coverage entry means the request was not a test; `run` listed as `not_projected` means this service cannot tell you either way. `actor` and `protection` carry their own `unavailable` state with a reason for the same purpose: the hosted reader does not know which credential admitted the request, or what the guards found, and says so instead of showing an anonymous request or a clean result. [Read request coverage and freshness](/docs/operate/request-coverage) covers the other half: coverage and freshness for the windowed reads, and what to do when the ID is not found yet. ## What can go wrong | What you see | What it means | What to do | |---|---|---| | HTTP 503, `type=dependency_unavailable`, a `Retry-After` header, and a detail saying the request "has not been projected into the analytics store yet" | the hosted index has no row for that ID in the range it searched. The detail names the arrival position and the date range searched | wait for the `Retry-After` interval (30 seconds on this build) and read again; see [Read request coverage and freshness](/docs/operate/request-coverage) | | the same 503 for an ID that ran long ago | the by-ID read searches the last 90 days, and the detail says which range | an older request cannot be read by ID here | | HTTP 404, `type=not_found` | the workspace is not one your key can see, or the record belongs to a different workspace than the path | check `--org` and `--workspace` against the ticket | | HTTP 403, `type=forbidden` | your key lacks `metadata.read` at that workspace | ask an administrator for the grant | | HTTP 400, `type=invalid_query` | the value is not a request ID (IDs are letters, digits, `_` and `-`, at most 128) | copy the header value again, without quotes or whitespace | An ID that belongs to another workspace is not answered `forbidden`, because that would confirm the ID exists somewhere. On the hosted service it gets the same retryable answer as an ID that has not been indexed yet, so a 503 that persists after several retries more often means the wrong workspace than a slow index.
The status codes, `type` values, `Retry-After` interval and detail wording in that table come from `internal/control/foundation/httpapi/requests.go` and `internal/control/cmd/controlplaned/foundation_insights.go` at this page's date. No failure response was captured for this page.
## Local ledger profile (legacy) This section is for the local, self-run gateway, which writes each request to a JSON Lines decision log rather than to the hosted service. Its request IDs look like `tt_…` rather than `lreq_…`, and none of the hosted reads above apply to it. ### What you need | You need | Because | |---|---| | the `x-tt-request-id` value | it is the join key; every other lookup starts from it | | read access to the decision log | that is where the row lives, by default `~/.tokentriage/decisions.jsonl` | | the `tokentriage` binary, or `curl` and the admin plane | either can read the ledger | You do not need the dashboard, an index rebuild, or a database. The ledger is a text file and the first step is a search. ### 1. Get the id off the response If you are reproducing the call rather than chasing somebody else's, `curl -si` shows it. This one is a sub-agent step inside a larger run, which is why it carries identity headers: ```console $ curl -si http://127.0.0.1:8787/v1/chat/completions \ -H 'content-type: application/json' \ -H 'x-tt-trace-id: run-7f3a91c2' -H 'x-tt-step-id: step-05' \ -H 'x-tt-parent-id: step-01' -H 'x-tt-agent: reviewer' \ -H 'x-tt-step-class: tool_loop' \ -H 'x-tt-tenant: acme-corp' -H 'x-tt-feature: nightly-summary' \ -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Review the draft."}]}' \ | head -5 HTTP/1.1 200 OK Content-Length: 415 Content-Type: application/json Date: Mon, 10 Aug 2026 05:52:21 GMT X-Tt-Request-Id: tt_oroicgol5mbg3lfxhka3wwccsq ``` The value is minted by TokenTriage on the way in. A client that sends its own inbound `x-tt-request-id` has it discarded, so the id on the response is always the proxy's own and always matches the row. ### 2. Find the row The decision log is JSON Lines: one row per request, one line each, appended in order. A search for the id finds it. ```console $ grep tt_oroicgol5mbg3lfxhka3wwccsq decisions.jsonl | head -c 200 {"v":1,"ts":"2026-08-10T05:52:21.257Z","req_id":"tt_oroicgol5mbg3lfxhka3wwccsq","trace":{"trace_id":"run-7f3a91c2","step_id":"step-05","parent_id":"step-01","agent":"reviewer","step_class":"tool_loop" ``` One line, roughly a hundred fields. The four blocks that answer the usual questions are `trace`, `decision`, `usage` and `cost`: ```json { "req_id": "tt_oroicgol5mbg3lfxhka3wwccsq", "ts": "2026-08-10T05:52:21.257Z", "trace": { "trace_id": "run-7f3a91c2", "step_id": "step-05", "parent_id": "step-01", "agent": "reviewer", "step_class": "tool_loop", "step_class_source": "header", "source": "x-tt" }, "decision": { "mode": "ledger", "tier": "", "rule": "default", "upstream": "mock" }, "usage": { "input_tokens": 768, "output_tokens": 512, "cache_read_tokens": 256, "source": "provider" }, "cost": { "usd": 0.0004416, "pricing_key": "openai/gpt-4o-mini", "pricing_snapshot": "litellm@2026-07-14#179affb", "estimated": false }, "dims": { "tenant": "acme-corp", "feature": "nightly-summary" } } ``` That block is the same line as the one preceding it, trimmed to those four blocks and formatted. Every capture in this section comes from a local run against a mock upstream, so the token counts are the mock's fixed numbers and `$0.0004416` is arithmetic over them rather than anyone's bill. The `mock` upstream name is this lab's; yours will be whatever you called your provider. Read the four blocks in this order. **`cost`** answers what it cost, and `estimated: false` plus a `pricing_key` says the figure came from a priced lookup rather than a guess. A row with no `usd` at all and `unpriced: true` is a model the price table does not know; [what a cost figure means](/docs/build/cost-figures) covers that case. **`usage.source`** says where the token counts came from. `provider` means the upstream reported them. Any other value means they were derived, and the cost inherits that. **`decision`** answers who chose the model: `rule` names the policy rule that fired, `tier` the tier it selected, `upstream` the provider that served it. In `ledger` mode, as here, `tier` is empty because nothing was routed. **`trace`** answers which step of which run this was, and `source` says how it knows. `x-tt` means the client set identity headers. `none` means it did not, and this request is a standalone row. ### 3. Walk the trace `trace.trace_id` opens the rest of the run: ```console $ tokentriage top --trace run-7f3a91c2 trace run-7f3a91c2 (5 steps) STEP CLASS AGENT MODEL COST IN OUT CACHE-R step-01 user_turn planner gpt-4o-mini $0.000442 768 512 256 step-02 tool_loop researcher gpt-4o-mini $0.000442 768 512 256 step-03 tool_loop writer gpt-4o-mini $0.000442 768 512 256 step-05 tool_loop reviewer gpt-4o-mini $0.000442 768 512 256 step-04 user_turn planner gpt-4o-mini $0.000442 768 512 256 TOTAL $0.002208 3840 2560 1280 ``` Indentation is the parent edge. `step-02`, `step-03` and your `step-05` sit under `step-01` because their requests carried `x-tt-parent-id: step-01`. `step-04` is flush left because it carried no parent, not because it came first. Your request is one row of that table, and now you can see what it cost relative to its siblings. ### 4. Ask why the model was chosen `decision.rule` names the rule. To see the walk that reached it, the same row feeds the policy explainer, which prints the conditions evaluated and the branch taken. That is [reading a routing decision](/docs/build/routing-explain)'s subject rather than this page's, and the `decision.explain.path` array on the row is the short version: it lists the rule names visited, in order. ### 5. Aggregate around it If the single row looks fine and you are chasing a pattern, the admin plane answers grouped questions over the same rows. Here is the one request, grouped by its trace: ```console $ curl -s -X POST http://127.0.0.1:9090/tt/api/v1/query \ -H 'content-type: application/json' \ -d '{"filter":{"f":"req_id","op":"eq","v":"tt_oroicgol5mbg3lfxhka3wwccsq"}, "group_by":["trace.trace_id"],"metrics":["cost_usd","requests"]}' ``` ```json { "groups": [ { "key": { "trace.trace_id": "run-7f3a91c2" }, "metrics": { "cost_usd": { "usd": 0.0004416, "estimated": false, "unpriced": false }, "requests": 1, "unpriced_rows": 0 } } ] } ``` That response is trimmed to the groups; the full body also carries a `provenance` block naming the pricing snapshots involved, the row count, and how many rows were excluded as unpriced. Widen the filter from `req_id` to `trace.trace_id`, `dim.tenant` or `request.model` and the same call answers the pattern question. ### What can go wrong in the local ledger **The id is not in the log.** Three causes, in the order they usually apply. The request went to a different replica, and each writes its own ledger file. The log has rotated, and the row is in a compressed sibling file next to the active one. Or the row was dropped on a full writer queue, which is counted rather than silent; check `tokentriage_declog_dropped_total` and read [what a drop means](/docs/build/observability-drops). **`trace.source` is `none` and you expected a tree.** The client sent no identity headers. Nothing can recover that after the fact, because nothing was inferred at the time. [Put identity on the wire](/docs/build/observability-headers) is the fix, and it applies to rows written after you deploy it. **`trace.source` is `traceparent`.** The caller sent W3C trace context and no `x-tt-*` headers. That value is recorded for correlation and is not used to build the tree, so the row is still standalone.
The local admin plane's query API is aggregation-shaped: it takes a filter, a grouping and a metric list, and returns groups. There is no `GET /requests/{id}` that returns a whole row, so a filtered query with `group_by` is how you reach one request through HTTP. For the raw row, the file is the interface. It is append-only JSON Lines with no index required, which is why a `grep` works and why the same row is readable from a backup, an archive, or a machine that cannot reach your admin plane.