# Investigate one application task Read every gateway request one task made as a graph, see which links it cannot draw, and add up what the task cost. Source: https://tokentriage.com/docs/build/investigate-application-task --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; `customer.traces.get` (`POST /organizations/{org}/workspaces/{ws}/traces/read`, scope `metadata.read` at the workspace) is registered. The examples on this page have not yet been executed against a deployment. In the meantime, read each request on its own with [Trace one request from its ID](/docs/build/observability-request-id). A customer reports that one task in your application went wrong: an agent run that classified, summarized and formatted a document, and billed more than it should have. Each of those steps was a separate gateway request. You want all of them at once, in the shape the task gave them, with what each cost. The trace read answers that. You name the trace ID your application sent, and it returns one page of a graph: a node per gateway request, an edge where a request named its parent step, a reason for every parent it cannot show, and a cost total for the page. It is a read. Nothing on this page sends a model request or spends money. If you have not put trace, step and parent IDs on your requests yet, start with [Put identity on the wire](/docs/build/observability-headers). [How steps become a tree](/docs/build/observability-trace-trees) explains the parent edge itself. ## What you need | You need | Because | |---|---| | the trace ID your application sent in `x-tt-trace-id` | it names the task; nothing else groups the requests | | the request's `namespace` | the same trace ID in two namespaces is two traces; copy it from any one request's receipt | | one application ID and one environment ID | a graph covers one application in one environment | | your organization and workspace IDs | the read runs inside one workspace | | a management key in `TOKENTRIAGE_MANAGEMENT_KEY` | the runtime key cannot read management records | | `metadata.read` on the workspace | it gates the whole read | | `cost.read` on the workspace | without it every cost reads `unavailable` with reason `scope_not_authorized` | | a gateway with `trace.links.enabled: true` | without link rows the read finds requests but draws no edges ([step 5](#5-read-what-the-page-cannot-show)) | `trace.links.enabled` is gateway configuration, off by default, and takes effect only after a restart. With it on, the gateway writes one link row per request that carried a trace ID and has a receipt, holding the trace, step and parent IDs as the request sent them (each cut to 128 bytes). The organization, workspace, application, environment and namespace on that row come from the receipt, not from your headers. ```yaml trace: links: enabled: true # default false; restart the gateway to apply ``` ## 1. Name the task The read body names the trace, where it ran, and a time window. Save it as `trace-read.json`: ```json { "schema": 1, "trace_id": "trace-7f3a", "namespace": "ns_1", "scope": { "organization_id": "org_1", "workspace_id": "ws_1", "application_ids": ["app_1"], "environment_ids": ["env_1"] }, "from": "2026-09-01T00:00:00Z", "to": "2026-09-08T00:00:00Z", "purposes": ["application"], "page_size": 5 } ``` That body is the contract's request fixture (`internal/central/insightcontract/testdata/trace-read-request.golden.json`) with `page_size` lowered from 100 to 5, so the example in step 3 spans two pages. The IDs are synthetic. The rules the body has to meet: - **One application and one environment.** `application_ids` and `environment_ids` each hold one ID, and `gateway_ids` is absent. A trace spans every gateway of its application and environment, so a gateway filter would hide nodes and misreport them as missing. - **A half-open UTC window of at most 31 days.** A request counts when its admitted instant is at or after `from` and before `to`. - **One or more distinct purposes**, from `application`, `playground`, `comparison`, `judge` and `connection_validation`. - **`page_size` from 1 to 500**, 100 when omitted. - **Labels of at most 128 bytes** with no control characters, for `trace_id`, `namespace` and every request and step ID. ## 2. Read the first page The operation is `POST {lane}/organizations/{org}/workspaces/{ws}/traces/read`, where `{lane}` is your deployment's origin followed by `/portal/api/v1/customer/v1`. Pick a channel. _Not yet executed against a deployment._ ```console $ LANE="$TOKENTRIAGE_BASE_URL/portal/api/v1/customer/v1" $ curl -s -X POST \ "$LANE/organizations/org_1/workspaces/ws_1/traces/read" \ -H "Authorization: Bearer $TOKENTRIAGE_MANAGEMENT_KEY" \ -H 'content-type: application/json' \ -d @trace-read.json ``` A success is HTTP 200 with an envelope whose `kind` is `trace-answer` and whose `data` is the page. From Python, the client takes the same body as a `dict` and returns the page from inside the envelope: _Not yet executed against a deployment._ ```python import json, os from tokentriage.customer import CustomerClient, UnsupportedError client = CustomerClient(os.environ["TOKENTRIAGE_BASE_URL"], os.environ["TOKENTRIAGE_MANAGEMENT_KEY"]) query = json.load(open("trace-read.json")) try: page = client.traces.read("org_1", "ws_1", query) except UnsupportedError as error: # OperationNotServedError until registered raise SystemExit(str(error)) print(page["truncated"], len(page["nodes"]), page["cost_rollup"]) ``` `sdk/python/examples/observability_task_graph.py` in the repository reads every page of a trace and prints it as a tree. From TypeScript: _Not yet executed against a deployment._ ```typescript import { readFileSync } from "node:fs"; import { CustomerClient } from "tokentriage/customer"; const client = new CustomerClient({ baseUrl: process.env.TOKENTRIAGE_BASE_URL!, managementApiKey: process.env.TOKENTRIAGE_MANAGEMENT_KEY!, }); const query = JSON.parse(readFileSync("trace-read.json", "utf8")); const page = await client.traces.read("org_1", "ws_1", query); console.log(page.truncated, page.nodes.length, page.cost_rollup); ``` From the CLI, the flags build the same body. `--from` and `--to` go together; without both, the CLI reads the 7 days before now, and `--purpose` defaults to `application`: _Not yet executed against a deployment._ ```console $ tokentriage cloud trace read \ --org org_1 --workspace ws_1 --app app_1 --env env_1 \ --trace trace-7f3a --namespace ns_1 \ --from 2026-09-01T00:00:00Z --to 2026-09-08T00:00:00Z \ --page-size 5 --base-url https://app.example.com ``` A successful read renders the graph the way step 3 shows: the table, the tree, any missing links and the cost rollup. `--json` prints the server's document unchanged. ## 3. Read the graph Every page lists its nodes in `(occurred_at, request_id)` order. A node is one gateway request: its `request_id`, `step_id` and `parent_step_id` as the request sent them, its `status`, its `attempt_count` and its `cost`. An edge joins two nodes of the same page where the child's `parent_step_id` is the parent's `step_id`. This is the first page of the example trace, rendered by the CLI: _Not yet executed against a deployment._ ```text trace trace-7f3a in namespace ns_1: 5 nodes, 2 edges on this page REQUEST STEP PARENT STATUS ATTEMPTS COST tt_a1 root - succeeded 1 $0.0012 tt_a2 classify tt_a1 failed 3 $0.0008 (adjusted) tt_a3 summarize tt_a1 partial 1 not stated (unpriced) tt_a5 (no step) - not_projected 0 not stated (not_observed) tt_a4 (no step) - succeeded 1 $0.0003 tree by parent: tt_a1 root succeeded tt_a2 classify failed tt_a3 summarize partial tt_a5 (no step) not_projected tt_a4 (no step) succeeded missing links: tt_a5: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed) tt_a4: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed) Not shown: some requests carried no step (step_not_observed) Not shown: some of this trace's links are not projected yet (trace_links_partial) Not shown: this page is part of a larger graph (truncated) cost rollup (this page): $0.0023 (2300000 nano_usd) across 3 known nodes; 2 unknown nodes; incomplete ``` That page and every rendering after it on this page were generated, not typed: a scratch test fed the contract's own five-row trace fixture (plus a later sixth row still to come) to the service's graph assembler with a page size of 5, and the CLI's own renderer printed the answer. No deployment and no ClickHouse were involved, and the dollar figures are fixture values, not anyone's bill. Read it this way: - **`tt_a2` failed after 3 attempts.** `attempt_count` counts the provider attempts of that one request. Retries are not separate nodes, and the node's cost already covers every attempt. - **`status` is one of five words.** `succeeded`, `partial`, `refused` and `failed` are the request receipt's own statuses. `not_projected` means a link row names the request but its facts have not reached the store yet, so it has `attempt_count` 0 and a cost of `unavailable` with reason `not_observed`, as `tt_a5` shows. - **The tree is by parent, not by clock.** Two siblings that ran at once and two that ran in turn look the same. - **`tt_a5` and `tt_a4` carry no step at all.** Neither named a `step_id`, so the table prints `(no step)`, neither can be anyone's parent, and both count toward the answer's `step_not_observed` coverage (step 5). ## 4. Read the links it cannot draw A node whose parent the page cannot show gets an entry in `missing_links` with one of four reasons, and no edge. The example page from step 3 has two: ```text missing links: tt_a5: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed) tt_a4: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed) ``` Neither `tt_a5` nor `tt_a4` named a `step_id` or a `parent_step_id`, so both fall under the one reason that also covers a node with no step and no parent at all. The four reasons: | reason | the node's parent | what to do | |---|---|---| | `parent_not_observed` | is not in this graph: never sent, not projected yet, removed by an exclusion, or hidden from you; also any node with no step and no parent | these causes share one reason, so an answer does not confirm that a request you cannot see exists; widen the window or read again later | | `parent_outside_scope` | carries a step seen in another application or environment of your workspace that you could read. This is a contract value: `customer.traces.get` resolves exactly one application and one environment, so through it such a parent appears as `parent_not_observed` and `parent_outside_scope` is never returned | read that application and environment as its own graph; the read does not cross the boundary | | `duplicate_step` | is ambiguous, because two or more requests in the trace claim the same step; the claimants get it too | give each call its own `x-tt-step-id` | | `cycle` | forms a loop with other nodes' parents; every member of the loop gets it | fix the parent IDs your application sends | Duplicates and cycles come from the IDs your application sends. In this second scratch trace, `tt_b1` and `tt_b2` both claimed step `fetch`, `tt_b3` named `fetch` as its parent, and `tt_b4` and `tt_b5` named each other: _Not yet executed against a deployment._ ```text missing links: tt_b1: another request in this trace carries the same step, so its parent link is ambiguous and was not drawn (duplicate_step) tt_b2: another request in this trace carries the same step, so its parent link is ambiguous and was not drawn (duplicate_step) tt_b3: another request in this trace carries the same step, so its parent link is ambiguous and was not drawn (duplicate_step) tt_b4: its parent links form a cycle, so no parent was drawn for it (cycle) tt_b5: its parent links form a cycle, so no parent was drawn for it (cycle) ``` The same request delivered twice to the store is not a duplicate step. It collapses to one node before the graph is built. ## 5. Read what the page cannot show The answer's `coverage` list, and a node's own, use four reasons. The set is closed: the contract rejects any other value. | reason | appears on | means | |---|---|---| | `step_not_observed` | the node | the request carried a trace ID and no step ID, so it cannot be anyone's parent | | `trace_links_partial` | the answer | some requests of this trace in the window reached the store without their link rows; those requests are absent from the graph until the rows arrive | | `trace_links_unavailable` | the node and the answer | no link rows exist for this trace, so the nodes were found through the receipt's `root_activity_id` and no edge is drawn | | `truncated` | the answer | more nodes follow this page; `next_cursor` continues it | `trace_links_unavailable` is what a gateway without `trace.links.enabled` produces. Every node is step-less, there are no edges and no missing links, and the page says why: _Not yet executed against a deployment._ ```text trace trace-7f3a in namespace ns_1: 2 nodes, 0 edges on this page REQUEST STEP PARENT STATUS ATTEMPTS COST tt_c1 (no step) - succeeded 1 $0.0000001 tt_c2 (no step) - succeeded 1 $0.0000001 tree by parent: tt_c1 (no step) succeeded tt_c2 (no step) succeeded Not shown: no links exist for this trace, so no edge is drawn (trace_links_unavailable) cost rollup (this page): $0.0000002 (200 nano_usd) across 2 known nodes; 0 unknown nodes; complete ``` ## 6. Add up what the task cost Each node carries the same cost cell the request receipt publishes: `priced` with `nano_usd`, `adjusted` with `adjusted_nano_usd`, `unpriced`, or `unavailable` with a reason. [How to read a cost figure](/docs/build/cost-figures) covers the four states. The page's `cost_rollup` adds them up node by node: | field | carries | |---|---| | `known_nano_usd` | the sum, as a decimal string of integer nano-USD, of each `priced` node's `nano_usd` and each `adjusted` node's `adjusted_nano_usd` | | `known_nodes` | how many nodes that sum counts, each once | | `unknown_nodes` | how many nodes on the page have no figure; they are not $0.00 | | `complete` | `true` only when no node is unknown and the page is not truncated | The page's rollup was `$0.0023` across 3 known nodes: `tt_a1`'s `1200000` plus `tt_a2`'s adjusted `800000`, which replaces its original `900000`, plus `tt_a4`'s `300000`. `tt_a3` was `unpriced` and `tt_a5` was `unavailable`, so both count as unknown. The rollup covers **this page only**. It is not a running total. To total a task that spans pages, add each page's `known_nano_usd` and `unknown_nodes` yourself. This page is truncated ([step 7](#7-continue-a-truncated-graph)), so a sixth request is still to come; until you read it, `2300000` nano-USD over 3 known nodes is a floor, not the task's total. Without `cost.read`, every node's cost reads `unavailable` with reason `scope_not_authorized` and the rollup states nothing: `"0"`, 0 known nodes, every node unknown, and `complete: false`. ## 7. Continue a truncated graph When `truncated` is `true`, the answer carries `next_cursor`. Send the same body again with two members added, both copied unchanged from the page you read last: - `cursor`: that page's `next_cursor` - `read_context`: that page's `read_context` The cursor is an opaque string of at most 512 bytes. It holds the trace ID and namespace in sealed form, a digest of the read context, and the `(occurred_at, request_id)` position the page ended at. It carries no permission: every page re-checks your access. The rules that follow from that: - **A cursor needs its read context.** Sent alone, it is rejected with 422 `validation_failed`. The SDKs and the CLI stop it before sending. - **A cursor belongs to one trace, namespace and read context.** Presented with any other, it is rejected with 400 `invalid_query`. - **An expired read context ends the walk.** When access, exclusions or projections change after the first page, the next page answers 410 `read_context_expired`. Start again without `cursor` and `read_context`. - **Pages meet without a gap or an overlap**, because the position is the complete `(occurred_at, request_id)` pair. The same graph-level answers hold on every page. The service resolves duplicates, missing parents and cycles over the whole trace in the window before it cuts a page, so two pages do not disagree about a node. That makes every page re-read the whole trace, and it has a bound: a trace with more than 5,000 requests in the window is rejected with 400 `invalid_query` rather than answered with gaps. Narrow the window and read again. In the SDKs, pass the continuation as arguments. From Python: _Not yet executed against a deployment._ ```python pages = [client.traces.read("org_1", "ws_1", query)] while pages[-1]["truncated"]: last = pages[-1] pages.append(client.traces.read( "org_1", "ws_1", query, cursor=last["next_cursor"], read_context=last["read_context"])) ``` From TypeScript, the same loop uses `options`: _Not yet executed against a deployment._ ```typescript const pages = [await client.traces.read("org_1", "ws_1", query)]; while (pages[pages.length - 1].truncated) { const last = pages[pages.length - 1]; pages.push(await client.traces.read("org_1", "ws_1", query, { cursor: last.next_cursor, readContext: last.read_context, })); } ``` Over HTTP, add `"cursor"` and `"read_context"` to `trace-read.json` and post it again. The CLI prints the continuation on stderr after a truncated page, as `more results: pass --cursor … with --read-context '…'`; rerun the step 2 command with those two flags added: _Not yet executed against a deployment._ ```console $ tokentriage cloud trace read \ --org org_1 --workspace ws_1 --app app_1 --env env_1 \ --trace trace-7f3a --namespace ns_1 \ --from 2026-09-01T00:00:00Z --to 2026-09-08T00:00:00Z \ --page-size 5 --base-url https://app.example.com \ --cursor "$NEXT_CURSOR" --read-context "$READ_CONTEXT" ``` ## What this read does not tell you - **How long the task took.** A node's `occurred_at` is when the gateway admitted that request. No gateway observes where a task starts or ends, so the answer has no task duration, no critical path and no latency. Node order is admission order, and the gap between two admissions says nothing about how long either request ran. - **What was said.** No prompt or completion text is in the answer, and no `content.read` grant changes that. - **Anything across the boundary.** Requests of the same trace in another application or environment are a separate graph. `customer.traces.get` resolves exactly one application and one environment, so a parent there shows up as `parent_not_observed`; the contract value `parent_outside_scope` is never returned through it. ## What can go wrong | What you see | What it means | What to do | |---|---|---| | HTTP 404, `type=not_found` | this trace is not visible in this workspace to your key: the application, environment or trace may not exist, or you may simply lack access to it; the answer is the same either way, so it confirms nothing | check the `--app`, `--env`, `--trace` and `--namespace` values against the receipt of one request in the task | | HTTP 422, `type=validation_failed` | the body broke a rule from step 1, named another organization or workspace than the path, or sent a cursor without its read context | fix the body; the detail lists the rules | | HTTP 403, `type=forbidden` | your key lacks the read permission (`metadata.read`) at this workspace | ask a workspace administrator for a binding that grants it | | HTTP 400, `type=invalid_query` | a cursor from another read, or more than 5,000 requests in the window | start without a cursor, or narrow the window | | HTTP 410, `type=read_context_expired` | access, exclusions or projections changed since the first page | read again from the first page | | HTTP 501, `type=not_implemented` | the deployment's analytics reader has no trace read | the operation exists and this deployment cannot serve it | | HTTP 503, `type=dependency_unavailable` | no analytics reader, the analytics service is unreachable (with `Retry-After: 30`), or an answer the lane would not publish | retry after the interval the header names | | every node reads `trace_links_unavailable` | the gateway that served the task has `trace.links.enabled` off | turn it on and restart; it applies to requests after the restart | | an empty graph for a trace you know ran | the window, the namespace or the purposes do not match the task | copy `namespace` and the admitted instant from one request's receipt |
The rules, bounds and status codes on this page were read from source on the `wip/obs-o02-t15` branch at `7b35ffcfd`, not captured from a deployment: `internal/central/insightcontract/trace_read.go` (the contract and its closed sets), `internal/central/analytics/chquery/trace_graph.go` and `trace_graph_assemble.go` (the 5,000-request bound, resolution, paging and the cursor), `internal/control/foundation/httpapi/traces.go` and `docs/manual-testing/request-observability/O02/proposed-registry-delta.md` (the path and the refusal table), `internal/cli/cloud_traces.go`, the Python and TypeScript clients under `sdk/`, and `internal/config/config.go` (`trace.links.enabled`).