Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Investigate one application task

Read every gateway request one task made as a graph, see which links it cannot draw, and add up what the task cost.

How-to guideLast updated

On this page

A customer reports that one task in your application went wrong: an agent run that classified, summarized and formatted a document, and billed more than it should have. Each of those steps was a separate gateway request. You want all of them at once, in the shape the task gave them, with what each cost.

The trace read answers that. You name the trace ID your application sent, and it returns one page of a graph: a node per gateway request, an edge where a request named its parent step, a reason for every parent it cannot show, and a cost total for the page. It is a read. Nothing on this page sends a model request or spends money.

If you have not put trace, step and parent IDs on your requests yet, start with Put identity on the wire. How steps become a tree explains the parent edge itself.

What you need

You need Because
the trace ID your application sent in x-tt-trace-id it names the task; nothing else groups the requests
the request’s namespace the same trace ID in two namespaces is two traces; copy it from any one request’s receipt
one application ID and one environment ID a graph covers one application in one environment
your organization and workspace IDs the read runs inside one workspace
a management key in TOKENTRIAGE_MANAGEMENT_KEY the runtime key cannot read management records
metadata.read on the workspace it gates the whole read
cost.read on the workspace without it every cost reads unavailable with reason scope_not_authorized
a gateway with trace.links.enabled: true without link rows the read finds requests but draws no edges (step 5)

trace.links.enabled is gateway configuration, off by default, and takes effect only after a restart. With it on, the gateway writes one link row per request that carried a trace ID and has a receipt, holding the trace, step and parent IDs as the request sent them (each cut to 128 bytes). The organization, workspace, application, environment and namespace on that row come from the receipt, not from your headers.

trace:
  links:
    enabled: true   # default false; restart the gateway to apply

1. Name the task

The read body names the trace, where it ran, and a time window. Save it as trace-read.json:

{
  "schema": 1,
  "trace_id": "trace-7f3a",
  "namespace": "ns_1",
  "scope": {
    "organization_id": "org_1",
    "workspace_id": "ws_1",
    "application_ids": ["app_1"],
    "environment_ids": ["env_1"]
  },
  "from": "2026-09-01T00:00:00Z",
  "to": "2026-09-08T00:00:00Z",
  "purposes": ["application"],
  "page_size": 5
}

That body is the contract’s request fixture (internal/central/insightcontract/testdata/trace-read-request.golden.json) with page_size lowered from 100 to 5, so the example in step 3 spans two pages. The IDs are synthetic.

The rules the body has to meet:

  • One application and one environment. application_ids and environment_ids each hold one ID, and gateway_ids is absent. A trace spans every gateway of its application and environment, so a gateway filter would hide nodes and misreport them as missing.
  • A half-open UTC window of at most 31 days. A request counts when its admitted instant is at or after from and before to.
  • One or more distinct purposes, from application, playground, comparison, judge and connection_validation.
  • page_size from 1 to 500, 100 when omitted.
  • Labels of at most 128 bytes with no control characters, for trace_id, namespace and every request and step ID.

2. Read the first page

The operation is POST {lane}/organizations/{org}/workspaces/{ws}/traces/read, where {lane} is your deployment’s origin followed by /portal/api/v1/customer/v1. Pick a channel.

Not yet executed against a deployment.

$ LANE="$TOKENTRIAGE_BASE_URL/portal/api/v1/customer/v1"
$ curl -s -X POST \
    "$LANE/organizations/org_1/workspaces/ws_1/traces/read" \
    -H "Authorization: Bearer $TOKENTRIAGE_MANAGEMENT_KEY" \
    -H 'content-type: application/json' \
    -d @trace-read.json

A success is HTTP 200 with an envelope whose kind is trace-answer and whose data is the page.

From Python, the client takes the same body as a dict and returns the page from inside the envelope:

Not yet executed against a deployment.

import json, os
from tokentriage.customer import CustomerClient, UnsupportedError

client = CustomerClient(os.environ["TOKENTRIAGE_BASE_URL"],
                        os.environ["TOKENTRIAGE_MANAGEMENT_KEY"])
query = json.load(open("trace-read.json"))
try:
    page = client.traces.read("org_1", "ws_1", query)
except UnsupportedError as error:  # OperationNotServedError until registered
    raise SystemExit(str(error))
print(page["truncated"], len(page["nodes"]), page["cost_rollup"])

sdk/python/examples/observability_task_graph.py in the repository reads every page of a trace and prints it as a tree.

From TypeScript:

Not yet executed against a deployment.

import { readFileSync } from "node:fs";
import { CustomerClient } from "tokentriage/customer";

const client = new CustomerClient({
  baseUrl: process.env.TOKENTRIAGE_BASE_URL!,
  managementApiKey: process.env.TOKENTRIAGE_MANAGEMENT_KEY!,
});
const query = JSON.parse(readFileSync("trace-read.json", "utf8"));
const page = await client.traces.read("org_1", "ws_1", query);
console.log(page.truncated, page.nodes.length, page.cost_rollup);

From the CLI, the flags build the same body. --from and --to go together; without both, the CLI reads the 7 days before now, and --purpose defaults to application:

Not yet executed against a deployment.

$ tokentriage cloud trace read \
    --org org_1 --workspace ws_1 --app app_1 --env env_1 \
    --trace trace-7f3a --namespace ns_1 \
    --from 2026-09-01T00:00:00Z --to 2026-09-08T00:00:00Z \
    --page-size 5 --base-url https://app.example.com

A successful read renders the graph the way step 3 shows: the table, the tree, any missing links and the cost rollup. --json prints the server’s document unchanged.

3. Read the graph

Every page lists its nodes in (occurred_at, request_id) order. A node is one gateway request: its request_id, step_id and parent_step_id as the request sent them, its status, its attempt_count and its cost. An edge joins two nodes of the same page where the child’s parent_step_id is the parent’s step_id.

This is the first page of the example trace, rendered by the CLI:

Not yet executed against a deployment.

trace trace-7f3a in namespace ns_1: 5 nodes, 2 edges on this page
REQUEST  STEP       PARENT  STATUS         ATTEMPTS  COST
tt_a1    root       -       succeeded      1         $0.0012
tt_a2    classify   tt_a1   failed         3         $0.0008 (adjusted)
tt_a3    summarize  tt_a1   partial        1         not stated (unpriced)
tt_a5    (no step)  -       not_projected  0         not stated (not_observed)
tt_a4    (no step)  -       succeeded      1         $0.0003

tree by parent:
  tt_a1 root succeeded
    tt_a2 classify failed
    tt_a3 summarize partial
  tt_a5 (no step) not_projected
  tt_a4 (no step) succeeded

missing links:
  tt_a5: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed)
  tt_a4: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed)
Not shown: some requests carried no step (step_not_observed)
Not shown: some of this trace's links are not projected yet (trace_links_partial)
Not shown: this page is part of a larger graph (truncated)
cost rollup (this page): $0.0023 (2300000 nano_usd) across 3 known nodes; 2 unknown nodes; incomplete

That page and every rendering after it on this page were generated, not typed: a scratch test fed the contract’s own five-row trace fixture (plus a later sixth row still to come) to the service’s graph assembler with a page size of 5, and the CLI’s own renderer printed the answer. No deployment and no ClickHouse were involved, and the dollar figures are fixture values, not anyone’s bill.

Read it this way:

  • tt_a2 failed after 3 attempts. attempt_count counts the provider attempts of that one request. Retries are not separate nodes, and the node’s cost already covers every attempt.
  • status is one of five words. succeeded, partial, refused and failed are the request receipt’s own statuses. not_projected means a link row names the request but its facts have not reached the store yet, so it has attempt_count 0 and a cost of unavailable with reason not_observed, as tt_a5 shows.
  • The tree is by parent, not by clock. Two siblings that ran at once and two that ran in turn look the same.
  • tt_a5 and tt_a4 carry no step at all. Neither named a step_id, so the table prints (no step), neither can be anyone’s parent, and both count toward the answer’s step_not_observed coverage (step 5).

A node whose parent the page cannot show gets an entry in missing_links with one of four reasons, and no edge. The example page from step 3 has two:

missing links:
  tt_a5: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed)
  tt_a4: its parent is not in this graph: not observed, not projected yet, or not visible to you (parent_not_observed)

Neither tt_a5 nor tt_a4 named a step_id or a parent_step_id, so both fall under the one reason that also covers a node with no step and no parent at all.

The four reasons:

reason the node’s parent what to do
parent_not_observed is not in this graph: never sent, not projected yet, removed by an exclusion, or hidden from you; also any node with no step and no parent these causes share one reason, so an answer does not confirm that a request you cannot see exists; widen the window or read again later
parent_outside_scope carries a step seen in another application or environment of your workspace that you could read. This is a contract value: customer.traces.get resolves exactly one application and one environment, so through it such a parent appears as parent_not_observed and parent_outside_scope is never returned read that application and environment as its own graph; the read does not cross the boundary
duplicate_step is ambiguous, because two or more requests in the trace claim the same step; the claimants get it too give each call its own x-tt-step-id
cycle forms a loop with other nodes’ parents; every member of the loop gets it fix the parent IDs your application sends

Duplicates and cycles come from the IDs your application sends. In this second scratch trace, tt_b1 and tt_b2 both claimed step fetch, tt_b3 named fetch as its parent, and tt_b4 and tt_b5 named each other:

Not yet executed against a deployment.

missing links:
  tt_b1: another request in this trace carries the same step, so its parent link is ambiguous and was not drawn (duplicate_step)
  tt_b2: another request in this trace carries the same step, so its parent link is ambiguous and was not drawn (duplicate_step)
  tt_b3: another request in this trace carries the same step, so its parent link is ambiguous and was not drawn (duplicate_step)
  tt_b4: its parent links form a cycle, so no parent was drawn for it (cycle)
  tt_b5: its parent links form a cycle, so no parent was drawn for it (cycle)

The same request delivered twice to the store is not a duplicate step. It collapses to one node before the graph is built.

5. Read what the page cannot show

The answer’s coverage list, and a node’s own, use four reasons. The set is closed: the contract rejects any other value.

reason appears on means
step_not_observed the node the request carried a trace ID and no step ID, so it cannot be anyone’s parent
trace_links_partial the answer some requests of this trace in the window reached the store without their link rows; those requests are absent from the graph until the rows arrive
trace_links_unavailable the node and the answer no link rows exist for this trace, so the nodes were found through the receipt’s root_activity_id and no edge is drawn
truncated the answer more nodes follow this page; next_cursor continues it

trace_links_unavailable is what a gateway without trace.links.enabled produces. Every node is step-less, there are no edges and no missing links, and the page says why:

Not yet executed against a deployment.

trace trace-7f3a in namespace ns_1: 2 nodes, 0 edges on this page
REQUEST  STEP       PARENT  STATUS     ATTEMPTS  COST
tt_c1    (no step)  -       succeeded  1         $0.0000001
tt_c2    (no step)  -       succeeded  1         $0.0000001

tree by parent:
  tt_c1 (no step) succeeded
  tt_c2 (no step) succeeded
Not shown: no links exist for this trace, so no edge is drawn (trace_links_unavailable)
cost rollup (this page): $0.0000002 (200 nano_usd) across 2 known nodes; 0 unknown nodes; complete

6. Add up what the task cost

Each node carries the same cost cell the request receipt publishes: priced with nano_usd, adjusted with adjusted_nano_usd, unpriced, or unavailable with a reason. How to read a cost figure covers the four states. The page’s cost_rollup adds them up node by node:

field carries
known_nano_usd the sum, as a decimal string of integer nano-USD, of each priced node’s nano_usd and each adjusted node’s adjusted_nano_usd
known_nodes how many nodes that sum counts, each once
unknown_nodes how many nodes on the page have no figure; they are not $0.00
complete true only when no node is unknown and the page is not truncated

The page’s rollup was $0.0023 across 3 known nodes: tt_a1’s 1200000 plus tt_a2’s adjusted 800000, which replaces its original 900000, plus tt_a4’s 300000. tt_a3 was unpriced and tt_a5 was unavailable, so both count as unknown.

The rollup covers this page only. It is not a running total. To total a task that spans pages, add each page’s known_nano_usd and unknown_nodes yourself. This page is truncated (step 7), so a sixth request is still to come; until you read it, 2300000 nano-USD over 3 known nodes is a floor, not the task’s total.

Without cost.read, every node’s cost reads unavailable with reason scope_not_authorized and the rollup states nothing: "0", 0 known nodes, every node unknown, and complete: false.

7. Continue a truncated graph

When truncated is true, the answer carries next_cursor. Send the same body again with two members added, both copied unchanged from the page you read last:

  • cursor: that page’s next_cursor
  • read_context: that page’s read_context

The cursor is an opaque string of at most 512 bytes. It holds the trace ID and namespace in sealed form, a digest of the read context, and the (occurred_at, request_id) position the page ended at. It carries no permission: every page re-checks your access. The rules that follow from that:

  • A cursor needs its read context. Sent alone, it is rejected with 422 validation_failed. The SDKs and the CLI stop it before sending.
  • A cursor belongs to one trace, namespace and read context. Presented with any other, it is rejected with 400 invalid_query.
  • An expired read context ends the walk. When access, exclusions or projections change after the first page, the next page answers 410 read_context_expired. Start again without cursor and read_context.
  • Pages meet without a gap or an overlap, because the position is the complete (occurred_at, request_id) pair.

The same graph-level answers hold on every page. The service resolves duplicates, missing parents and cycles over the whole trace in the window before it cuts a page, so two pages do not disagree about a node. That makes every page re-read the whole trace, and it has a bound: a trace with more than 5,000 requests in the window is rejected with 400 invalid_query rather than answered with gaps. Narrow the window and read again.

In the SDKs, pass the continuation as arguments. From Python:

Not yet executed against a deployment.

pages = [client.traces.read("org_1", "ws_1", query)]
while pages[-1]["truncated"]:
    last = pages[-1]
    pages.append(client.traces.read(
        "org_1", "ws_1", query,
        cursor=last["next_cursor"], read_context=last["read_context"]))

From TypeScript, the same loop uses options:

Not yet executed against a deployment.

const pages = [await client.traces.read("org_1", "ws_1", query)];
while (pages[pages.length - 1].truncated) {
  const last = pages[pages.length - 1];
  pages.push(await client.traces.read("org_1", "ws_1", query, {
    cursor: last.next_cursor,
    readContext: last.read_context,
  }));
}

Over HTTP, add "cursor" and "read_context" to trace-read.json and post it again. The CLI prints the continuation on stderr after a truncated page, as more results: pass --cursor … with --read-context '…'; rerun the step 2 command with those two flags added:

Not yet executed against a deployment.

$ tokentriage cloud trace read \
    --org org_1 --workspace ws_1 --app app_1 --env env_1 \
    --trace trace-7f3a --namespace ns_1 \
    --from 2026-09-01T00:00:00Z --to 2026-09-08T00:00:00Z \
    --page-size 5 --base-url https://app.example.com \
    --cursor "$NEXT_CURSOR" --read-context "$READ_CONTEXT"

What this read does not tell you

  • How long the task took. A node’s occurred_at is when the gateway admitted that request. No gateway observes where a task starts or ends, so the answer has no task duration, no critical path and no latency. Node order is admission order, and the gap between two admissions says nothing about how long either request ran.
  • What was said. No prompt or completion text is in the answer, and no content.read grant changes that.
  • Anything across the boundary. Requests of the same trace in another application or environment are a separate graph. customer.traces.get resolves exactly one application and one environment, so a parent there shows up as parent_not_observed; the contract value parent_outside_scope is never returned through it.

What can go wrong

What you see What it means What to do
HTTP 404, type=not_found this trace is not visible in this workspace to your key: the application, environment or trace may not exist, or you may simply lack access to it; the answer is the same either way, so it confirms nothing check the --app, --env, --trace and --namespace values against the receipt of one request in the task
HTTP 422, type=validation_failed the body broke a rule from step 1, named another organization or workspace than the path, or sent a cursor without its read context fix the body; the detail lists the rules
HTTP 403, type=forbidden your key lacks the read permission (metadata.read) at this workspace ask a workspace administrator for a binding that grants it
HTTP 400, type=invalid_query a cursor from another read, or more than 5,000 requests in the window start without a cursor, or narrow the window
HTTP 410, type=read_context_expired access, exclusions or projections changed since the first page read again from the first page
HTTP 501, type=not_implemented the deployment’s analytics reader has no trace read the operation exists and this deployment cannot serve it
HTTP 503, type=dependency_unavailable no analytics reader, the analytics service is unreachable (with Retry-After: 30), or an answer the lane would not publish retry after the interval the header names
every node reads trace_links_unavailable the gateway that served the task has trace.links.enabled off turn it on and restart; it applies to requests after the restart
an empty graph for a trace you know ran the window, the namespace or the purposes do not match the task copy namespace and the admitted instant from one request’s receipt
Where these answers were read from

The rules, bounds and status codes on this page were read from source on the wip/obs-o02-t15 branch at 7b35ffcfd, not captured from a deployment: internal/central/insightcontract/trace_read.go (the contract and its closed sets), internal/central/analytics/chquery/trace_graph.go and trace_graph_assemble.go (the 5,000-request bound, resolution, paging and the cursor), internal/control/foundation/httpapi/traces.go and docs/manual-testing/request-observability/O02/proposed-registry-delta.md (the path and the refusal table), internal/cli/cloud_traces.go, the Python and TypeScript clients under sdk/, and internal/config/config.go (trace.links.enabled).