How steps become a tree

A trace is not a list of calls. It is a shape, and the shape is where the money is: a planner that costs a tenth of a cent and fans out to three sub-agents that cost forty times that is a different problem from one expensive call.

This page is about how three headers produce that shape, what the shape can and cannot tell you, and the handful of ways it comes out looking wrong.

Three headers, one tree

Read it left to right. Your agent sets the headers per call; equal trace ids collect the rows into one trace; the parent id draws the edge from a step to the step that spawned it; and the same rows feed the per-step tree, the rollups and the span export.

The rules are short enough to state completely.

1. Rows sharing an x-tt-trace-id are one trace. Nothing else joins them.
2. Within a trace, a row is a child of the row whose x-tt-step-id equals its x-tt-parent-id.
3. A row with no parent id renders at the root, whatever its position in time.
4. Nothing validates the edge. A parent id naming a step that does not exist leaves the row at the root rather than failing.

Rendered, that is one command:

$ tokentriage top --trace run-7f3a91c2
trace run-7f3a91c2  (5 steps)
STEP       CLASS      AGENT       MODEL        COST       IN    OUT   CACHE-R
step-01    user_turn  planner     gpt-4o-mini  $0.000442  768   512   256
  step-02  tool_loop  researcher  gpt-4o-mini  $0.000442  768   512   256
  step-03  tool_loop  writer      gpt-4o-mini  $0.000442  768   512   256
  step-05  tool_loop  reviewer    gpt-4o-mini  $0.000442  768   512   256
step-04    user_turn  planner     gpt-4o-mini  $0.000442  768   512   256
TOTAL                                          $0.002208  3840  2560  1280

Every capture on this page is from a local run against a mock upstream with fixed token counts, so the dollars are arithmetic over those counts and every step costs the same. In your traffic the column that matters is COST, and the reason to draw a tree is that it usually is not flat.

The tree is ordered by structure rather than by clock. step-04 appears last and is a root, because it carried no parent; step-05 was sent after it and appears above it, indented, because it named step-01 as its parent.

The hosted service has a registered read of the same shape, customer.traces.get, not yet executed against a deployment. It returns one trace in one application and environment as a graph, names a reason for every parent it cannot draw instead of placing the row at the root, and totals the cost per page. Investigate one application task walks through it.

A tree and a rollup answer different questions

The tree answers what did this one run do. A rollup answers where does the money go across all runs, and it is the one you look at first when the bill moves:

$ tokentriage top --by agent
AGENT       REQS  INPUT  OUTPUT  CACHE-R  CACHE-W  REASON  CACHE-SAV  COST       P50        P90        P99
(none)      61    46848  31232   15616    0        0       25.0%      $0.026938  $0.000442  $0.000442  $0.000442
planner     2     1536   1024    512      0        0       25.0%      $0.000883  $0.000442  $0.000442  $0.000442
researcher  1     768    512     256      0        0       25.0%      $0.000442  $0.000442  $0.000442  $0.000442
reviewer    1     768    512     256      0        0       25.0%      $0.000442  $0.000442  $0.000442  $0.000442
writer      1     768    512     256      0        0       25.0%      $0.000442  $0.000442  $0.000442  $0.000442
TOTAL       66    50688  33792   16896    0        0       25.0%      $0.029146

(none) is the row worth staring at. It is spend from requests that carried no x-tt-agent, and it is rendered as its own line rather than spread across the named agents. In that lab run it is 92% of the total, which is what an uninstrumented service looks like from the inside. A large (none) line is the first thing to chase, and the fix is putting identity on the wire in the caller that produced it.

The same command groups by model, step-class, trace or tool.

Four ways a tree comes out wrong

It is flat. No request carried x-tt-parent-id. The trace is real and every step is costed; only the shape is missing. Set the parent id where one step spawns another.

It is one giant trace. The trace id was minted once per process rather than once per run. A week of traffic in a single trace is technically correct and useless. Mint it where the run begins.

Two runs are tangled together. Two runs chose the same id. Ids are opaque strings compared for equality, and nothing checks for collisions, so a short or sequential id merges unrelated work. Use something with entropy in it.

A step is missing. Either that call did not go through TokenTriage at all, or its row was dropped on a full writer queue. The second case is counted, so check tokentriage_declog_dropped_total before assuming the first. What a drop means has the arithmetic.

What the tree does when a parent arrives after its child

Nothing special, and that is worth knowing. The rows are grouped by id, not replayed in order, so a child logged before its parent still renders underneath it once both rows exist. Streaming responses, retries and out-of-order writes therefore do not reshape the tree.

The corollary is that a tree read while a run is still in flight is a snapshot. A child whose parent has not been written yet renders at the root and moves when the parent lands.