# How steps become a tree

What the parent edge builds, how a tree reads next to a rollup by agent, and the four reasons a tree comes out looking wrong.

Source: https://tokentriage.com/docs/build/observability-trace-trees

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import TraceTreeDiagram from '../../../components/TraceTreeDiagram.astro';

A trace is not a list of calls. It is a shape, and the shape is where the money is: a planner
that costs a tenth of a cent and fans out to three sub-agents that cost forty times that is a
different problem from one expensive call.

This page is about how three headers produce that shape, what the shape can and cannot tell you,
and the handful of ways it comes out looking wrong.

## Three headers, one tree

<TraceTreeDiagram />

Read it left to right. Your agent sets the headers per call; equal trace ids collect the rows
into one trace; the parent id draws the edge from a step to the step that spawned it; and the
same rows feed the per-step tree, the rollups and the span export.

The rules are short enough to state completely.

1. Rows sharing an `x-tt-trace-id` are one trace. Nothing else joins them.
2. Within a trace, a row is a child of the row whose `x-tt-step-id` equals its `x-tt-parent-id`.
3. A row with no parent id renders at the root, whatever its position in time.
4. Nothing validates the edge. A parent id naming a step that does not exist leaves the row at
   the root rather than failing.

Rendered, that is one command:

```console
$ tokentriage top --trace run-7f3a91c2
trace run-7f3a91c2  (5 steps)
STEP       CLASS      AGENT       MODEL        COST       IN    OUT   CACHE-R
step-01    user_turn  planner     gpt-4o-mini  $0.000442  768   512   256
  step-02  tool_loop  researcher  gpt-4o-mini  $0.000442  768   512   256
  step-03  tool_loop  writer      gpt-4o-mini  $0.000442  768   512   256
  step-05  tool_loop  reviewer    gpt-4o-mini  $0.000442  768   512   256
step-04    user_turn  planner     gpt-4o-mini  $0.000442  768   512   256
TOTAL                                          $0.002208  3840  2560  1280
```

Every capture on this page is from a local run against a mock upstream with fixed token counts,
so the dollars are arithmetic over those counts and every step costs the same. In your traffic
the column that matters is `COST`, and the reason to draw a tree is that it usually is not flat.

The tree is ordered by structure rather than by clock. `step-04` appears last and is a root,
because it carried no parent; `step-05` was sent after it and appears above it, indented, because
it named `step-01` as its parent.

The hosted service has a registered read of the same shape, `customer.traces.get`, not yet
executed against a deployment. It returns one trace in one application and environment as a
graph, names a reason for every parent it cannot draw instead of placing the row at the root,
and totals the cost per page.
[Investigate one application task](/docs/build/investigate-application-task) walks through it.

## A tree and a rollup answer different questions

The tree answers *what did this one run do*. A rollup answers *where does the money go across all
runs*, and it is the one you look at first when the bill moves:

```console
$ tokentriage top --by agent
AGENT       REQS  INPUT  OUTPUT  CACHE-R  CACHE-W  REASON  CACHE-SAV  COST       P50        P90        P99
(none)      61    46848  31232   15616    0        0       25.0%      $0.026938  $0.000442  $0.000442  $0.000442
planner     2     1536   1024    512      0        0       25.0%      $0.000883  $0.000442  $0.000442  $0.000442
researcher  1     768    512     256      0        0       25.0%      $0.000442  $0.000442  $0.000442  $0.000442
reviewer    1     768    512     256      0        0       25.0%      $0.000442  $0.000442  $0.000442  $0.000442
writer      1     768    512     256      0        0       25.0%      $0.000442  $0.000442  $0.000442  $0.000442
TOTAL       66    50688  33792   16896    0        0       25.0%      $0.029146
```

`(none)` is the row worth staring at. It is spend from requests that carried no `x-tt-agent`, and
it is rendered as its own line rather than spread across the named agents. In that lab run it is
92% of the total, which is what an uninstrumented service looks like from the inside. A large
`(none)` line is the first thing to chase, and the fix is
[putting identity on the wire](/docs/build/observability-headers) in the caller that produced it.

The same command groups by `model`, `step-class`, `trace` or `tool`.

## Four ways a tree comes out wrong

**It is flat.** No request carried `x-tt-parent-id`. The trace is real and every step is costed;
only the shape is missing. Set the parent id where one step spawns another.

**It is one giant trace.** The trace id was minted once per process rather than once per run. A
week of traffic in a single trace is technically correct and useless. Mint it where the run
begins.

**Two runs are tangled together.** Two runs chose the same id. Ids are opaque strings compared
for equality, and nothing checks for collisions, so a short or sequential id merges unrelated
work. Use something with entropy in it.

**A step is missing.** Either that call did not go through TokenTriage at all, or its row was
dropped on a full writer queue. The second case is counted, so check
`tokentriage_declog_dropped_total` before assuming the first. [What a drop
means](/docs/build/observability-drops) has the arithmetic.

<Callout type="note">
A cost tree is not a latency waterfall. The columns are dollars and tokens, and the indentation
is the delegation edge rather than a timeline: two sibling steps that ran concurrently and two
that ran in sequence look identical here. Per-request timings are on the row in `latency`, and
distributions are in [the metrics](/docs/build/observability-metrics-reference).
</Callout>

<Details summary="What the tree does when a parent arrives after its child">

Nothing special, and that is worth knowing. The rows are grouped by id, not replayed in order, so
a child logged before its parent still renders underneath it once both rows exist. Streaming
responses, retries and out-of-order writes therefore do not reshape the tree.

The corollary is that a tree read while a run is still in flight is a snapshot. A child whose
parent has not been written yet renders at the root and moves when the parent lands.

</Details>

<NextSteps items={[
  {title: "Trace one request end to end", href: "/docs/build/observability-request-id", description: "The other direction: start from one request id and work outwards to its trace, decision and dollar."},
  {title: "Put identity on the wire", href: "/docs/build/observability-headers", description: "Set the trace, step, parent and agent headers so the tree has a shape at all."},
  {title: "Export spans to your collector", href: "/docs/build/observability-otlp", description: "The same trace as OTLP spans, including what the parent edge does and does not survive."},
  {title: "The trust dashboard", href: "/docs/build/dashboard", description: "The Traces surface renders the same tree as a cost waterfall, with the sub-agent fan-out."},
]} />