# Anomaly detection
A named, recomputable detector for spend that broke its own pattern, the double gate that stops it paging you, and its affirmative negative.
Source: https://tokentriage.com/docs/build/anomaly-detection
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import BubbleUpDiagram from '../../../components/BubbleUpDiagram.astro';
import Screenshot from '../../../components/Screenshot.astro';
Spend jumped at three in the morning and nobody noticed until the invoice. The obvious fix is a
threshold alert, and the obvious problem with a threshold alert is that spend is seasonal. Your
Tuesday afternoon is not your Sunday night, so any single number is either deaf on your busy
hours or screaming on your quiet ones.
TokenTriage's detector compares each hour against *that same hour in previous weeks*, which is
the comparison a threshold cannot make. It also names its method on every finding and carries
every parameter it used, so you can recompute a flag by hand and disagree with it.
Detection needs no analytics configuration — it runs wherever the dashboard runs, reading the
decision log directly. It does need history: the baseline is built from previous weeks, so a
fresh install detects nothing for a fortnight.
## How a finding is decided
The detector is called `robust-seasonal-v1`, and the id is printed verbatim on every finding.
Five steps, no machine learning:
1. **Bucket spend hourly**, per watched series. A series is one value of one watched dimension —
`decision.model_served = "openai/gpt-4o"` is a series, plus a grand-total series.
2. **Build the baseline** from the same hour of the week over the previous **4** weeks. A week
whose hour has no data at all is *missing* and shrinks the baseline population rather than
contributing a fabricated zero.
3. **Score in robust-z.** `(x − median) ÷ max(1.4826 × MAD, $0.01)`. The median and the median
absolute deviation are used rather than the mean and standard deviation because one previous
spike would otherwise inflate the baseline and hide the next one. The one-cent floor stops a
perfectly flat series dividing by nearly nothing.
4. **Apply both gates.** A bucket is anomalous only when `|z| ≥ 4` **and** the dollar difference
from the median is at least **$1.00**.
5. **Merge consecutive hours** into one finding, carrying the peak z and the summed excess
dollars.
The dollar gate is what makes this usable. Eleven cents on a perfectly flat series is
statistically enormous (the z-score can run into the thousands) and it is not worth anyone's
night. Statistical surprise alone pages nobody here.
A bucket needs at least **2** populated baseline weeks to be scored at all. A series where no
bucket clears that is reported as *unevaluable* rather than silently skipped, which matters in
week one when that is every series you have.
## What it watches, and what it does not
Four dimensions by default, plus the grand total:
```text
decision.model_served · trace.agent · dim.tenant · mcp.tool · total
```
Eight more are watchable and not watched: `decision.tier`, `decision.upstream`,
`decision.rule`, `dim.user`, `dim.feature`, `mcp.server`, `request.model` and `http.dialect`.
Both lists travel on every response, so the blind spots are stated rather than left for you to
discover.
Unpriced rows are excluded from the dollar buckets rather than summed as zero, and the count of
them inside the evaluated window is disclosed, so a spike hiding inside spend nobody could price
is not invisible to someone reading this surface alone.
The evaluation window is the last 48 hours, ending at the last **complete** hour. The current
wall-clock hour is still accumulating, and admitting it would make every active series look like
a collapse on every scan.
## Ranking what drove it
A finding that says "spend was anomalous" has done a fraction of the work. The interesting
question is which model, which tenant, which agent.
The naive answer, ranking values by how much they spent, returns your most expensive model
every single time, spike or no spike. So each value is scored on how much it spent *against how
much its own history predicted it would*.
Read it left to right: what the value spent over the anomalous hours, minus what the baseline
said to expect from it, is its contribution. Values whose contribution is zero or negative are
not causes and are dropped; the rest are ranked largest first. A value that is *always* expensive
predicts its own large share and therefore contributes little, which is exactly why this ranking
surfaces what changed rather than what is big.
Here is a live finding, from a sandbox whose synthetic traffic contains a planted spike:
```json
{
"id": "robust-seasonal-v1|total||2026-08-09T00:00:00Z",
"dimension": "total",
"window": { "since": "2026-08-09T00:00:00Z", "until": "2026-08-09T01:00:00Z" },
"peak_z": 5987.800125,
"impact": { "usd": 59.87800125, "estimated": false, "unpriced": false },
"causes": [
{ "dimension": "decision.model_served", "value": "anthropic/claude-opus-4-8",
"contribution": { "usd": 59.863763824481126 },
"filter": "decision.model_served = \"anthropic/claude-opus-4-8\"" },
{ "dimension": "dim.tenant", "value": "spike-co",
"contribution": { "usd": 59.80750132448112 },
"filter": "dim.tenant = \"spike-co\"" },
{ "dimension": "trace.agent", "value": "batch-runner",
"contribution": { "usd": 59.80750132448112 },
"filter": "trace.agent = \"batch-runner\"" },
{ "dimension": "mcp.tool", "value": "code_exec",
"contribution": { "usd": 0.036342597021828925 },
"filter": "mcp.tool = \"code_exec\"" }
],
"method": "robust-seasonal-v1",
"repro": { "method": "robust-seasonal-v1", "w": 4, "z_threshold": 4,
"impact_floor_usd": 1, "sigma_min": 0.01, "baseline_population": 4 }
}
```
One hour, $59.88 above its usual, and the first three causes tell one story from three angles:
one tenant, running one agent, against one expensive model. The fourth cause is four cents and is
ordinary traffic.
Two details to take from that block. The `peak_z` of 5,987 is not a typo. It is what happens
when a nearly flat baseline meets a large spike, and it is why the dollar gate does the work
that matters. And every cause carries a **`filter`** built from the query language's own syntax, so the
one-click drill-down cannot be handed a string the query engine will reject.
## Recomputing a flag by hand
Every finding embeds the parameters it was decided under:
| Field | Default | What it does |
|---|---|---|
| `w` | 4 | trailing weeks in the baseline |
| `z_threshold` | 4 | the robust-z gate |
| `impact_floor_usd` | 1.00 | the dollar gate |
| `sigma_min` | 0.01 | the sigma floor, in dollars |
| `baseline_population` | varies | how many of the `w` weeks had data |
`baseline_population` is the one to read first. Four means the baseline was full. Two means the
median behind that flag came from two numbers, and a median of two numbers is an average wearing
a disguise.
Changing any parameter or the algorithm changes the method id, and findings under different
method ids are not merged.
## When nothing fires
An empty findings list means one of two very different things: "we watched these series over
this window and nothing was unusual", or "we watched nothing". This surface will not let them
look alike. Here is an empty one, from a sandbox holding six requests and no history:
```json
{
"findings": [],
"evaluation": {
"window_evaluated": { "since": "2026-08-08T01:00:00Z", "until": "2026-08-10T02:00:00Z" },
"series_evaluated": 0,
"series_unevaluable": 0,
"unpriced_excluded": 0,
"dimensions_watched": ["decision.model_served", "trace.agent", "dim.tenant", "mcp.tool", "total"],
"dimensions_not_watched": ["decision.tier", "decision.upstream", "decision.rule", "dim.user", "dim.feature", "mcp.server", "request.model", "http.dialect"],
"clean": true,
"method": "robust-seasonal-v1",
"repro": { "method": "robust-seasonal-v1", "w": 4, "z_threshold": 4, "impact_floor_usd": 1, "sigma_min": 0.01, "baseline_population": 0 }
},
"scope": "watching: decision.model_served, trace.agent, dim.tenant, mcp.tool, total — not watching: decision.tier, decision.upstream, decision.rule, dim.user, dim.feature, mcp.server, request.model, http.dialect"
}
```
`clean: true` and `series_evaluated: 0` together are the sentence that matters: **nothing was
found because nothing was examined.** No series had two populated baseline weeks behind it, so
none could be scored. A tool that printed a green tick here would be telling you your spend is
fine on the strength of having looked at none of it.
Compare the same block from the ledger that produced the finding above: 21 series evaluated,
none unevaluable, two unpriced rows excluded, `clean: false`. Same shape, opposite meaning, and
you can tell them apart at a glance.
## What changed near the spike
Every response carries a **change correlation** block beside the findings. A second method,
`change-correlation-v1`, joins each finding's window against the audit log (configuration
applies, policy generation changes and pricing-snapshot changes within a 6-hour lookback) and
ranks what it finds by how close in time it was.
```json
{
"correlations": {
"method": "change-correlation-v1",
"lookback_minutes": 360,
"changes_considered": 0,
"candidates": [],
"reason": "no configuration apply, policy generation change or pricing-table change is recorded anywhere in reach of these findings, so nothing is offered as a candidate"
}
}
```
It is served as a sibling block and is not merged into a finding. The method ids differ so a
correlation cannot be mistaken for a detection, and the block is labelled correlation rather
than cause. When nothing lines up it says so, as above, instead of reaching for the nearest
change, and it distinguishes that from having had nothing to correlate against in the first
place, which reads `no anomalies were found, so there was nothing to
correlate a change against`.
## Turning it on
```yaml
dashboard:
enabled: true # that is the whole requirement
```
The detector is composed alongside the dashboard and reads the joined decision log off the
request path. It needs neither `analytics.enabled` nor the derived index, which is worth knowing
because the other surfaces in this section need both — a deployment with analytics off still
answers `GET /tt/api/v1/anomalies` with findings and a full evaluation block.
Findings persist in an append-only store with acknowledgement state, and appear on the standalone
Anomalies destination as well as embedded in the Overview and Cost surfaces.
## What can go wrong
**Nothing has fired and you conclude it is broken.** Check `series_evaluated` and
`series_unevaluable`. On a ledger under two weeks old every series is unevaluable, which is the
correct answer and not a fault.
**A spike you saw is not in the list.** Three candidates, in order: it was under $1.00 above the
median; it was on a dimension in the not-watched list; or its hour was inside the window's open
edge, since scoring stops at the last complete hour.
**A finding you disagree with.** Recompute it. Everything needed is in the `repro` block and your
own ledger, and disagreeing with a flag you can recompute is a much better position than
disagreeing with one you cannot.
The score is `(x − median) ÷ max(1.4826 × MAD, 0.01)`. The `1.4826` converts a median absolute
deviation into something comparable to a standard deviation for normally distributed data.
The floor in that denominator is doing more work than the constant. On a series that spends
almost exactly the same amount every Tuesday at 3pm, the MAD is near zero, so without the floor
the score would be near infinite for any deviation at all. With the floor, a series that barely
varies produces enormous scores for small absolute changes, which is why the flag also has to
clear a dollar bar before anyone hears about it.
Treat z as *how unusual* and the impact dollars as *how much you care*. They are different
questions and the detector requires both.