Anomaly detection

Spend jumped at three in the morning and nobody noticed until the invoice. The obvious fix is a threshold alert, and the obvious problem with a threshold alert is that spend is seasonal. Your Tuesday afternoon is not your Sunday night, so any single number is either deaf on your busy hours or screaming on your quiet ones.

TokenTriage’s detector compares each hour against that same hour in previous weeks, which is the comparison a threshold cannot make. It also names its method on every finding and carries every parameter it used, so you can recompute a flag by hand and disagree with it.

Detection needs no analytics configuration — it runs wherever the dashboard runs, reading the decision log directly. It does need history: the baseline is built from previous weeks, so a fresh install detects nothing for a fortnight.

How a finding is decided

The detector is called robust-seasonal-v1, and the id is printed verbatim on every finding. Five steps, no machine learning:

1. Bucket spend hourly, per watched series. A series is one value of one watched dimension — decision.model_served = "openai/gpt-4o" is a series, plus a grand-total series.
2. Build the baseline from the same hour of the week over the previous 4 weeks. A week whose hour has no data at all is missing and shrinks the baseline population rather than contributing a fabricated zero.
3. Score in robust-z. (x − median) ÷ max(1.4826 × MAD, $0.01). The median and the median absolute deviation are used rather than the mean and standard deviation because one previous spike would otherwise inflate the baseline and hide the next one. The one-cent floor stops a perfectly flat series dividing by nearly nothing.
4. Apply both gates. A bucket is anomalous only when |z| ≥ 4 and the dollar difference from the median is at least $1.00.
5. Merge consecutive hours into one finding, carrying the peak z and the summed excess dollars.

A bucket needs at least 2 populated baseline weeks to be scored at all. A series where no bucket clears that is reported as unevaluable rather than silently skipped, which matters in week one when that is every series you have.

What it watches, and what it does not

Four dimensions by default, plus the grand total:

decision.model_served · trace.agent · dim.tenant · mcp.tool · total

Eight more are watchable and not watched: decision.tier, decision.upstream, decision.rule, dim.user, dim.feature, mcp.server, request.model and http.dialect. Both lists travel on every response, so the blind spots are stated rather than left for you to discover.

Unpriced rows are excluded from the dollar buckets rather than summed as zero, and the count of them inside the evaluated window is disclosed, so a spike hiding inside spend nobody could price is not invisible to someone reading this surface alone.

The evaluation window is the last 48 hours, ending at the last complete hour. The current wall-clock hour is still accumulating, and admitting it would make every active series look like a collapse on every scan.

Ranking what drove it

A finding that says “spend was anomalous” has done a fraction of the work. The interesting question is which model, which tenant, which agent.

The naive answer, ranking values by how much they spent, returns your most expensive model every single time, spike or no spike. So each value is scored on how much it spent against how much its own history predicted it would.

Read it left to right: what the value spent over the anomalous hours, minus what the baseline said to expect from it, is its contribution. Values whose contribution is zero or negative are not causes and are dropped; the rest are ranked largest first. A value that is always expensive predicts its own large share and therefore contributes little, which is exactly why this ranking surfaces what changed rather than what is big.

Here is a live finding, from a sandbox whose synthetic traffic contains a planted spike:

{
  "id": "robust-seasonal-v1|total||2026-08-09T00:00:00Z",
  "dimension": "total",
  "window": { "since": "2026-08-09T00:00:00Z", "until": "2026-08-09T01:00:00Z" },
  "peak_z": 5987.800125,
  "impact": { "usd": 59.87800125, "estimated": false, "unpriced": false },
  "causes": [
    { "dimension": "decision.model_served", "value": "anthropic/claude-opus-4-8",
      "contribution": { "usd": 59.863763824481126 },
      "filter": "decision.model_served = \"anthropic/claude-opus-4-8\"" },
    { "dimension": "dim.tenant", "value": "spike-co",
      "contribution": { "usd": 59.80750132448112 },
      "filter": "dim.tenant = \"spike-co\"" },
    { "dimension": "trace.agent", "value": "batch-runner",
      "contribution": { "usd": 59.80750132448112 },
      "filter": "trace.agent = \"batch-runner\"" },
    { "dimension": "mcp.tool", "value": "code_exec",
      "contribution": { "usd": 0.036342597021828925 },
      "filter": "mcp.tool = \"code_exec\"" }
  ],
  "method": "robust-seasonal-v1",
  "repro": { "method": "robust-seasonal-v1", "w": 4, "z_threshold": 4,
             "impact_floor_usd": 1, "sigma_min": 0.01, "baseline_population": 4 }
}

One hour, $59.88 above its usual, and the first three causes tell one story from three angles: one tenant, running one agent, against one expensive model. The fourth cause is four cents and is ordinary traffic.

Two details to take from that block. The peak_z of 5,987 is not a typo. It is what happens when a nearly flat baseline meets a large spike, and it is why the dollar gate does the work that matters. And every cause carries a filter built from the query language’s own syntax, so the one-click drill-down cannot be handed a string the query engine will reject.

localhost:9090/ui/anomaliesDemo data

Recomputing a flag by hand

Every finding embeds the parameters it was decided under:

Field	Default	What it does
w	4	trailing weeks in the baseline
z_threshold	4	the robust-z gate
impact_floor_usd	1.00	the dollar gate
sigma_min	0.01	the sigma floor, in dollars
baseline_population	varies	how many of the w weeks had data

baseline_population is the one to read first. Four means the baseline was full. Two means the median behind that flag came from two numbers, and a median of two numbers is an average wearing a disguise.

Changing any parameter or the algorithm changes the method id, and findings under different method ids are not merged.

When nothing fires

An empty findings list means one of two very different things: “we watched these series over this window and nothing was unusual”, or “we watched nothing”. This surface will not let them look alike. Here is an empty one, from a sandbox holding six requests and no history:

{
  "findings": [],
  "evaluation": {
    "window_evaluated": { "since": "2026-08-08T01:00:00Z", "until": "2026-08-10T02:00:00Z" },
    "series_evaluated": 0,
    "series_unevaluable": 0,
    "unpriced_excluded": 0,
    "dimensions_watched": ["decision.model_served", "trace.agent", "dim.tenant", "mcp.tool", "total"],
    "dimensions_not_watched": ["decision.tier", "decision.upstream", "decision.rule", "dim.user", "dim.feature", "mcp.server", "request.model", "http.dialect"],
    "clean": true,
    "method": "robust-seasonal-v1",
    "repro": { "method": "robust-seasonal-v1", "w": 4, "z_threshold": 4, "impact_floor_usd": 1, "sigma_min": 0.01, "baseline_population": 0 }
  },
  "scope": "watching: decision.model_served, trace.agent, dim.tenant, mcp.tool, total — not watching: decision.tier, decision.upstream, decision.rule, dim.user, dim.feature, mcp.server, request.model, http.dialect"
}

clean: true and series_evaluated: 0 together are the sentence that matters: nothing was found because nothing was examined. No series had two populated baseline weeks behind it, so none could be scored. A tool that printed a green tick here would be telling you your spend is fine on the strength of having looked at none of it.

Compare the same block from the ledger that produced the finding above: 21 series evaluated, none unevaluable, two unpriced rows excluded, clean: false. Same shape, opposite meaning, and you can tell them apart at a glance.

What changed near the spike

Every response carries a change correlation block beside the findings. A second method, change-correlation-v1, joins each finding’s window against the audit log (configuration applies, policy generation changes and pricing-snapshot changes within a 6-hour lookback) and ranks what it finds by how close in time it was.

{
  "correlations": {
    "method": "change-correlation-v1",
    "lookback_minutes": 360,
    "changes_considered": 0,
    "candidates": [],
    "reason": "no configuration apply, policy generation change or pricing-table change is recorded anywhere in reach of these findings, so nothing is offered as a candidate"
  }
}

It is served as a sibling block and is not merged into a finding. The method ids differ so a correlation cannot be mistaken for a detection, and the block is labelled correlation rather than cause. When nothing lines up it says so, as above, instead of reaching for the nearest change, and it distinguishes that from having had nothing to correlate against in the first place, which reads no anomalies were found, so there was nothing to correlate a change against.

Turning it on

dashboard:
  enabled: true # that is the whole requirement

The detector is composed alongside the dashboard and reads the joined decision log off the request path. It needs neither analytics.enabled nor the derived index, which is worth knowing because the other surfaces in this section need both — a deployment with analytics off still answers GET /tt/api/v1/anomalies with findings and a full evaluation block.

Findings persist in an append-only store with acknowledgement state, and appear on the standalone Anomalies destination as well as embedded in the Overview and Cost surfaces.

What can go wrong

Nothing has fired and you conclude it is broken. Check series_evaluated and series_unevaluable. On a ledger under two weeks old every series is unevaluable, which is the correct answer and not a fault.

A spike you saw is not in the list. Three candidates, in order: it was under $1.00 above the median; it was on a dimension in the not-watched list; or its hour was inside the window’s open edge, since scoring stops at the last complete hour.

A finding you disagree with. Recompute it. Everything needed is in the repro block and your own ledger, and disagreeing with a flag you can recompute is a much better position than disagreeing with one you cannot.

What a large z-score does and does not tell you

The score is (x − median) ÷ max(1.4826 × MAD, 0.01). The 1.4826 converts a median absolute deviation into something comparable to a standard deviation for normally distributed data.

The floor in that denominator is doing more work than the constant. On a series that spends almost exactly the same amount every Tuesday at 3pm, the MAD is near zero, so without the floor the score would be near infinite for any deviation at all. With the floor, a series that barely varies produces enormous scores for small absolute changes, which is why the flag also has to clear a dollar bar before anyone hears about it.

Treat z as how unusual and the impact dollars as how much you care. They are different questions and the detector requires both.