# Alert on what the ledger sees

Nine bounded rule types over your own decision log, delivered to Slack, a webhook or email. Plus a backtest that shows what would have fired.

Source: https://tokentriage.com/docs/build/dashboard-alerts

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import Screenshot from '../../../components/Screenshot.astro';

You want to hear about spend spikes, error rates and budget breaches without watching a
screen. The `alerts:` block does that from the ledger the dashboard already reads, so a rule
sees the same rows and the same dollars a surface does.

This is a small, closed system: nine rule types, three delivery channels, no expression
language. If you already run Prometheus and want the gateway in it,
[metrics](/docs/build/observability-metrics) is the other route and the two are complementary.

## Define a channel and a rule

A channel is a delivery destination. Credentials are environment variable *names* rather than
values, because the running config is served verbatim on the admin plane:

```yaml
alerts:
  channels:
    - id: ops-webhook
      type: webhook              # slack | webhook | email
      url_env: OPS_WEBHOOK_URL   # the NAME of the variable, not the URL
  rules:
    - id: spend-spike
      type: spend_threshold
      filter: dim.team = "ml"    # empty matches all traffic
      window: 15m
      threshold_usd: 10.00
      min_requests: 5            # floor before a sampled rule may fire
      consecutive: 2             # windows in a row before it fires
      notify: [ops-webhook]
```

`filter` uses the same query language as the Requests explorer. `consecutive` is the
anti-noise control: a rule that needs two windows in a row will not page you for a single
spiky minute.

## The nine rule types

Seven of the nine fold a window of ledger rows into one number and compare it against a
threshold. Two are driven by something else entirely.

| Type | Reduces | Fires when | Threshold field |
|---|---|---|---|
| `spend_threshold` | sum | priced spend in the window reaches the figure | `threshold_usd` |
| `error_rate` | rate | the fraction of errored requests reaches the percentage | `threshold_pct` |
| `dropped_rows` | rate | the fraction of client-disconnected responses reaches it | `threshold_pct` |
| `shadow_cap` | rate | the fraction of requests shadow-duplicated reaches it | `threshold_pct` |
| `unpriced_rows` | rate | the fraction of rows with no price reaches it | `threshold_pct` |
| `provider_cooldown` | rate | the fraction of requests that skipped a cooled upstream reaches it | `threshold_pct` |
| `verdict_transition` | rate | the fraction of graded rows judging the chosen tier not-ok reaches it | `threshold_pct` |
| `budget_threshold` | — | a named budget crosses one of its own thresholds | `budget_id` |
| `anomaly` | — | the detector publishes a finding in scope | `filter` |

Two names in that table mean something narrower than they sound, and both have caught people
out.

**`dropped_rows` is about clients, not the ledger.** It watches responses where the caller
disconnected. The ledger writer's own discards are a different counter with a different
subject; [what a drop means](/docs/build/observability-drops) owns that one, and it is worth
alerting on through metrics.

**`verdict_transition` is a rate, not a transition.** It fires on the fraction of graded rows
whose shadow verdict judged the chosen tier not-ok. It does not fire when the honesty verdict
changes state. Its `filter` is also ignored, because verdicts live on companion rows carrying
no filterable dimensions, so every graded row is in scope whatever you write.

The last two rows are the **latched** rules: they track a condition rather than a window, so
they have no sample count.

<Callout type="warning">
Setting `min_requests` on `budget_threshold` silently stops it firing, because a latched
condition has no sample count and the floor can never be met. Config validation accepts it,
checking only that the value is non-negative, so nothing warns you. The shipped demo config
omits it.
</Callout>

## Read the surface

<Screenshot
  stem="dashboard-alerts"
  dark={true}
  url="localhost:9090/ui/alerts"
  alt="The Alerts surface: two rules with their type, filter scope, window and notification channel, each showing an OK state; a silence form requiring an expiry; an empty transition history; and a burn-alert backtest that replayed 46 historical records and reports that neither rule is backtestable, with the reason for each."
  caption="Two rules, a silence form that will not accept a permanent mute, and a backtest that declines to fabricate markers for rules it cannot replay (real dashboard, demo mode)."
/>

Four panels, and the last two are the interesting ones.

**Rules** shows each rule's current state, its scope and window, and where it delivers.
**History** shows transitions as the evaluator recorded them, including delivery failures — a
rule that fired and could not be delivered is a different fact from a rule that did not fire.

**Silences must expire.** There is no permanent mute, and the server rejects a silence without
an expiry. That is a deliberate constraint against the most common way alerting rots, which is a
mute somebody adds during an incident and nobody removes.

**The backtest replays history through the same evaluator** that fires live rules, so a
would-have-fired marker is computed by the production code path rather than a re-implementation
of it. Where a rule cannot be replayed, the panel says so and names the reason instead of
producing a marker:

```text
budget-threshold rules are not backtestable — they fire from a latched LIVE budget status
(a burn fraction is not a per-window sample), which historical ledger rows cannot reconstruct.
Never fabricated from rows.
```

That is the same latched-versus-sampled distinction as the `min_requests` trap, showing up
where you can see it.

<Details summary="What a backtest is and is not">

It answers one question: given these rules and this history, when would each have fired? It
replays a bounded window of ledger rows through the live rule evaluator and marks the windows
that would have crossed a threshold, with the sample count and the estimated and unpriced
counts behind each marker.

It is not a simulation of delivery, and it grants the evaluator no new authority — it is a
read-only replay. Two rule kinds have nothing to replay from: a budget threshold latches on
live budget state, and an anomaly rule fires from the detector's own findings feed. Both are
listed as not backtestable rather than being given markers that would look like evidence.

Use it to size a threshold before you turn a rule on. A rule that would have fired forty times
last month is a rule that will page you forty times next month.

</Details>

## Where governance alerts go

Governance has its own alerting for limit thresholds, and it fans out to the same channels
while being independent of `alerts.rules`. So a channel defined here also carries limit
alerts; a rule defined here does not govern them.
[Governance operations](/docs/build/governance-operations) owns that side.

## What can go wrong

**A rule never fires.** Check for `min_requests` on a latched type first, then check the
filter against the Requests explorer, then check `consecutive`.

**Alerts fire constantly.** Raise `consecutive` before raising the threshold. A threshold sized
for the peak stops catching the sustained problem.

**Nothing is delivered.** The channel names an environment variable that must exist in the
process environment. Delivery failures appear in the history panel with the failure, which is
where to look before suspecting the rule.

**You want alerting without the dashboard.** The alert evaluator is part of the dashboard
composition. If you would rather alert from your existing monitoring, scrape the metrics
endpoint instead: [metrics](/docs/build/observability-metrics) ships three rules with the
chart.

<NextSteps items={[
  {title: "Cap spend with a budget", href: "/docs/build/dashboard-budgets", description: "The ceiling a budget_threshold rule watches, and the trap the two share."},
  {title: "Anomaly detection", href: "/docs/build/anomaly-detection", description: "What the detector behind an anomaly rule looks for, and what it will not claim."},
  {title: "Scrape the metrics endpoint", href: "/docs/build/observability-metrics", description: "The other route into alerting, through the monitoring you already run."},
  {title: "What a drop means", href: "/docs/build/observability-drops", description: "The counter behind a dropped_rows rule, and why it is worth alerting on."}
]} />