Alert on what the ledger sees

You want to hear about spend spikes, error rates and budget breaches without watching a screen. The alerts: block does that from the ledger the dashboard already reads, so a rule sees the same rows and the same dollars a surface does.

This is a small, closed system: nine rule types, three delivery channels, no expression language. If you already run Prometheus and want the gateway in it, metrics is the other route and the two are complementary.

Define a channel and a rule

A channel is a delivery destination. Credentials are environment variable names rather than values, because the running config is served verbatim on the admin plane:

alerts:
  channels:
    - id: ops-webhook
      type: webhook              # slack | webhook | email
      url_env: OPS_WEBHOOK_URL   # the NAME of the variable, not the URL
  rules:
    - id: spend-spike
      type: spend_threshold
      filter: dim.team = "ml"    # empty matches all traffic
      window: 15m
      threshold_usd: 10.00
      min_requests: 5            # floor before a sampled rule may fire
      consecutive: 2             # windows in a row before it fires
      notify: [ops-webhook]

filter uses the same query language as the Requests explorer. consecutive is the anti-noise control: a rule that needs two windows in a row will not page you for a single spiky minute.

The nine rule types

Seven of the nine fold a window of ledger rows into one number and compare it against a threshold. Two are driven by something else entirely.

Type	Reduces	Fires when	Threshold field
spend_threshold	sum	priced spend in the window reaches the figure	threshold_usd
error_rate	rate	the fraction of errored requests reaches the percentage	threshold_pct
dropped_rows	rate	the fraction of client-disconnected responses reaches it	threshold_pct
shadow_cap	rate	the fraction of requests shadow-duplicated reaches it	threshold_pct
unpriced_rows	rate	the fraction of rows with no price reaches it	threshold_pct
provider_cooldown	rate	the fraction of requests that skipped a cooled upstream reaches it	threshold_pct
verdict_transition	rate	the fraction of graded rows judging the chosen tier not-ok reaches it	threshold_pct
budget_threshold	—	a named budget crosses one of its own thresholds	budget_id
anomaly	—	the detector publishes a finding in scope	filter

Two names in that table mean something narrower than they sound, and both have caught people out.

dropped_rows is about clients, not the ledger. It watches responses where the caller disconnected. The ledger writer’s own discards are a different counter with a different subject; what a drop means owns that one, and it is worth alerting on through metrics.

verdict_transition is a rate, not a transition. It fires on the fraction of graded rows whose shadow verdict judged the chosen tier not-ok. It does not fire when the honesty verdict changes state. Its filter is also ignored, because verdicts live on companion rows carrying no filterable dimensions, so every graded row is in scope whatever you write.

The last two rows are the latched rules: they track a condition rather than a window, so they have no sample count.

Read the surface

localhost:9090/ui/alertsDemo data

Four panels, and the last two are the interesting ones.

Rules shows each rule’s current state, its scope and window, and where it delivers. History shows transitions as the evaluator recorded them, including delivery failures — a rule that fired and could not be delivered is a different fact from a rule that did not fire.

Silences must expire. There is no permanent mute, and the server rejects a silence without an expiry. That is a deliberate constraint against the most common way alerting rots, which is a mute somebody adds during an incident and nobody removes.

The backtest replays history through the same evaluator that fires live rules, so a would-have-fired marker is computed by the production code path rather than a re-implementation of it. Where a rule cannot be replayed, the panel says so and names the reason instead of producing a marker:

budget-threshold rules are not backtestable — they fire from a latched LIVE budget status
(a burn fraction is not a per-window sample), which historical ledger rows cannot reconstruct.
Never fabricated from rows.

That is the same latched-versus-sampled distinction as the min_requests trap, showing up where you can see it.

What a backtest is and is not

It answers one question: given these rules and this history, when would each have fired? It replays a bounded window of ledger rows through the live rule evaluator and marks the windows that would have crossed a threshold, with the sample count and the estimated and unpriced counts behind each marker.

It is not a simulation of delivery, and it grants the evaluator no new authority — it is a read-only replay. Two rule kinds have nothing to replay from: a budget threshold latches on live budget state, and an anomaly rule fires from the detector’s own findings feed. Both are listed as not backtestable rather than being given markers that would look like evidence.

Use it to size a threshold before you turn a rule on. A rule that would have fired forty times last month is a rule that will page you forty times next month.

Where governance alerts go

Governance has its own alerting for limit thresholds, and it fans out to the same channels while being independent of alerts.rules. So a channel defined here also carries limit alerts; a rule defined here does not govern them. Governance operations owns that side.

What can go wrong

A rule never fires. Check for min_requests on a latched type first, then check the filter against the Requests explorer, then check consecutive.

Alerts fire constantly. Raise consecutive before raising the threshold. A threshold sized for the peak stops catching the sustained problem.

Nothing is delivered. The channel names an environment variable that must exist in the process environment. Delivery failures appear in the history panel with the failure, which is where to look before suspecting the rule.

You want alerting without the dashboard. The alert evaluator is part of the dashboard composition. If you would rather alert from your existing monitoring, scrape the metrics endpoint instead: metrics ships three rules with the chart.