Alerting on suspensions

Evidence & alerting says alert on suspended[]. This page is how that actually happens, what you can tune, and — mostly — what you cannot.

There is no guardrails.alerts: block

The one thing you control is whether it is armed at all, and that is implied by two other decisions:

{"level":"INFO","msg":"guardrail alerting armed","channels":0,"reused_dispatcher":true}

That line prints only when guardrails are enabled and the dashboard is composed. Both gates are real:

• guardrails.enabled: false registers no alerting at all. Nothing to observe, nothing to latch.
• The dashboard being off means no channels, no shared dispatcher and no SSE publisher were built — so the wire is not made. A deployment running without the dashboard still gets guardrail suspensions on its rows, which is the durable record and is unconditional; it just gets no push notification.

Where the signal comes from

The feed is the ledger tee, not a call from the request path. It fires post-write, in the sink’s own goroutine, after a successful append — so an alert can never describe a row the file does not contain. Ingest is a copy-and-enqueue with drop-and-count, so a slow dispatcher can never apply backpressure to the ledger writer.

A guardrail suspension is an event, not a status. There is no “is binding X suspended right now?” to poll: the engine emits a suspension on the one request that faulted, and then forgets. So the source latches the event itself — fire on the Nth observation, resolve after a quiet period.

The anti-noise contract

Knob	Default	What it does
MinSuspensions	1	How many suspensions for the same (binding, phase, reason) key before the alert fires.
ResolveAfter	15m	How long a firing key must go without a new suspension before it resolves.
SweepInterval	1m	The resolve cadence. It only bounds how late a resolve is noticed.
QueueSize	4096	The ingest queue bound, past which events are dropped and counted rather than blocking.
DeliverTimeout	10s	Bounds one delivery attempt.

The key includes the scope of the outage, which is what keeps the page accurate about the failure mode: a binding that inspected nothing and a binding whose one detector fell over are different keys and read differently. Inferring the scope from the detector map instead once produced a page for a binding-wide gap that named a single detector — the wrong failure mode.

What a resolve does not claim

Nil-safe sinks, and channels: 0

Every sink is independently optional, and an absent one degrades rather than breaks:

• No dispatcher → no delivery. If you have no config alert rules at all, a private dispatcher is built so guardrail alerts still deliver, and closed again on shutdown.
• No publisher → no SSE publish.
• channels: 0 → SSE only. The source still latches and still reports Firing. An armed line with zero channels is a working alert source with nowhere to send; the state is real even when the delivery is not.

The one thing you cannot do with it

Guardrail alerts are not backtestable, and the rule says so. The raw fact is on disk — every suspension is written to the row that produced it — but the backtest surface drives rules compiled from config, and this is not one of those. Claiming otherwise would have the backtest compute an empty result: “0 would have fired” over a window that contained a hundred suspensions is worse than “it cannot look.”