Watch cost arrive live

Some questions are answered by a graph and some by watching. Whether your headers are arriving, whether a new deployment is hitting the model you expected, whether the first requests after a config change are being priced: those are all easier to see in a live tail than to query for.

The dashboard’s Live surface is that tail. Each costed request appears as a row a moment after it completes, with its model, its dollar and its provenance.

Turn it on

The Live feed is part of the dashboard, which is off in a config file and on in the zero-config quickstarts.

dashboard:
  enabled: true # off by default in a config file
  stream:
    client_queue: 1024 # per-viewer buffered events; default 1024
    max_clients: 32 # concurrent viewers; default 32

Then open the admin plane and go to the Live surface.

localhost:9090/ui/liveDemo data

What it can and cannot show

The stream is fed by a tee that fires after a row has been appended to the ledger, which gives it one property worth relying on: every row you see in the feed is already in the file. The feed cannot show you a row that failed to write, cannot read ahead, and cannot synthesize one.

The consequence runs the other way too. A row can be in the file and not in your feed, if your viewer fell behind. So the feed is a sample of the ledger rather than a second copy of it, and it is not the thing to count from.

Alongside the rows, the stream carries a periodic health event. That event is where declog_dropped_total surfaces outside the metrics endpoint, and it is what the byte-fidelity indicator reads: it is green when the fail-open counter and the passthrough-error counter are both zero, and those numbers come from the running proxy rather than from a constant.

What it costs

Two bounded resources, both configurable.

A queue per viewer. Each connected browser gets a buffer of client_queue events. Rows are handed to it without blocking; when the buffer is full the event is discarded for that viewer, counted on tokentriage_dash_stream_dropped_total, and disclosed to that viewer as an in-stream dropped notice. A viewer that cannot keep up degrades visibly rather than slowing anything down.

A cap on viewers. max_clients bounds concurrent connections, and a connection past the cap is turned away with a message saying so rather than being accepted and starved.

Neither one can reach your traffic. The tee runs after the write, it cannot block the writer, and a dead subscriber is dropped rather than backpressuring anything.

Reconnecting without gaps or duplicates

The stream supports Last-Event-ID. Each row event’s id is the row’s coordinate in the ledger file, so a viewer that drops its connection reconnects with the last id it saw and the server replays the byte range after it, straight out of the file. A row that arrives live during the replay is de-duplicated against the replayed range rather than shown twice.

One case cannot be replayed exactly, and it says so instead of pretending. If the log rotated while you were disconnected, the coordinate names a file that is now compressed and has no seekable offset, so the server emits a notice describing the gap rather than silently resuming from the new file’s start.

What can go wrong

The surface is missing. The dashboard is off, or the stream is not wired into this build. A route that exists without its feature answers with a message naming the cure rather than a bare 404.

Rows appear with no tenant or agent. The feed shows what is on the row. Empty attribution means the headers did not arrive, not that the feed dropped them; check identity on the wire.

Nothing appears at all and traffic is flowing. Requests are reaching a different replica, or the dashboard is reading a different decision-log path from the one the proxy is writing. Both paths are in the same config file, so compare them directly.

Why the feed is not a substitute for the metrics endpoint

They fail differently and that is the point. The feed is per-viewer, unbounded in what it shows and bounded in how much it can buffer, so it degrades under load exactly when you most want to be watching. The counters are aggregated, cheap, and scraped on somebody else’s schedule.

Use the feed for the first five minutes after a change, when you want to see individual rows and you know what you are looking for. Use the metrics for anything you want to be told about while you are asleep.