What a drop means

Four counters in the catalogue end in _dropped_total. Their presence is a promise and a warning at once: nothing is silently discarded, and something can be discarded.

The promise is short. Every writer that could block a request instead has a bounded queue, and when the queue is full it discards the item and adds one to a counter rather than making your traffic wait. A logging problem cannot become a latency problem, and it cannot become a fabricated row either.

The warning takes longer, because the four drops cost four different things, and only one of them can change a number somebody bills from.

What each one loses

Counter	Full queue	What is lost	Can it move a spend total
tokentriage_declog_dropped_total	the ledger writer, 8,192 rows by default	the whole row	yes
tokentriage_otlp_dropped_total	the span-export batch, 4,096 rows	one span at your collector	no
tokentriage_dash_stream_dropped_total	one Live-feed viewer’s queue	one event in one browser	no
tokentriage_dims_dropped_total	not a queue; a bound	one tag, or one metric label	no

Read the last column first. Three of the four lose a mirror of something the ledger still holds, so the row, its dollar and every total computed from it are unaffected. You lose a span in a trace view, a line in a live tail, or a series on a graph, and you can recover all three by re-reading the ledger.

The first one is different, and the rest of this page is about it.

The one that matters

The decision ledger is the record. Cost totals, rollups, exports, savings reports and the dashboard all read it. A row that was dropped was never appended, so it is not in the file, and every figure computed over the file excludes it.

This is the part worth being precise about, because it is the one place in the product where a total is incomplete and cannot say so. Everywhere else, an exclusion travels with the number: an unpriced row is counted and disclosed under the total it is missing from, and a query response carries the rows it excluded. A dropped row leaves nothing behind in the file to count. The absence is only visible from outside.

The same count is also on the dashboard’s health event as declog_dropped_total, so a Live feed you are watching carries it without Prometheus. Two places, one number, and both are outside the file.

When it actually happens

A drop needs the writer to fall 8,192 records behind. The writer marshals one line and appends it with no sync, so it keeps up with a great deal of traffic; the realistic causes are a stalled disk, a filesystem that has filled, or a sustained burst far above what the volume can absorb.

ledger:
  decision_log:
    path: ~/.tokentriage/decisions.jsonl
    queue: 8192 # bounded writer queue; overflow drops and counts
    rotate_mb: 256

Raising queue buys a longer burst at the cost of memory, and it does not help a writer that is permanently slower than the traffic. If the counter moves at all, treat it as a storage problem rather than a tuning problem, and alert on it: the chart’s TokenTriageDeclogDrops rule fires on rate(tokentriage_declog_dropped_total[5m]) > 0, which is to say on any drop at all, because the correct number of dropped rows is zero.

The two exclusions that are not drops

Two more things leave a surface without incrementing a drop counter, and both surprise people.

Rows with no trace id are not exported. The span exporter derives an OTLP trace id from the row’s trace.trace_id, and a row that carried no x-tt-trace-id has none to derive from, so it is skipped. This is not counted on tokentriage_otlp_dropped_total, which counts queue overflow only. In practice it means your collector sees the instrumented share of your traffic and your ledger sees all of it, and the two totals differ by however much of your traffic is uninstrumented. In the lab run behind these pages, 66 requests produced 5 spans.

A restart resets every counter. Prometheus counters live in the process. After a restart a total starts from zero while the ledger file keeps every row it already had, so a ledger total and a counter total diverge permanently and neither is wrong. Use rate() and increase() rather than raw totals when you compare the two.

What to do about each one

declog_dropped_total moved. Investigate storage. The rows are gone; no rebuild recovers them. Note the window in whatever you report from that period, because the totals across it are short by an amount you know the count of and not the value of.

otlp_dropped_total moved. Your collector is slow, unreachable, or rejecting batches. The ledger is intact, so this is a trace-view gap rather than a data-loss event. Check the collector before touching the queue.

dash_stream_dropped_total moved. A Live-feed viewer could not keep up, usually a browser tab left open on a busy instance. Nothing is wrong with your data. Reload the tab.

dims_dropped_total moved. Either a request sent more tags than max_tags, or a promoted dimension exceeded fifty distinct values. What a promoted dimension costs covers both, and neither loses a dollar.

Why the writer drops rather than blocking, and what the alternative would cost

The alternative is backpressure: hold the request until the row can be written. That makes the ledger complete, and it also makes disk latency into request latency and a full disk into an outage. A gateway that stops serving traffic because it cannot write a log line has chosen the wrong failure.

The drop is the other choice, and it is only defensible because the drop is counted. An uncounted drop would give you an incomplete ledger that looks complete, which is worse than either alternative.

The same discipline runs through the other three queues. Each is bounded, each is non-blocking, and each counts. The span exporter and the Live feed additionally sit behind the ledger write, so they only ever see rows that were successfully appended, which is why a row can be missing from your collector while being present in the file, and never the other way round.