# What a drop means

Four counters say something was thrown away. Only one can change a spend total, and the ledger cannot tell you about that one.

Source: https://tokentriage.com/docs/build/observability-drops

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

Four counters in the catalogue end in `_dropped_total`. Their presence is a promise and a warning
at once: nothing is silently discarded, and something can be discarded.

The promise is short. Every writer that could block a request instead has a bounded queue,
and when the queue is full it discards the item and adds one to a counter rather than making your
traffic wait. A logging problem cannot become a latency problem, and it cannot become a fabricated
row either.

The warning takes longer, because the four drops cost four different things, and only one of them
can change a number somebody bills from.

## What each one loses

| Counter | Full queue | What is lost | Can it move a spend total |
|---|---|---|---|
| `tokentriage_declog_dropped_total` | the ledger writer, 8,192 rows by default | the whole row | yes |
| `tokentriage_otlp_dropped_total` | the span-export batch, 4,096 rows | one span at your collector | no |
| `tokentriage_dash_stream_dropped_total` | one Live-feed viewer's queue | one event in one browser | no |
| `tokentriage_dims_dropped_total` | not a queue; a bound | one tag, or one metric label | no |

Read the last column first. Three of the four lose a *mirror* of something the ledger still holds,
so the row, its dollar and every total computed from it are unaffected. You lose a span in a trace
view, a line in a live tail, or a series on a graph, and you can recover all three by re-reading
the ledger.

The first one is different, and the rest of this page is about it.

## The one that matters

The decision ledger is the record. Cost totals, rollups, exports, savings reports and the
dashboard all read it. A row that was dropped was never appended, so it is not in the file, and
every figure computed over the file excludes it.

This is the part worth being precise about, because it is the one place in the product where a
total is incomplete and cannot say so. Everywhere else, an exclusion travels with the number: an
unpriced row is counted and disclosed under the total it is missing from, and a query response
carries the rows it excluded. A dropped row leaves nothing behind in the file to count. The
absence is only visible from outside.

<Callout type="key">
The ledger cannot disclose a dropped row, because the row is what the disclosure would have been
written on. `tokentriage_declog_dropped_total` is the disclosure, and it lives on the metrics
endpoint. A deployment that scrapes nothing has no way to learn that its spend figures are short.
</Callout>

The same count is also on the dashboard's health event as `declog_dropped_total`, so a Live feed
you are watching carries it without Prometheus. Two places, one number, and both are outside the
file.

## When it actually happens

A drop needs the writer to fall 8,192 records behind. The writer marshals one line and appends it
with no sync, so it keeps up with a great deal of traffic; the realistic causes are a stalled
disk, a filesystem that has filled, or a sustained burst far above what the volume can absorb.

```yaml
ledger:
  decision_log:
    path: ~/.tokentriage/decisions.jsonl
    queue: 8192 # bounded writer queue; overflow drops and counts
    rotate_mb: 256
```

Raising `queue` buys a longer burst at the cost of memory, and it does not help a writer that is
permanently slower than the traffic. If the counter moves at all, treat it as a storage problem
rather than a tuning problem, and alert on it: the chart's `TokenTriageDeclogDrops` rule fires on
`rate(tokentriage_declog_dropped_total[5m]) > 0`, which is to say on any drop at all, because the
correct number of dropped rows is zero.

## The two exclusions that are not drops

Two more things leave a surface without incrementing a drop counter, and both surprise people.

**Rows with no trace id are not exported.** The span exporter derives an OTLP trace id from the
row's `trace.trace_id`, and a row that carried no `x-tt-trace-id` has none to derive from, so it
is skipped. This is not counted on `tokentriage_otlp_dropped_total`, which counts queue overflow
only. In practice it means your collector sees the instrumented share of your traffic and your
ledger sees all of it, and the two totals differ by however much of your traffic is uninstrumented.
In the lab run behind these pages, 66 requests produced 5 spans.

**A restart resets every counter.** Prometheus counters live in the process. After a restart a
total starts from zero while the ledger file keeps every row it already had, so a ledger total and
a counter total diverge permanently and neither is wrong. Use `rate()` and `increase()` rather
than raw totals when you compare the two.

## What to do about each one

**`declog_dropped_total` moved.** Investigate storage. The rows are gone; no rebuild recovers
them. Note the window in whatever you report from that period, because the totals across it are
short by an amount you know the count of and not the value of.

**`otlp_dropped_total` moved.** Your collector is slow, unreachable, or rejecting batches. The
ledger is intact, so this is a trace-view gap rather than a data-loss event. Check the collector
before touching the queue.

**`dash_stream_dropped_total` moved.** A Live-feed viewer could not keep up, usually a browser tab
left open on a busy instance. Nothing is wrong with your data. Reload the tab.

**`dims_dropped_total` moved.** Either a request sent more tags than `max_tags`, or a promoted
dimension exceeded fifty distinct values. [What a promoted dimension
costs](/docs/build/observability-cardinality) covers both, and neither loses a dollar.

<Details summary="Why the writer drops rather than blocking, and what the alternative would cost">

The alternative is backpressure: hold the request until the row can be written. That makes the
ledger complete, and it also makes disk latency into request latency and a full disk into an
outage. A gateway that stops serving traffic because it cannot write a log line has chosen the
wrong failure.

The drop is the other choice, and it is only defensible because the drop is counted. An
uncounted drop would give you an incomplete ledger that looks complete, which is worse than
either alternative.

The same discipline runs through the other three queues. Each is bounded, each is non-blocking,
and each counts. The span exporter and the Live feed additionally sit behind the ledger write, so
they only ever see rows that were successfully appended, which is why a row can be missing from
your collector while being present in the file, and never the other way round.

</Details>

<NextSteps items={[
  {title: "Scrape the metrics endpoint", href: "/docs/build/observability-metrics", description: "Get the drop counters somewhere you will see them, and load the alert that fires on the first one."},
  {title: "The metrics catalogue", href: "/docs/build/observability-metrics-reference", description: "The four counters with their exact names and types, alongside the rest of the catalogue."},
  {title: "Export spans to your collector", href: "/docs/build/observability-otlp", description: "The export queue, what reaches a collector, and why untraced rows never do."},
  {title: "What a cost figure means", href: "/docs/build/cost-figures", description: "The other kind of incompleteness: unpriced rows, which are excluded from a total and disclosed under it."},
]} />