What a promoted dimension costs

Every distinct combination of label values on a Prometheus metric is a separate time series with its own memory, its own index entries and its own cost at query time. Ten tenants is ten series. A hundred thousand user ids is a hundred thousand series, and it is the classic way to take an observability stack down with a config change that looked like one word.

TokenTriage records your business dimensions on every ledger row, where the number of distinct values costs nothing. Turning one of them into a Prometheus label is a separate, opt-in decision, and this page is about making it well.

Why the ledger and the metrics disagree on purpose

The two surfaces have opposite economics, and the design leans into it rather than splitting the difference.

A ledger row is a line in a file. Writing a hundred thousand distinct user ids across a hundred thousand rows costs a hundred thousand rows, which is what a hundred thousand requests costs anyway. Querying it is a scan, and a scan is slow rather than fatal.

A Prometheus label value is a permanent series. Each new value adds a series that stays resident for the retention window, whether or not it is ever queried, and the cost lands on the monitoring system rather than on the writer.

So the rule is: put unbounded things on the row, and put only enumerable things on a label. That is why agent is not a metric label anywhere in the catalogue, even though it would be a convenient one. Agent names come from whatever your code passes, so the set has no bound, and agent-level analysis lives in the ledger where the cost of a distinct value is a row.

The two guards

Read it left to right. The dimension value goes to the ledger row on the upper path unconditionally. Promotion runs on the lower path and passes two checks.

Guard one is the name. ledger.dimensions.promote_to_metrics is an allowlist of dimension names, and it is empty out of the box, so nothing is promoted until you say so. A dimension not on the list is recorded on the row and is not a label, with no way to override that per request. The list itself takes at most sixteen entries.

Guard two is the value. Each promoted dimension admits at most fifty distinct values as label series. The fifty-first new value is turned away, counted, and not minted, so a dimension you allowlisted by mistake costs you fifty series and a counter rather than an incident. The cap is fixed in the build rather than configurable, which is what makes it a guard rather than a setting somebody raises under pressure.

Names on the allowlist are the short forms, matching what lands on the row: tenant, feature, user, or a tag key. They are not the dim.-prefixed field names a query uses. A tag key that collides with one of the three first-class names is ignored for promotion rather than allowed to win, so a promoted label cannot disagree with the row it came from.

ledger:
  dimensions:
    enabled: true # off by default; see the dimensions how-to
    promote_to_metrics: [tenant] # allowlist; empty by default, at most 16 entries

Only promote_to_metrics belongs to this page. The flag and the headers that populate a dimension in the first place are turn on business dimensions’s.

Both guards, on real traffic

A lab run against a mock upstream sent requests carrying sixty-one distinct x-tt-tenant values, with promote_to_metrics: [tenant], plus one request carrying nine tags against a max_tags of eight. The scrape afterwards:

tokentriage_dim_requests_total{dim="tenant",value="acme-corp"} 4
tokentriage_dims_dropped_total{reason="max_tags"} 1
tokentriage_dims_dropped_total{reason="promote_cardinality"} 11

Counting the tokentriage_dim_requests_total series gives exactly 50. Sixty-one distinct tenants arrived, fifty became series, eleven were turned away and counted, and one tag past the budget was turned away and counted separately.

The two dropped numbers cost different things, and neither costs a dollar.

The eleven lost a label. Those requests are on the ledger with their tenant recorded, their spend is in tokentriage_cost_usd_total, and a query grouped by dim.tenant returns them. What they cannot do is appear as a Prometheus series.

The one lost a tag. That tag key is not on its row, so a query grouping by it will not find it. The rest of the row, including its cost and its other eight tags, is intact. Tags are admitted in sorted key order, so the tag that disappears is deterministic rather than arbitrary: it is the last one alphabetically.

The one unbounded label in the catalogue

Promotion is the guarded path. There is one label that is not guarded, and it is worth knowing about before somebody points a scanner at your data plane.

tokentriage_requests_total carries a path label, and its value is the request path as the client sent it. Every distinct path mints a series, including paths the proxy does not serve. Driving four unknown paths at a lab instance took the metric from one series to five:

tokentriage_requests_total{dialect="",mode="ledger",path="/v1/nope",status="404"} 1
tokentriage_requests_total{dialect="openai",mode="ledger",path="/v1/chat/completions",status="200"} 66

In normal traffic this is bounded by the handful of endpoints your clients call, which is why it has not been a problem. It is not bounded by anything in the build, so a data plane reachable by untrusted callers can grow the series count with 404s.

Two mitigations, neither of which needs a code change. Keep the data plane behind whatever already fronts it, so unknown paths are turned away before they reach the proxy. And if you cannot, drop or rewrite the label at scrape time with metric_relabel_configs, which is the standard Prometheus answer to a label you cannot fix at the source. This is filed.

Choosing what to promote

Ask one question: can you write down every value it will ever take?

Promote environment, plan, region, or a tenant list you can name. Tens of values, changing when you change something, and useful on a graph because you want to see them side by side over time.

Do not promote user, session, request or trace ids, or a tag whose value comes from user input. These are exactly what the ledger is for.

Think about tenant. It is the common case and the answer depends on your business. Tens of tenants is a good label. Thousands is not, and the guard will hold at fifty while your chart quietly stops representing your fleet. If you have thousands of tenants, group them by plan and promote that.

Think about feature. Usually a good label, because features are things you ship rather than things users create, so the set is bounded by your own roadmap.

What happens to the fifty-first value

Admission is first-come rather than most-valuable. The first fifty distinct values seen for a dimension since the process started become series, and the rest are counted. There is no eviction and no ranking, so a restart re-races the set.

That is a trade in favour of predictability: an evicting cache would keep the series count bounded too, and it would also make a graph’s membership change under you for reasons unrelated to your traffic. A fixed set plus a visible counter is easier to reason about at three in the morning.

If tokentriage_dims_dropped_total{reason="promote_cardinality"} is climbing, the promoted dimension is the wrong one. Take it off the allowlist and query the ledger instead; nothing is lost by doing so, because the row always had the value.