# What a promoted dimension costs
Turning a tenant into a Prometheus label is one line of config. The two guards that stop it melting your Prometheus, and how to choose.
Source: https://tokentriage.com/docs/build/observability-cardinality
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import DimensionPromotionDiagram from '../../../components/DimensionPromotionDiagram.astro';
import NextSteps from '../../../components/NextSteps.astro';
Every distinct combination of label values on a Prometheus metric is a separate time series with
its own memory, its own index entries and its own cost at query time. Ten tenants is ten series.
A hundred thousand user ids is a hundred thousand series, and it is the classic way to take an
observability stack down with a config change that looked like one word.
TokenTriage records your business dimensions on every ledger row, where the number of distinct
values costs nothing. Turning one of them into a Prometheus label is a separate, opt-in decision,
and this page is about making it well.
## Why the ledger and the metrics disagree on purpose
The two surfaces have opposite economics, and the design leans into it rather than splitting the
difference.
A ledger row is a line in a file. Writing a hundred thousand distinct user ids across a hundred
thousand rows costs a hundred thousand rows, which is what a hundred thousand requests costs
anyway. Querying it is a scan, and a scan is slow rather than fatal.
A Prometheus label value is a permanent series. Each new value adds a series that stays resident
for the retention window, whether or not it is ever queried, and the cost lands on the monitoring
system rather than on the writer.
So the rule is: put unbounded things on the row, and put only enumerable things on a label. That
is why `agent` is not a metric label anywhere in the catalogue, even though it would be a
convenient one. Agent names come from whatever your code passes, so the set has no bound, and
agent-level analysis lives in the ledger where the cost of a distinct value is a row.
## The two guards
Read it left to right. The dimension value goes to the ledger row on the upper path
unconditionally. Promotion runs on the lower path and passes two checks.
**Guard one is the name.** `ledger.dimensions.promote_to_metrics` is an allowlist of dimension
names, and it is empty out of the box, so nothing is promoted until you say so. A dimension not on
the list is recorded on the row and is not a label, with no way to override that per request. The
list itself takes at most sixteen entries.
**Guard two is the value.** Each promoted dimension admits at most fifty distinct values as label
series. The fifty-first new value is turned away, counted, and not minted, so a dimension you
allowlisted by mistake costs you fifty series and a counter rather than an incident. The cap is
fixed in the build rather than configurable, which is what makes it a guard rather than a setting
somebody raises under pressure.
Names on the allowlist are the short forms, matching what lands on the row: `tenant`, `feature`,
`user`, or a tag key. They are not the `dim.`-prefixed field names a query uses. A tag key that
collides with one of the three first-class names is ignored for promotion rather than allowed to
win, so a promoted label cannot disagree with the row it came from.
```yaml
ledger:
dimensions:
enabled: true # off by default; see the dimensions how-to
promote_to_metrics: [tenant] # allowlist; empty by default, at most 16 entries
```
Only `promote_to_metrics` belongs to this page. The flag and the headers that populate a dimension
in the first place are [turn on business dimensions](/docs/build/cost-dimensions)'s.
## Both guards, on real traffic
A lab run against a mock upstream sent requests carrying sixty-one distinct `x-tt-tenant` values,
with `promote_to_metrics: [tenant]`, plus one request carrying nine tags against a `max_tags` of
eight. The scrape afterwards:
```text
tokentriage_dim_requests_total{dim="tenant",value="acme-corp"} 4
tokentriage_dims_dropped_total{reason="max_tags"} 1
tokentriage_dims_dropped_total{reason="promote_cardinality"} 11
```
Counting the `tokentriage_dim_requests_total` series gives exactly 50. Sixty-one distinct tenants
arrived, fifty became series, eleven were turned away and counted, and one tag past the budget was
turned away and counted separately.
The two dropped numbers cost different things, and neither costs a dollar.
The eleven lost a **label**. Those requests are on the ledger with their tenant recorded, their
spend is in `tokentriage_cost_usd_total`, and a query grouped by `dim.tenant` returns them. What
they cannot do is appear as a Prometheus series.
The one lost a **tag**. That tag key is not on its row, so a query grouping by it will not find
it. The rest of the row, including its cost and its other eight tags, is intact. Tags are admitted
in sorted key order, so the tag that disappears is deterministic rather than arbitrary: it is the
last one alphabetically.
A dropped label costs you a chart. A dropped tag costs you one field on one row. Neither loses
money from a total, because the cost is on the row either way, and the ledger is what the spend
figures are computed from.
## The one unbounded label in the catalogue
Promotion is the guarded path. There is one label that is not guarded, and it is worth knowing
about before somebody points a scanner at your data plane.
`tokentriage_requests_total` carries a `path` label, and its value is the request path as the
client sent it. Every distinct path mints a series, including paths the proxy does not serve.
Driving four unknown paths at a lab instance took the metric from one series to five:
```text
tokentriage_requests_total{dialect="",mode="ledger",path="/v1/nope",status="404"} 1
tokentriage_requests_total{dialect="openai",mode="ledger",path="/v1/chat/completions",status="200"} 66
```
In normal traffic this is bounded by the handful of endpoints your clients call, which is why it
has not been a problem. It is not bounded by anything in the build, so a data plane reachable by
untrusted callers can grow the series count with 404s.
Two mitigations, neither of which needs a code change. Keep the data plane behind whatever already
fronts it, so unknown paths are turned away before they reach the proxy. And if you cannot, drop
or rewrite the label at scrape time with `metric_relabel_configs`, which is the standard Prometheus
answer to a label you cannot fix at the source. This is filed.
## Choosing what to promote
Ask one question: can you write down every value it will ever take?
**Promote** environment, plan, region, or a tenant list you can name. Tens of values, changing
when you change something, and useful on a graph because you want to see them side by side over
time.
**Do not promote** user, session, request or trace ids, or a tag whose value comes from user
input. These are exactly what the ledger is for.
**Think about `tenant`.** It is the common case and the answer depends on your business. Tens of
tenants is a good label. Thousands is not, and the guard will hold at fifty while your chart
quietly stops representing your fleet. If you have thousands of tenants, group them by plan and
promote that.
**Think about `feature`.** Usually a good label, because features are things you ship rather than
things users create, so the set is bounded by your own roadmap.
Admission is first-come rather than most-valuable. The first fifty distinct values seen for a
dimension since the process started become series, and the rest are counted. There is no eviction
and no ranking, so a restart re-races the set.
That is a trade in favour of predictability: an evicting cache would keep the series count bounded
too, and it would also make a graph's membership change under you for reasons unrelated to your
traffic. A fixed set plus a visible counter is easier to reason about at three in the morning.
If `tokentriage_dims_dropped_total{reason="promote_cardinality"}` is climbing, the promoted
dimension is the wrong one. Take it off the allowlist and query the ledger instead; nothing is
lost by doing so, because the row always had the value.