Quality on the wire
The exporter emits evaluation results as well as cost spans, and a score that cannot be stated arrives with a reason instead of a number.
On this page
A token count is a fact about the world. A quality score is the output of a method, and the method decides whether the number may be averaged, thresholded or alerted on. Two evaluation results that look identical on a dashboard can be a measurement against ground truth and an unvalidated model’s opinion, and averaging them together produces something that means nothing.
The span exporter carries quality as well as cost, and the attributes it puts on the wire exist to make that distinction survive the trip into your monitoring backend.
What gets exported besides cost
Enabling export.otlp.enabled turns on two producers, not one.
The first is the decision span, one per costed ledger row, which the attribute mapping documents.
The second emits gen_ai.evaluation.result, the OpenTelemetry GenAI convention’s evaluation
event, from two sources:
| Subject | What it is |
|---|---|
eval_outcome |
a result measured after the fact by an evaluator, a human, or a business signal |
served_estimate |
the quality estimate the router was handed at decision time |
Keeping them apart matters because they are different facts about the same request. One is what was later found to be true; the other is what was believed when the choice was made. Summing them together double-counts.
Each event is carried on its own span, kind SPAN_KIND_INTERNAL, parented to the decision span
for the request it grades, because an evaluation is an observation about a call rather than a call
of its own. When the trace id is unknown the event still joins through gen_ai.response.id, which
carries the request id, and the span is emitted without a parent rather than pointing at a trace
it does not belong to.
Three bases, and what each licenses
Every event carries tokentriage.quality.basis from a closed set of three values. It is appended
unconditionally, so there is no event on the wire without one.
| Basis | Means | What you may do with it |
|---|---|---|
measured |
an evaluator ran against ground truth, or a proxy validated against human labels | average it, threshold it, route on it |
reported |
an unvalidated proxy: a judge score, a model’s self-assessment, a heuristic | read it, watch it move, do not treat it as ground truth |
insufficient |
not enough evidence to state a score at all | count it; there is no number to use |
The reason a bare score is not enough is the same reason a bare dollar is not enough. A backend that receives two evaluation results cannot legitimately average them without knowing whether it is averaging two measurements, two guesses, or one of each, so without the basis it averages them anyway.
A score that cannot be stated arrives with a reason
An insufficient estimate is emitted with no score attribute under any spelling. Not zero, not
blank, not omitted-and-unexplained. What travels instead is the reason, in three places a
different kind of consumer will look:
error.typecarries the closed reason code, which is the convention’s own slot for an evaluation that could not be computed.tokentriage.quality.insufficient.reason,.needand.havecarry the typed version: what was required, and what there was.gen_ai.evaluation.explanationrenders the whole thing as one deterministic sentence, because free text is the only standard slot that can carry a basis to a consumer reading nothing but the convention.
tokentriage.quality.n is emitted on every event, including n: 0, so “no evidence” and “not
disclosed” stay distinguishable.
An interval travels with its calibration or not at all
A confidence interval is a probability statement, and a probability statement is worth what the fit behind it is worth. So the interval attributes are emitted as a group that travels whole:
tokentriage.quality.interval.lo
tokentriage.quality.interval.hi
tokentriage.quality.interval.confidence
tokentriage.quality.calibration.artifact_id
tokentriage.quality.calibration.method
tokentriage.quality.calibration.error
tokentriage.quality.calibration.error_kind
tokentriage.quality.calibration.cal_n
tokentriage.quality.calibration.holdout_m
tokentriage.quality.calibration.fitted_at
tokentriage.quality.calibration.fingerprintAn estimate whose interval is not licensed by a calibration artifact emits no interval, and an
estimate whose basis is reported emits none either, so a proxy score does not arrive dressed as
a probability. confidence is the nominal coverage the artifact was fitted at; the measured
deviation is calibration.error, with error_kind saying which quantity was measured, because an
error without its kind cannot be interpreted.
holdout_m is the one that looks skippable. Without it, a producer that ran no holdout reports a
calibration error at its language’s zero value and thereby claims perfect calibration from the
least possible evidence. An error of 0.03 is a failure at a holdout of 2,000 and inside the
noise at a holdout of 25.
A basis that changes is an event, not an attribute
Everything above rides the span exporter and describes one evaluation at the instant it was made. A basis is also a thing that changes, and a change has no span to sit on. Two entries in the dashboard’s closed event vocabulary carry those changes, and both are webhook types you can subscribe to as well.
| Event | What it records |
|---|---|
quality.judge.promoted |
a judge subject cleared the validation gates, so its outcomes may carry a measured basis from that change onward |
quality.judge.demoted |
a standing promotion for a subject was withdrawn, carrying the previous statistic, the new one, the floor it fell under and the refusals that decided it |
Read the second with its producer in view, because the name suggests a watcher and the watcher is
the part that is missing. What withdraws a promotion in this build is a re-run of the validation
check: point tokentriage evals validate at a subject whose promotion still stands at the same
version, and a record falling under the agreement floor withdraws it, emits the notice and revokes
that subject’s accept region in the same write. Automatic post-promotion monitoring is planned
rather than shipped. Gate.ObserveAgreement has no call site outside its tests,
so no shipped binary can fire a drift demotion,
and nothing arrives on this event between the runs you start yourself.
How a judge reaches a measured basis is the walk both events report on.
One property matters if you are wiring an alert to the demotion event: the notice is handed to the
event sink at the moment it happens and is not retained anywhere, so the delivery you receive is
the copy. What persists is thinner. The subject’s history keeps a demoted entry on its timeline
with the statistic and the N behind it, and the recalibration log keeps the rest.
Where this extends the convention, and why
The convention’s evaluation event defines a name, a score value, an optional label and free-form explanation. It defines no attribute for how the score was obtained and none for its uncertainty, so an LLM judge’s guess, a human label and an execution-verified result arrive as the same double.
TokenTriage’s maintainers filed a proposal to add both: a score.basis from a closed set, a
sample size, an interval group that travels with a named calibration, and a reason attribute for
an evaluation that states no score. Until something lands, the typed facts travel in the
tokentriage.quality.* namespace and the convention slots carry what they can.
One attribute is dual-written under a deprecated spelling as well as the current one. That is
routine for a pre-stable convention, and one of the two aliases in the table deserves a note: the
exporter emits gen_ai.evaluation.score alongside gen_ai.evaluation.score.value, and the source
marks that alias as unverified against the registry. Checking the published registry in August
2026 found gen_ai.evaluation.score.value and gen_ai.evaluation.score.label with no
gen_ai.evaluation.score deprecation listed, so treat the extra attribute as harmless duplication
rather than as a convention key.
What happens to an event that cannot be emitted honestly
The emitter takes a typed quality value rather than a number, and runs it through its own rules before it builds a single attribute. An estimate with no basis, an interval with no licence, or a stated refusal that also carries a score is turned away at that gate and counted, and no partial event goes out.
That is a stronger guarantee than validating attributes on the way out, because there is no exported function in the package that accepts a bare score. A caller holding loose numbers has nothing to pass.
An event turned away stays in the ledger. A later re-export can carry it once the estimate is complete.