# Quality on the wire The exporter emits evaluation results as well as cost spans, and a score that cannot be stated arrives with a reason instead of a number. Source: https://tokentriage.com/docs/build/observability-quality-events --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; A token count is a fact about the world. A quality score is the output of a method, and the method decides whether the number may be averaged, thresholded or alerted on. Two evaluation results that look identical on a dashboard can be a measurement against ground truth and an unvalidated model's opinion, and averaging them together produces something that means nothing. The span exporter carries quality as well as cost, and the attributes it puts on the wire exist to make that distinction survive the trip into your monitoring backend. ## What gets exported besides cost Enabling `export.otlp.enabled` turns on two producers, not one. The first is the decision span, one per costed ledger row, which [the attribute mapping](/docs/build/observability-otel-mapping) documents. The second emits `gen_ai.evaluation.result`, the OpenTelemetry GenAI convention's evaluation event, from two sources: | Subject | What it is | |---|---| | `eval_outcome` | a result measured after the fact by an evaluator, a human, or a business signal | | `served_estimate` | the quality estimate the router was handed at decision time | Keeping them apart matters because they are different facts about the same request. One is what was later found to be true; the other is what was believed when the choice was made. Summing them together double-counts. Each event is carried on its own span, kind `SPAN_KIND_INTERNAL`, parented to the decision span for the request it grades, because an evaluation is an observation about a call rather than a call of its own. When the trace id is unknown the event still joins through `gen_ai.response.id`, which carries the request id, and the span is emitted without a parent rather than pointing at a trace it does not belong to. ## Three bases, and what each licenses Every event carries `tokentriage.quality.basis` from a closed set of three values. It is appended unconditionally, so there is no event on the wire without one. | Basis | Means | What you may do with it | |---|---|---| | `measured` | an evaluator ran against ground truth, or a proxy validated against human labels | average it, threshold it, route on it | | `reported` | an unvalidated proxy: a judge score, a model's self-assessment, a heuristic | read it, watch it move, do not treat it as ground truth | | `insufficient` | not enough evidence to state a score at all | count it; there is no number to use | The reason a bare score is not enough is the same reason a bare dollar is not enough. A backend that receives two evaluation results cannot legitimately average them without knowing whether it is averaging two measurements, two guesses, or one of each, so without the basis it averages them anyway. ## A score that cannot be stated arrives with a reason An `insufficient` estimate is emitted with no score attribute under any spelling. Not zero, not blank, not omitted-and-unexplained. What travels instead is the reason, in three places a different kind of consumer will look: - `error.type` carries the closed reason code, which is the convention's own slot for an evaluation that could not be computed. - `tokentriage.quality.insufficient.reason`, `.need` and `.have` carry the typed version: what was required, and what there was. - `gen_ai.evaluation.explanation` renders the whole thing as one deterministic sentence, because free text is the only standard slot that can carry a basis to a consumer reading nothing but the convention. Zero is the wrong answer here in a way it is not for cost. An unknown price sums low; an unknown quality score renders as the worst possible result, and it is the value a dashboard pages somebody about. A terrible output and an unprovable one are different facts. `tokentriage.quality.n` is emitted on every event, including `n: 0`, so "no evidence" and "not disclosed" stay distinguishable. ## An interval travels with its calibration or not at all A confidence interval is a probability statement, and a probability statement is worth what the fit behind it is worth. So the interval attributes are emitted as a group that travels whole: ```text tokentriage.quality.interval.lo tokentriage.quality.interval.hi tokentriage.quality.interval.confidence tokentriage.quality.calibration.artifact_id tokentriage.quality.calibration.method tokentriage.quality.calibration.error tokentriage.quality.calibration.error_kind tokentriage.quality.calibration.cal_n tokentriage.quality.calibration.holdout_m tokentriage.quality.calibration.fitted_at tokentriage.quality.calibration.fingerprint ``` An estimate whose interval is not licensed by a calibration artifact emits no interval, and an estimate whose basis is `reported` emits none either, so a proxy score does not arrive dressed as a probability. `confidence` is the nominal coverage the artifact was fitted at; the measured deviation is `calibration.error`, with `error_kind` saying which quantity was measured, because an error without its kind cannot be interpreted. `holdout_m` is the one that looks skippable. Without it, a producer that ran no holdout reports a calibration error at its language's zero value and thereby claims perfect calibration from the least possible evidence. An error of `0.03` is a failure at a holdout of 2,000 and inside the noise at a holdout of 25. ## A basis that changes is an event, not an attribute Everything above rides the span exporter and describes one evaluation at the instant it was made. A basis is also a thing that *changes*, and a change has no span to sit on. Two entries in the dashboard's closed event vocabulary carry those changes, and both are webhook types you can subscribe to as well. | Event | What it records | |---|---| | `quality.judge.promoted` | a judge subject cleared the validation gates, so its outcomes may carry a `measured` basis from that change onward | | `quality.judge.demoted` | a standing promotion for a subject was withdrawn, carrying the previous statistic, the new one, the floor it fell under and the refusals that decided it | Read the second with its producer in view, because the name suggests a watcher and the watcher is the part that is missing. What withdraws a promotion in this build is a **re-run of the validation check**: point `tokentriage evals validate` at a subject whose promotion still stands at the same version, and a record falling under the agreement floor withdraws it, emits the notice and revokes that subject's accept region in the same write. Automatic post-promotion monitoring is planned rather than shipped. `Gate.ObserveAgreement` has no call site outside its tests, so no shipped binary can fire a drift demotion, and nothing arrives on this event between the runs you start yourself. [How a judge reaches a measured basis](/docs/judges/agreement) is the walk both events report on. One property matters if you are wiring an alert to the demotion event: the notice is handed to the event sink at the moment it happens and is not retained anywhere, so the delivery you receive is the copy. What persists is thinner. The subject's history keeps a `demoted` entry on its timeline with the statistic and the N behind it, and the recalibration log keeps the rest. ## Where this extends the convention, and why The convention's evaluation event defines a name, a score value, an optional label and free-form explanation. It defines no attribute for how the score was obtained and none for its uncertainty, so an LLM judge's guess, a human label and an execution-verified result arrive as the same double. TokenTriage's maintainers filed a proposal to add both: a `score.basis` from a closed set, a sample size, an interval group that travels with a named calibration, and a reason attribute for an evaluation that states no score. Until something lands, the typed facts travel in the `tokentriage.quality.*` namespace and the convention slots carry what they can. One attribute is dual-written under a deprecated spelling as well as the current one. That is routine for a pre-stable convention, and one of the two aliases in the table deserves a note: the exporter emits `gen_ai.evaluation.score` alongside `gen_ai.evaluation.score.value`, and the source marks that alias as unverified against the registry. Checking the published registry in August 2026 found `gen_ai.evaluation.score.value` and `gen_ai.evaluation.score.label` with no `gen_ai.evaluation.score` deprecation listed, so treat the extra attribute as harmless duplication rather than as a convention key.
The emitter takes a typed quality value rather than a number, and runs it through its own rules before it builds a single attribute. An estimate with no basis, an interval with no licence, or a stated refusal that also carries a score is turned away at that gate and counted, and no partial event goes out. That is a stronger guarantee than validating attributes on the way out, because there is no exported function in the package that accepts a bare score. A caller holding loose numbers has nothing to pass. An event turned away stays in the ledger. A later re-export can carry it once the estimate is complete.