Features at a glance

Use this page to decide which capabilities you need and what will change when you enable them. TokenTriage starts as a passive cost ledger: it forwards requests to your existing model providers, records token usage, and estimates the cost of each request.

Routing, caching, guardrails, budgets, analytics, and cross-provider translation stay off until you enable them. If you are new to model gateways, start with What is TokenTriage?.

What happens on first start

A fresh installation records traffic without making product decisions for you. It does not change the requested model, reuse an earlier answer, inspect content, or enforce a spending limit.

capability	default	what changes when you enable it
request forwarding	on	sends supported API requests to their configured provider
cost ledger	on in ledger mode	records token usage and estimated cost for each request
routing	off	can choose a different model or provider for a request
response caching	off	can return a stored answer instead of calling the provider
governance	off	can require virtual keys and enforce budgets, quotas, and rate limits
guardrails	off	can inspect prompts and responses for sensitive or unsafe content
analytics and anomaly detection	off	can produce rollups, forecasts, and unusual-spend alerts
cross-provider translation	off	can translate OpenAI- or Anthropic-formatted requests for another provider
trace export	off	can send OpenTelemetry traces to your observability backend
dashboard	on in zero-config quickstarts; otherwise off	serves the local web interface
internal processing errors	fail open	forwards the original request instead of losing a provider response

The exact configuration keys and their defaults are in the configuration reference.

Application traffic enters on 127.0.0.1:8787. The dashboard, metrics, health checks, and management API use a separate admin port on 127.0.0.1:9090, outside the request path. Some management endpoints change state, including endpoints that purge cache entries or decommission a fleet member.

Understand usage and cost

Cost ledger and attribution

The cost ledger gives each request a local record you can trace and group. It stores token counts, pricing status, and estimated cost in ~/.tokentriage/decisions.jsonl. Prices come from a table compiled into the binary, so pricing does not require a network call.

When TokenTriage cannot price a model, it marks the request unpriced. It excludes that request from cost totals instead of treating the missing price as $0.

Add dimensions for a user, tenant, feature, or agent step when you need to attribute shared provider spending. The cost analytics guide explains the workflow, and the ledger field reference defines every stored value.

Starts enabled. Ledger mode records and prices requests without choosing a different model.

Dashboard and analytics

The dashboard is a local web interface for exploring requests, cost, and operational status. Its data stays on the host, and its SQLite index can be discarded and rebuilt from the ledger.

Analytics can roll ledger records up, forecast spending, and flag unusual changes. Anomaly detection also feeds alert rules. Both analytics and anomaly recommendations are opt-in.

The dashboard guide covers setup and access. The anomaly detection guide explains what the detector looks for.

Reduce cost safely

Routing

Routing sends a request to a cheaper or different model when a rule matches. Rules can consider the requested model, whether tools are present, the calling agent, request headers, and signals from the built-in complexity classifier.

You can use an ordered rule list or the policy expression language. In ledger mode, neither one changes traffic. The routing overview explains how to preview and introduce a policy, and the policy language reference covers its syntax.

Starts off. Routing begins only after you move beyond ledger mode.

Response caching

Response caching avoids paying for the same answer twice. It can match an identical request or a semantically similar one, depending on the rule you choose.

The cache can use memory, a local SQLite file, or an optional Redis service. Redis requires explicit permission for external network access. TokenTriage reports cache savings separately from routing savings so the same avoided cost is not counted twice.

Starts off. See response caching for rollout guidance and cache accounting for the savings model.

Control and protect traffic

Governance and virtual keys

Governance shows which organization, team, or user is spending money and lets you cap future usage. You issue scoped virtual keys and attach limits at any level of that hierarchy. Available controls include dollar budgets, token and request quotas, rate limits, and concurrency limits.

TokenTriage stores virtual keys as SHA-256 hashes rather than plaintext. You also decide whether anonymous traffic is allowed, optional, or rejected.

Starts off. The governance guide covers virtual keys, inheritance, and safe rollout.

Guardrails

Guardrails inspect content rather than cost. Built-in detectors cover areas such as personal data, secrets, prompt injection, toxicity, code, and multimodal input. Each detector declares whether it can run on input, output, streaming content, or a subset of those phases.

Local detectors share a time budget so inspection cannot stall a request indefinitely. Optional model-based evaluation can send content to another model and therefore requires explicit permission for that external call.

Starts off. Begin with the guardrails overview, which explains the available actions and how to test a policy before enforcing it.

Evaluate answer quality

Cost reduction only helps when the resulting answers remain good enough for their job. TokenTriage includes separate tools for tuning the router, running experiments, and checking automated judges against human labels.

• calibrate fits the complexity thresholds used by the router.
• quality estimates the likelihood of a satisfactory outcome and reports its reliability.
• evals manages datasets, runs experiments, and compares results.
• eval checks whether the router beat a baseline with no routing.

An automated judge can report results before it is trusted to influence routing. It must first meet its agreement threshold against human labels. TokenTriage does not automatically demote a judge when its quality later drifts, so you must choose and run a revalidation schedule.

Outcome ingestion from your own systems starts off. The calibration and evals overview covers the end-to-end workflow, and the LLM judges overview covers judge validation.

Connect and operate

Providers and request formats

TokenTriage includes 95 provider descriptors, but support is published per endpoint rather than as one blanket provider claim. The provider matrix distinguishes endpoints covered by an end-to-end mock test from endpoints that are only declared. Those tests use provider-shaped mocks, not live provider services.

Requests can use a provider’s native API format. Optional cross-provider translation accepts OpenAI- and Anthropic-formatted requests. You can also define a provider that is not built in. Configuration stores the name of a credential environment variable, not the credential itself.

Check the provider matrix and setup guide for the endpoint you plan to use.

SDKs

The Python and TypeScript SDKs add attribution headers and read the receipt returned by the proxy. They do not calculate cost on the client. Both are optional, and the proxy works without them.

See the SDK overview for installation and the shared behavior contract.

Observability

The admin port exposes health checks and Prometheus metrics for requests, tokens, cost, pricing misses, governance decisions, and proxy overhead. OpenTelemetry trace export and outcome ingestion are separate opt-in features.

See observability for metric definitions, trace setup, and data-loss signals.

Deployment and command line

TokenTriage ships as a single static binary and a minimal container image. The default ledger is a local JSONL file, so a single instance needs no external database. Deployment assets include Docker, Helm, Kustomize, and Terraform for AWS.

Replicas that share state use Redis and require explicit permission for that external connection. The deployment guide covers one instance, and scaling out covers shared state and multiple replicas.

The CLI starts and validates the gateway, diagnoses setup problems, previews policies, and manages features such as virtual keys and guardrails. The CLI reference lists every command and flag.

Known limitations

TokenTriage is beta software, and you run it yourself. It is not a hosted model service or a reseller of provider access.

• Provider coverage varies by endpoint. Some built-in descriptors have no end-to-end gated endpoint, so check the provider matrix before adopting one.
• Load balancing currently supports weighted pools, least-latency selection, and passive cooldown. It does not provide active health probing, hedged requests, consistent hashing, or configurable ordered fallback chains.
• Streaming guardrails are bounded and best-effort. A slow stream can deliver a short secret before a detector has enough context to identify it.
• Cross-provider translation is limited to OpenAI- and Anthropic-formatted requests.

The core ships under the TokenTriage commercial licence. See How TokenTriage compares for the fuller gap list and the trade-offs against hosted and broader provider gateways.