# Features at a glance

See what TokenTriage can do, what starts enabled, which capabilities you opt into, and where to learn more.

Source: https://tokentriage.com/docs/introduction/features

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import SystemMapDiagram from '../../../components/SystemMapDiagram.astro';

Use this page to decide which capabilities you need and what will change when you enable them.
TokenTriage starts as a passive cost ledger: it forwards requests to your existing model providers,
records token usage, and estimates the cost of each request.

Routing, caching, guardrails, budgets, analytics, and cross-provider translation stay off until you
enable them. If you are new to model gateways, start with
[What is TokenTriage?](/docs/introduction/what-is-tokentriage).

## What happens on first start

A fresh installation records traffic without making product decisions for you. It does not change
the requested model, reuse an earlier answer, inspect content, or enforce a spending limit.

| capability | default | what changes when you enable it |
|---|---|---|
| request forwarding | on | sends supported API requests to their configured provider |
| cost ledger | on in `ledger` mode | records token usage and estimated cost for each request |
| routing | off | can choose a different model or provider for a request |
| response caching | off | can return a stored answer instead of calling the provider |
| governance | off | can require virtual keys and enforce budgets, quotas, and rate limits |
| guardrails | off | can inspect prompts and responses for sensitive or unsafe content |
| analytics and anomaly detection | off | can produce rollups, forecasts, and unusual-spend alerts |
| cross-provider translation | off | can translate OpenAI- or Anthropic-formatted requests for another provider |
| trace export | off | can send OpenTelemetry traces to your observability backend |
| dashboard | on in zero-config quickstarts; otherwise off | serves the local web interface |
| internal processing errors | fail open | forwards the original request instead of losing a provider response |

The exact configuration keys and their defaults are in the
[configuration reference](/docs/reference/configuration).

<Callout type="warning">
Governance starts off, but a limit you add starts in enforcement mode. Use `mode: shadow` for a new
limit if you want to observe its decisions before it can reject traffic.
</Callout>

Application traffic enters on `127.0.0.1:8787`. The dashboard, metrics, health checks, and
management API use a separate admin port on `127.0.0.1:9090`, outside the request path. Some
management endpoints change state, including endpoints that purge cache entries or decommission a
fleet member.

<SystemMapDiagram />

## Understand usage and cost

### Cost ledger and attribution

The cost ledger gives each request a local record you can trace and group. It stores token counts,
pricing status, and estimated cost in `~/.tokentriage/decisions.jsonl`. Prices come from a table
compiled into the binary, so pricing does not require a network call.

When TokenTriage cannot price a model, it marks the request `unpriced`. It excludes that request
from cost totals instead of treating the missing price as `$0`.

Add dimensions for a user, tenant, feature, or agent step when you need to attribute shared provider
spending. The [cost analytics guide](/docs/build/cost-analytics) explains the workflow, and the
[ledger field reference](/docs/reference/ledger-fields) defines every stored value.

**Starts enabled.** Ledger mode records and prices requests without choosing a different model.

### Dashboard and analytics

The dashboard is a local web interface for exploring requests, cost, and operational status. Its
data stays on the host, and its SQLite index can be discarded and rebuilt from the ledger.

Analytics can roll ledger records up, forecast spending, and flag unusual changes. Anomaly
detection also feeds alert rules. Both analytics and anomaly recommendations are opt-in.

The [dashboard guide](/docs/build/dashboard) covers setup and access. The
[anomaly detection guide](/docs/build/anomaly-detection) explains what the detector looks for.

## Reduce cost safely

### Routing

Routing sends a request to a cheaper or different model when a rule matches. Rules can consider the
requested model, whether tools are present, the calling agent, request headers, and signals from the
built-in complexity classifier.

You can use an ordered rule list or the policy expression language. In `ledger` mode, neither one
changes traffic. The [routing overview](/docs/build/routing) explains how to preview and introduce a
policy, and the [policy language reference](/docs/build/policy-language) covers its syntax.

**Starts off.** Routing begins only after you move beyond ledger mode.

### Response caching

Response caching avoids paying for the same answer twice. It can match an identical request or a
semantically similar one, depending on the rule you choose.

The cache can use memory, a local SQLite file, or an optional Redis service. Redis requires explicit
permission for external network access. TokenTriage reports cache savings separately from routing
savings so the same avoided cost is not counted twice.

**Starts off.** See [response caching](/docs/build/caching) for rollout guidance and
[cache accounting](/docs/concepts/cache-accounting) for the savings model.

## Control and protect traffic

### Governance and virtual keys

Governance shows which organization, team, or user is spending money and lets you cap future usage.
You issue scoped virtual keys and attach limits at any level of that hierarchy. Available controls
include dollar budgets, token and request quotas, rate limits, and concurrency limits.

TokenTriage stores virtual keys as SHA-256 hashes rather than plaintext. You also decide whether
anonymous traffic is allowed, optional, or rejected.

**Starts off.** The [governance guide](/docs/build/governance) covers virtual keys, inheritance, and
safe rollout.

### Guardrails

Guardrails inspect content rather than cost. Built-in detectors cover areas such as personal data,
secrets, prompt injection, toxicity, code, and multimodal input. Each detector declares whether it
can run on input, output, streaming content, or a subset of those phases.

Local detectors share a time budget so inspection cannot stall a request indefinitely. Optional
model-based evaluation can send content to another model and therefore requires explicit permission
for that external call.

**Starts off.** Begin with the [guardrails overview](/docs/guardrails/overview), which explains the
available actions and how to test a policy before enforcing it.

## Evaluate answer quality

Cost reduction only helps when the resulting answers remain good enough for their job. TokenTriage
includes separate tools for tuning the router, running experiments, and checking automated judges
against human labels.

- `calibrate` fits the complexity thresholds used by the router.
- `quality` estimates the likelihood of a satisfactory outcome and reports its reliability.
- `evals` manages datasets, runs experiments, and compares results.
- `eval` checks whether the router beat a baseline with no routing.

An automated judge can report results before it is trusted to influence routing. It must first meet
its agreement threshold against human labels. TokenTriage does not automatically demote a judge
when its quality later drifts, so you must choose and run a revalidation schedule.

Outcome ingestion from your own systems starts off. The
[calibration and evals overview](/docs/calibration/overview) covers the end-to-end workflow, and the
[LLM judges overview](/docs/judges/overview) covers judge validation.

## Connect and operate

### Providers and request formats

TokenTriage includes 95 provider descriptors, but support is published per endpoint rather than as
one blanket provider claim. The provider matrix distinguishes endpoints covered by an end-to-end
mock test from endpoints that are only declared. Those tests use provider-shaped mocks, not live
provider services.

Requests can use a provider's native API format. Optional cross-provider translation accepts
OpenAI- and Anthropic-formatted requests. You can also define a provider that is not built in.
Configuration stores the name of a credential environment variable, not the credential itself.

Check the [provider matrix and setup guide](/docs/build/providers) for the endpoint you plan to use.

### SDKs

The Python and TypeScript SDKs add attribution headers and read the receipt returned by the proxy.
They do not calculate cost on the client. Both are optional, and the proxy works without them.

See the [SDK overview](/docs/sdks/overview) for installation and the shared behavior contract.

### Observability

The admin port exposes health checks and Prometheus metrics for requests, tokens, cost, pricing
misses, governance decisions, and proxy overhead. OpenTelemetry trace export and outcome ingestion
are separate opt-in features.

See [observability](/docs/build/observability) for metric definitions, trace setup, and data-loss
signals.

### Deployment and command line

TokenTriage ships as a single static binary and a minimal container image. The default ledger is a
local JSONL file, so a single instance needs no external database. Deployment assets include
Docker, Helm, Kustomize, and Terraform for AWS.

Replicas that share state use Redis and require explicit permission for that external connection.
The [deployment guide](/docs/operate/deployment) covers one instance, and
[scaling out](/docs/operate/scale-out) covers shared state and multiple replicas.

The CLI starts and validates the gateway, diagnoses setup problems, previews policies, and manages
features such as virtual keys and guardrails. The [CLI reference](/docs/reference/cli) lists every
command and flag.

## Known limitations

TokenTriage is beta software, and you run it yourself. It is not a hosted model service or a
reseller of provider access.

- Provider coverage varies by endpoint. Some built-in descriptors have no end-to-end gated
  endpoint, so check the provider matrix before adopting one.
- Load balancing currently supports weighted pools, least-latency selection, and passive cooldown.
  It does not provide active health probing, hedged requests, consistent hashing, or configurable
  ordered fallback chains.
- Streaming guardrails are bounded and best-effort. A slow stream can deliver a short secret before
  a detector has enough context to identify it.
- Cross-provider translation is limited to OpenAI- and Anthropic-formatted requests.

The core ships under the TokenTriage commercial licence. See
[How TokenTriage compares](/docs/introduction/how-tokentriage-compares) for the fuller gap list and
the trade-offs against hosted and broader provider gateways.

<NextSteps items={[
  {
    title: "Why TokenTriage",
    href: "/docs/introduction/why-tokentriage",
    description: "Understand the product choices behind these capabilities and the teams they suit."
  },
  {
    title: "How TokenTriage compares",
    href: "/docs/introduction/how-tokentriage-compares",
    description: "Compare the self-hosted approach, feature coverage, and known gaps with other gateways."
  },
  {
    title: "Quickstart",
    href: "/docs/get-started/quickstart",
    description: "Start the passive ledger locally with mock traffic and no provider account."
  },
  {
    title: "Configuration reference",
    href: "/docs/reference/configuration",
    description: "Look up the exact keys and defaults for the capabilities you want to enable."
  }
]} />