# LLM gateways, proxies, and routers
Understand the difference between an LLM proxy, gateway, and router; what each helps with; and where TokenTriage fits.
Source: https://tokentriage.com/docs/introduction/llm-gateways-explained
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import GatewayPositionDiagram from '../../../components/GatewayPositionDiagram.astro';
Once more than one application or model shares an AI budget, you need somewhere to see and control
the traffic. Start looking for that software and you will meet several overlapping labels: LLM
proxy, AI gateway, model router, observability platform, and FinOps for AI.
Vendors use these terms differently. This page gives you working definitions, shows which problems
each kind of tool can solve, and explains where TokenTriage fits. You do not need to have installed
anything. If model pricing is also new to you, read
[Tokens, prices, and where the money goes](/docs/introduction/the-cost-problem) first.
**The short version:** a proxy forwards model requests, a gateway applies shared controls to them,
and a router chooses which model receives them. One product can do all three jobs, but the jobs are
not the same.
## One app, one model, one key
Early LLM features often looked like this: one application, one provider SDK, one API key, and one
model name in the code. Nothing about that setup needed a gateway. Each request had one place to go,
and the provider sent one invoice.
Then a second model arrived, perhaps better at one task or much cheaper. Now there was another API
format, another key to rotate, another rate limit, and another bill. Multiply that across several
teams and every application has to solve the same integration and cost-tracking problems for itself.
Read the diagram from left to right. On the left, three applications calling three providers can
mean nine integration paths to maintain. The provider invoices show totals, but they do not connect
those totals to individual requests. On the right, the same traffic passes through one process that
records each request as it happens. A gateway occupies that box.
## What the words mean
These labels are not standards. No RFC defines an AI gateway, and vendors draw the boundaries to
match their products. Treat the definitions below as common usage rather than strict categories.
**LLM proxy.** A server between your application and a model provider. It forwards requests and may
reshape them, substitute provider keys, record usage, or return a cached response. A proxy can stay
small and apply no company-wide rules.
**LLM gateway or AI gateway.** A proxy that also gives you one place to apply shared controls such
as budgets, access rules, guardrails, routing, and failover. “AI gateway” is often the wider label
because it can include embeddings and agent traffic as well as text-generation calls.
**LLM router or model router.** The part that decides which model should serve a request. A router
can live inside a gateway, inside an application, or as a separate service. A proxy does not have to
route, and a router does not have to proxy traffic.
### Related terms
**Semantic router.** It compares a request with labelled examples and sends it to the closest
matching intent or handler. It usually chooses what kind of work to do, rather than which model to
use.
**LLM observability platform.** It records what happened around a model call: traces, prompts and
responses, token counts, latency, and evaluation results. It usually reports after the event. A
gateway sits in the live request path and can also block or change a request.
**FinOps for AI.** The practice of allocating, forecasting, and reducing AI spending. It is a way
for engineering and finance teams to work with the usage data produced by gateways and
observability tools, not a particular place in the request path.
One line to carry away: proxy, gateway, and router describe **what software does to a request**.
Observability describes **what you learn about the request**. FinOps describes **what the business
does with that information**.
These definitions follow the general usage in
[Cloudflare's AI Gateway documentation](https://developers.cloudflare.com/ai-gateway/),
[Portkey's comparison of proxies and gateways](https://portkey.ai/blog/llm-proxy-vs-ai-gateway/),
and [the FinOps Foundation's AI guidance](https://www.finops.org/framework/technology-categories/ai/).
The semantic-router definition follows common usage rather than a formal specification.
## The ten problems these tools solve
You probably do not need every gateway feature. Use these ten problems as a shopping list: identify
the ones your team has, then check how a product handles those specific jobs.
### Understand usage and spending
- **Cost visibility and attribution.** Provider bills arrive later and group many requests
together. Per-request records let you see which team, customer, feature, or agent spent the money.
- **Observability.** Traces and evaluations help you understand what was sent, what came back, how
long it took, and whether the result was useful.
TokenTriage records the cost and routing decision for each request and can export metrics and trace
data. Its decision ledger does not store prompts or responses, and it does not replace a full
observability platform. Many teams use both. See [cost analytics](/docs/build/cost-analytics) and
[observability](/docs/build/observability).
### Standardize provider access
- **Model sprawl.** Without a shared layer, every provider API becomes another integration for each
application to maintain.
- **Key management.** A gateway can keep provider credentials out of application configuration and
substitute them when it forwards a request.
- **Vendor lock-in.** A common entry point reduces the amount of application code that must change
when you add or replace a provider.
Applications using the OpenAI or Anthropic SDK can point their base URL at TokenTriage and keep
their existing client code. Exact provider, endpoint, and API-format coverage lives in the
[provider matrix](/docs/build/providers), where the verification level is shown beside each entry.
### Control access and risk
- **Rate limits.** Shared limits stop one busy team or customer from using all available capacity.
- **Governance.** Budgets and model-access rules can be checked before a request spends money.
- **Guardrails.** Request and response checks can catch secrets, personal data, prompt injection,
or unsafe content.
TokenTriage supports per-tenant rate limits, budgets, access rules, and guardrails. These controls
are off when you first start the gateway, so you opt into each one separately. See
[governance](/docs/build/governance) and [guardrails](/docs/guardrails/overview).
### Improve reliability and efficiency
- **Failover.** When one provider is unavailable, traffic may need another place to go.
- **Caching.** A stored response can avoid paying for the same request more than once.
TokenTriage's response cache is off by default. Its failover support is narrower: it can steer later
requests away from an upstream that has been failing, but it does not retry the failed request on a
second provider. Your application receives that error. If same-request failover is your main reason
for adopting a gateway, read [How TokenTriage compares](/docs/introduction/how-tokentriage-compares)
before choosing it.
## A useful analogy, and where it stops
Think of a gateway as a universal remote. Your application gets one familiar interface while the
gateway handles devices that speak different protocols. Adding or replacing a provider becomes a
change in one shared layer rather than a change in every application.
The analogy stops because a remote is passive. A gateway can also hold a request that would exceed
a budget, remove sensitive data before it leaves your network, or choose a different model. Calling
it a load balancer also gets only part of the way there: model backends differ in capability and
price, so choosing one can change the answer your user receives.
## Does routing to a cheaper model actually work?
Sometimes, but only if the router is tested on work that resembles your own. A cheap model can be
good enough for one kind of request and poor at another. The router has to identify that difference
reliably enough to save money without giving up more quality than you intended.
Routing and cascading make that trade-off at different times:
```text
routing decide first: read the request, pick a model, make one call
cascading decide after: call the cheap model, judge its answer, escalate if needed
```
Routing makes one call, but cannot inspect the answer before choosing. A cascade sees the cheap
model's answer before deciding whether to spend more, but pays for the first call again whenever it
escalates.
There is also a trap in measuring a router. Imagine a blind policy that sends 20% of requests to an
expensive model and 80% to a cheap one at random. A learned router using the same 20/80 split should
produce better results at the same average cost. If it does not, looking at the request before
choosing has added no value. Research calls that blind comparison a **zero router**.
Compare a router with a random mix of the same models at the same average cost. That tests whether
the routing decision adds value, rather than whether using the cheaper model sometimes reduces the
bill.
**RouteLLM** ([arXiv:2406.18665](https://arxiv.org/abs/2406.18665), from LMSYS and Berkeley) trained
routers on human preference data. On MT-Bench, an LLM evaluation suite, the authors reported 95% of
GPT-4's quality while sending 14% of queries to GPT-4. Routers trained on one model pair also
transferred to new pairs without retraining.
**FrugalGPT** ([arXiv:2305.05176](https://arxiv.org/abs/2305.05176)) describes three ways to reduce
cost: adapt the prompt, approximate an expensive model with a cheaper one, or use a cascade. The
large price differences between models are what make these strategies worth testing.
**RouterBench** ([arXiv:2403.12031](https://arxiv.org/abs/2403.12031)) shows why a single savings
number is incomplete. Change a router's threshold and both its cost and quality move. Comparing
routers therefore means comparing their results at similar costs, including against the zero-router
baseline.
This is the comparison `tokentriage eval` makes. It builds a zero-router baseline from the models'
measured cost and quality, then checks whether routing performs better at the same cost. When the
available labels cannot support that comparison, it reports what is missing instead of claiming
that routing won. [How a verdict is reached](/docs/concepts/honesty-model) explains those outcomes.
## Where TokenTriage sits on this map
By these definitions, TokenTriage is a **self-hosted LLM gateway with an optional router**. You run
it between your applications and your provider accounts. It does not host models or resell model
capacity.
Every proxied request adds one row to the decision ledger with its model, token counts, and cost.
In the default `ledger` mode, the router is off: the model your application asks for is the model it
gets. Routing, caching, governance, guardrails, and trace export remain off until you enable them.
TokenTriage is not a full prompt-tracing platform or a hosted service. You bring your own provider
accounts, and the price table is compiled into the binary, so costing a request needs no network
call. [Why TokenTriage](/docs/introduction/why-tokentriage) explains those design choices, while
[How TokenTriage compares](/docs/introduction/how-tokentriage-compares) covers the alternatives and
the places where they are stronger.