LLM gateways, proxies, and routers

Once more than one application or model shares an AI budget, you need somewhere to see and control the traffic. Start looking for that software and you will meet several overlapping labels: LLM proxy, AI gateway, model router, observability platform, and FinOps for AI.

Vendors use these terms differently. This page gives you working definitions, shows which problems each kind of tool can solve, and explains where TokenTriage fits. You do not need to have installed anything. If model pricing is also new to you, read Tokens, prices, and where the money goes first.

The short version: a proxy forwards model requests, a gateway applies shared controls to them, and a router chooses which model receives them. One product can do all three jobs, but the jobs are not the same.

One app, one model, one key

Early LLM features often looked like this: one application, one provider SDK, one API key, and one model name in the code. Nothing about that setup needed a gateway. Each request had one place to go, and the provider sent one invoice.

Then a second model arrived, perhaps better at one task or much cheaper. Now there was another API format, another key to rotate, another rate limit, and another bill. Multiply that across several teams and every application has to solve the same integration and cost-tracking problems for itself.

Read the diagram from left to right. On the left, three applications calling three providers can mean nine integration paths to maintain. The provider invoices show totals, but they do not connect those totals to individual requests. On the right, the same traffic passes through one process that records each request as it happens. A gateway occupies that box.

What the words mean

These labels are not standards. No RFC defines an AI gateway, and vendors draw the boundaries to match their products. Treat the definitions below as common usage rather than strict categories.

LLM proxy. A server between your application and a model provider. It forwards requests and may reshape them, substitute provider keys, record usage, or return a cached response. A proxy can stay small and apply no company-wide rules.

LLM gateway or AI gateway. A proxy that also gives you one place to apply shared controls such as budgets, access rules, guardrails, routing, and failover. “AI gateway” is often the wider label because it can include embeddings and agent traffic as well as text-generation calls.

LLM router or model router. The part that decides which model should serve a request. A router can live inside a gateway, inside an application, or as a separate service. A proxy does not have to route, and a router does not have to proxy traffic.

Related terms

Semantic router. It compares a request with labelled examples and sends it to the closest matching intent or handler. It usually chooses what kind of work to do, rather than which model to use.

LLM observability platform. It records what happened around a model call: traces, prompts and responses, token counts, latency, and evaluation results. It usually reports after the event. A gateway sits in the live request path and can also block or change a request.

FinOps for AI. The practice of allocating, forecasting, and reducing AI spending. It is a way for engineering and finance teams to work with the usage data produced by gateways and observability tools, not a particular place in the request path.

One line to carry away: proxy, gateway, and router describe what software does to a request. Observability describes what you learn about the request. FinOps describes what the business does with that information.

These definitions follow the general usage in Cloudflare’s AI Gateway documentation, Portkey’s comparison of proxies and gateways, and the FinOps Foundation’s AI guidance. The semantic-router definition follows common usage rather than a formal specification.

The ten problems these tools solve

You probably do not need every gateway feature. Use these ten problems as a shopping list: identify the ones your team has, then check how a product handles those specific jobs.

Understand usage and spending

• Cost visibility and attribution. Provider bills arrive later and group many requests together. Per-request records let you see which team, customer, feature, or agent spent the money.
• Observability. Traces and evaluations help you understand what was sent, what came back, how long it took, and whether the result was useful.

TokenTriage records the cost and routing decision for each request and can export metrics and trace data. Its decision ledger does not store prompts or responses, and it does not replace a full observability platform. Many teams use both. See cost analytics and observability.

Standardize provider access

• Model sprawl. Without a shared layer, every provider API becomes another integration for each application to maintain.
• Key management. A gateway can keep provider credentials out of application configuration and substitute them when it forwards a request.
• Vendor lock-in. A common entry point reduces the amount of application code that must change when you add or replace a provider.

Applications using the OpenAI or Anthropic SDK can point their base URL at TokenTriage and keep their existing client code. Exact provider, endpoint, and API-format coverage lives in the provider matrix, where the verification level is shown beside each entry.

Control access and risk

• Rate limits. Shared limits stop one busy team or customer from using all available capacity.
• Governance. Budgets and model-access rules can be checked before a request spends money.
• Guardrails. Request and response checks can catch secrets, personal data, prompt injection, or unsafe content.

TokenTriage supports per-tenant rate limits, budgets, access rules, and guardrails. These controls are off when you first start the gateway, so you opt into each one separately. See governance and guardrails.

Improve reliability and efficiency

• Failover. When one provider is unavailable, traffic may need another place to go.
• Caching. A stored response can avoid paying for the same request more than once.

TokenTriage’s response cache is off by default. Its failover support is narrower: it can steer later requests away from an upstream that has been failing, but it does not retry the failed request on a second provider. Your application receives that error. If same-request failover is your main reason for adopting a gateway, read How TokenTriage compares before choosing it.

A useful analogy, and where it stops

Think of a gateway as a universal remote. Your application gets one familiar interface while the gateway handles devices that speak different protocols. Adding or replacing a provider becomes a change in one shared layer rather than a change in every application.

The analogy stops because a remote is passive. A gateway can also hold a request that would exceed a budget, remove sensitive data before it leaves your network, or choose a different model. Calling it a load balancer also gets only part of the way there: model backends differ in capability and price, so choosing one can change the answer your user receives.

Does routing to a cheaper model actually work?

Sometimes, but only if the router is tested on work that resembles your own. A cheap model can be good enough for one kind of request and poor at another. The router has to identify that difference reliably enough to save money without giving up more quality than you intended.

Routing and cascading make that trade-off at different times:

routing     decide first:  read the request, pick a model, make one call
cascading   decide after:  call the cheap model, judge its answer, escalate if needed

Routing makes one call, but cannot inspect the answer before choosing. A cascade sees the cheap model’s answer before deciding whether to spend more, but pays for the first call again whenever it escalates.

There is also a trap in measuring a router. Imagine a blind policy that sends 20% of requests to an expensive model and 80% to a cheap one at random. A learned router using the same 20/80 split should produce better results at the same average cost. If it does not, looking at the request before choosing has added no value. Research calls that blind comparison a zero router.

What published routing research found

RouteLLM (arXiv:2406.18665, from LMSYS and Berkeley) trained routers on human preference data. On MT-Bench, an LLM evaluation suite, the authors reported 95% of GPT-4’s quality while sending 14% of queries to GPT-4. Routers trained on one model pair also transferred to new pairs without retraining.

FrugalGPT (arXiv:2305.05176) describes three ways to reduce cost: adapt the prompt, approximate an expensive model with a cheaper one, or use a cascade. The large price differences between models are what make these strategies worth testing.

RouterBench (arXiv:2403.12031) shows why a single savings number is incomplete. Change a router’s threshold and both its cost and quality move. Comparing routers therefore means comparing their results at similar costs, including against the zero-router baseline.

This is the comparison tokentriage eval makes. It builds a zero-router baseline from the models’ measured cost and quality, then checks whether routing performs better at the same cost. When the available labels cannot support that comparison, it reports what is missing instead of claiming that routing won. How a verdict is reached explains those outcomes.

Where TokenTriage sits on this map

By these definitions, TokenTriage is a self-hosted LLM gateway with an optional router. You run it between your applications and your provider accounts. It does not host models or resell model capacity.

Every proxied request adds one row to the decision ledger with its model, token counts, and cost. In the default ledger mode, the router is off: the model your application asks for is the model it gets. Routing, caching, governance, guardrails, and trace export remain off until you enable them.

TokenTriage is not a full prompt-tracing platform or a hosted service. You bring your own provider accounts, and the price table is compiled into the binary, so costing a request needs no network call. Why TokenTriage explains those design choices, while How TokenTriage compares covers the alternatives and the places where they are stronger.