Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Self-hosted runtimes

Serve Ollama models through TokenTriage
You point TokenTriage at Ollama.

Ollama is an upstream, not a client. You add one upstream block and your local models land in the same ledger as hosted ones.

# tokentriage.yaml — this page points TokenTriage AT Ollama.
upstreams:
  ollama:
    provider: ollama                   # asserted $0, stamped descriptor@ollama
    base_url: http://127.0.0.1:11434   # the host only. No /v1. Auth is none.

tiers:
  - name: local
    upstream: ollama
    model: llama3.1:8b                 # the tag your Ollama already serves
Paste this into your TokenTriage config. Nothing changes on the Ollama side. provider: ollama is what makes the row an asserted zero with a descriptor stamp, rather than an unpriced row with a dash.
The setting
upstreams.ollama.provider: ollama
The base URL
http://127.0.0.1:11434 — the bare host. A /v1 suffix forwards /v1/v1 and 404s

How to tell it worked

Send one OpenAI-dialect request through the gateway. Then read the last ledger row. An asserted zero carries a descriptor stamp; an unpriced row carries unpriced true and a reason instead.

tail -n 1 ~/.tokentriage/decisions.jsonl

What you should see

"cost": {"estimated": false, "pricing_key": "", "pricing_snapshot": "descriptor@ollama#78964d3", "usd": 0}

How sure we are

Verified

A named test in the product repo exercises this integration and asserts the result. What it drives is our side of it: no test here installs the third-party client.

Verified against Ollama 0.32.15 on

Two CI gates that never skip: provider_sweep_test.go TestProviderSweepRouteAndPrice/ollama and catalog_test.go TestGoldens/ollama

The gotcha

Ollama publishes a base URL ending in /v1. Yours must not.

Here you are not the client, so that /v1 belongs to whoever calls TokenTriage, not to your upstream. TokenTriage joins base_url with the inbound path, so it forwards /v1/v1/chat/completions and Ollama answers 404 page not found. Our own docker-compose.yml gets this wrong.

# docker-compose.yml:45 and :59 tell you to write this. It 404s, loudly:
#   base_url: http://ollama:11434/v1   ->   POST /v1/v1/chat/completions
upstreams:
  ollama:
    provider: ollama
    base_url: http://ollama:11434      # the host, and nothing else

“base_url = 'http://localhost:11434/v1',”

ollama.com/blog/openai-compatibility · read 2026-08-25

What you get now

What a local model adds to the ledger.

Three things that land the first time a request reaches your box. Each links the page that evidences it. Read those before you trust this one.

2 CI gates
both unconditional: no skip, no build tag, no job condition
0 Ollama-native routes
no /api/chat, /api/generate or /api/tags is registered here
8 declared-free descriptors
ollama, vllm, lmstudio, llamacpp, tgi, localai, lemonade, local
91 MB of model
smollm:135m ran the real requests here; llama3.1:8b would be 4.9 GB

What does not carry across

Seven edges, and where each one is.

Two of these change what lands in your ledger. Two are about the direction of traffic. Three are about the edges of our evidence.

  • TokenTriage speaks no Ollama-native inbound dialect

    The gateway registers no /api/chat, /api/generate or /api/tags. A client speaking Ollama-native takes the unmatched-path passthrough. It reaches your box and returns real text. Nothing parses, prices or routes it.

    Which inbound dialects the gateway speaks

  • That passthrough row carries no unpriced flag

    The key is absent, not false. Code that renders a dash when unpriced is true and a dollar otherwise prints a confident zero on every one of those rows. The discriminator is an empty http.dialect.

    Telling an asserted zero from a hole

  • The legacy dialect: openai shape gives you a dash

    An upstream with dialect: openai and no provider: key compiles to the OpenAI descriptor, so your local model is looked up in the OpenAI catalog and misses. The row reads unpriced, model_unknown. Write provider: ollama.

    The declared-free upstream shape

  • Chat is what we drove, and only chat

    Chat is all we drove. The descriptor declares no tools and no vision, and our conformance matrix marks streaming, tool calls and structured output untested for this provider. The model we used declares completion only.

    Which endpoints a descriptor declares

  • Two documented paths here were never driven

    Two shapes were measured, not three. The generic openai_compatible descriptor with pricing: free also reaches Ollama and was not driven. GET /v1/models is declared by this descriptor and was not called either.

  • Whatever your client sends as Authorization lands on your box

    This upstream names no api_key_env, so TokenTriage forwards the caller credential verbatim. It removes only its own ttv_ namespace. The golden file lists four strip headers, and under scheme none they are inert.

    What a scheme none upstream does with credentials

  • Both CI gates drive a mock, not your Ollama

    The gates prove our routing and our costing through the real proxy engine, against an OpenAI-shaped mock upstream. Real Ollama 0.32.15 was driven separately, by hand, on the date beside the chip. That half is a reproduction.

Point it at your box, then read one row.

Adding the upstream is four lines. Nothing your clients send changes. The row it writes is the part worth reading, and the quickstart is there if the gateway is not up yet.

Also: Routing a tier to your own boxWhen an upstream answers 404Every integration

Install it and run itThe declared-free upstream recipe

These are independent integrations. No product named here has endorsed, reviewed or sponsored TokenTriage; every name is its owner's trademark, used only to say what works with what.