Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Agent frameworks

Point LlamaIndex at TokenTriage
The argument is not called base_url.

LlamaIndex spells it api_base, never base_url. Set it once. Every call your index and your agent make lands in one priced ledger.

from llama_index.core.llms import ChatMessage
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="gpt-4o-mini",
    api_base="http://127.0.0.1:8787/v1",  # api_base, and the /v1 is exact
    api_key="fake",      # TokenTriage holds the real key
    is_chat_model=True,  # default False calls /v1/completions
    max_retries=0,       # default 3 becomes twelve upstream requests
)
print(llm.chat([ChatMessage(role="user", content="ping")]).message.content)
Two arguments carry the swap: api_base and is_chat_model. max_retries=0 is optional. The gotcha below says why. Everything else is the constructor you already write.
The setting
OpenAILike(api_base=…) — base_url= is dropped in silence
The base URL
http://127.0.0.1:8787/v1 — env var OPENAI_API_BASE, never OPENAI_BASE_URL

How to tell it worked

Send one chat call, then read the last row of the decision log. Four fields decide it. The response header will not: x-tt-decision is emitted in shadow and route mode only.

tail -1 ~/.tokentriage/decisions.jsonl | jq -r \
  '[.http.path, .http.dialect, .cost.pricing_key, .usage.source] | join(" ")'

What you should see

/v1/chat/completions openai openai/gpt-4o-mini provider

How sure we are

Reproduced

A person ran it by hand against a real gateway on a stated date, and the transcript is on the page.

Verified against llama-index 0.14.24 · llama-index-llms-openai 0.7.10 · openai 2.54.0 on 

[LAB RUN · MOCK UPSTREAM]  C1 — OpenAILike(api_base=root + '/v1') · llm.chat()
  content: 'Hello from the mock upstream.'
  [0] path='/v1/chat/completions' dialect='openai' stream=False status=200
      model='gpt-4o-mini' tier='fast' tokens=100/20 usd=2.7e-05
      pricing_key='openai/gpt-4o-mini'

[LAB RUN · MOCK UPSTREAM]  C9 — ★★ THE RETRY MULTIPLIER
  llm.max_retries (left at the DEFAULT): 3
  CLIENT RAISED: openai.InternalServerError
  --- LEDGER ROWS PRODUCED [LAB RUN · MOCK UPSTREAM]: 12 ---
  [0..11] path='/v1/chat/completions' dialect='openai' status=500
          model='gpt-4o-mini' tier='fast' tokens=0/0 usd=0 pricing_key=''
  ★ ONE llm.chat() call produced this many ledger rows: 12
Excerpted from a run of llama-index 0.14.24 against the real proxy engine in front of a mock upstream: ten cases, twenty rows, zero probe errors

The gotcha

One llm.chat() call, twelve upstream requests and twelve rows

max_retries defaults to 3 and LlamaIndex applies it twice: once as a tenacity wrapper, once inside the openai client. Three times four is twelve. TokenTriage runs no retry chain, so every attempt is its own usd=0 row.

# max_retries is read twice: llm_retry_decorator wraps the method,
# and the same value is handed to the openai client underneath.
llm = OpenAILike(model="gpt-4o-mini", api_base="http://127.0.0.1:8787/v1",
                 api_key="fake",
                 is_chat_model=True, max_retries=0)

“How many times to retry the API call if it fails.” — max_retries, default 3

developers.llamaindex.ai/python/framework-api-reference/llms/openai/ · read 2026-08-25

What you get now

What the ledger holds once the index runs.

Three things that switch on the first time a call arrives. Each item links its evidence. Click through before you believe any of it.

1 argument
api_base, and it is the only line of yours that changes
12 ledger rows
from one llm.chat() at the stock max_retries of three
20 rows in one run
ten cases against a mock upstream, zero probe errors
2 header seams
default_headers on the client, extra_headers on the call

What the swap does not fix

Seven things LlamaIndex decides for you.

Most of these are the framework deciding for you. The remedy is usually one keyword. The last item is about our evidence, not your code.

  • tokentriage wrap cannot reach LlamaIndex

    LlamaIndex reads OPENAI_API_BASE. It never reads OPENAI_BASE_URL, which is the only OpenAI base URL wrap sets. The wrapped command succeeds and the ledger stays empty. Pass api_base instead; the constructor outranks every environment variable.

    Base URL and path, per client

  • The class and the argument name are both traps

    base_url= is not a field: pydantic drops it, nothing raises, and api_base keeps api.openai.com. OpenAI accepts api_base too, then refuses your model name late, from .metadata. OpenAILike is the class that takes yours.

  • llm.chat() can land on /v1/completions

    is_chat_model defaults to False, so OpenAILike turns your chat call into a completion. Your code still says llm.chat. The row lands on /v1/completions with an empty pricing_key, and a server that implements it answers 200.

    Which endpoints are registered

  • Drop the /v1 and the row stops costing

    api_base without /v1 calls /chat/completions. No dialect matches it. The call is relayed to your upstream with the path as written, and a permissive upstream answers 200 with no dialect, no model and no price on the row.

    Why the path match is exact

  • Settings.llm moves one slot, not two

    Settings.embed_model is separate and defaults to OpenAIEmbedding pointed at api.openai.com. Building an index calls it. With no key you get a ValueError naming OPENAI_API_KEY; with any key your embedding traffic leaves the gateway.

    Getting every step onto one ledger

  • context_window defaults to 3900 tokens

    OpenAILike sets context_window to 3900 unless you say otherwise. That is a default, not a discovery. LlamaIndex chunks and truncates against it, so give it the real window of the model your tier serves.

  • What the run covered, and what it did not

    One machine, one Python, a mock upstream. No vendor endpoint was called, no async path was driven, and no index was built end to end. The retry and endpoint findings are library behaviour, not gateway behaviour.

    The headers we did drive

Point it, then read one row.

The swap is one argument. It changes nothing about your index. Read the first row before you change anything else, then set max_retries and watch the count stop multiplying.

Also: What the ledger recordsThe trace headersEvery integration

Get the gateway runningAttribute cost per agent step

These are independent integrations. No product named here has endorsed, reviewed or sponsored TokenTriage; every name is its owner's trademark, used only to say what works with what.