Provider clients and streaming

Install the local SDK update and set your environment. These configured APIs are unreleased.

The snippets below pass a provider key to the client, which is the pre-Foundation managed shape. On a saved-mode gateway there is no provider key to pass: drop that argument and give the binding instead, connection.async_openai(gateway_id="gw_...", environment_id="env_...") in Python or connection.openai({ gatewayId, environmentId }) in TypeScript, with "gateway" as the model. Your first routed request walks one call end to end.

Async Python

import asyncio
import os
import sys

from tokentriage.client import TokenTriage

NAMES = ("TOKENTRIAGE_GATEWAY_BASE", "TOKENTRIAGE_RUNTIME_KEY",
         "TOKENTRIAGE_GATEWAY_ID", "TOKENTRIAGE_ENVIRONMENT_ID")
missing = [name for name in NAMES if not os.environ.get(name)]
if missing:
    sys.exit(f"Set {', '.join(missing)} before running this example.")
base, runtime_key, gateway_id, environment_id = (os.environ[name] for name in NAMES)


async def main() -> None:
    async with TokenTriage(base_url=base, api_key=runtime_key) as connection:
        try:
            client = connection.async_openai(gateway_id=gateway_id, environment_id=environment_id)
            response = await client.chat.completions.with_raw_response.create(
                model="gateway",
                messages=[{"role": "user", "content": "Say hello in one sentence."}],
            )
        except Exception as error:
            details = connection.error_details(error)
            sys.exit(f"Request failed: code={details.code} status={details.status} "
                     f"request_id={details.request_id}")
        print(response.parse().choices[0].message.content)
        print("Request ID:", connection.metadata(response).request_id)


asyncio.run(main())

Python streaming

Headers carry the request ID before the stream finishes. The context manager closes it when the loop exits early.

import asyncio
import os
import sys

from openai import APIError
from tokentriage.client import TokenTriage, TokenTriageError, error_details

NAMES = ("TOKENTRIAGE_GATEWAY_BASE", "TOKENTRIAGE_RUNTIME_KEY",
         "TOKENTRIAGE_GATEWAY_ID", "TOKENTRIAGE_ENVIRONMENT_ID")
missing = [name for name in NAMES if not os.environ.get(name)]
if missing:
    sys.exit(f"Set {', '.join(missing)} before running this example.")
base, runtime_key, gateway_id, environment_id = (os.environ[name] for name in NAMES)


async def main() -> None:
    try:
        async with TokenTriage(base_url=base, api_key=runtime_key) as connection:
            client = connection.async_openai(gateway_id=gateway_id, environment_id=environment_id)
            async with client.chat.completions.with_streaming_response.create(
                model="gateway",
                messages=[{"role": "user", "content": "Say hello in one sentence."}],
                stream=True,
            ) as response:
                print("Request ID:", connection.metadata(response).request_id)
                stream = await response.parse()
                async for chunk in stream:
                    if chunk.choices:
                        print(chunk.choices[0].delta.content or "", end="", flush=True)
            print()
    except (TokenTriageError, APIError) as error:
        # A configuration error is raised locally and never reaches the gateway.
        details = error_details(error)
        sys.exit(f"Request failed: category={details.category} code={details.code} "
                 f"status={details.status} request_id={details.request_id}")
    except Exception as error:
        sys.exit(f"Request failed: {error!r}")


asyncio.run(main())

TypeScript streaming

import OpenAI from "openai";
import { TokenTriage, TokenTriageError, errorDetails, gatewayModel } from "tokentriage/client";

const names = ["TOKENTRIAGE_GATEWAY_BASE", "TOKENTRIAGE_RUNTIME_KEY",
  "TOKENTRIAGE_GATEWAY_ID", "TOKENTRIAGE_ENVIRONMENT_ID"];
const missing = names.filter((name) => !process.env[name]);
if (missing.length > 0) {
  console.error(`Set ${missing.join(", ")} before running this example.`);
  process.exit(1);
}
const [baseUrl, apiKey, gatewayId, environmentId] = names.map((name) => process.env[name] ?? "");

let connection: TokenTriage | undefined;
try {
  connection = new TokenTriage({ baseUrl, apiKey });
  const client = await connection.openai({ gatewayId, environmentId });
  const { data: stream, response } = await client.chat.completions.create({
    model: gatewayModel,
    messages: [{ role: "user", content: "Say hello in one sentence." }],
    stream: true,
  }).withResponse();
  console.log("Request ID:", connection.metadata(response).requestId);
  try {
    for await (const chunk of stream) {
      process.stdout.write(chunk.choices[0]?.delta.content ?? "");
    }
  } finally {
    stream.controller.abort();
  }
  console.log();
} catch (error) {
  if (error instanceof TokenTriageError || error instanceof OpenAI.APIError) {
    // A configuration error is raised locally and never reaches the gateway.
    const details = errorDetails(error);
    console.error(`Request failed: category=${details.category} code=${details.code} `
      + `status=${details.status} request_id=${details.requestId}`);
  } else {
    console.error("Request failed:", error);
  }
  process.exitCode = 1;
} finally {
  connection?.close();
}

Provider chunks and tool-call fields keep their official types. Do not replay a stream after delivering part of it to your application.

Keep your constructor or transport

Python provides openai_options(http_client=...) and async_openai_options(http_client=...). TypeScript provides openaiOptions({ fetch }). Pass the result to the official OpenAI constructor.

Both options APIs take the saved-mode binding beside your transport: openai_options(gateway_id=..., environment_id=..., http_client=...) and openaiOptions({ gatewayId, environmentId, fetch }).

The connection borrows your supplied transport. Your application owns its cleanup.

wrap_openai() and wrapOpenAI() support factory-created clients. Arbitrary third-party instances may hide custom transports; unsupported wrapping raises a configuration error. Construct them with the options API instead.

Choose how to activate

Mode	Routing	Observation
Configured factory/options with a saved-mode binding	the gateway’s applied release picks the model	response metadata, including the release and model it chose
Configured factory/options	Assigned gateway	Response metadata and optional local observations
Legacy init() and headers	Your existing destination	Labels and receipts
Local router	In-process policy	Local observations and optional Central export

Automatic bootstrap for arbitrary provider clients is unsupported. Use the factory or options API for explicit activation.