Configure, test and promote a gateway

A cheaper model for streaming requests is a candidate, not a decision. In this tutorial we’ll take one gateway from that idea to a deployed release: add the model, branch on whether the caller asked for a stream, check which branch a request would take without calling a provider, and save the whole thing under the revision you read.

The artefact you end up looking at is one gateway-config/v1 document with two model choices, two routes and a pinned policy revision, and a release id that pins it.

This tutorial takes the command-line path, because it is the one that shows the document. The gateway’s Configure and Test tabs in the portal drive the same operations over the same draft, so each step below ends by naming what the portal calls that step and where the two differ.

What you need

Thing	Needed?	Why
an application with one LLM gateway	yes	Create your company and first application makes both
`config.edit` on that application	yes	reading a draft needs `gateway.read`; saving one needs `config.edit`
`deploy` in the target environment	for the last step only	a protected environment also needs `deploy.protected`
a saved provider connection for the environment	before any paid step	Add a saved provider connection
a provider API key of your own	no	you bind a connection by id; the key stays with TokenTriage
a config file on disk	no	the draft is a document the control plane stores

Steps 1 to 7 make no provider call, so you can walk them before anybody has added a key.

No time budget is published for this path. These steps have not been walked end to end against a deployment, so there is no measured number to print, and a guessed one would be worse than none. Every code block below carries the same note for the same reason.

The commands are tokentriage cloud, which needs --base-url and reads the management key from TOKENTRIAGE_MANAGEMENT_KEY. Each example omits the repeated --org/--workspace/--app flags after the first.

1. Open the draft

The draft is the gateway’s own document in one environment, and reading it is how you get the revision number every later save has to echo.

$ tokentriage cloud draft get --base-url https://app.example.com \
    --org org_… --workspace t_… --app app_… \
    --gateway gw_… --env env_… --json > draft.json

That command has not been run against the private deployment; its verbs and flags come from the CLI’s own definitions.

A gateway nobody has edited yet is seeded on that read: one model choice on the single eligible inventory target, and one default route on the baseline policy, so the first thing you edit is a working configuration rather than an empty file. Seeding writes a row, so it happens only for a caller who also holds config.edit.

You know it worked when draft.json carries a document whose schema_version is gateway-config/v1, and an expected_revision you can read. A gateway with nothing eligible, or with several eligible targets and no way to choose, comes back with saved: false, a warning and revision 0: a seed that guessed would put your money behind a model nobody picked.

If the draft comes back not_found

You hold gateway.read and not config.edit, so there is nothing to read and nothing this call may create. That answer is the truth rather than a permission error: the draft does not exist yet. Ask for config.edit on the application, or have somebody who holds it open the gateway once.

Availability. Landed.

2. Add a model choice

Add the candidate before the route, so the branch you write next has something to point at.

A model choice names an entry of the operator inventory. Read the inventory first, because it carries a verdict per target and the reason when one cannot be used.

$ tokentriage cloud gateway targets --gateway gw_… --env env_…

That command has not been run against the private deployment.

Then add the choice to draft.json. The id is yours to mint: mc_ plus twelve base58 characters, unique inside this document.

{
  "id": "mc_7Qd2fLm9Xk3T",
  "label": "Economical classifier",
  "target_id": "tgt_…",
  "settings": { "max_output_tokens": 512 }
}

That fragment is a request document, not captured output; it follows the field list in the gateway configuration reference.

Adding a model makes it available to the draft. It sends nothing to the provider and changes nothing that is running.

You know it worked at the next save: a target this environment may not use is declined 422 inventory_target_ineligible naming the target, rather than accepted and discovered later.

Adding a model also adds a slot to bind. A binding is keyed on the target’s upstream, not on the model, because several models of one upstream share one credential. The document carries one connection_bindings entry per upstream, and a slot nobody bound is what 422 connection_slot_unbound reports when you create a release.

Availability. Landed. In the portal this is the Configure tab’s Model choices list, which offers only the targets the inventory calls eligible, and its Provider connections list, which shows one row per upstream with either the bound connection or No connection bound. Adding a model there sends nothing to the provider and binds no connection.

3. Add a conditional route, and keep the default

A gateway with one model needs no routes. You add one when two kinds of request should be treated differently and the answer has to be predictable.

The one condition v1 registers is request.stream, a boolean saying whether the caller asked for a streamed response. It is always present, so the branch cannot be skipped by a missing value.

A route pins a policy revision rather than a policy, so read the revision you mean to pin.

$ tokentriage cloud policy get --policy rtp_… --revision 4

That command has not been run against the private deployment.

Then write the routes. Order is the semantics: first match wins, and the unconditional route is last.

[
  {
    "id": "rt_5Hb8xNq2Wme1",
    "label": "Streaming",
    "predicate": { "fact": "request.stream", "op": "is", "value": "true" },
    "policy_id": "rtp_…",
    "policy_revision": 4,
    "model_choice_id": "mc_7Qd2fLm9Xk3T"
  },
  {
    "id": "rt_3Jc9yPr4Znf2",
    "label": "Everything else",
    "predicate": null,
    "policy_id": "rtp_…",
    "policy_revision": 4,
    "model_choice_id": "mc_1Ab2cDe3Fgh4"
  }
]

Those are request documents rather than captured output.

Routes resolve in four steps, and each step can only narrow the answer.

Evaluate the conditional routes in document order.
Select the first route whose predicate matches. Its pinned policy chooses the model.
If no predicate matched, select the route whose predicate is null.
Execute only that route’s own retry and fallback behaviour.

Step 4 is the one readers get wrong. A sibling route does not run because a provider call failed; a fallback belongs to the selected route and is visible in fallback_model_choice_id.

You know it worked when a save is accepted. Three shapes are declined by name: no unconditional route, an unconditional route that is not last, and two routes with the same predicate.

Availability. Landed. request.stream is the whole registered fact set for v1, so treat the reference table as the list rather than an example.

4. Reuse a shared policy, or copy it

Three things you can do to a policy look similar and differ in who else they affect, which is why this step comes before you touch one.

Action	What it does	Who else is affected
publish a revision	appends a new revision to the shared policy	nobody immediately; each consumer stays pinned until its own draft adopts it
copy the policy	starts an independent policy at that revision’s content	nobody, now or later
adopt a revision	repoints one named draft at a revision	nobody else; that draft still has to be released

Pinning is what stops your edit from rerouting a colleague’s traffic. Before publishing, read who holds the revision you are changing.

$ tokentriage cloud policy impact --policy rtp_… --revision 4
$ tokentriage cloud policy publish --policy rtp_… --file policy.json \
    --expected-revision 4
$ tokentriage cloud policy adopt --policy rtp_… --revision 5 \
    --gateway gw_… --env env_… --expected-revision 3

Those three commands have not been run against the private deployment. The impact report is filtered to scopes you may read, and consumers outside them appear as counts rather than names.

Use tokentriage cloud policy copy --policy rtp_… --revision 4 --display-name "…" instead when your change should affect nothing else.

You know it worked when the draft’s route names the revision you adopted. A pin on a revision nobody published, or one in another company, is the same answer: 404 policy_revision_not_found.

Availability. Landed.

5. Edit a protection component

An inherited rule you cannot see is the most common cause of “the builder says one thing and the gateway does another”, so read the palette before editing a card.

$ tokentriage cloud gateway components --gateway gw_…

That command has not been run against the private deployment.

Each card carries its kind, its placement, its bounds, its fields with types, which fields are inherited and from where, and its docs path. The palette is filtered rather than annotated: a card this gateway’s profile cannot execute, or one whose action you do not hold, is absent, so a tool iterating the list cannot offer one by accident.

Protection itself is an attachment in the document, pinned to a profile revision, with an optional list of inherited rules turned off here.

{
  "profile_id": "grp_…",
  "profile_revision": 2,
  "disabled_rule_ids": ["inherited-pii-monitor"]
}

That is a request fragment rather than captured output.

A required company or workspace rule is not in that list’s reach. Mandatory bindings come first in each phase, and where two rules disagree the more restrictive action wins, ordered block, modify, monitor, simulate.

You know it worked when the next inspect shows the rule in the phase you expected, with the mandatory ones you did not touch still there.

Availability. The palette is landed. The guardrail composition it feeds lands with the release-derivation increment. The portal’s Configure tab is one-way here: it renders each stage’s cards and lists the inherited rules this draft already has switched off, with a Switch back on control for each. It offers no control for switching an inherited rule off, because no customer operation publishes a profile’s rule ids and the page would have no id to send. Write disabled_rule_ids from the command line, as above, until that read exists.

6. Inspect which route a request would take

This is the step that costs nothing. Inspect compiles the subject in memory, evaluates one described request through the routing engine, and contacts no provider, so a broken reference or an unreachable branch surfaces before anything is charged.

Send the subject and the request facts to customer.gateways.inspect.

{
  "subject": { "kind": "saved_draft_revision", "draft_revision": 4 },
  "request": { "stream": true, "model": "…" }
}

That body has not been sent against the private deployment. kind is saved_draft_revision or applied_release, and it has to agree with the identity beside it.

The answer names the matched route, the policy revision it selected, the tier and the fallback plan, plus highlights that map each decision back to the card that caused it. It returns no model answer, because no model was called.

You know it worked when a stream: true request matches your streaming route and a stream: false request matches the default one. If both match the default, the predicate value is a string in this document: "true", not true.

The portal’s Test tab offers Inspect routing without a provider call. The CLI addresses the same operation:

$ tokentriage cloud gateway inspect --base-url https://app.example.com \
    --org org_… --workspace t_… --app app_… --gateway gw_… \
    --env env_… --draft-revision 4 --json

Use --release rel_… instead to inspect an applied release. Export the management key as TOKENTRIAGE_MANAGEMENT_KEY; the command does not need browser cookies. The key’s owner still needs access to the gateway and environment.

The shipped sdk/ts/examples/iterate_routing.ts and sdk/python/examples/iterate_routing.py execute read, CAS save, inspect and destination promotion review. They add a model choice and a first non-streaming branch using the default route’s pinned policy; they refuse a draft that already has that branch. Choose a fresh canonical model-choice ID (mc_ followed by 12 base58 characters). A saved choice must be referenced by a route before it can be released.

The programs stop after review. Call their exported iterateRouting / iterate_routing function with explicit destination bindings and confirmPromote: true / confirm_promote=True to apply. An unbound slot remains a blocker. A first deployment has no head row: only the exact 404 not_found after successful destination review becomes expected head revision zero; other refusals remain errors.

7. Compare Draft with Applied

Before any paid call, settle which of the two configurations you are reading. Applied is what the environment serves now; the draft is what you are preparing.

customer.gateways.diff takes two subjects and returns the change set.

{
  "from": { "kind": "applied_release", "release_id": "rel_…" },
  "to": { "kind": "saved_draft_revision", "draft_revision": 4 }
}

That body has not been sent against the private deployment.

The answer groups changes the way the document does: model choices, routes with their old and new positions, protections by placement, and bindings by slot. It carries both effective digests and an identical flag, so “no difference” is an answer rather than an empty screen.

You know it worked when the routes group names your new route as added and the digests differ. Identical digests with a changed document means you diffed the draft against itself.

The diff operation is available through the SDK and customer HTTP lane. The portal’s Changes panel answers two questions and keeps them apart. “Not saved yet, in this browser” is the editor’s own comparison of what you have typed against what you opened, because no server operation can be asked about bytes it has never seen. “The saved draft, against what is applied” is this diff, computed by the control plane, which is why a reordered route reads “moved” and a re-pinned policy reads “re-pinned” instead of both reading “changed”.

8. Save it, and recover from a stale draft

A save replaces the draft with the document you send, and nothing is merged into it. That is what makes “read it, edit the file, send it back” mean what it looks like.

$ tokentriage cloud draft put --gateway gw_… --env env_… \
    --file draft.json --expected-revision 3

That command has not been run against the private deployment.

--expected-revision is the draft revision you read in step 1, not an internal counter. If somebody saved while you were editing, the save comes back 412 draft_revision_stale carrying the winner’s revision and their whole document, so you can show the conflict without a second request.

The same fence is in the TypeScript client, where that answer arrives as a typed error.

import { CustomerClient, StaleDraftError } from "tokentriage/customer";

const client = new CustomerClient({
  baseUrl: "https://app.example.com",
  managementApiKey: process.env.TOKENTRIAGE_MANAGEMENT_KEY ?? "",
});

try {
  await client.gateways.saveDraft(org, ws, app, gw, env, {
    document,
    expected_revision: 3,
  });
} catch (err) {
  if (!(err instanceof StaleDraftError)) throw err;
  // err.headRevision is the winner's revision; err.latest is their document.
  showConflict(err.headRevision, err.latest);
}

That snippet has not been run against the private deployment; it uses the shipped gateways.saveDraft signature and the StaleDraftError fields.

Recovering means rebasing, not retrying: apply your change to the winner’s document, then send it with their revision as expected_revision. Both versions stay in the append-only history, which tokentriage cloud draft history reads, so neither edit is lost.

You know it worked when the response carries a revision above the one you sent. Deleting a model choice a route still selects is declined 409 model_choice_referenced with the routes listed.

Availability. Landed. The portal’s Save this draft panel states the revision it was built on, and a lost race offers Rebase onto revision N built from the refusal body. Pressing it adopts the winner’s document whole and says in as many words that your unsaved edits are not in it, because a three-way merge of somebody else’s routing decisions would be a guess about traffic.

9. Create a release, then deploy it

These are two decisions and two grants, which is the vocabulary this step exists to teach. customer.releases.create pins the configuration under config.edit: the settings revision, the derived policy and profile revisions, the mandatory generation, every slot’s connection, and the effective digest. Creating one changes nothing about what the gateway serves.

customer.deployments.deploy makes one release the desired one under deploy, echoing the deployment head’s revision as expected_revision, or 0 for a target nobody has deployed. A protected environment also needs deploy.protected. Cells observe the head and the state moves to applied.

Two refusals belong to the create rather than to the first live request: a compiled artifact with no unconditional route is 422 default_route_missing, and one dispatching to a slot nothing resolves is 422 connection_slot_unbound.

You know it worked when the deployment head names your release as both desired and applied, and one permitted request through that environment returns a request id whose record names the same release. If it does not, roll that environment back to the prior release.

Availability. Release, deployment and promotion operations are declared in the lane and land across this private test. A deployment with no release ledger behind them answers 503 with “deployments not configured”, so a wired path and a misspelled one stay distinguishable. Move a gateway between environments covers promotion, the deploy gate and rollback.

The portal reaches the same place by a different route, and the difference is worth knowing before you look for a button. There is no create-a-release panel and no deploy panel on the Configure tab. What it offers is Promote this configuration, which reviews against a destination environment and then applies, and applying is what derives the release and deploys it. The tab’s Applied view is a read of the release ledger rather than a second canvas: it shows the release’s identity, which is the revisions it pins, its effective digest, its required-controls generation and the connections it binds. It draws no routes and no model choices, because the release API publishes neither, and it says so on the panel rather than redrawing the draft under an “Applied” label.

What this tutorial does not cover

How a policy picks a model inside the route you selected: How a routing decision is made.
Writing a policy’s content: Routing policy language.
Every field, status and operation named here: Gateway configuration reference.
Running a question through the draft against a provider, and keeping the answer where your team can review it: Test a gateway draft and save the run for your team.