# Test a gateway draft and save the run for your team Run a question through a draft or the applied release, see what it would cost first, and keep the answer for your team. Source: https://tokentriage.com/docs/build/test-and-share-gateway-runs --- import Callout from '../../../components/Callout.astro'; import CodeTabs from '../../../components/CodeTabs.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; {/* Provenance of every block below marked as program output, so a later editor can regenerate rather than guess: the four `tokentriage cloud` transcripts — the shipped command tree driven against a loopback HTTP server by the harness in internal/cli/cloud_test.go (cloudServer + runCloud), captured 2026-09-22. The documents the server answered with are the shapes internal/control/foundation/ httpapi answers with; every figure in them is a fixture constant and is nobody's bill. The `cloud:` diagnostic lines carry the loopback port and are omitted. the script's request sequence and its result object — sdk/ts/test/unit/customer_tests_example.test.ts, which drives sdk/ts/examples/test_and_save_run.ts through the client's fake transport. the state and content wording — ui/src/features/foundation/tests/vocabulary.ts and ui/src/features/foundation/builder/refusals.tsx, the strings the portal renders. the portal frames each state was reviewed in — docs/manual-testing/screenshots/foundation/S08-20260922T1050Z-01-local: 01 the preflight and the two buttons, 02 the capture choice, 03 the history, 04 a run read out, 05 review/rerun/share, 06 unknown_effect, 07 content_purged, 08 an ineligible slot, 09 tests_not_enabled. */} You changed a routing rule and you want to know what the gateway now answers, before anyone's users find out. Typing the question into a chat window somewhere else tells you what that window's model says. It tells you nothing about your compiled configuration, your guardrails, your budget or the connection your slots bind. The playground runs your question through the gateway itself: the same compiled document, the same protection stages, the same admission and budget checks, the same saved provider connection. Choose scratch or team capture before dispatch. Team capture turns the spot check into a saved run colleagues can read, review, and cite when they deploy. This page is the task. Look at what a test would do, run it, follow it, read it, and leave a verdict on it. For the configuration you are testing, start with [Configure, test and promote a gateway](/docs/build/gateway-builder). ## What you need | Thing | Needed? | Why | |---|---|---| | a gateway with a saved draft revision or an applied release | yes | a test runs against a subject that exists, not the bytes in your editor | | `tests.run` at the gateway, in that environment | for anything paid | the preflight and the history need `metadata.read` only | | `content.read` at the gateway | to read prompts and answers | an explicit, audited grant; owning the company does not carry it | | an active saved provider connection for each bound slot | yes | [Add a saved provider connection](/docs/build/provider-connections) | | an active environment | yes | a retiring or retired one is declined before anything is spent | Two words carry the whole page. A **case** is one question, stored under the gateway and versioned, so the same question can be asked of staging and of production. A **run** is one execution of one case version in one environment, which is what makes it evidence about that environment and nothing else. ## Look before you spend Every paid control on this lane sits under the same five facts, and the read that produces them calls no provider and costs nothing. Ask for them first: ```console $ tokentriage cloud tests inspect --base-url https://app.example.com \ --org org_acme00000001 --workspace t_acme0000001 --app app_userportal01 \ --gateway gw_ticketclass1 --env env_staging00001 --draft-revision 7 # … two `cloud:` diagnostic lines naming the scope and the operation URL subject saved_draft_revision 7 (digest sha256:4f1ac0f0f4b2) slot primary pconn_9mQ2xR7tLbVd version 3: eligible slot fallback pconn_5kT8yH3wNcJr version 1: NOT ELIGIBLE (connection_purpose_not_granted) ceiling up to $0.05 (spend ceiling) retention kept for 30 days (standard) audience everyone in this workspace with content.read ``` That capture came from the shipped command tree driven against a loopback server in the CLI's own test suite, so the ids and the figures are fixtures rather than an invoice. Against your deployment the shapes are the same and the numbers are yours. Read it as five answers: | Line | The answer it gives | |---|---| | `subject` | which configuration would execute, and the `effective_digest` it compiles to | | `slot` | the saved connection each destination slot would bind, and whether its grant admits a playground test | | `ceiling` | the most this one run may spend, published before it runs | | `retention` | how long a saved answer is kept, in the workspace's own sentence and class | | `audience` | who would be able to open the run, and who could open its text | A slot marked `NOT ELIGIBLE` is a run that will be declined once it starts. The slug names which door to open: `connection_purpose_not_granted` means the connection's grant does not admit the `playground` purpose for this application and environment, so grant it on the connections page or bind one that already carries it. The portal's **Test** tab issues that same read when it opens, and prints the five facts above both buttons that spend money. You do not have to press anything to learn what pressing something would cost. ## Choose scratch or keep, before anything runs The choice between the two buttons is the product, and it is made before dispatch. No flag anywhere promotes a finished scratch run into a saved one; asking for one answers `capture_choice_immutable`. | | Run once and keep nothing | Run and save to team | |---|---|---| | operation | `customer.tests.run` | `customer.cases.create` then `customer.runs.create` | | CLI | `cloud tests run --file …` | `cloud cases create` then `cloud runs create` | | what is written | no case, no run row, no content object | a case version, a run, a result, and the input the server observed | | where the answer lives | the response body, and nowhere else | a content object under the gateway, for its retention class | | when it answers | synchronously, with the completion in the reply | `202` with a `queued` row; a worker dispatches it | | who can read it afterwards | nobody, including you | the audience the preflight named | Both spend money. The paid verbs put the preflight in front of you rather than trusting a sentence in a help screen: ```console $ tokentriage cloud tests run --gateway gw_ticketclass1 --env env_staging00001 \ --draft-revision 7 --file prompt.txt # … the same five preflight lines as above, on stderr Error: `cloud tests run` was NOT sent: it causes a real provider call against a real saved credential, and the ceiling above is what it may cost. Re-run with --yes to send it (or with --json, which is the scripted path and skips this confirmation) ``` Add `--yes` to send it. `--json` also sends it directly, because a confirmation prompt inside a pipeline is a process that stops responding where nobody can see it, and `cloud tests inspect --json` answers the same question for nothing. Repeating `cloud runs create` dispatches a second paid run. These verbs carry no idempotency key, because one would promise a deduplication the server does not make. Read the run you already have before you ask for another. ### What a saved run records A saved run is a row you can point a colleague at months later, so it keeps the parts that make the answer mean something: - the case and the version it asked, pinned, so a question that moves later cannot rewrite what this run was about - the input bytes as the **server** observed them, stored as a content object with its own retention class - the subject: the configuration's digest, its draft revision or release id, and the version of each connection it bound - the result: a terminal state, the receipt id, the token counts, the latency, and the cost in integer nano-USD - every verdict anyone recorded on it, in order, with the note each one carried Nothing of a scratch run survives it, and that falls out of the shape rather than a policy: the scratch path writes no case, no run and no content object, so a later read finds nothing. Asking for a scratch answer after the event says `scratch_output_not_retainable`. ## Follow the run to a terminal word `runs create` answers `202` with a `queued` row and no bytes. A control-plane worker re-resolves your authority, mints a short-lived execution ticket and only then dispatches, which is why the answer is not in the reply. Poll the run: ```console $ tokentriage cloud runs get --gateway gw_ticketclass1 --env env_staging00001 \ --run trun_3hF7kM2rTyWb --json { "schema": "customer/v1", "request_id": "req_6bN4pS1vZxQe", "kind": "test-run", "data": { "run": { "run_id": "trun_3hF7kM2rTyWb", "state": "succeeded", "case_id": "tcase_2wK1nVQ8bdHp", "case_version": 1, "capture": "team" }, "result": { "terminal_state": "succeeded", "cost_state": "priced", "cost_nano_usd": 41200, "output_ref": "cobj_8dL4vC6qEnHs" } } } ``` State moves forward and stops. Six words, and the three terminal ones are alternatives rather than a sequence: | `state` | The portal's word | What it says | |---|---|---| | `queued` | Queued | the question, the run and its job are committed; nothing dispatched and nothing spent | | `dispatched` | Running | a cell has it and a provider call may be in flight | | `succeeded` | Answered | a receipt came back and the answer is kept for this run's retention | | `failed` | Failed | this ended in a decline or a provider failure the build can name, and `failure_code` says which | | `expired` | Expired | the execution ticket's window closed before the cell answered; rerun the same case version | | `unknown_effect` | Effect unknown | the dispatch was committed and then the build lost sight of it | `unknown_effect` is terminal and is not a failure. The provider may have been called and charged. Check the receipt before running the case again, because running it again may be the second charge rather than the first. Cost has three states of its own, and none of them is zero. `priced` carries the settled integer, `pending` means it has not settled yet, and `unknown` means this build cannot tell. A run whose cost is unsettled prints "Not settled yet", not `$0.00`. ## Read the answer, or the reason you cannot A run and its text are two reads, and the second re-checks `content.read` at retrieval rather than at the reference. So a person who can open every run here can still be declined its prompt, and holding an object id is not holding the bytes: ```console $ tokentriage cloud content get --gateway gw_ticketclass1 --object cobj_8dL4vC6qEnHs # … two `cloud:` diagnostic lines naming the scope and the operation URL Error: customer.content.get was refused (HTTP 409); detail: scopedcontent: content purged: cobj_8dL4vC6qEnHs was purged on 2026-10-22T00:00:00Z; type=content_purged; request_id=req_1cV9jD5uXpKa ``` **Content withheld** is the portal's name for that outcome: the run stands, its cost, its states and its verdicts read normally, and the text alone is unavailable. Which of the three it is matters, because they have different recoveries: | Slug | Status | What happened | What to do | |---|---|---|---| | `content_read_denied` | 403 | you may read this run, not its text | ask for `content.read` at this gateway; it is granted explicitly and audited | | `content_excluded` | 409 | somebody withdrew this object; the bytes still exist | whoever excluded it can say why | | `content_purged` | 409 | retention ended and the bytes are gone | nothing brings them back, and the run's own record stands | Collapsing the three into "unavailable" would tell a reader their prompt might come back when it cannot, and would hide the one case where asking a colleague works. An object addressed through a gateway that does not own it is a concealed `404`, the same answer as one nobody ever wrote. ## Review it, rerun it, share it A saved run becomes evidence once somebody says what they think of it. Three operations close the loop, and each keeps the original intact. **Review** records one verdict on a terminal run: `approved`, `rejected` or `needs_work`, with an optional note stored as its own content object under its own retention class. A later verdict *supersedes* an earlier one rather than editing it, so the history reads as a sequence of opinions instead of one mutable field. A verdict on a run that has not finished answers `run_not_terminal`; one written from a page that has gone stale answers `review_superseded`. **Rerun** asks the same case version again and writes a new run with its own evidence. The subject is not copied from the earlier run, which is the part people get wrong: "the same as last time" is a configuration that may have moved since, so name `--release` or `--draft-revision`, or neither to use what the environment serves now. A rerun cannot change the capture choice its parent made. **Share** answers with the run's own path and two sentences about who can open it. It mints no token and publishes no URL. Whoever follows the link authenticates as themselves and reads what their own grants allow, which is why a colleague without `content.read` opens the run and not the prompt.
A protected environment can require provider evidence before a deploy. The newest succeeded saved run matching the release's effective digest, its environment and its connection fingerprint is the one stamped onto the release. A new draft revision, a change to the required controls, a different environment or a changed connection flips that evidence to `provider_test_stale`, and the fix is a rerun against the subject as it is now. [Move a gateway between environments](/docs/build/environments-and-promotion) covers the deploy side.
## The same lane from a script Both SDKs carry a worked example of the whole sequence: inspect, create the case, create the run, poll it, then read the bytes. The scripts stop before spending when a slot is ineligible, the spend ceiling is missing or malformed, or the published ceiling is above what the caller allowed. Their default limit is 50,000 micro-USD ($0.05); set `TOKENTRIAGE_TEST_CEILING_MICRO_USD` to a non-negative integer only when you intend to allow a different maximum. A missing cost figure is never treated as zero. Use a management key in `TOKENTRIAGE_MANAGEMENT_KEY`, plus the explicit base URL, organization, workspace, application, gateway and environment variables shown in each example. No browser cookie is needed. These programs omit a subject selector and test what the environment currently serves. A management key cannot select a saved draft or an explicit release; use the authenticated browser playground for that workflow. The input is a gateway request body, not plain prompt text: ```json {"model":"gateway","messages":[{"role":"user","content":"say hello"}],"max_tokens":64} ``` Python passes these JSON bytes to the SDK; TypeScript supplies their base64 encoding. The saved case retains exactly those request bytes. The programs print the run ID even when polling ends before completion; read that run again rather than restarting the program, which would create another paid run. ```python # sdk/python/examples/test_and_save_run.py, trimmed to the five calls preflight = _data(client.tests.inspect(org, ws, app, gw, env, subject=plan.subject)) created = _data(client.cases.create(org, ws, app, gw, input=plan.input, title=plan.title, origin="authored")) run = _data(client.runs.create(org, ws, app, gw, env, case_id=case_id, case_version=case_version, capture="team")) view = _data(client.runs.get(org, ws, app, gw, env, run_id)) # until terminal content = _data(client.content.get(org, ws, app, gw, output_ref)) ``` ```typescript // sdk/ts/examples/test_and_save_run.ts, trimmed to the five calls const preflight = data( await client.tests.inspect(org, ws, app, gw, env, plan.subject)); const created = data( await client.cases.create(org, ws, app, gw, { title: plan.title, origin: "authored", input: plan.input })); const run = data( await client.runs.create(org, ws, app, gw, env, { case_id: caseId, case_version: caseVersion, capture: "team" })); const view = data(await client.runs.get(org, ws, app, gw, env, runId)); const content = data(await client.content.get(org, ws, app, gw, outputRef)); ``` {/* The shipped Python and TypeScript programs and CLI inspect/get/share/review/create/rerun commands also run in the local PostgreSQL integration proof `TestS08CustomerExamplesAgainstDurableLocalAPI`. It uses the real management-key ledger, current scoped grants, customer HTTP handler, saved-run ledger, ticket and durable worker with a loopback cell. Compiler, release storage, budget and object bytes are explicit fixtures; this verifies client integration without a live provider call. It also rejects management-key draft selection and a rerun after `tests.run` is revoked. */} Every answer on this lane is `{schema, request_id, kind, data}` and the resource sits under `data`. The unit test that drives the TypeScript example through a fake transport asserts the request sequence, which is the part of an example that rots: ```text POST …/environments/{env}/tests/inspect POST …/gateways/{gw}/tests/cases POST …/environments/{env}/tests/runs GET …/environments/{env}/tests/runs/run_1 GET …/environments/{env}/tests/runs/run_1 GET …/gateways/{gw}/tests/content/cobj_9 ``` The same test pins what the script returns when the answer is withheld: `state` reads `succeeded`, `output` is null, and `outputAbsentBecause` carries the slug the server sent, such as `content_read_denied`. Neither example records a verdict, because a verdict belongs to a person, and a script that stamped "approved" on its own output would make the team's history worthless. ## What this lane does not do Four boundaries, each of which someone has expected the other way round: 1. **A saved run does not stream to you live.** The worker owns the dispatch, so the run row moves `queued` to `dispatched` to a terminal word and you read the answer once it is stored. A scratch run answers synchronously, with the completion in the reply body. 2. **A streamed answer is not schema-checked.** `format.jsonschema` declares the output phase alone and no binding can name `stream` beside it, so a test that asks for a stream against an armed gateway arrives unvalidated, with `stream_unsupported` published on the protection read. [Check a response against a JSON Schema](/docs/guardrails/response-schema#streaming-is-not-schema-checked) has the detail. 3. **An application key cannot choose what it runs against.** A management key naming a subject answers `app_key_cannot_select_subject` with a `403`. Selecting a draft revision or a release is something a person does while signed in; an application-class credential always gets what the environment is serving. 4. **A deployment can be built without this lane at all.** Then every row answers `tests_not_enabled` with a `501`, by name rather than as a `404`, so a missing feature and a mistyped path stay distinguishable. Nothing you did caused it and no retry changes it; whoever runs the deployment can turn it on. Two neighbouring tasks live elsewhere. Comparing two configurations on one question, and judging quality rather than reading it, are not on this lane. What your company is billed for these runs, split from your product's traffic, is on [What your company is billed for](/docs/build/company-billing).