# Upgrade and roll back Change a value or an image version, watch the pod roll, and get back to where you were. The ledger is untouched throughout. Source: https://tokentriage.com/docs/operate/upgrades --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; Every change to a running deployment on Kubernetes goes through one command, and every one of them is reversible. There is no separate procedure for a config edit, an image bump or a mode change: you edit values, preview, upgrade, and the pod rolls onto the new configuration. This page runs the cycle end to end, changing, verifying and rolling back, on a live instance holding 2,303 ledger rows, and shows what each step does to that data. **Prerequisites:** an installed release and permission to upgrade it. Nothing here needs a maintenance window. ## Preview before you apply Two commands answer "what would this do" without touching the cluster: ```console $ helm template tt deploy/helm/tokentriage -n tokentriage -f my-values.yaml $ helm upgrade tt deploy/helm/tokentriage -n tokentriage -f my-values.yaml --dry-run ``` `helm template` renders every manifest; `--dry-run` renders what the upgrade would send. Take the rendered ConfigMap through the binary's own validator before you go further. Helm cannot run a binary during templating, so a config that renders fine can still abort at boot: ```console $ helm template tt deploy/helm/tokentriage -f my-values.yaml \ | python -c "import sys,yaml;[print(d['data']['tokentriage.yaml']) for d in yaml.safe_load_all(sys.stdin) if d and d.get('kind')=='ConfigMap']" \ | tokentriage validate --config - mode: ledger listen: 0.0.0.0:8787 admin: 0.0.0.0:9090 upstreams: 1 - anthropic anthropic http://mock-upstream:8080 tiers: 0 (cheapest -> most capable) rules: 0 OK: configuration is valid ``` That upstream is a local mock, which is why it is an `http://` service name rather than a provider host. [Configuration in production](/docs/operate/configuration) covers what validate does and does not check. ## Change a value Any change to the config body changes the rendered ConfigMap, which changes the `checksum/config` annotation on the pod template, which rolls the pod. You never have to work out whether a key needs a restart: ```console $ kubectl get statefulset tt-tokentriage -n tokentriage \ -o jsonpath='{.spec.template.metadata.annotations.checksum/config}' aa599e69e1355a17b19de4083460a7cc51688ce0709177e893918841e7923f64 $ helm upgrade tt deploy/helm/tokentriage -n tokentriage -f my-values-v2.yaml Release "tt" has been upgraded. Happy Helming! NAME: tt NAMESPACE: tokentriage STATUS: deployed $ kubectl rollout status statefulset/tt-tokentriage -n tokentriage partitioned roll out complete: 1 new pods have been updated... $ kubectl get statefulset tt-tokentriage -n tokentriage \ -o jsonpath='{.spec.template.metadata.annotations.checksum/config}' b52fda19b9532749beea8b5d357e445c8389ca98a0bc05b887949833a7e369ee ``` The only difference between those two releases was `retention.max_total_mb`, from `512` to `256`. The checksum moved, so the pod rolled. This is a strict superset of what the binary itself would require: it always rolls on any config change, which is always safe, at the cost of a restart for edits the running process could have absorbed. ## The ledger is not part of the change The volume is not touched by an upgrade. Same instance, immediately after the roll: ```console $ kubectl exec -n tokentriage tt-tokentriage-0 -- tokentriage top --log /data/.tokentriage/ TOTAL 2303 2760000 805000 $6.785000 ``` The same 2,303 rows and the same $6.79. A roll replaces the process, and the process is the only thing that was ever ephemeral. What a roll *does* end is every browser session, because the session store is in memory. Minted tokens and users live on the volume and keep working, so nobody re-mints; people log in again. ## Move the image The image tag defaults to the chart's `appVersion`. Pin a version explicitly, or a digest when you are running from a mirror: ```console $ helm upgrade tt deploy/helm/tokentriage -n tokentriage --reuse-values \ --set image.tag=0.2.0 ``` `image.digest` wins over `image.tag` when both are set, and `latest` is refused by the values schema, because a mutable tag is not a deployment you can reason about. The pull policy defaults to `IfNotPresent`. If you have an external backend enabled, the chart fails the render when the effective image tag parses below that backend's minimum version, naming the floor. An older binary strict-decodes the backend's config keys as unknown and crash-loops at load, so the refusal at render time is the cheap version of that failure. Raise `image.tag` to the named floor. ## Roll back Helm keeps release history. Rolling back is one command, and it restores the previous configuration byte for byte: ```console $ helm history tt -n tokentriage REVISION UPDATED STATUS CHART DESCRIPTION 1 Mon Aug 10 06:01:59 2026 superseded tokentriage-0.1.0 Install complete 2 Mon Aug 10 06:09:06 2026 deployed tokentriage-0.1.0 Upgrade complete $ helm rollback tt 1 -n tokentriage Rollback was a success! Happy Helming! $ kubectl get statefulset tt-tokentriage -n tokentriage \ -o jsonpath='{.spec.template.metadata.annotations.checksum/config}' aa599e69e1355a17b19de4083460a7cc51688ce0709177e893918841e7923f64 ``` That is revision 1's checksum, recovered exactly. The ledger read `2303` rows and `$6.785000` after the rollback as well. The volume is outside the release, so neither direction of the cycle touches it. A rollback is itself a pod roll, with the same session consequence and the same data consequence, which is to say none. ## The field an upgrade cannot change A StatefulSet's `volumeClaimTemplates` is immutable. Kubernetes rejects an upgrade that changes it, which means two values behave differently from every other: - **`persistence.size`** cannot be grown by upgrading. Use your `StorageClass`'s volume expansion, editing the claim's `spec.resources.requests.storage` directly, where `allowVolumeExpansion` is true. - **`persistence.storageClass`** cannot be changed on an existing release. Moving storage classes is a [backup and restore](/docs/operate/backup-restore) onto a fresh claim. Ordinary version upgrades are unaffected. The chart keeps the version-bearing labels off the volume claim template so bumping `appVersion` does not trip the immutability rule.
The config arrives as a ConfigMap projected read-only at `/etc/tokentriage/tokentriage.yaml`. The dashboard's **validate** and **diff** surfaces still work, because they compute and do not write. **Apply** cannot write, and the chart does not fake a writable copy to pretend otherwise. The binary refuses before it gets that far. An apply carrying any restart-required change returns `422 restart_required` with the exact field list, writes an audit record marking the attempt rejected, and changes nothing. Every deployment-critical key is restart-required by the binary's classification, so on Kubernetes the answer is always the same one: change the value in `values.yaml` and upgrade. On a host or in Compose, where the config file is writable, that same surface can apply hot-reloadable edits. [Configuration in production](/docs/operate/configuration) covers the difference.
## After the upgrade Re-run the bundled smoke test. It asserts the two facts an upgrade could plausibly break — that the process is up, and that the admin API is still closed to an unauthenticated caller: ```console $ helm test tt -n tokentriage NAME: tt STATUS: deployed REVISION: 2 TEST SUITE: tt-tokentriage-test-connection Phase: Succeeded ``` Then check the surfaces that report state rather than liveness: `/tt/api/status` for upstream health, and `doctor` for the config and the ledger. [Diagnose a running instance](/docs/operate/diagnostics) has both. ## What this does not cover Moving between trust-ladder modes is the same mechanic, a value change plus an upgrade, but it carries an evidence gate, and [the first routing rollout](/docs/get-started/first-routing-rollout) owns that. Upgrading a multi-replica fleet has one extra consideration around rate-counter identity, covered in [multi-replica & scale-out](/docs/operate/scale-out). Compose and host deployments replace the binary or the image and restart; there is no release history to roll back to, so keep the previous artifact.