Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Upgrade and roll back

Change a value or an image version, watch the pod roll, and get back to where you were. The ledger is untouched throughout.

How-to guideLast updated

On this page

Every change to a running deployment on Kubernetes goes through one command, and every one of them is reversible. There is no separate procedure for a config edit, an image bump or a mode change: you edit values, preview, upgrade, and the pod rolls onto the new configuration.

This page runs the cycle end to end, changing, verifying and rolling back, on a live instance holding 2,303 ledger rows, and shows what each step does to that data. Prerequisites: an installed release and permission to upgrade it. Nothing here needs a maintenance window.

Preview before you apply

Two commands answer “what would this do” without touching the cluster:

$ helm template tt deploy/helm/tokentriage -n tokentriage -f my-values.yaml
$ helm upgrade tt deploy/helm/tokentriage -n tokentriage -f my-values.yaml --dry-run

helm template renders every manifest; --dry-run renders what the upgrade would send. Take the rendered ConfigMap through the binary’s own validator before you go further. Helm cannot run a binary during templating, so a config that renders fine can still abort at boot:

$ helm template tt deploy/helm/tokentriage -f my-values.yaml \
  | python -c "import sys,yaml;[print(d['data']['tokentriage.yaml']) for d in yaml.safe_load_all(sys.stdin) if d and d.get('kind')=='ConfigMap']" \
  | tokentriage validate --config -
mode:      ledger
listen:    0.0.0.0:8787   admin: 0.0.0.0:9090
upstreams: 1
  - anthropic    anthropic http://mock-upstream:8080
tiers:     0 (cheapest -> most capable)
rules:     0
OK: configuration is valid

That upstream is a local mock, which is why it is an http:// service name rather than a provider host. Configuration in production covers what validate does and does not check.

Change a value

Any change to the config body changes the rendered ConfigMap, which changes the checksum/config annotation on the pod template, which rolls the pod. You never have to work out whether a key needs a restart:

$ kubectl get statefulset tt-tokentriage -n tokentriage \
    -o jsonpath='{.spec.template.metadata.annotations.checksum/config}'
aa599e69e1355a17b19de4083460a7cc51688ce0709177e893918841e7923f64

$ helm upgrade tt deploy/helm/tokentriage -n tokentriage -f my-values-v2.yaml
Release "tt" has been upgraded. Happy Helming!
NAME: tt
NAMESPACE: tokentriage
STATUS: deployed

$ kubectl rollout status statefulset/tt-tokentriage -n tokentriage
partitioned roll out complete: 1 new pods have been updated...

$ kubectl get statefulset tt-tokentriage -n tokentriage \
    -o jsonpath='{.spec.template.metadata.annotations.checksum/config}'
b52fda19b9532749beea8b5d357e445c8389ca98a0bc05b887949833a7e369ee

The only difference between those two releases was retention.max_total_mb, from 512 to 256. The checksum moved, so the pod rolled.

This is a strict superset of what the binary itself would require: it always rolls on any config change, which is always safe, at the cost of a restart for edits the running process could have absorbed.

The ledger is not part of the change

The volume is not touched by an upgrade. Same instance, immediately after the roll:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- tokentriage top --log /data/.tokentriage/
TOTAL             2303  2760000  805000  $6.785000

The same 2,303 rows and the same $6.79. A roll replaces the process, and the process is the only thing that was ever ephemeral.

What a roll does end is every browser session, because the session store is in memory. Minted tokens and users live on the volume and keep working, so nobody re-mints; people log in again.

Move the image

The image tag defaults to the chart’s appVersion. Pin a version explicitly, or a digest when you are running from a mirror:

$ helm upgrade tt deploy/helm/tokentriage -n tokentriage --reuse-values \
    --set image.tag=0.2.0

image.digest wins over image.tag when both are set, and latest is refused by the values schema, because a mutable tag is not a deployment you can reason about. The pull policy defaults to IfNotPresent.

Roll back

Helm keeps release history. Rolling back is one command, and it restores the previous configuration byte for byte:

$ helm history tt -n tokentriage
REVISION  UPDATED                   STATUS      CHART              DESCRIPTION
1         Mon Aug 10 06:01:59 2026  superseded  tokentriage-0.1.0  Install complete
2         Mon Aug 10 06:09:06 2026  deployed    tokentriage-0.1.0  Upgrade complete

$ helm rollback tt 1 -n tokentriage
Rollback was a success! Happy Helming!

$ kubectl get statefulset tt-tokentriage -n tokentriage \
    -o jsonpath='{.spec.template.metadata.annotations.checksum/config}'
aa599e69e1355a17b19de4083460a7cc51688ce0709177e893918841e7923f64

That is revision 1’s checksum, recovered exactly. The ledger read 2303 rows and $6.785000 after the rollback as well. The volume is outside the release, so neither direction of the cycle touches it.

A rollback is itself a pod roll, with the same session consequence and the same data consequence, which is to say none.

The field an upgrade cannot change

A StatefulSet’s volumeClaimTemplates is immutable. Kubernetes rejects an upgrade that changes it, which means two values behave differently from every other:

  • persistence.size cannot be grown by upgrading. Use your StorageClass’s volume expansion, editing the claim’s spec.resources.requests.storage directly, where allowVolumeExpansion is true.
  • persistence.storageClass cannot be changed on an existing release. Moving storage classes is a backup and restore onto a fresh claim.

Ordinary version upgrades are unaffected. The chart keeps the version-bearing labels off the volume claim template so bumping appVersion does not trip the immutability rule.

Why the dashboard cannot apply a config change on Kubernetes

The config arrives as a ConfigMap projected read-only at /etc/tokentriage/tokentriage.yaml. The dashboard’s validate and diff surfaces still work, because they compute and do not write. Apply cannot write, and the chart does not fake a writable copy to pretend otherwise.

The binary refuses before it gets that far. An apply carrying any restart-required change returns 422 restart_required with the exact field list, writes an audit record marking the attempt rejected, and changes nothing. Every deployment-critical key is restart-required by the binary’s classification, so on Kubernetes the answer is always the same one: change the value in values.yaml and upgrade.

On a host or in Compose, where the config file is writable, that same surface can apply hot-reloadable edits. Configuration in production covers the difference.

After the upgrade

Re-run the bundled smoke test. It asserts the two facts an upgrade could plausibly break — that the process is up, and that the admin API is still closed to an unauthenticated caller:

$ helm test tt -n tokentriage
NAME: tt
STATUS: deployed
REVISION: 2
TEST SUITE:     tt-tokentriage-test-connection
Phase:          Succeeded

Then check the surfaces that report state rather than liveness: /tt/api/status for upstream health, and doctor for the config and the ledger. Diagnose a running instance has both.

What this does not cover

Moving between trust-ladder modes is the same mechanic, a value change plus an upgrade, but it carries an evidence gate, and the first routing rollout owns that. Upgrading a multi-replica fleet has one extra consideration around rate-counter identity, covered in multi-replica & scale-out. Compose and host deployments replace the binary or the image and restart; there is no release history to roll back to, so keep the previous artifact.