Back up and restore

All of TokenTriage’s state lives under one directory: /data on Kubernetes and in the container images, ~/.tokentriage for a binary on a host. Copy that directory and you have copied everything.

What follows is the procedure, executed end to end on a Kubernetes cluster: a fleet of one holding 2,303 rows and $6.79 of attributed spend was backed up, its volume was deleted, and it was restored to the same row count and the same dollar. The transcripts are from that run.

Before you start. You need kubectl against the cluster, permission to scale the workload, and somewhere off-cluster to put the archive. You do not need a shell in the application container. The image is FROM scratch and has none, which shapes the whole procedure below.

What is worth backing up

Three classes of file live under the data directory, and they are not equally valuable.

The figure separates the two kinds. Everything on the left is a record of something that happened and exists nowhere else; the read-model on the right is a projection you can throw away and recompute. Backing up the whole directory is fine and simplest; the index is overwritten or rebuilt.

Storage the ledger must not sit on

/data must be ReadWriteOnce block storage. The chart’s volumeClaimTemplates hardcodes accessModes: ["ReadWriteOnce"] for this reason, and the reason is rotation.

Rotating the ledger unlinks the active file. On a shared or ReadWriteMany filesystem such as NFS or EFS, or anything mountable by two writers, a rotation on one pod can unlink a file another pod’s handle still holds, and that writer keeps appending to an inode nobody will ever read. The ledger loses decisions and says nothing. RWO block storage, the default class on every major cloud, makes that impossible: one pod holds the volume at a time.

The chart also refuses a ledger path that resolves inside a shared volume you mounted yourself, so the rule is enforced rather than advised.

Take a backup

Scale the workload down first. The volume is ReadWriteOnce, so a helper pod cannot mount it while the application pod is running, and a copy taken from a stopped writer is consistent rather than merely crash-consistent.

$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=0
statefulset.apps/tt-tokentriage scaled
$ kubectl wait -n tokentriage --for=delete pod/tt-tokentriage-0 --timeout=120s
pod/tt-tokentriage-0 condition met

kubectl scale returns before the pod is gone, so the wait is not optional. Schedule the helper too early and it sits in ContainerCreating on a multi-attach error.

Now start a helper pod that mounts the same claim by name and does nothing else, then stream the archive out of it:

$ kubectl run tt-helper -n tokentriage --restart=Never \
    --image=busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028 \
    --overrides='{"spec":{"containers":[{"name":"h","image":"busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028","command":["sleep","1800"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data-tt-tokentriage-0"}}]}}'
pod/tt-helper created
$ kubectl wait -n tokentriage --for=condition=Ready pod/tt-helper --timeout=180s
pod/tt-helper condition met
$ kubectl exec -n tokentriage tt-helper -- tar czf - -C /data .tokentriage > tt-data-2026-08-10.tgz

Then verify it, and only then delete the helper:

$ tar tzf tt-data-2026-08-10.tgz
.tokentriage/
.tokentriage/dash/
.tokentriage/dash/audit.jsonl
.tokentriage/dash/tokens.json
.tokentriage/dash/reconcile/
.tokentriage/decisions-20260810T130318.jsonl.gz
.tokentriage/decisions-20260810T130317.jsonl.gz
.tokentriage/decisions.jsonl
.tokentriage/index/
.tokentriage/index/decisions.sqlite
.tokentriage/decisions-20260810T130239.jsonl.gz
$ kubectl delete pod -n tokentriage tt-helper
$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=1

That listing is trimmed to the interesting entries; the real one also carries the empty dash/ subdirectories. Store the archive off-cluster.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: tt-data-snapshot
  namespace: tokentriage
spec:
  # volumeSnapshotClassName: <your-csi-snapshot-class>
  source:
    persistentVolumeClaimName: data-tt-tokentriage-0

Restore

Restoring reverses the backup: get the files onto a claim under the name the StatefulSet expects, then let it adopt them.

$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=0
$ kubectl wait -n tokentriage --for=delete pod/tt-tokentriage-0 --timeout=120s
pod/tt-tokentriage-0 condition met
$ kubectl run tt-restore -n tokentriage --restart=Never \
    --image=busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028 \
    --overrides='{"spec":{"containers":[{"name":"r","image":"busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028","command":["sleep","1800"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data-tt-tokentriage-0"}}]}}'
pod/tt-restore created
$ kubectl wait -n tokentriage --for=condition=Ready pod/tt-restore --timeout=180s
pod/tt-restore condition met
$ kubectl exec -i -n tokentriage tt-restore -- tar xzf - -C /data < tt-data-2026-08-10.tgz
$ kubectl exec -n tokentriage tt-restore -- wc -l /data/.tokentriage/decisions.jsonl
484 /data/.tokentriage/decisions.jsonl
$ kubectl delete pod -n tokentriage tt-restore
$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=1

The helper is started first and entered with exec rather than fed through kubectl run -i. With the one-shot form tar reaches end-of-file on stdin before the client finishes attaching, the command exits 1 reporting a timeout, and nothing is restored, leaving an empty volume behind while you believe you recovered.

That 484 is the active file only. The rest of the history is in the companions, which is why the next step matters.

Check the restore, on both surfaces

Ask the CLI and the dashboard the same question. They read the same ledger and they should agree:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage top --log /data/.tokentriage/
MODEL             REQS  INPUT    OUTPUT  COST
claude-haiku-4-5  2300  2760000  805000  $6.785000
TOTAL             2303  2760000  805000  $6.785000

$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq .provenance
{
  "rows": 2303,
  "source": "index",
  "indexed_through": { "file": "decisions.jsonl", "offset": 837805 }
}

2,303 on both, matching what the instance held before its volume was deleted. Compare against a figure you recorded before the backup, not against whatever the restore produces. A restore that silently lost history still returns a number.

The trap: a rebuilt index loses your rotated history

The restore above agreed because the archive carried the SQLite index along with the ledger, so nothing was rebuilt. Delete or lose the index and the two surfaces stop agreeing.

Here is that case on the same cluster. The index directory was removed from the restored volume and the pod restarted, which rebuilds the read-model from the configured ledger path:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- tokentriage top --log /data/.tokentriage/
TOTAL             2303  2760000  805000  $6.785000

$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq '.provenance.rows'
484

The CLI joins the rotated companions; the read-model rebuilt from a file path does not. 1,819 rows, every row that had ever rotated, are missing from the dashboard permanently, and the dashboard reports the shortfall as a normal total.

The fix is one command plus a restart, pointed at the directory:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage index rebuild --index /data/.tokentriage/index/decisions.sqlite \
                              --log /data/.tokentriage/
rebuilt index /data/.tokentriage/index/decisions.sqlite from /data/.tokentriage/
  rows indexed:    2303
  indexed_through: file=decisions.jsonl offset=837805 ts=2026-08-10T13:03:18.741Z
$ kubectl delete pod -n tokentriage tt-tokentriage-0

After the pod comes back the query surface reports 2,303 again.

What survives a restore, and what does not

Minted tokens and users keep working. They live in the auth stores on the volume, so a restored deployment accepts the credentials it accepted before. You do not re-mint.

Browser sessions do not. The session store is in memory, so any restart at all, whether a restore, an upgrade or a reschedule, logs every browser out. That is expected, and it is not a sign the restore failed.

The volume is not deleted for you. Claims are retained on helm uninstall and on scale-down; the ledger is not removed for you. Reclaiming the storage is an explicit kubectl delete pvc, which is exactly what the drill above used to simulate the disaster.

Outside Kubernetes

On a host or in Compose the same three classes live under one directory and the same rules apply: copy the whole thing while the process is stopped, verify the archive, and rebuild the index from the directory afterwards. There is no helper pod involved, so the two kubectl races above cannot happen; everything else, including the rebuild trap, is identical.