# Back up and restore

One volume holds everything. Take a copy you have verified, and get it back with the row count and the dollars intact.

Source: https://tokentriage.com/docs/operate/backup-restore

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import BackupClassesDiagram from '../../../components/BackupClassesDiagram.astro';

All of TokenTriage's state lives under one directory: `/data` on Kubernetes and in the
container images, `~/.tokentriage` for a binary on a host. Copy that directory and you have
copied everything.

What follows is the procedure, executed end to end on a Kubernetes cluster: a fleet of one
holding 2,303 rows and $6.79 of attributed spend was backed up, its volume was deleted, and it
was restored to the same row count and the same dollar. The transcripts are from that run.

**Before you start.** You need `kubectl` against the cluster, permission to scale the
workload, and somewhere off-cluster to put the archive. You do **not** need a shell in the
application container. The image is `FROM scratch` and has none, which shapes the whole
procedure below.

## What is worth backing up

Three classes of file live under the data directory, and they are not equally valuable.

| Class | Files | Rebuildable | Back it up |
|---|---|---|---|
| canonical ledger | `decisions.jsonl` and its `.jsonl.gz` companions | no | yes — this is the one you cannot lose |
| authoritative stores | `governance.db`, `dash/tokens.json`, `dash/users.yaml`, `dash/share-tokens.json`, `dash/audit.jsonl` | no | yes |
| derived index | `index/decisions.sqlite` | yes, from the ledger | optional |

<BackupClassesDiagram />

The figure separates the two kinds. Everything on the left is a record of something that
happened and exists nowhere else; the read-model on the right is a projection you can throw
away and recompute. Backing up the whole directory is fine and simplest; the index is overwritten or rebuilt.

<Callout type="key">
The ledger is the system of record. Losing the auth stores logs everyone out until you mint
again; losing `governance.db` loses governance state; losing the ledger loses history that
nothing can reconstruct.
</Callout>

## Storage the ledger must not sit on

`/data` must be **ReadWriteOnce block storage**. The chart's `volumeClaimTemplates` hardcodes
`accessModes: ["ReadWriteOnce"]` for this reason, and the reason is rotation.

Rotating the ledger unlinks the active file. On a shared or ReadWriteMany filesystem such as
NFS or EFS, or anything mountable by two writers, a rotation on one pod can unlink a file
another pod's handle still holds, and that writer keeps appending to an inode nobody will ever read. The
ledger loses decisions and says nothing. RWO block storage, the default class on every major
cloud, makes that impossible: one pod holds the volume at a time.

The chart also refuses a ledger path that resolves inside a shared volume you mounted
yourself, so the rule is enforced rather than advised.

## Take a backup

Scale the workload down first. The volume is ReadWriteOnce, so a helper pod cannot mount it
while the application pod is running, and a copy taken from a stopped writer is consistent
rather than merely crash-consistent.

```console
$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=0
statefulset.apps/tt-tokentriage scaled
$ kubectl wait -n tokentriage --for=delete pod/tt-tokentriage-0 --timeout=120s
pod/tt-tokentriage-0 condition met
```

`kubectl scale` returns before the pod is gone, so the `wait` is not optional. Schedule the
helper too early and it sits in `ContainerCreating` on a multi-attach error.

Now start a helper pod that mounts the same claim by name and does nothing else, then stream
the archive out of it:

```console
$ kubectl run tt-helper -n tokentriage --restart=Never \
    --image=busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028 \
    --overrides='{"spec":{"containers":[{"name":"h","image":"busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028","command":["sleep","1800"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data-tt-tokentriage-0"}}]}}'
pod/tt-helper created
$ kubectl wait -n tokentriage --for=condition=Ready pod/tt-helper --timeout=180s
pod/tt-helper condition met
$ kubectl exec -n tokentriage tt-helper -- tar czf - -C /data .tokentriage > tt-data-2026-08-10.tgz
```

Then verify it, and only then delete the helper:

```console
$ tar tzf tt-data-2026-08-10.tgz
.tokentriage/
.tokentriage/dash/
.tokentriage/dash/audit.jsonl
.tokentriage/dash/tokens.json
.tokentriage/dash/reconcile/
.tokentriage/decisions-20260810T130318.jsonl.gz
.tokentriage/decisions-20260810T130317.jsonl.gz
.tokentriage/decisions.jsonl
.tokentriage/index/
.tokentriage/index/decisions.sqlite
.tokentriage/decisions-20260810T130239.jsonl.gz
$ kubectl delete pod -n tokentriage tt-helper
$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=1
```

That listing is trimmed to the interesting entries; the real one also carries the empty
`dash/` subdirectories. Store the archive off-cluster.

<Callout type="warning">
A backup you have not listed is a hope. On the run behind this page the same archive taken with
`kubectl run -i --rm` instead of a long-lived helper plus `kubectl exec` came out one byte
short, 296,390 bytes against the 296,391 the pod produced, and `tar tzf` failed with a CRC
error while `ls -la` showed a plausible file. The stdin-attached one-shot form races the
client's detach. Use the two-step form above, and run `tar tzf` every time.
</Callout>

<Details summary="If your storage class supports CSI volume snapshots, prefer that">

A `VolumeSnapshot` is crash-consistent, needs no downtime and no helper pod, and is the better
option wherever your `StorageClass` offers it:

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: tt-data-snapshot
  namespace: tokentriage
spec:
  # volumeSnapshotClassName: <your-csi-snapshot-class>
  source:
    persistentVolumeClaimName: data-tt-tokentriage-0
```

Restoring means creating a claim from the snapshot **under the name the StatefulSet expects**
(`data-tt-tokentriage-0`) before you install or scale up, so the workload adopts it. The
file-level path in the main procedure is the fallback for classes without snapshot support —
which includes the local-path provisioner the run behind this page used, and is why the
transcripts show the fallback rather than this.

</Details>

## Restore

Restoring reverses the backup: get the files onto a claim under the name the StatefulSet
expects, then let it adopt them.

```console
$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=0
$ kubectl wait -n tokentriage --for=delete pod/tt-tokentriage-0 --timeout=120s
pod/tt-tokentriage-0 condition met
$ kubectl run tt-restore -n tokentriage --restart=Never \
    --image=busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028 \
    --overrides='{"spec":{"containers":[{"name":"r","image":"busybox:1.37.0@sha256:9532d8c39891ca2ecde4d30d7710e01fb739c87a8b9299685c63704296b16028","command":["sleep","1800"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data-tt-tokentriage-0"}}]}}'
pod/tt-restore created
$ kubectl wait -n tokentriage --for=condition=Ready pod/tt-restore --timeout=180s
pod/tt-restore condition met
$ kubectl exec -i -n tokentriage tt-restore -- tar xzf - -C /data < tt-data-2026-08-10.tgz
$ kubectl exec -n tokentriage tt-restore -- wc -l /data/.tokentriage/decisions.jsonl
484 /data/.tokentriage/decisions.jsonl
$ kubectl delete pod -n tokentriage tt-restore
$ kubectl scale -n tokentriage statefulset/tt-tokentriage --replicas=1
```

The helper is started first and entered with `exec` rather than fed through `kubectl run -i`.
With the one-shot form `tar` reaches end-of-file on stdin before the client finishes attaching,
the command exits `1` reporting a timeout, and **nothing is restored**, leaving an empty
volume behind while you believe you recovered.

That `484` is the active file only. The rest of the history is in the companions, which is why
the next step matters.

## Check the restore, on both surfaces

Ask the CLI and the dashboard the same question. They read the same ledger and they should
agree:

```console
$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage top --log /data/.tokentriage/
MODEL             REQS  INPUT    OUTPUT  COST
claude-haiku-4-5  2300  2760000  805000  $6.785000
TOTAL             2303  2760000  805000  $6.785000
```

```console
$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq .provenance
{
  "rows": 2303,
  "source": "index",
  "indexed_through": { "file": "decisions.jsonl", "offset": 837805 }
}
```

2,303 on both, matching what the instance held before its volume was deleted. Compare against a
figure you recorded *before* the backup, not against whatever the restore produces. A restore
that silently lost history still returns a number.

## The trap: a rebuilt index loses your rotated history

The restore above agreed because the archive carried the SQLite index along with the ledger, so
nothing was rebuilt. Delete or lose the index and the two surfaces stop agreeing.

Here is that case on the same cluster. The index directory was removed from the restored volume
and the pod restarted, which rebuilds the read-model from the configured ledger **path**:

```console
$ kubectl exec -n tokentriage tt-tokentriage-0 -- tokentriage top --log /data/.tokentriage/
TOTAL             2303  2760000  805000  $6.785000

$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq '.provenance.rows'
484
```

The CLI joins the rotated companions; the read-model rebuilt from a file path does not. 1,819
rows, every row that had ever rotated, are missing from the dashboard permanently, and the
dashboard reports the shortfall as a normal total.

The fix is one command plus a restart, pointed at the **directory**:

```console
$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage index rebuild --index /data/.tokentriage/index/decisions.sqlite \
                              --log /data/.tokentriage/
rebuilt index /data/.tokentriage/index/decisions.sqlite from /data/.tokentriage/
  rows indexed:    2303
  indexed_through: file=decisions.jsonl offset=837805 ts=2026-08-10T13:03:18.741Z
$ kubectl delete pod -n tokentriage tt-tokentriage-0
```

After the pod comes back the query surface reports 2,303 again.

<Callout type="warning">
Run `index rebuild` with `--log` pointing at the ledger **directory**, not at the file. The
flag's default is the configured log file, so a bare `tokentriage index rebuild` on any install
that has ever rotated rebuilds a read-model containing only the active file. Reported as a
product defect; until the default changes, pass the directory.
</Callout>

## What survives a restore, and what does not

**Minted tokens and users keep working.** They live in the auth stores on the volume, so a
restored deployment accepts the credentials it accepted before. You do not re-mint.

**Browser sessions do not.** The session store is in memory, so any restart at all, whether a
restore, an upgrade or a reschedule, logs every browser out. That is expected, and it is not a sign the
restore failed.

**The volume is not deleted for you.** Claims are retained on `helm uninstall` and on
scale-down; the ledger is not removed for you. Reclaiming the storage is an explicit
`kubectl delete pvc`, which is exactly what the drill above used to simulate the disaster.

## Outside Kubernetes

On a host or in Compose the same three classes live under one directory and the same rules
apply: copy the whole thing while the process is stopped, verify the archive, and rebuild the
index from the directory afterwards. There is no helper pod involved, so the two `kubectl`
races above cannot happen; everything else, including the rebuild trap, is identical.

<NextSteps items={[
  {title: "The decision ledger on disk", href: "/docs/operate/ledger-lifecycle", description: "What the companion files are, how fast they accumulate, and what retention deletes."},
  {title: "Upgrade and roll back", href: "/docs/operate/upgrades", description: "The other day-two procedure that touches the volume, and why an upgrade never does."},
  {title: "Runbooks", href: "/docs/operate/runbooks", description: "Symptom-first procedures, including the ones that end in a restore."},
  {title: "Sizing and capacity", href: "/docs/operate/capacity", description: "How large the archive gets, and what the volume needs to hold."}
]} />