# Read several instances at once

Point one dashboard at its peers and read merged totals. What is merged, what is disclosed instead of merged, and what never crosses a boundary.

Source: https://tokentriage.com/docs/build/dashboard-federation

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

You run TokenTriage in three regions. Each keeps its own ledger on its own volume, which is
what makes each one fast and independent, and it means no single dashboard shows you the
fleet's spend.

Federation fixes that at read time. One instance becomes a hub, fans a query out to its peers,
and merges the answers. Nothing is copied and nothing is centralised; the merge happens per
query and is thrown away.

## Point a hub at its peers

```yaml
instances:
  - name: us-east
    url: https://tt-use.internal        # https, or a loopback address
    token_env: TT_PEER_USE_TOKEN        # the NAME of an environment variable
  - name: eu-west
    url: https://tt-euw.internal
    token_env: TT_PEER_EUW_TOKEN
```

Each peer needs a bearer token on its own admin plane; the hub presents one when it calls.
`token_env` names an environment variable and the hub reads it at call time, so no peer
credential is written into your config or served back by the config API.

Configuring at least one peer adds the Fleet and Instances entries to the hub's sidebar. An
instance with no peers has neither, and a direct visit to `/fleet` reports a single node
rather than an error.

The peers themselves need no special configuration. A peer is an ordinary instance whose admin
plane the hub can reach and authenticate against.

## What merging does and does not do

The useful part of federation is what it declines to do quietly.

**A disagreement about prices is disclosed, not averaged.** If peers are running different
pricing snapshots, a merged total carries a mixed-snapshot disclosure. You see that the figure
spans two price tables rather than receiving a number that silently blends them.

**A peer that does not answer is a labelled exclusion.** A dead or slow peer produces a merged
total that says how many of how many replicas answered, and which one is missing. A smaller
total is never presented as a complete one.

**Identity does not cross a boundary.** There is no cross-instance trace stitching. A trace
belongs to the instance that saw it, and a fan-out that appears to span regions is two traces
rather than one. Do not expect a waterfall to follow a request across a peer.

**A peer cannot take the hub down.** A panic handling one peer's response is contained, and a
loop guard on the fan-out stops a mesh of hubs re-fanning to each other forever.

<Callout type="key">
A federated figure carries its coverage: how many peers answered, which did not, and whether
the rows behind it span more than one pricing snapshot. Read the coverage line before you read
the total.
</Callout>

## Read the Fleet surface

Fleet shows the roster by heartbeat age, the measured clock-skew bound between replicas, and
the coverage of the last merged read. Liveness is the age of the last heartbeat rather than a
green dot somebody chose to draw.

<Callout type="warning">
The coverage panel on Fleet currently renders a fixed example rather than this instance's
coverage, so it reports two of three replicas answering regardless of your topology. The
roster above it is live and correct. This is filed; [what a new dashboard
shows](/docs/build/dashboard-first-run#panels-that-are-not-yours) lists the other panels
affected.
</Callout>

Operate's [multi-replica and scale-out](/docs/operate/scale-out) page owns running a fleet:
provisioning, the shared coordination store, `fleet list` and `fleet decommission`, and what is
not coordinated across replicas. This page is only the read side.

## Choosing between federation and a shared store

They solve different problems and are often both wanted.

| You want | Use |
|---|---|
| one view of spend across regions | federation, on a hub |
| a limit that counts across replicas | a shared coordination store |
| a replica to survive another's outage | neither — each is already independent |
| one trace spanning two regions | not available |

Federation is read-time and needs nothing shared. A coordinated limit needs replicas writing to
common state, which is [distributed governance](/docs/build/governance-distributed).

<Details summary="What a hub actually asks its peers">

The hub calls the same management-API read endpoints you would call yourself, with a bearer
token, and merges the envelopes. Which means three things follow.

The peer's role model applies. A hub token with the viewer role reads what a viewer reads.

The peer's query budget applies. A hub's fan-out is subject to each peer's deadline and row
cap, which is why a slow peer becomes an exclusion rather than a stall.

Nothing is stored on the hub. There is no replica of a peer's ledger and no cached copy of its
rows; ask again and the hub asks again.

</Details>

## What can go wrong

**A peer is permanently excluded.** Check reachability first, then the token. The hub reads
`token_env` at call time, so a variable that is missing from the hub's environment produces a
credential-less call and a `401` from the peer.

**Totals look low.** Read the coverage line. An excluded peer is disclosed rather than
silently dropped, and that disclosure is the answer.

**A mixed-snapshot chip appears and will not go away.** The peers are on different pricing
snapshots, which usually means different binary versions. Align them, or accept the disclosure
as accurate.

**Fleet is absent from the sidebar.** No peer is configured. Federation is opt-in and a single
instance has nothing to fan out to.

<NextSteps items={[
  {title: "Multi-replica and scale-out", href: "/docs/operate/scale-out", description: "Running the fleet this page reads: topology, the shared store, and what is not coordinated."},
  {title: "Distributed governance", href: "/docs/build/governance-distributed", description: "The other multi-replica problem: a limit that has to count across nodes."},
  {title: "Sign in to the dashboard", href: "/docs/build/dashboard-auth", description: "Minting the bearer token a hub presents to each peer."},
  {title: "How to read a number", href: "/docs/build/dashboard-reading-a-number", description: "The mixed-snapshot chip and the other disclosures a merged figure carries."}
]} />