Fleet configuration: epochs, pushes, rotation on · pushing needs an author key

Documents the atlasd 2.0.0 release lineView as Markdown

A fleet's shared policy changes through signed epochs that travel over the mesh's own gossip, with no controller to hand them out. This page covers what an epoch is, how a canary rollout gates a change, how update classes decide which nodes a release waits for, how the node that pushed follows the change to every member, and how keys and the mesh secret rotate on a schedule that no node has to be online for.

Epochs and canary rollouts

An epoch is a signed envelope: a strictly increasing number, the author, optional role or node targeting, an optional minimum daemon version, the confirm window and the policy TOML, signed with Ed25519. Only the policy sections — scheduler, link_monitor, qos, traversal and per-role overlays — may travel in one; identity, peers, auth and mesh settings are refused at signing and at verification. Each node checks the signature against the authors on its disk, keeps a high-water epoch so a replayed or older envelope is rejected, and runs an accepted epoch through the commit-confirmed apply cycle. Gossip delivers it, and a node that was offline catches up when it reconnects.

A canary rollout is two epochs. Phase 1 carries the new policy to a canary set (canary_nodes, canary_roles, or both intersected); phase 2 releases the same bytes to everyone, or an abort signs the previous policy as a newer epoch aimed at the canaries. The gate allows a release only when every canary has applied and none has failed; a failed canary blocks it unless the release is forced. Start, release and abort are under /api/config/rollout/*, and the dashboard drives the same calls.

Following a push: states, delivery and replay

Every node gossips its config status back, and two fields say how the epoch reached it: received_via, the short id of the neighbour whose gossip delivered it (self on the node that pushed), and received_at, when it arrived. They sit with the rest of each node's status under mesh.config_status.<short id> in GET /api/stats, and the journal's epoch.received event carries via.

The dashboard turns each node's status into one push state for the rollout in hand:

applied — running the epoch.
awaiting confirm, staged, received — on its way through the commit-confirmed cycle, latest first.
pending — targeted, nothing reported for this epoch yet.
silent — targeted, and nothing heard past the rollout's watch deadline: the canary deadline during phase 1, and after release twice the confirm window, never less than two minutes.
failed — rejected (signature, rollback, version or validation) or reverted by the confirm cycle.
not targeted — outside the epoch's targeting.
old version — running a daemon below the epoch's min_daemon_version.

The node that pushed records a timeline of every node's status changes for the rollout's epochs, stamping each arrival with that node's own received_at. Once the rollout has ended and every node has settled on the final epoch, or fifteen minutes after it ended, the timeline is archived. The last fifty archives are kept, sealed when data at rest is enabled.

GET /api/config/rollout?interface=nexus0          # live record, gate, canaries, timeline
GET /api/config/rollouts?interface=nexus0         # archives, newest first
GET /api/config/rollouts/<rollout_id>?interface=nexus0   # one archive with its timeline

A timeline entry is { at, node_id_short, state, via, epoch }; an archive summary carries rollout_id, phase, started_at, finished_at, canary_epoch, final_epoch, note, nodes, events and closed. Two views read them. The Configure tab's Fleet view adds Via and Received columns and a push-state chip, sorts the rows worst first during a rollout, and lists the archive. The Topology tab's Propagation toggle rings each node in its push state, runs a pulse along the link each copy arrived on, draws a dashed arc from the pusher when the deliverer is not on the canvas, marks nodes off the fleet's majority build with a version chip, and adds a scrubber for replay. Follow a rollout from the dashboard walks through both.

Update classes: required, best-effort, left-behind

Every node record carries an update_class, and a rollout reads it. The class is in the record, not in the epoch: an envelope's signed bytes are the same whatever the classes of the fleet, so the daemon on every node accepts every epoch exactly as before.

required — gates the release. The rollout is not complete until every required node has confirmed; a required node that reports failed aborts the release at once, and one still unconfirmed at the release deadline aborts it then. A required canary that is silent past the canary deadline aborts in the canary phase.
best_effort — tracked, never awaited. It is listed in the gate and applies when it is next seen; a release completes without it.
left_behind, with pinned = true|false — outside targeting and both gates. The node reads its own record: it adopts the envelope for anti-entropy (its neighbours still catch up through it), records its state as left_behind, and does not stage or apply while the rollout runs. Unpinned, it applies the newest epoch left_behind_grace_secs after first sight, so it catches up late and never during the watch; pinned, it never applies. The Fleet view shows it as running epoch N−k.

Releasing opens a release watch. The rollout record moves to released with a release_deadline, and every intake tick joins the targeted set — roles and nodes, minus the left-behind — against the gossiped config status to compute a release_gate: required_total, required_confirmed, required_silent, required_failed, the deadline, holding (required nodes still to confirm), stragglers (best-effort nodes not yet confirmed) and a nodes list with each node's class, state and epochs_behind. All required confirmed → phase = "complete", a terminal phase that lists the best-effort stragglers. Any required node failed, or unconfirmed at the deadline → an automatic abort: the pre-rollout policy signed forward as a newer epoch to everyone the release reached, and abort_cause = { node_id_short, name, reason, phase } kept on the record — reason is silent or failed, phase is release, or canary for a required canary silent past the canary deadline.

[config_mgmt]
release_watch_secs = 0        # per rollout unless set here; 0 = 3 × confirm_secs + 120
left_behind_grace_secs = 600  # an unpinned left-behind node applies this long after first sight

Journal: rollout.release_watch when the watch opens, rollout.complete, and rollout.auto_abort with rollout_id, node and reason. Each node's gossiped status gains update_class and state_epoch (the epoch its state speaks for, so an older rollout's rejection never counts against this one), so the Fleet view's Class column and atlasd fleet rollout status show the class beside the state without a lookup. See Push a release the fleet cannot half-take.

Rotation: publish, activate, retire

A node's transport key, its signing key and the fleet's mesh secret rotate on one kind of schedule, carried in the signed fleet list under rotations[]. Three instants, publish_at ≤ activate_at ≤ retire_at, and what each means:

kind	publish	activate	retire
`node_transport`	peers accept the node's next key at the handshake; the node keeps using its current key	the node initiates with the next key; peers re-handshake make-before-break; the node id is unchanged, so the LSDB has no churn	peers refuse the old key
`node_signing`	verifiers accept adverts and lists under either key	the node signs with the next key once its config carries it as `[interface] signing_key` — a config apply, not the instant; the schedule still gates and retires it like the others	the old key is refused
`mesh_secret`	nodes add the new secret to the accepted list	nodes send gossip under the new secret	the old secret is removed from the accepted list

# one schedule in the signed fleet list
[[rotations]]
id          = "rot-1791900000-mesh_secret"
kind        = "mesh_secret"          # node_transport | node_signing | mesh_secret
# subject   = "<record uuid>"        # node kinds only: the record whose key rotates
# next_key_id = "<key id>"           # node kinds only: the next key, already in the record
publish_at  = 1791900000
activate_at = 1791903600
retire_at   = 0                      # 0 = not yet signed: nodes treat retire as never
state       = "published"            # planned | published | active | retired | cancelled
wrapped     = [{ recipient = "<node id>", blob = "<the new secret sealed to that record's current transport key>" }]

Material travels only over authenticated sessions, on the fleet-list channel, never in membership gossip: a node still on the old secret receives the new one through a session it already has, and during the overlap every node opens gossip under each accepted secret in turn. A node reconciles the schedules on every list intake, computes each phase from the clock against the instants, applies idempotently and persists its rotation state encrypted at rest. A node without a trusted clock treats activate_at as on receipt and journals rotation.clock_untrusted. Each node reports its state per rotation in its gossiped status (rotations: { <id>: published | active | retired | missed | blocked }; blocked means the node cannot take part — no blob sealed to it, or the subject without the next private key), which is where the dashboard's "N nodes still on the old key" and the retire gate come from.

Retire is a signed step. A later list revision sets retire_at; until then nothing retires. The operator's node computes the retire gate — every required node, and a node rotation's subject whatever its class, has reported active; best-effort and left-behind listed, not awaited — and atlasd fleet rotate retire, like the dashboard button, refuses unless the gate passes or --force is given, which is recorded (rotation.retire_forced). Retire reaches members first; records flagged lifeline = true (the bootstrap peers) keep accepting the old material for lifeline_secs longer, and a retired transport key's valid_to (retire plus the lifeline) ends it for everyone. The relay registry is not on the schedule: relays learn revocations, not retires. A rotation cancelled before activate leaves every node on its current material; after activate only the retire step ends it.

[rotation]
default_publish_to_activate_secs = 3600      # 1 h from publish to activate, for `fleet rotate plan`
default_activate_to_retire_secs  = 604800    # 7 days: the earliest retire the operator is offered
lifeline_secs                    = 86400     # 24 h: lifeline nodes keep the old material this long after retire
require_trusted_clock_for_scheduled_activation = true

Journal on every node: rotation.published, rotation.active, rotation.retired, rotation.blocked and rotation.missed, plus rotation.clock_untrusted and rotation.retire_forced where they apply. A node holding only retired material is refused at every handshake, and a peer still on a retired key is evicted the moment a node reaches the retire; its own atlasd status says which rotation it missed and that it re-enrols with an invitation. Only retired keys are refused: a node that missed a mesh-secret rotation but still holds a valid transport key is admitted, takes the list over that session and catches up. The per-node mechanics of two keys in one record are under Identity: rotation with overlap; the fleet-wide procedure is Rotate keys and the mesh secret without stranding anyone.

Nodes that were offline

Three answers, depending on what the node missed.

A push. A best-effort node applies the newest epoch when it reconnects, through gossip anti-entropy, and the archived rollout shows its late arrival. A required node that was offline at release aborted the release at the deadline; it comes back to the baseline like everyone else. A left-behind node adopts the envelope, then applies after the grace (unpinned) or never (pinned).
A rotation, back inside the window. The node rejoins on its old material — its peers still accept it until retire, and the lifeline nodes for lifeline_secs after — receives the list over its first session, opens its copy of the secret or activates its next key, and switches with no operator action. The others lose nothing.
A rotation, back after retire and after the lifeline window. Refused at every handshake. There is no back door by design: it comes back through admission with a fresh invitation as a new record, and the operator revokes its old one.