Rotate keys and the mesh secret without stranding anyone shipped

A rotation across a fleet that is never fully online at once. The schedule — publish, activate, retire — is signed into the fleet list; every node acts on it when it holds it; retire is signed only when the nodes that matter have moved. The mechanism is under Rotation: publish, activate, retire; rotating one node's key by hand is its own guide.

0 — Name the lifeline nodes once

The bootstrap peers keep answering retired material for lifeline_secs (24 h by default) after every retire, so a straggler always finds a node that speaks its key. Flag them once:

$ atlasd fleet set-lifeline gs1
`gs1`: lifeline on — it keeps old material for the lifeline window after each retire (unsigned — run `atlasd fleet sign`, then push)
$ atlasd fleet set-lifeline relay-van
`relay-van`: lifeline on — it keeps old material for the lifeline window after each retire (unsigned — run `atlasd fleet sign`, then push)
$ atlasd fleet sign && atlasd fleet push -c /etc/atlas/config.toml
fleet `field-1` seq 13 signed by 3c24f045 (6 records, 2d8e91f4)
fleet `field-1` seq 13 handed to the atlas0 daemon — it floods it over its sessions

1 — Plan the rotation

The mesh secret alone, a node key alone, or both in one list. The secret is minted on the operator host and sealed to every record's current transport key; a node key names the subject and its next key:

# the mesh secret, published now, activating in an hour (the [rotation] default)
$ atlasd fleet rotate plan --kind mesh-secret --push /etc/atlas/config.toml
  rot-1791900000-mesh_secret: new mesh secret sealed to 6 of 6 record(s); published 1791900000, activates 1791903600
fleet `field-1` seq 14 signed by 3c24f045 (5be2a1c0)
fleet `field-1` seq 14 handed to the atlas0 daemon — it floods it over its sessions

# one node's transport key, with the secret, activating twelve hours after publish
$ atlasd fleet rotate plan --kind node-transport,mesh-secret --subject drone-a \
    --next-key "$(cat drone-a.next.pub)" --activate-after 43200 --push /etc/atlas/config.toml

Publish is immediate unless deferred with --publish-in. From publish, every node accepts the new material beside the old; nothing is used yet. Journal on each node: rotation.published. A signing key rotates the same way (--kind node-signing --next-signing-key …), with one difference: the subject starts signing with the next key when its config carries it as [interface] signing_key — a config apply — not at the instant.

2 — Watch it activate

$ atlasd fleet rotate status -c /etc/atlas/config.toml
rot-1791900000-mesh_secret · mesh_secret · Published · publish 3600 s ago · activate 12 s ago · retire —
  retire gate: every required node reports the rotation active (2 of 2); 2 node(s) still on the old secret
  2 node(s) still on the old secret: drone-b, drone-c
    drone-a        d1a0..7c21   best_effort  active
    drone-b        d1b4..0e93   best_effort  published
    drone-c        d1c9..55af   best_effort  unknown
    gs1            9b1e..40aa   required     active
    relay-van      4f02..c8d1   required     active
    test-bench     e77a..1b06   left_behind  active

The list's own state stays published until a signed step moves it; what each node reports is the row. A node with a trusted clock switches at activate_at; one without switches on receipt and says so (rotation.clock_untrusted); unknown is a node that has not reported this rotation at all. atlasd fleet rotate activate <id> brings the activation forward to now and signs it. For a node key the subject starts initiating with the next key and its peers re-handshake make-before-break; routes and the topology do not move, because the node id is the record's, not the key's. The same table is the Configure tab's Rotations view and GET /api/fleet/rotations.

3 — Retire when the gate passes

Retire is a signed revision of its own, and it is refused until every required node — and, for a node key, its subject — has reported active. Best-effort and left-behind nodes are listed, not awaited:

$ atlasd fleet rotate retire rot-1791900000-mesh_secret -c /etc/atlas/config.toml --push /etc/atlas/config.toml
  retire gate: every required node reports the rotation active (2 of 2); 2 node(s) still on the old secret
  rot-1791900000-mesh_secret: members refuse the old secret from 1791990000, lifeline nodes from 1792076400
fleet `field-1` seq 15 signed by 3c24f045 (c0ffee12)
fleet `field-1` seq 15 handed to the atlas0 daemon — it floods it over its sessions

If a required node has not moved, the command names it and stops (retire waits for 1: relay-van — wait for them, or pass --force). --force --reason "…" retires anyway and leaves an audit entry on the node whose view was overridden (rotation.retire_forced). After retire, members refuse the old secret or key; the lifeline nodes keep answering it for the lifeline window, then stop too. The relay registry is not part of this: relays learn revocations, not retires. A rotation that has not activated can be withdrawn with atlasd fleet rotate cancel <id>; every node stays on its current material.

4 — The straggler, inside and after

drone-c returns inside the window. Its peers still accept the old material, so its first handshake succeeds; the fleet list arrives over that session; it opens its copy of the secret, or activates its next key, and switches. Nobody does anything. If drone-c had been required, retire would simply have waited for it.

drone-c returns after retire and after the lifeline window. Every handshake is refused. On drone-c, atlasd status says this node missed rotation rot-… — it holds only retired material; re-enrol with an invitation, and its journal has rotation.missed. Revoke its record and bring it back through admission with a fresh invitation. There is no back door by design. Only retired keys are refused: had drone-c missed the mesh secret but kept a valid transport key, it would have been let in, taken the list over that session and caught up.

Boundaries

• The retire gate reads what nodes gossip. A node that is up but whose gossip does not reach the operator's node reads as not reported; check its path before forcing.
• A node that is not yet a record — joining by invitation, holding no list — takes part in no rotation until it holds the list.
• Scheduled activation needs a disciplined clock; without one a node activates on receipt, which may be before or after the others. If simultaneity matters, make sure the clocks are.
• Retiring is permanent for the material: a node that returns after the lifeline window re-enrols; its old key cannot be re-admitted.
• Rotation steps from the dashboard need a session; a write-scope API token reaches update-class only.