Mesh & multi-hop routing

Three or more nodes stop being pairs and become a fleet: every node learns the whole topology, computes shortest paths, and forwards for peers it cannot reach directly — with no controller anywhere. This page covers the routing machinery, what you actually configure for a relay chain, and one crypto boundary you must design around.

Link-state routing, no controller

Every node floods link-state advertisements (LSAs) to its established peers: who it is, which neighbors it hears and at what cost (best alive RTT per link), which tunnel prefixes it serves, and optionally its name, role and position. LSAs age out after 100 seconds, refresh every 30, and departed nodes leave tombstones so a stale advertisement cannot resurrect them. On top of the database, Dijkstra computes routes; forwarding carries a TTL'd relay header whose hop budget is [routing] max_hops: 8 hops by default, up to 255. The origin stamps its budget and every relay honours that stamp, so nodes with different settings agree on how far a packet may go.

Two cost modes: measured cost (default — routes prefer low-RTT paths and shift as quality changes) or hop count ([scheduler] strategy = "shortest_path" — every alive link costs 1, so routes hold steady under probe jitter). Cost changes only trigger a re-flood when they clear a hysteresis threshold — max(5 ms, 25% of the link's cost) — so a wobbling link does not chat the fleet to death.

What a 3-node relay chain actually requires

For A — B — C, each node peers only with its direct neighbors: A lists B, B lists A and C, C lists B. That is the whole trick — A and C need no configuration for each other. LSAs advertise every node's tunnel address, SPF resolves A→B→C, and B forwards. The middle node's config is ordinary:

# Node B — the relay. IPs are documentation addresses; use your own.
[interface]
name = "nexus0"
address = "10.0.100.11/24"
private_key = "NODE_B_PRIVATE_KEY"
listen_port = 51820
node_role = "relay"              # advertised in LSAs; display-only

[[link]]
name = "primary"
bind_device = "eth0"
bind_address = "0.0.0.0"

[[peer]]                          # node A
public_key = "NODE_A_PUBLIC_KEY"
endpoint = ["192.0.2.1:51820"]
allowed_ips = ["10.0.100.1/32"]

[[peer]]                          # node C
public_key = "NODE_C_PUBLIC_KEY"
endpoint = ["198.51.100.2:51820"]
allowed_ips = ["10.0.100.2/32"]

Want more than SPF gives you? [routing] path_redundancy = 2 computes the top-2 next-hops and fans every fragment across both with receive-side dedup — killing any single link in a diamond topology becomes loss-free because the alternate path is already carrying live traffic. Or pin routing entirely with [[routing.static_path]] entries (a named hop list with a stable path_id stamped into the relay header) — SPF keeps measuring for telemetry, but forwarding obeys the operator; automatic failover is deliberately off in that mode. The two knobs are mutually exclusive, and the static table wins.

The encryption boundary — read this one twice

Static versus discovered peers

Peers written in the config are static; peers learned at runtime — through mesh membership, an auth provider extending trust, or relay learning — are discovered, and the difference matters when things go quiet:

• Static peers are never deleted. A configured peer that stops answering is marked unreachable and kept — its sessions, probes and your explicit intent survive the outage, and journal events fire on both edges of the transition.
• Discovered peers expire after [mesh] expiry_secs (default 60) of gossip silence, destroying their sessions and any punched endpoints. Deployments that lean on discovery should size this knob generously — and know that under the default static auth provider, an expired discovered peer cannot re-add itself.

The [mesh] section is membership, not routing

A subtlety that saves confusion: LSA flooding and multi-hop SPF run unconditionally for established peers — a statically peered chain routes multi-hop with no [mesh] table at all. What [mesh] enabled = true adds is the membership plane: secret-authorized join (mesh_secret), a replicated member registry, heartbeats and expiry, anti-entropy gossip with a random member it can reach directly each interval, and bootstrap via bootstrap_peers — so nodes can find each other without anyone editing peer lists. Gossip is encrypted and replay-damped, sized to fit single datagrams (large rosters page across several), periodically re-synced in full to repair losses, and re-anchored to the seed list on a slow timer so a partitioned mesh heals instead of staying split forever.

Route flooding on narrow bearers

A link that carries 250 kbit/s cannot afford the same routing chatter as a gigabit port — so the flooding cadence is per link: [[link]] route_advert_interval_secs overrides the global [mesh] route_advert_interval_secs (default 2) for that bearer only, clamped so a paced link still delivers at least three LSAs inside every aging window. A narrowband radio and a fast backbone can carry the same mesh without the radio drowning in updates.

Position adverts

Range prediction and the SPF's terrain weights read advertised positions, so every node weighs an edge alike and routing stays loop-free. A node's position rides in its LSA, which used to be re-originated only on a link change or the 30 s refresh, so a moving node's position — and every reroute that depends on it — lagged by up to 30 s. A moving node now re-advertises when it moves. Three [link_monitor] keys, all applied without a restart:

• position_advert_distance_m (25) — moving that far from the position in the last advert sends one at once; 0 turns it off.
• position_advert_interval_secs (10) — a node with a fix that has moved at all (at least 1 m) re-advertises at least this often; 0 turns it off; more than 30 is warned, because the refresh already carries the position.
• position_advert_min_interval_secs (1.0, from 0.1 to 3600) — the fewest seconds between two position adverts, node-wide.

A third trigger falls out of terrain: when a neighbour path changes terrain state and the node has moved since its last advert, its position goes out too. The fix is checked every 250 ms. The rate limit lives inside the bearer pacing: a position advert is an ordinary flood that marks every link as owing one; a link with its own route_advert_interval_secs carries it only when that interval is due, coalesced with the newest fix, and every other link at most once per minimum interval. Position adverts are not held to the global [mesh] route_advert_interval_secs spacing. Snapshot block position_adverts: the live settings, sent, by_distance / by_period / by_terrain, rate_limited, paced_held, moved_m, and per link floods, position_floods, paced_held. A 220 m move reaches the other nodes in 136–210 ms; behind a ridge the route moves in about 0.5 s instead of about 23 s. Tuning is in Tune position adverts for fast movers.

What else rides the LSAs

• Node identity for humans: node_name (defaults to the hostname) and node_role — ground-station, uav, ugv, usv, dismount, relay, or your own value — advertised so topology views can label and symbolize nodes. Purely descriptive; routing never reads it — but with Mesh DNS enabled, node_name also becomes the node's resolvable name, answered from every node's own copy of this database.
• Capabilities: the advert's caps list carries the names the node advertises in its handshake — seal, multicast, piggyback and the rest (see capability negotiation) — which is how a node learns what a destination it has no session with supports.
• Position and velocity: from static config ([interface] position), a live feed on the telemetry socket, or an operator's PUT /api/position. A moving node re-advertises it as it moves (position adverts). Fleets that move use it for predictive scheduling and range-aware route costs, and nodes holding elevation tiles judge each advertised position against the terrain.

Headroom-aware routing for Control

Priority at a hop cannot reorder packets already inside a radio's buffer; path choice can. Each node advertises how loaded each of its links is, and the classes in [routing] headroom_classes (default ["Control"]) take the path with room even when a shorter one exists.

• The advert. Every link entry of the LSA can carry util_pct (bytes sent over load_window_secs against the declared capacity, or the measured drain rate when lower) in 10-point steps, and queue_ms from the standing-queue signal in 50 ms steps. A step counts once the reading stays a quarter step past the boundary for one probe interval; zero fields are omitted, so an idle mesh floods what it flooded before.
• The second shortest path. A second Dijkstra over the same graph and rules uses edge cost RTT + queue_ms + RTT·((util−70)/30)²·4 above 70 % utilisation: a saturated hop costs up to five times its RTT plus its queue.
• Who follows it. The packet's class byte selects the map at the origin and at every transit hop, so a packet stays on one map end to end. "*" sends every class by headroom; [] switches the second computation off. Unclassified traffic (0xFF, a sender without QoS) keeps the default map. With path_redundancy ≥ 2 the top-K paths come from whichever map the class selects.
• Why it cannot loop. Every node computes the same map from the same flooded figures, never from a fresher local reading, so each next hop's shortest path is the rest of the path.

[routing]
headroom_classes = ["Control", "Voice"]  # default ["Control"]

[[link]]
name = "uhf"
capacity = "250 kbps"   # utilisation is read against this

Seeing it. The snapshot gives topology.links[].util_pct and .queue_ms, routes[] entries with class: "default" | "control", and a headroom block (classes, originated, forwarded, forwarded_default). atlasd status lists a control route under its prefix where it leaves by another next hop; the dashboard's Mesh & Routes does the same, and loaded topology edges carry a 90% · q150 badge. When a hop fills with video, the Control route moves to the path with room within about three seconds.

Boundaries. Advertised figures lag by one advert interval. Utilisation needs an honest capacity: a slow bearer left at the default reads idle until its queue builds. Load is per underlay link, shared by every neighbour behind it. Static paths and multicast trees ignore the headroom map; Control clamped to a lower class by a sender's record takes the default map; the control channel's own messages are unclassified.