Multicast video over the mesh

Several stations watching the same drone feed used to mean the scarce hops carried it once per viewer. With [multicast] enabled, listed groups are carried as source-rooted distribution trees: every shared segment carries the stream once, copies are made only where the watchers' paths diverge, and the tree re-forms when the fleet moves — no IGMP, no PIM, no controller. This guide builds the reference topology out of eight atlasd nodes and three config archetypes, reads the proof out of the counters, and states what our own verification rig measured on exactly this shape.

1 — The topology

The shape every drone unit recognises: an ISR aircraft deep in the airspace, a relay layer of two drones, a forward layer of two more, and three ground stations that all want the video. Three segments separate the layers — the aircraft's long-haul link, the inter-drone layer, and the ground distribution network — and the scarce, expensive hop is the first one.

Any lab that can stand up three segments reproduces this; the drawing above is also the healthy steady state you will read out of the counters. Every node runs the same binary with its own small config — atlasd run -c <config>, eight times — and nothing about the tree is configured anywhere: subscriptions travel in the LSA flood, and every node derives its own role — reverse-path parent, subscriber-pruned children — from the topology database it already holds.

2 — Three configs, eight nodes

Each node's identity is a Curve25519 keypair: mint one per node, keep the private half on that machine, hand the public half to every direct neighbour's [[peer]] block. Files live at /etc/atlas/config.toml in production, any path on a bench — and validate before running anything:

$ atlasd keygen
private key: 4Ai…kEY=        # stays in this node's config — mode 0600
public key:  9Xp…bQc=        # goes to every direct neighbour's [[peer]] block
$ atlasd config check -c relay1.toml
relay1.toml is valid

The whole fleet needs only three config shapes. The addresses below follow one convention — tunnel 10.90.0.n and underlay host .n match — and the device names are whatever your lab hands each node; adjust both freely.

[interface]
name = "nexus0"                      # the TUN device this daemon owns
address = "10.90.0.1/24"             # this node's address inside the tunnel
private_key = "<node-private-key>"   # from: atlasd keygen
listen_port = 51820

[[link]]
name = "far"                         # the scarce ISR ⇄ relay segment
bind_device = "eth0"                 # whichever device carries net-far here
bind_address = "0.0.0.0"             # survives the device being re-addressed

[[peer]]                             # relay1
public_key = "<peer-public-key>"
endpoint = ["172.20.10.2:51820"]
allowed_ips = ["10.90.0.2/32"]
persistent_keepalive = 25

[[peer]]                             # relay2
public_key = "<peer-public-key>"
endpoint = ["172.20.10.3:51820"]
allowed_ips = ["10.90.0.3/32"]
persistent_keepalive = 25

[multicast]
enabled = true
groups = ["239.31.0.1"]              # tree-distributed — identical list fleet-wide

[interface]
name = "nexus0"
address = "10.90.0.2/24"
private_key = "<node-private-key>"
listen_port = 51820

[[link]]
name = "far"                         # up: toward the ISR
bind_device = "eth0"
bind_address = "0.0.0.0"

[[link]]
name = "mid"                         # down: toward the forward drones
bind_device = "eth1"
bind_address = "0.0.0.0"

[link_monitor]
probe_interval_ms = 250              # the default, written out: five missed probes
                                     # declare a dead carrier — the repair clock

[[peer]]                             # the ISR — reachable on "far" only
public_key = "<peer-public-key>"
endpoint_per_link = { "far" = "172.20.10.1:51820" }
allowed_ips = ["10.90.0.1/32"]
persistent_keepalive = 25

[[peer]]                             # uav1 — reachable on "mid" only
public_key = "<peer-public-key>"
endpoint_per_link = { "mid" = "172.20.20.4:51820" }
allowed_ips = ["10.90.0.4/32"]
persistent_keepalive = 25

[[peer]]                             # uav2
public_key = "<peer-public-key>"
endpoint_per_link = { "mid" = "172.20.20.5:51820" }
allowed_ips = ["10.90.0.5/32"]
persistent_keepalive = 25

[multicast]
enabled = true                       # same groups list — and no subscribe:
groups = ["239.31.0.1"]              # this node forwards, it never watches

[interface]
name = "nexus0"
address = "10.90.0.6/24"
private_key = "<node-private-key>"
listen_port = 51820

[[link]]
name = "ground"
bind_device = "eth0"
bind_address = "0.0.0.0"

[[peer]]                             # uav1
public_key = "<peer-public-key>"
endpoint = ["172.20.30.4:51820"]
allowed_ips = ["10.90.0.4/32"]
persistent_keepalive = 25

[[peer]]                             # uav2
public_key = "<peer-public-key>"
endpoint = ["172.20.30.5:51820"]
allowed_ips = ["10.90.0.5/32"]
persistent_keepalive = 25

[multicast]
enabled = true
groups = ["239.31.0.1"]
subscribe = ["239.31.0.1"]           # the one line that makes this node a watcher

The forward drones are the relay archetype shifted one segment down — links mid and ground, peers to both relays above and all three stations below. Two things are deliberately absent. There is no route to anything two hops away: allowed_ips only ever names the direct neighbour, and multi-hop reachability is learned, not configured. And there is no description of the tree: the [multicast] block names groups, never shapes.

Groups not listed in groups keep the historical flat fan-out — source-side replication, once per peer, right for mDNS-sized chatter and wrong for video — and with enabled = false the data path is byte-identical to a build without the feature. The configuration reference covers all three keys.

3 — Bring it up and read the tree

$ sudo atlasd run -c relay1.toml
…  handshake complete with peer 1
…  all peer handshakes established

Convergence is observable, and it happens before any video exists. Each daemon rewrites its stats snapshot to /tmp/atlas/nexus0.json every second — the same JSON the REST API serves at /api/stats and the dashboard (atlasd web -i nexus0) renders. The moment the three stations' subscriptions have crossed both drone layers, the ISR's snapshot shows "subscribers": { "239.31.0.1": 3 } — three watchers it was never told about, learned from the flood, typically within seconds of the handshakes completing.

Now give it traffic. Any process on the ISR that writes UDP to 239.31.0.1 on the tunnel device is a source — §6 has real camera pipelines — and every node's multicast block starts telling its role:

$ atlasd stats -i nexus0 -f json-pretty
…
"multicast": {
  "groups":         ["239.31.0.1"],
  "subscribed":     [],
  "subscribers":    { "239.31.0.1": 3 },
  "forwarded":      18744,
  "delivered":      0,
  "rpf_drops":      0,
  "off_tree_drops": 0
}

• groups / subscribed — the tree-distributed list this node carries, and the subset it asked to have delivered locally.
• subscribers — per-group watcher counts learned from the LSA flood; every node on the tree's path knows the number without being told.
• forwarded — copies this node put on the wire for tree groups; delivered — packets handed to this node's own TUN for its subscriptions.
• rpf_drops — arrivals rejected by the reverse-path check, because they came in on a link that is not on this node's shortest path back toward the source.
• off_tree_drops — arrivals for a tree group this node has no subscription and no downstream branch for.

What healthy looks like, role by role: on the trunk — the ISR and whichever relay carries it — forwarded climbs at exactly the frame rate, 1×. On the forward layer it climbs at 3×, one copy per diverging station. On the stations, delivered climbs and forwarded stays at zero — leaves fork nothing. And rpf_drops and off_tree_drops sit at zero everywhere on a stable tree; a brief flicker of either during a topology change is the protection doing its job, and a hop limit caps how far any packet can travel while the fleet's view is still settling.

4 — What the verification rig measured

The ratios above you can watch on any bench. The absolute numbers below are ours — the verification rig this feature ships against is an eight-node containerised fleet in exactly the topology drawn in §1, and before publication we drove five minutes of Full-HD-rate video through it — 6.00 Mbit/s, 625 packets per second of 1,200-byte payloads, 187,500 packets, ≈225 MB — then swept every node's counters after the last packet:

sent by the ISR:                    187,500 packets · 300.0 s · 6.00 Mbit/s

received gs1:                       187,466 of 187,500
received gs2:                       187,454 of 187,500
received gs3:                       187,466 of 187,500

ISR → relay layer (the trunk):      187,500 copies — exactly one per packet
relay layer → forward layer:        187,500 copies
forward layer → 3 ground stations:  562,500 copies (the fork — exactly 3×)
reverse-path drops anywhere:              0

same trunk, per-viewer unicast:     562,500 copies

Read the trunk line twice — it is the whole feature. Three watchers at full-HD rate for five minutes, and the scarce ISR-to-relay hop carried each packet exactly once: 187,500 copies where per-viewer unicast would have put 562,500 on that same hop. The tripling happened only at the layer where the three paths diverge — exactly 562,500 copies, not one more — and the handful of packets that went missing (at most 46 of 187,500 per station) did so downstream of the fork at delivery, not in the tree: the counters balance. A gentler 2 Mbit/s run on the same rig delivers loss-free; at sustained 625 pps the residual receiver-side loss is precisely the case the Video class's forward error correction exists for.

The invariant is what you should take away: the trunk count equals the frame count — at any stream duration and for any number of watchers. Add ground stations and the trunk cost does not move; only the fork widens, on the layer that has the fan-out capacity anyway.

5 — Kill the carrying drone

Find the forward drone whose forwarded counter is climbing at 3× — that one is carrying the branches — and, mid-stream, take it away: stop its daemon, or power the node off; the mesh cannot tell the difference. Five missed probes later its neighbours declare the links dead, the LSA flood carries the news, and every node re-derives the tree from its updated topology database — no election, no signalling round. The branches re-form through the surviving forward drone, and on the ground stations delivered keeps climbing through the kill. Start the daemon again and the tree heals back the same way.

With equal-cost paths the fork need not sit on one node: after a re-derivation — or from the start — different stations' branches may ride different forward drones. The accounting is indifferent to that: the layer's summed copies stay 3× the trunk either way.

6 — Point a real camera at it

Nothing in the mesh knows or cares what the frames are: any producer that writes UDP to a multicast group works unmodified, and receivers just join the group on the atlas interface — the kernel's own multicast filtering applies exactly as on a LAN.

# sender (the ISR node) — RTP/H.265 from a test source, or swap in your camera
gst-launch-1.0 videotestsrc is-live=true ! x265enc bitrate=4000 tune=zerolatency \
  ! rtph265pay ! udpsink host=239.31.0.1 port=5000 multicast-iface=nexus0 ttl-mc=8

# each watching station
gst-launch-1.0 udpsrc address=239.31.0.1 port=5000 multicast-iface=nexus0 \
  caps="application/x-rtp,media=video,encoding-name=H265" \
  ! rtph265depay ! h265parse ! avdec_h265 ! autovideosink sync=false

The same pattern holds for ffmpeg -f mpegts udp://239.31.0.1:5000?localaddr=…, MAVLink camera streams, or anything else datagram-shaped.

7 — Boundaries, stated plainly

Delivery is deliberately unreliable-with-FEC: no ACK trees, no retransmit fan-in — pair the group with the Video transport class (deadline drops, Reed–Solomon repair) when the trunk is lossy, and keep reliable one-to-many on store-carry-forward where it belongs. Transient re-rooting can double-deliver for a moment; the receive-side dedup window absorbs it. Tree traffic is hop-by-hop encrypted like all mesh relaying — relays are trusted, authenticated members of your fleet. And two fleet rules before this leaves the lab: enable [multicast] only once every node runs a build that has it — an unupgraded relay drops tree traffic as unroutable — and keep the groups list identical everywhere, because it is what sources and forwarders consult.