Transport classes & FEC opt-in · default off

On a wide link, fairness is free. On a 64 kbit/s bearer shared by a command channel and a picture feed that fits it — greyscale stills, one frame a second at most — something still decides which one waits when both have data ready, and if you didn't decide, the queue did. The transport-class path makes that decision explicit: classify traffic, give every class a policy, and let the narrow moments degrade the right things first. With [qos] enabled = false (the default) none of this machinery is constructed, and the data path is byte-identical to a build without it.

Six classes, one honest queue

Traffic is sorted into Control, Voice, Position, Telemetry, Video and Bulk — highest priority first — by a classifier that needs no application changes: DSCP marks win when present (CS6/CS7 → Control, EF → Voice, AF41 → Video, CS1 → Bulk), well-known ports fill the gaps (MAVLink's UDP 14550/14551 → Control, RTP ranges → Video), and everything else defaults sanely (TCP → Bulk). Class and reliability mode ride in the class byte of every wire-5 inner header; a node with [qos] disabled reads it as no class and forwards the traffic as usual. What needs both ends is negotiated in the handshake: FEC is used only towards a next hop that advertises it, and a Fec class takes the plain path towards one that does not. Store-carry-forward needs nothing from the far end, because a flushed bundle leaves as an ordinary frame.

The queue is strict-priority with two safeties: per-class reserved-rate floors so even Bulk trickles rather than starving outright, and a byte-budget AQM (backlog_bytes, default 4 MB) that sheds from the lowest class when the backlog fills. A 4 Mbps video stream can never crowd out a 2 kbit/s command channel — that sentence is the feature.

What each kind of data needs

The lane is the contract; the primitive is how the contract is kept. The second line of the Lane column names what selects each primitive: a class's reliability or multipath value or its repeat setting in the policy document (GET/PUT /api/qos), or the level of a [[qos.flow]] rule in config.toml. The shipped tactical_default table already sets the first four rows this way.

In the policy document those values read "reliability": "DeadlineBounded", "multipath": {"kind": "Duplicate", "copies": 2} and "repeat": {"copies": 3, "spacing": "rtt/3"} — the Control default. Fec{n} also needs fec_enabled, and the short lifetime is bundle_lifetime_ms. Repeats arm only when the class's plan resolves to a single link and the peer has fewer than two alive: the same encrypted fragments are queued again copies − 1 times, spacing apart (a fixed "40ms" also works, null switches them off), every copy before the deadline, and the receiver's replay window drops the extra copies. [qos] repeat_budget_pct (20) caps them as a share of that link's capacity.

Precedence levels inside a lane

Two streams of one class otherwise compete as equals. Each packet also carries a precedence level — 0 primary, 1 secondary, 2 tertiary — in two inner-header flag bits (PREC_LO|PREC_HI; the reserved value 3 reads as 2). A node with [qos] disabled leaves the bits clear and ignores them, and relays carry them through.

• Where levels come from. DSCP: AF41/AF42/AF43 → Video 0/1/2, AF31/AF32/AF33 → Telemetry 0/1/2. Operator rules, which are tried before the DSCP and port rules and so win over a mark: [[qos.flow]] with src (IPv4 prefix), dst_port (UDP or TCP), or both, plus class and level (default 0). atlasd config check rejects an unknown class, a level above 2, a bad prefix and a rule with neither match key.
• Service and shedding. Each lane of the class queue holds one FIFO per level; it serves the lowest level first and sheds the highest first.
• Where drops really happen. Under saturation the lower levels also give way at the link: a secondary packet is refused once only a third of a link's 32-slot queue is free (tertiary: two thirds), and for the feasibility check a secondary packet counts on a third of the primary's time left (tertiary: a sixth), so a growing queue sheds it well before the primary's window closes.
• Budgets. budget_by_level_ms = [200, 400, 800] in the policy file or under [qos.classes.Video]; without it, the class budget × 1, 1.5 and 2.
• Counters. GET /api/qos and the stats file add classes[].levels — three entries with enqueued, sent, dropped_late, dropped_aqm, dropped_infeasible, dropped_queue_full — once a lower level has carried traffic; Prometheus atlas_qos_level_dropped_packets_total with level="primary"|"secondary"|"tertiary" and reason; a level table on the QoS tab.

The video governor reads the same counters: started with --camera-level 0|1|2, one instance per camera, it holds its camera while only lower levels are being shed and steps it down when its own level is. The guide A two-camera vehicle walks through both ways of setting levels.

Deadline budgets: late data is a lie

Real-time classes carry a per-packet deadline derived from the bond itself — deadline_k (default 3.5) × the worst alive link's RTT, clamped between deadline_floor_ms (80) and deadline_ceil_ms (1000). A packet that exceeds its budget while queued is dropped at dequeue: a video frame that would arrive 2 seconds late is worthless, and transmitting it anyway would delay the frames that still matter. Each class's reliability mode says what lateness means for it — reliable-bounded, deadline-bounded, latest-value-only, best-effort, or store-carry-forward.

Per-class multipath

Each class chooses how it uses the bond, overriding the global scheduler for its own traffic:

Links that miss a class's latency budget are excluded from its stripe set before weights are computed; an emptied set falls back to the single best link, and an all-dead bond falls back to transmitting on everything configured. The default policy table ships sensible tactics — Control duplicated ×2 with an 80 ms budget and a reserved floor, Video striped with a 200 ms budget, Bulk stored when the peer is away — and everything is editable live.

Forward error correction

FEC batches K data packets (fec_block_packets, default 8), computes M parity shards with a systematic Reed-Solomon code, and stripes all K+M across the aggregate link set. Any K surviving shards reconstruct the block — loss on any link, recovered at the receiver, with no retransmission round trip. An idle flush (fec_flush_ms, 60) bounds latency when traffic is sparse, and incomplete blocks are abandoned after fec_block_timeout_ms (200), delivering whatever data shards arrived. It is doubly gated: the global fec_enabled switch and a per-class Fec policy must both be set. A node with both on advertises fec in its handshake (see capability negotiation); towards a next hop that does not, a Fec-policy class goes as the plain class, striped and unbatched, so no peer is handed a shard it cannot decode. Worked example: 1-in-8 packet loss becomes invisible to the application for 12.5% overhead.

Store-carry-forward

Bulk traffic addressed to a peer with no alive links is not dropped — it is stored (memory, or a file-backed write-ahead log via bundle_backend = "file") and flushed priority-then-FIFO the moment any link to that peer returns. Bounded by count, bytes and lifetime (bundle_max_count 10 000, bundle_max_bytes 16 MB, bundle_lifetime_ms 10 minutes). A vehicle that drives out of coverage delivers its backlog when it drives back in.

Complete footage by backfill shipped

The live view of a video stream drops what arrives too late, which is right for the pilot and wrong for the archive a search or an incident review needs. A packet goes missing in two ways: the sender's budget dropped it as late, or the air lost it. Backfill closes both: the sender keeps every packet of a retained class, the receiver asks for exactly the holes it has, and only those packets cross again, below every live class.

• Retention. retain = true on a class ([qos.classes.Video], the policy file or the QoS API) journals it at the sender. Each packet to a unicast destination gets a stream sequence at TUN ingress, before the class queue, so a packet the live path later drops still has a number. The journal (<state>/<iface>.retain/) is a ring bounded by [qos.retain] max_bytes ("2 GiB") and max_secs (86400), sealed per record under the store key when [storage] at_rest is on. A full write channel drops the journal copy, counted, never the live packet. retain is refused on a store-carry-forward class, which is already reliable.
• Gap requests. The destination writes every packet of a journaled stream once to its capture, tracks holes per origin, class and stream, and after gap_request_delay_ms (2000, so reordering and FEC get their chance) sends a GapRequest (control type 0x11) for the ranges: at most one per gap_request_interval_ms per origin, 64 ranges and 4096 packets outstanding, backing off while the origin is silent. It travels on the control channel, so it is routed, relayed and end-to-end sealed like data.
• The Bulk lane. The origin answers with the stream's highest sequence and the ranges its journal no longer holds (marked unrecoverable), and re-sends the rest only when no live class has anything queued, only over next-hop links with a short send queue and no standing queue, at most max_inflight (1024) per destination, optionally capped by backfill_max_kbps. Toward a destination with no path nothing is sent; the receiver asks again when the path returns, during the flight or after it.
• The capture. <iface>.capture/<origin>/<Class>/<stream>.cap, packet-complete and sealed the same way. A backfilled packet keeps its original send time and goes to the capture only, never to the TUN. atlasd capture export writes the stream in sequence order with its gaps annotated, and tools/atlas-capture, outside the daemon, turns that into pcapng or a playable file.

Seeing it. atlasd retain status on the sender, atlasd capture status on the receiver; backfill.retain and backfill.capture in the snapshot and GET /api/qos (per stream: completeness_pct, missing, unrecoverable, oldest_missing_secs); a Retention card, an Archive completeness card and a "complete footage" chip on the QoS tab. Five minutes of a lossy link complete about 14 s after the loss stops. The guide Complete footage from a SAR drone sets it up end to end.

Boundaries. Frames are packets to Atlas; making them playable is the converter's job. The receiver captures only with [qos] enabled. A capture that reaches max_bytes stops growing (full_dropped) until atlasd capture clear, and never deletes footage itself. A stream whose source is an interconnect-mapped address is journaled but not backfilled. A power cut loses up to the last ~64 KiB of journal writes, which come back unrecoverable. POST /api/capture/export refuses captures over 256 MiB; export those on the node.

Adaptive per-class escalation

Optionally, individual classes escalate their own protection as conditions degrade — Duplicate adds copies, Aggregate steps into FEC and thickens it — with hysteresis and dwell on both directions, and two global guards that keep self-protection from becoming self-harm: an escalation bandwidth budget (default 30% of the peer's alive capacity) and a queue-depth gate so congestion loss can never trigger load amplification. Every decision is visible in the per-interface QoS stats document.

Drop accounting

Every packet the daemon discards is counted once, by reason, at the place the decision was taken. The class queue counts per class; each link's send queue counts per link and per class, and those figures are also summed per class. Link-level counting runs with QoS off too, under the unclassified bucket. Per-class arrays in the snapshot are indexed 0 Control … 5 Bulk, 6 unclassified; links[].drops is omitted while every counter is zero, and link series appear in Prometheus once they are non-zero.

In the class queue

• Late — the budget expired while the packet waited; dropped at dequeue. GET /api/qos classes[].stats.dropped_late; atlas_qos_dropped_packets_total with reason="late".
• Shed from the backlog — the newest arrival pushed the shared backlog past backlog_bytes and the lowest lane gave way. classes[].stats.dropped_aqm; reason="aqm".
• Infeasible — no link could deliver the packet before its deadline, so it was dropped at planning ([qos] feasibility, on by default). classes[].dropped_infeasible; reason="infeasible".

In a link's send queue

• Late at the link — the packet's budget ran out in the link's queue; dropped there instead of being sent late. Snapshot links[].drops.late_by_class; classes[].stats.dropped_late_link; atlas_link_dropped_packets_total with reason="late_link" (labels link, class) and the same reason on atlas_qos_dropped_packets_total.
• Queue full — the link's queue refused the packet. links[].drops.queue_full_by_class, which sums to the link's send_dropped; classes[].stats.dropped_queue_full; reason="queue_full" on both families.
• Abandoned before send — a refused fragment ended a multi-fragment packet before any piece was accepted. links[].drops.abandoned_unsent; atlas_link_abandoned_packets_total with phase="unsent".
• Abandoned after send — part of the packet was already accepted; the airtime is lost. links[].drops.abandoned_sent; classes[].stats.abandoned_sent; phase="sent".
• Cancelled — fragments the link worker skipped because a sibling was refused (fragments, not packets). links[].drops.cancelled.

With precedence levels in use, classes[].levels splits the class-queue and planning drops and the link refusals by level. The dashboard reads the same figures: the link card's Drops line by reason, the QoS tab by class. Troubleshooting has an entry per family. The engineering site states the reasons as a table.

Turning it on

[qos]
enabled = true              # master switch — off means none of this exists
fec_enabled = true          # second gate for FEC; classes still need a Fec policy
rx_release_buffer = true    # optional Video jitter/reorder buffer (default off)
bundle_backend = "file"     # durable store-carry-forward across restarts
bundle_path = "/var/lib/atlas/nexus0.bundles.wal"

Class policies, per-class escalation and the guards live in a JSON policy document beside the daemon — re-read every reload_secs (default 2) and editable through GET/PUT /api/qos with merge semantics — so policy changes land mid-session with no restart and no packet loss. The configuration reference documents every key and default.