Troubleshooting

The failures a bench run actually produces, written as symptom → cause → fix. Every entry ends at the page that explains the mechanism — this page exists so you don't have to know in advance which page that is. If your problem isn't here, it may be documented behavior: check Known limitations before assuming a bug.

Read the daemon before guessing

Four commands answer most questions, in this order:

1. atlasd config check — is the configuration even valid? Run it before every start; it catches the errors below that are catchable statically.
2. atlasd status — the one-line truth: interface, strategy, how many links are alive, how many peers established.
3. atlasd stats — per-link measurements as JSON: aliveness, RTT, loss. This is what the scheduler sees, so it is what your traffic experiences.
4. atlasd journal — the event history: handshakes, link transitions, peer changes, with timestamps. Almost every symptom below leaves its cause here.

The same data is available remotely via the REST API (/api/stats, /api/journal, /api/health) and continuously via Prometheus metrics.

The daemon exits immediately at startup

Symptom: atlasd run (or the systemd unit) terminates within a second of starting.

Causes, most common first:

• Invalid configuration. The daemon refuses to run on a config it cannot fully parse — it does not skip the parts it dislikes. atlasd config check prints exactly what is wrong and where; fix until it reports the file is valid.
• Missing privilege. Creating the TUN device and programming routes needs root or CAP_NET_ADMIN. A bench run as an ordinary user fails here; the packaged systemd unit grants that capability, and CAP_NET_BIND_SERVICE for the mesh DNS resolver’s port 53, to an unprivileged service user — use it for anything beyond a bench.
• The listen port is taken. Another process — or another atlasd instance you forgot about — already owns the configured listen_port. One daemon per interface; check with ss -ulpn.

Fix: run atlasd config check first, always. If the config is valid and it still exits, read the lines it printed on the way out — and under systemd, journalctl -u holds them.

Config validation rejects two links on one address

Symptom: atlasd config check refuses a config in which two [[link]] entries share a device and bind address.

Cause: this is the validator saving you from a genuinely nasty failure. Two UDP sockets on the same address and port make the kernel load-balance inbound datagrams between them — each link would silently lose half its traffic while both look "up". The architecture page explains the mechanism.

Fix: give each link on a shared device and address its own listen_port. Two links on different devices need no special handling — this rule exists specifically for the one-device, one-address case (a second path over the same NIC, or the traversal pattern of one link per bearer).

Peers never establish

Symptom: both daemons run, atlasd status shows links alive, but the peer count stays at zero establishments and the journal shows no handshake complete line.

Causes, in the order worth checking:

• Crossed keys. Each config carries its own private key and the other node's public key. The classic miss is pasting a node's own public key into its own [[peer]] block — re-derive from atlasd keygen output and check both files side by side, as in the quickstart.
• Wrong or unreachable endpoint. Every address in the peer’s endpoint list must be one the underlay can actually deliver to — test with a plain ping of the underlay address itself. Tunnel establishment never performs a DNS lookup; endpoints are IPs.
• A firewall eating UDP. The handshake is UDP to the peer's listen_port. Host firewalls that default-drop inbound UDP produce exactly this symptom: both sides initiate, neither ever answers.
• Protocol version skew. Every packet carries a protocol version byte; two nodes on different wires fail loudly at decode rather than negotiating. A node accepts wire 5 only (the relay servers accept 5 and 4); a wire-5 node that receives wire 1, 2, 3 or 4 counts the refusals in protocol_refused and names the sender's endpoint and version once in the log and the journal (peer.protocol_refused); atlasd --version on both ends prints the wire each one speaks. When the counter rises, the two speak different wires and will not establish: every node of a fleet must run the same wire — release notes flag every wire change and Versioning explains the contract; upgrade a deployment as a deployment.
• An auth provider saying no. Under [auth] providers beyond the default, a peer that fails the allowlist or the external command is rejected by policy, not by fault — see Peer authentication.

Handshake complete, but nothing flows through the tunnel

Symptom: the journal shows handshake complete, yet a ping of the peer's tunnel address gets nothing back.

Causes:

• Tunnel addressing. Both [interface] address values must sit in the same subnet — 10.0.100.1/24 and 10.0.100.2/24 in the quickstart. A typo'd netmask or mismatched subnets means the host never routes the ping into the TUN device at all.
• IPv6 inside the tunnel. The inner tunnel is IPv4-only in the current release line (the underlay may be IPv6). Pinging a peer's IPv6 address through the tunnel cannot work — see Known limitations.
• A host firewall on the TUN device. The tunnel interface (nexus0) is an ordinary network device to the host; a default-drop INPUT policy applies to it like any other. Allow the tunnel subnet explicitly.

Fix: confirm addressing first — it accounts for most of these — then test with the tunnel-side addresses from each end while watching atlasd stats to confirm packets are actually entering a link.

A link is declared dead while the bearer looks fine

Symptom: a link the underlying network says is healthy keeps dropping out of the bond and rejoining.

Cause: liveness is decided by probes, not by carrier state — by default a probe every 250 ms per link, and a link is dead after 5 consecutive misses. A bearer whose round trip or scheduling jitter regularly exceeds the probe budget (a congested cellular link, a long-latency or half-duplex radio) will miss probes while carrying real traffic.

Fix: tune the probing to the bearer's honest characteristics — the probe interval is a per-link setting (probe_interval_ms), and declaring an honest capacity per link keeps the scheduler's cost model from overdriving narrow links (the default assumption is 80 Mbps). Watch atlasd stats after each change: the numbers there are the decision inputs, so when they stabilise, the flapping stops.

The abandoned-after-send counter is growing

Symptom: a link card's Drops line shows "abandoned after send" in amber, and links[].drops.abandoned_sent (in Prometheus, atlas_link_abandoned_packets_total with phase="sent") keeps climbing.

What it means: part of a packet was handed to that link and a later fragment was refused because the link's send queue was full. The daemon withdrew the rest of the packet from the link, but the fragments already accepted still go out, on a packet the far end can never reassemble. Its sibling, abandoned_unsent, costs nothing; this one is wasted airtime.

Cause: the link is offered more multi-fragment traffic than it drains. Every link queue holds 32 packets whatever the bearer: a few milliseconds of traffic on Ethernet, seconds on a 64 kbit/s radio. Once that queue is full, a large packet straddles the boundary. The usual sources:

• A capacity declared too high — or not at all, which assumes 80 Mbps — so the delivery-time arithmetic keeps sending bulky packets to a narrow link.
• A class policy that puts a copy on the slow link — Duplicate{n} copies go to the n best links, which on a small bond includes the narrow one, and broadcast uses every link.
• A heavy class allowed onto a narrowband bearer — Video or Bulk on a serial radio that should carry only Control and Position.

Fix: declare an honest capacity for the link first; it is what keeps large packets off thin links without further tuning. Then keep the heavy classes off the narrow bearer with a link group and [qos.group.<name>] rules, or give the duplicating class a policy that does not include it. The counter should flatten while abandoned_unsent and the queue-full refusals carry any remaining overload. The mechanism is on the architecture page.

Packets of a class are dropped as late

Symptom: dropped_late or dropped_late_link climbs for a class in GET /api/qos, or the link card's Drops line shows "late".

Cause: the packet's budget ran out before it could leave. In the class queue (dropped_late) it waited behind higher lanes; for Video under a flood that is the design working. In a link's queue (dropped_late_link, per link in links[].drops.late_by_class) the link drains more slowly than the scheduler believed: a declared capacity above what the bearer delivers, or a bearer that is saturated.

Fix: for class-queue drops, compare the class budget with the paths — a fixed budget below the worst usable round trip can never be met, and a derived one follows deadline_k × the worst alive RTT. For link-queue drops, look at the link's queue delay first (a link that reads alive but is late), declare an honest capacity and leave feasibility on, so a packet that cannot make its deadline is dropped at planning instead of occupying the link. The counters are listed in Drop accounting.

A link refuses packets as queue full

Symptom: links[].drops.queue_full_by_class and the link's send_dropped rise together; the Drops line shows "queue full".

Cause: the link's send queue (32 packets) was full when the packet arrived — the link is offered more than it drains. On a narrow bearer this is normal at the edges of a burst; constant refusals mean the scheduler keeps choosing the link for traffic it cannot carry.

Fix: the same levers as for late drops: an honest capacity, and keeping heavy classes off the narrow bearer with a link group and [qos.group.<name>] rules. Look at which class the refusals belong to before changing anything: refusals of Bulk on a radio are a policy question; refusals of Control are a capacity problem.

Packets are counted as abandoned

Symptom: links[].drops.abandoned_unsent or abandoned_sent is non-zero.

Cause: a queue-full refusal hit a packet split into fragments, and the rest of that packet was withdrawn from the link. abandoned_unsent is harmless on its own — the queue-full figure, seen as whole packets. abandoned_sent means part of the packet had already gone out.

Fix: treat abandoned_unsent as queue-full refusals (above). For a growing abandoned_sent, see the abandoned-after-send entry.

A link reads alive but everything on it is late

Symptom: the link answers probes and stays alive, yet traffic on it arrives seconds late, or its class shows late drops. The link card's Queue row (snapshot links[].queue_delay_ms) reads hundreds of milliseconds or more.

Cause: a buffer below the daemon is full. links[].egress_queue says which one: qdisc_bytes for a shaper or a deep device queue in front of a modem, serial_bytes for a serial radio whose line cannot keep up, rising qdisc_drops when that queue overflows. Most often the bearer's real rate has fallen below what the link declares — a busy cell, a radio that dropped to a lower rate — and the scheduler keeps feeding it.

Fix: leave standing_queue on (the default), so the queue delay counts in the link's delivery time and traffic moves away while the figure stays high. Declare an honest capacity, and keep heavy classes off a narrow bearer. If the device queue itself is deep, shorten it — a queue sized for a fast Ethernet card holds a long time on a slow modem. If the Queue row stays near zero while the RTT climbs, the backlog is inside the radio or modem, where the host cannot read it. On a SiK-family radio with Atlas on both ends, a native serial link reads the radio's buffer and paces on it. The figures are described under Queues below the tunnel.

A link shows the standing-queue flag

Symptom: the link card carries a yellow standing-queue chip, atlasd status prints queue N ms standing after the link, and links[].standing_queue is true.

What it means: for at least one whole interval, even the luckiest packet on that link waited longer than the target — a tenth of the link's RTT, at least 5 ms. The queue in front of the link is not draining: the link is being offered more than it carries right now. queue_delay_ms is the wait that adds to every packet.

What it is not: loss. Nothing has been dropped because of the flag, and the link is still alive. It is the earlier warning — the delay is already in the link's delivery time, so the scheduler moves traffic away before the queue overflows. A flag that comes and goes with bursts is the signal working.

What to check: whether it is the daemon's queue or the kernel's — links[].egress_queue showing large qdisc_bytes or serial_bytes puts it below the daemon (see a link that reads alive but is late). If it stays set, the bearer's real rate is below what it is asked to carry: compare its declared capacity with what it delivers, and check whether a duplicating policy or a heavy class keeps feeding it. With standing_queue = false the flag still shows; only the scheduler stops reacting. The rule is under The standing-queue signal.

Traversal connects, but the path never upgrades past the relay

Symptom: two NATed peers communicate — through a relay — and hole punching never yields a direct path.

Causes:

• That may be the correct outcome. A pair of endpoint-dependent ("hard") NATs is not punchable without port prediction, which Atlas deliberately does not attempt — such pairs relay indefinitely by design. See Known limitations.
• NAT classification is starved. Distinguishing the NAT types that predict punch success requires STUN observations from two distinct server addresses — two ports on one IP cannot do it. Every Atlas relay answers STUN on both of its public addresses precisely for this; if you run your own stun_servers list, it must contain two different IPs, or classification silently degrades and punching is skipped where it might have worked. The traversal page covers the whole ladder.

Fix: check what the classifier concluded (journal + /api/stats) before blaming the puncher. Remember the relay path staying up is the designed floor, not a failure — the scheduler treats a punched path as one more link and shifts to it only when it is genuinely better.

A discovered mesh peer disappears after about a minute

Symptom: a peer that joined via mesh gossip works, then vanishes from the roster roughly a minute after its traffic stops.

Cause: runtime-discovered peers expire after [mesh] expiry_secs (default 60 s) of gossip silence — expiry destroys their sessions and punched endpoints, and under the default static auth provider an expired discovered peer cannot re-add itself. Statically configured peers are exempt.

Fix: peers that matter belong in the config as [[peer]] entries — discovery is for churn, static configuration is for infrastructure. If a longer memory for discovered peers is genuinely what you want, raise expiry_secs; see Mesh & multi-hop routing for what membership does and does not govern.

An applied configuration reverted on its own

Symptom: a config change applied over the REST API was live, and some minutes later the previous configuration was back.

Cause: that is commit-confirmed apply doing its job. A staged configuration must be confirmed within the timeout window after being committed; if the confirmation never arrives — typically because the change broke the very connectivity you were managing the node over — the daemon reverts to the last known-good configuration rather than staying wedged.

Fix: none needed for the revert itself — it is the safety mechanism. Re-apply the change, and send the confirm step over a path the change cannot sever (a second link is exactly the right tool). The full stage → commit → confirm lifecycle, with the timeout-revert outcome, is documented on the API page.

A serial link stays closed

Symptom: the link card's serial chip is amber and reads “closed — reopening”, links[].serial.open is false, open_errors keeps climbing, and the link never comes alive.

Cause: last_error names it. open: Device or resource busy means another process holds the port — most often a SiK adapter or pppd left running from before; once the link holds the port it holds it exclusively, but a process that opened it exclusively first keeps it. No such file or directory means the path does not exist: the radio is unplugged, or a /dev/ttyUSB number moved after re-enumeration. Permission denied means the daemon's user may not open the device.

Fix: stop and disable the adapter and pppd for that radio (fuser -v on the device shows who holds it), name the radio by its /dev/serial/by-id/ path, and give the daemon's user access to the device. The link retries every 500 ms on its own; no restart is needed once the port is free. The serial radio guide covers taking a radio off pppd.

frames_rx stays flat while the error counters climb

Symptom: the device is open and bytes arrive (wire_rx_bytes rises), but frames_rx does not move while crc_errors, framing_errors and resync_bytes climb; on a real serial device uart_frame_errors climbs too.

Cause: the line rate does not match. The link's baud and the radio's SERIAL_SPEED (ATS1) differ, so every byte reads as noise and no frame survives the CRC. A few errors among many good frames are a different thing: a noisy line, a radio dropping bytes on overflow (see radio_overflow_bytes_est), or frames from two senders interleaving on a shared channel.

Fix: set baud to the radio's serial speed, or the radio to the link's, on both ends. The counters stop rising as soon as the rates match; nothing has to be renegotiated.

No radio status or RSSI on a serial link

Symptom: the link carries traffic, but radio_status_frames stays at zero, radio_buffer_free_pct and radio_rssi_dbm are absent, and the link score has no signal figures.

Cause: the radio is not asked, or cannot answer. With radio = "generic" — the default — or heartbeat_ms = 0, the link writes no heartbeat, and the radio reports only when it sees one. With MAVLink framing off on the radio (ATS6=0) it never reports at all.

Fix: set radio = "rfd900" or "sik" (heartbeats every 250 ms by default) and MAVLINK to 1 on the radio. heartbeats_tx should climb at once and radio_status_frames with it. Without reports the link still paces on the serial line and assumes the radio drains at air_rate.

Voice is late or broken on a serial radio

Symptom: calls over a serial link arrive hundreds of milliseconds late, or lose words, while video or bulk traffic shares the radio.

Causes and fixes:

• Pacing is off. With buffer_pacing = false the link writes as fast as the port takes it, video fills the radio's buffer, and voice waits behind it. Leave it at the default, true.
• The radio's buffer is larger than the pacer assumes. radio_overflow_bytes_est grows or radio_buffer_free_pct sits near zero: set radio_buffer_bytes to your radio's figure (default 2048).
• The call does not fit. Fifty 20 ms frames a second are about 43 kbit/s on the wire, and carrying the previous packet raises that to about 69 kbit/s — more than a 64 kbit/s radio holds. On such a radio, turn the class's redundancy off or use 40 ms frames. On a 57 600-baud line the line itself carries about 46 kbit/s; 115 200 baud leaves the air rate in charge.
• Heavy classes allowed onto the radio. Keep Video and Bulk off a link that should carry Control, Voice and Position.

The mechanism is under Pacing on the radio's buffer.

A node on a shared channel cannot reach a peer

Symptom: three or more nodes share one radio channel; some pairs work, another stays dead.

Causes: two nodes share an address, so each refuses the other's frames as coming from itself (counted in framing_errors); one node on the channel has no address while the others do; or a node's peer_addresses points a peer at the wrong address, so its frames go to a node that drops them. not_for_us climbing on a node that should not be receiving them is the sign of a wrong destination. If no node hears another, check the radios: multipoint firmware, a node id each, destination broadcast.

Fix: give every node a distinct address from 1 to 254, set it on every node of the channel, and correct or drop the peer_addresses entry — a peer not listed is learned from its first authenticated frame. See a third node on the same channel.

A node isolated or wiped itself

Symptom: a node dropped every session and refuses handshakes, or it exited and will not start again.

Cause: its record carries a geofence and its own fix was outside for grace_secs. With isolate, atlasd status prints isolated by its geofence — sessions stopped, handshakes refused until back inside and the journal has security.geofence_isolated. With zeroize, or after a remote zeroize command, the journal keeps one line, security.zeroized, naming the cause — outside the authorised area for N s (grace M s), or the command's signer and fleet sequence — and a restart refuses. A node with no fix never fires, so an outage of the GPS alone does not cause this; a fix that put the node outside does, including a wrong [interface] position or a position injected over the API.

Fix: an isolated node returns by itself once its fix is back inside; if the fence is wrong, correct it with atlasd fleet set-geofence (or --clear), sign and push. A zeroized node is gone as an identity: re-provision it with atlasd keygen --identity and a new record.

A push aborted on its own

Symptom: a released rollout shows aborted with a cause naming a node, and the fleet is back on the previous policy as a newer epoch.

Cause: a required node was silent at the release deadline or reported failed. The record's abort_cause, the Fleet view and the journal's rollout.auto_abort name it. Silent means its config status never reached the pushing node in time — it was offline, or its gossip has no path there; failed means it rejected the epoch (its own journal's epoch.rejected says why: a version below min_daemon_version, validation, a signature it does not trust).

Fix: for a node that was out of reach, decide whether it belongs in required at all (atlasd fleet set-update-class <node> best-effort if a release should not wait for it), or give the rollout a longer release_watch_secs. For a rejection, fix the cause on the node. Then start the rollout again; the abort cost nothing but time. See Push a release the fleet cannot half-take.

A node returned after the retire window

Symptom: every handshake from a node is refused, and on the node atlasd status prints this node missed rotation rot-… — it holds only retired material; re-enrol with an invitation, with rotation.missed in its journal.

Cause: the node was offline through a rotation's retire step and past the lifeline window, so no node, lifeline included, accepts its key any more. This is by design: there is no back door past a retire. (A node that missed only a mesh-secret rotation and still holds a valid transport key is not refused: it takes the list over a session and catches up.)

Fix: revoke its record (atlasd fleet revoke <node>) and bring the device back through admission with a fresh invitation; it joins as a new record. To keep a node that is routinely away from being stranded, do not retire while it is still on old material if it is required (the gate holds for it), or lengthen [rotation] lifeline_secs for the next rotation.

A peer counts sink drops

Symptom: on the feed gateway host, caps.sink_local_dropped climbs and something there sees no replies through the tunnel; or caps.sink_dropped climbs on a node next to it.

Cause: the gateway host's record carries sink = true. Its own daemon holds back whatever the host's TUN offers — a ping, a stray service, a forwarding rule between its internet leg and the fleet leg — and counts it caps.sink_local_dropped; a sink cannot answer a ping or reach a peer's dashboard through the tunnel by design. caps.sink_dropped on a peer means data was nevertheless originated by that host's daemon, which a stock build never does.

Fix: sink_local_dropped is the record doing its job: find the writer on the host and stop it, make sure nothing forwards between the two legs, and manage the host over its own leg. sink_dropped climbing on a peer means the daemon on the gateway host is not the one you installed: treat the host as compromised and atlasd fleet revoke it — the removal touches no operational node. If the host genuinely must send, it is not a sink: give it an ordinary record with the classes it needs and accept that the public side then touches a full member.

Still stuck

atlasd support-bundle packages the diagnostic state — configuration with secrets redacted, journal, per-link statistics — into one archive an engineer can actually work from. Send it with a description of the bench (hosts, bearers, what you expected, what happened) to office@nexusatlas.io. Approved evaluators get engineering support as part of the evaluation — reporting a documentation gap counts too: if a failure took you longer than this page, that is a bug in this page.