Troubleshooting

The failures a bench run actually produces, written as symptom → cause → fix. Every entry ends at the page that explains the mechanism — this page exists so you don't have to know in advance which page that is. If your problem isn't here, it may be documented behavior: check Known limitations before assuming a bug.

Read the daemon before guessing

Four commands answer most questions, in this order:

1. atlasd config check — is the configuration even valid? Run it before every start; it catches the errors below that are catchable statically.
2. atlasd status — the one-line truth: interface, strategy, how many links are alive, how many peers established.
3. atlasd stats — per-link measurements as JSON: aliveness, RTT, loss. This is what the scheduler sees, so it is what your traffic experiences.
4. atlasd journal — the event history: handshakes, link transitions, peer changes, with timestamps. Almost every symptom below leaves its cause here.

The same data is available remotely via the REST API (/api/stats, /api/journal, /api/health) and continuously via Prometheus metrics.

The daemon exits immediately at startup

Symptom: atlasd run (or the systemd unit) terminates within a second of starting.

Causes, most common first:

• Invalid configuration. The daemon refuses to run on a config it cannot fully parse — it does not skip the parts it dislikes. atlasd config check prints exactly what is wrong and where; fix until it reports the file is valid.
• Missing privilege. Creating the TUN device and programming routes needs root or CAP_NET_ADMIN. A bench run as an ordinary user fails here; the packaged systemd unit grants exactly that one capability to an unprivileged service user — use it for anything beyond a bench.
• The listen port is taken. Another process — or another atlasd instance you forgot about — already owns the configured listen_port. One daemon per interface; check with ss -ulpn.

Fix: run atlasd config check first, always. If the config is valid and it still exits, read the lines it printed on the way out — and under systemd, journalctl -u holds them.

Config validation rejects two links on one address

Symptom: atlasd config check refuses a config in which two [[link]] entries share a device and bind address.

Cause: this is the validator saving you from a genuinely nasty failure. Two UDP sockets on the same address and port make the kernel load-balance inbound datagrams between them — each link would silently lose half its traffic while both look "up". The architecture page explains the mechanism.

Fix: give each link on a shared device and address its own listen_port. Two links on different devices need no special handling — this rule exists specifically for the one-device, one-address case (a second path over the same NIC, or the traversal pattern of one link per bearer).

Peers never establish

Symptom: both daemons run, atlasd status shows links alive, but the peer count stays at zero establishments and the journal shows no handshake complete line.

Causes, in the order worth checking:

• Crossed keys. Each config carries its own private key and the other node's public key. The classic miss is pasting a node's own public key into its own [[peer]] block — re-derive from atlasd keygen output and check both files side by side, as in the quickstart.
• Wrong or unreachable endpoint. Every address in the peer’s endpoint list must be one the underlay can actually deliver to — test with a plain ping of the underlay address itself. Tunnel establishment never performs a DNS lookup; endpoints are IPs.
• A firewall eating UDP. The handshake is UDP to the peer's listen_port. Host firewalls that default-drop inbound UDP produce exactly this symptom: both sides initiate, neither ever answers.
• Protocol version skew. Every packet carries a protocol version byte; mixed versions fail loudly at decode rather than negotiating. If one node was upgraded across a protocol bump and its peer wasn't, they will not establish — release notes flag every bump; upgrade a deployment as a deployment.
• An auth provider saying no. Under [auth] providers beyond the default, a peer that fails the allowlist or the external command is rejected by policy, not by fault — see Peer authentication.

Handshake complete, but nothing flows through the tunnel

Symptom: the journal shows handshake complete, yet a ping of the peer's tunnel address gets nothing back.

Causes:

• Tunnel addressing. Both [interface] address values must sit in the same subnet — 10.0.100.1/24 and 10.0.100.2/24 in the quickstart. A typo'd netmask or mismatched subnets means the host never routes the ping into the TUN device at all.
• IPv6 inside the tunnel. The inner tunnel is IPv4-only in the current release line (the underlay may be IPv6). Pinging a peer's IPv6 address through the tunnel cannot work — see Known limitations.
• A host firewall on the TUN device. The tunnel interface (nexus0) is an ordinary network device to the host; a default-drop INPUT policy applies to it like any other. Allow the tunnel subnet explicitly.

Fix: confirm addressing first — it accounts for most of these — then test with the tunnel-side addresses from each end while watching atlasd stats to confirm packets are actually entering a link.

A link is declared dead while the bearer looks fine

Symptom: a link the underlying network says is healthy keeps dropping out of the bond and rejoining.

Cause: liveness is decided by probes, not by carrier state — by default a probe every 250 ms per link, and a link is dead after 5 consecutive misses. A bearer whose round trip or scheduling jitter regularly exceeds the probe budget (a congested cellular link, a long-latency or half-duplex radio) will miss probes while carrying real traffic.

Fix: tune the probing to the bearer's honest characteristics — the probe interval is a per-link setting (probe_interval_ms), and declaring an honest capacity per link keeps the scheduler's cost model from overdriving narrow links (the default assumption is 80 Mbps). Watch atlasd stats after each change: the numbers there are the decision inputs, so when they stabilise, the flapping stops.

Traversal connects, but the path never upgrades past the relay

Symptom: two NATed peers communicate — through a relay — and hole punching never yields a direct path.

Causes:

• That may be the correct outcome. A pair of endpoint-dependent ("hard") NATs is not punchable without port prediction, which Atlas deliberately does not attempt — such pairs relay indefinitely by design. See Known limitations.
• NAT classification is starved. Distinguishing the NAT types that predict punch success requires STUN observations from two distinct server addresses — two ports on one IP cannot do it. Every Atlas relay answers STUN on both of its public addresses precisely for this; if you run your own stun_servers list, it must contain two different IPs, or classification silently degrades and punching is skipped where it might have worked. The traversal page covers the whole ladder.

Fix: check what the classifier concluded (journal + /api/stats) before blaming the puncher. Remember the relay path staying up is the designed floor, not a failure — the scheduler treats a punched path as one more link and shifts to it only when it is genuinely better.

A discovered mesh peer disappears after about a minute

Symptom: a peer that joined via mesh gossip works, then vanishes from the roster roughly a minute after its traffic stops.

Cause: runtime-discovered peers expire after [mesh] expiry_secs (default 60 s) of gossip silence — expiry destroys their sessions and punched endpoints, and under the default static auth provider an expired discovered peer cannot re-add itself. Statically configured peers are exempt.

Fix: peers that matter belong in the config as [[peer]] entries — discovery is for churn, static configuration is for infrastructure. If a longer memory for discovered peers is genuinely what you want, raise expiry_secs; see Mesh & multi-hop routing for what membership does and does not govern.

An applied configuration reverted on its own

Symptom: a config change applied over the REST API was live, and some minutes later the previous configuration was back.

Cause: that is commit-confirmed apply doing its job. A staged configuration must be confirmed within the timeout window after being committed; if the confirmation never arrives — typically because the change broke the very connectivity you were managing the node over — the daemon reverts to the last known-good configuration rather than staying wedged.

Fix: none needed for the revert itself — it is the safety mechanism. Re-apply the change, and send the confirm step over a path the change cannot sever (a second link is exactly the right tool). The full stage → commit → confirm lifecycle, with the timeout-revert outcome, is documented on the API page.

Still stuck

atlasd support-bundle packages the diagnostic state — configuration with secrets redacted, journal, per-link statistics — into one archive an engineer can actually work from. Send it with a description of the bench (hosts, bearers, what you expected, what happened) to office@nexusatlas.io. Approved evaluators get engineering support as part of the evaluation — reporting a documentation gap counts too: if a failure took you longer than this page, that is a bug in this page.