> Source: https://nexusatlas.dev/docs/2.0.0/concepts/architecture — Nexus Atlas developer documentation (Concepts). Converted from the HTML page; the page is canonical. This is the 2.0.0 release line; the current line is 3.0.0 (https://nexusatlas.dev/docs/concepts/architecture).

# Architecture & core tunnel

One daemon owns one TUN interface and a set of UDP sockets, one per underlay link. Everything between the TUN and the sockets — encryption, fragmentation, scheduling, probing — is designed around a single question: what happens when a link misbehaves? This page walks the data path and answers it at each stage.

## The data path

An application writes an IPv4 packet to the tunnel interface (`[interface] name`, e.g. `nexus0`). The daemon reads it, decides which peer owns the destination (longest-prefix match over each peer's `allowed_ips`), encrypts it for that peer, fragments it if it exceeds the link budget, hands the fragments to the scheduler, and the scheduler places them on a link. On the far side the same machinery runs in reverse. The inner MTU is `[interface] mtu` (default 1416, so a full-size packet still fits a 1500-byte link whole when a relay adds its header); packets above the fragment budget are split into fragments sharing an ID and reassembled at the peer, with a janitor expiring half-arrived buffers so they can never leak memory.

Two properties of the path are worth designing around:

- **All fragments of one packet ride one link.** The scheduler places packets, not fragments. Striping one packet's fragments across lossy links multiplies its loss probability (lose any fragment, lose the packet) and forces cross-path reordering — so it deliberately never happens. Cross-link distribution happens per packet, or per FEC shard in the [transport-class path](https://nexusatlas.dev/docs/2.0.0/concepts/qos), where parity makes it safe.

- **A wedged link cannot block the others.** Each link has its own send workers and a bounded queue (32 packets, drop-on-full, non-blocking): a saturated slow link sheds its own load instead of back-pressuring the TUN reader and stalling the whole bond.

### A packet is sent whole or abandoned on that link

A fragmented packet is worth nothing at the far end unless every fragment arrives. So when a link's send queue refuses one fragment of a multi-fragment packet, the daemon stops spending that link on it: the remaining fragments are not enqueued there, and fragments of the same packet already waiting in the queue are skipped by the link worker, through a cancel flag shared per (packet, link). Other links keep their own copy, so under `broadcast` or a `Duplicate` policy the fast link still delivers. Single-fragment packets are never abandoned, nothing changes on the wire, and no configuration is involved.

The snapshot counts each case per link under `links[].drops` (omitted while every counter is zero):

- `abandoned_unsent` — packets abandoned before any fragment was accepted onto the link. No airtime spent.

- `abandoned_sent` — packets abandoned after at least one fragment was accepted: airtime already spent on a packet that can no longer be reassembled. The figure to watch.

- `cancelled` — fragments the worker skipped because a sibling had been refused.

- `queue_full_by_class` — the refusals themselves, per class (index 6 is unclassified traffic); they sum to the link's `send_dropped`.

Prometheus exports the two abandonment figures as `atlas_link_abandoned_packets_total` with `phase="unsent"` or `phase="sent"`; with QoS on, `GET /api/qos` carries `abandoned_sent` per class. When the after-send figure keeps growing, see [the troubleshooting entry](https://nexusatlas.dev/guides/troubleshooting#abandoned-after-send).

## Encryption

Each peer relationship runs the **Noise IK** handshake — X25519 key agreement, ChaCha20-Poly1305 AEAD, BLAKE2s hashing — the same peer-reviewed construction family used by modern commercial VPNs. Node identity is a static Curve25519 keypair from `atlasd keygen`; decoded private keys live in zeroizing memory and a config file looser than mode 0600 draws a load-time warning.

- **Rekeying is make-before-break:** driven by `[crypto] rekey_interval_secs` (default 120) and/or `rekey_data_limit` (default 1 GiB), a fresh handshake is established beside the live session and both sides keep the previous keys warm for in-flight packets — measured at zero packet loss across rekeys in the end-to-end suite.

- **Simultaneous initiation is resolved deterministically** (no duelling sessions), stalled handshakes retry on a budget, and an initiator that goes unanswered for 10 seconds rolls its session and re-initiates — recovering from a lost handshake response without a restart.

- **Anti-replay** is a 1024-bit sliding window per session, checked after authentication. It doubles as the deduplication layer for the redundancy modes that intentionally send copies of a packet on several links.

- A protocol version byte is checked on every packet decode. The current wire is 4, and the version itself is never negotiated: a node on wire 4 refuses a wire-3 node (counted `protocol_refused`, journaled `peer.protocol_refused` with the sender's endpoint and version) rather than corrupting silently, so plan fleet upgrades as an upgrade, not a drift — same major, same fleet. What is negotiated, inside one wire, is the set of optional features.

### Capability negotiation in the handshake

Both handshake messages open their encrypted Noise payload with the sender's capability bitset, 8 bytes big-endian: the initiator's message carries the bitset followed by its hello, the responder's carries the bitset alone. Because the bits ride inside Noise they are authenticated, so nothing on the path can strip one. A payload without bit 0 is refused. A node advertises a bit when the feature is compiled in (every feature is, in every wire-4 build) and switched on in its configuration; the set is computed at start, since every key involved is restart-class.

| Bit | Name | Advertised when | What happens towards a node without it |
|-----|------|-----------------|----------------------------------------|
| 0 | understood | Always, on wire 4 | The handshake is refused. |
| 1 | qos | [qos] enabled | Nothing: informational. The class byte is part of every wire-4 inner header, and a node without QoS reads it as no class. |
| 2 | fec | [qos] enabled and fec_enabled | Gated per next hop: a class with an Fec policy goes as the plain class, striped and unbatched. |
| 3 | bundle | [qos] enabled and bundle_enabled | Nothing: store-carry-forward is local to the sender, and a flushed bundle leaves as an ordinary frame. |
| 4 | seal | [crypto] e2e or e2e_require | Gated per destination: unicast goes hop-encrypted only (e2e.unsealed_no_cap), unless this node runs e2e_require, which never downgrades and seals as before. |
| 5 | multicast | [multicast] enabled | Gated per tree child: the neighbour is left out of every distribution tree this node forks, as source or transit (multicast.child_without_cap). |
| 6 | interconnect | An Atlas Interconnect seam is configured | Gated per seam peer: nothing crosses the seam (interconnect[].dropped_no_cap). |
| 7 | piggyback | Always: compiled into every wire-4 build | Gated per direct peer, by its handshake (units ride only a direct send): packets go without the previous packet riding along. |

A missing `seal` bit never refuses a session: a relay without `[crypto] e2e` still forwards sealed traffic as an opaque blob. For a destination this node has no session with, the answer comes from that node's link-state advert, whose `caps` list carries the same names; a direct peer that has not advertised yet is judged by its handshake. Bits from a newer peer that this build does not know are kept and shown as `bit<N>`. Both sides of the negotiation are in the snapshot: `release.caps` for this node and `peers[].caps` for each peer (see the [API reference](https://nexusatlas.dev/docs/2.0.0/api#stats-release)).

### Wire versions

A capability bit switches a feature on or off inside one wire. A change to what every node must parse or verify takes a new wire number instead, and with it a new major release ([Versioning](https://nexusatlas.dev/releases/versioning)).

| Wire | Release | What changed | Rollout |
|------|---------|--------------|---------|
| 4 current | 2.0.0 | Link-state adverts carry the prefixes a seam host imports for Interconnect prefix mode, inside the signed body. A wire-3 node cannot verify a wire-4 advert. | A fleet rolls together; the relay servers carry 4 and 3 meanwhile. |
| 3 | 1.0.0 | Both handshake messages open with the capability bitset above. | A fleet rolls together. |

## Links, endpoints and the art of staying reachable

Each `[[link]]` binds a UDP socket to a device (`SO_BINDTODEVICE`) and address. Peers are addressed positionally — `endpoint[i]` reaches the peer over link *i* — or explicitly via `endpoint_per_link` when each radio has its own point-to-point addressing. Then reality intervenes, and three mechanisms deal with it:

- **Endpoint roaming** (`[interface] endpoint_roaming`, default on, WireGuard semantics per link): an *authenticated* packet arriving from a new source address updates that peer's endpoint on that link. A node whose address changes mid-session — a vehicle re-associating to a different access point — keeps its tunnel without anyone restarting anything. Unauthenticated probes and keepalives can never steer an endpoint.

- **The rebind watchdog:** after three consecutive send errors a link's socket is torn down and rebuilt. A USB radio that un-enumerates and returns recovers without a daemon restart; if the configured bind address is temporarily gone, the rebind falls back to the device's current address and returns to the configured one the moment it reappears.

- **Persistent keepalive** (`[[peer]] persistent_keepalive`, seconds) keeps NAT pinholes and idle radio paths warm.

> **One rule worth memorising:** two links sharing the same `bind_device` and `bind_address` must set distinct `listen_port` values — the sockets both set `SO_REUSEPORT`, and the kernel would otherwise load-balance inbound datagrams between them, silently costing each link half its traffic. `atlasd config check` catches it.

## Address families

The *underlay* speaks both families: a link may bind an IPv6 address and peer endpoints may be bracketed IPv6 literals; probes, handshakes and roaming are family-agnostic, and a family-mismatched link/peer pair is warned about and skipped without disturbing other links. The *inner* tunnel data path is IPv4-only today — stated plainly on the [limitations page](https://nexusatlas.dev/limitations).

## Who is allowed to connect

Peers listed in the config are trusted by definition. What happens when a handshake arrives from an *unknown* key is the job of `[auth]` — default: reject. A hot-reloaded keyfile allowlist and an external decision hook are available for fleets that outgrow hand-edited peer lists, and every failure mode fails closed. The [peer authentication guide](https://nexusatlas.dev/guides/auth) covers all three providers.

## The operational surface around the tunnel

The daemon assumes it will run unattended and be asked awkward questions later. It writes a 1 Hz stats snapshot (`/tmp/atlas/<iface>.json`) that the CLI, dashboard and your tooling all read; keeps a persistent, size-capped [event journal](https://nexusatlas.dev/docs/2.0.0/api#journal) so "what happened at 14:32?" has an answer after the fact; feeds the systemd watchdog only while the data path is genuinely moving; and captures crashes locally — redacted, and never uploaded without an operator's explicit action. The details live in [Telemetry & observability](https://nexusatlas.dev/docs/2.0.0/concepts/telemetry).

## Embedding note

The engine is a library first: it can adopt an externally created TUN file descriptor and per-link sockets (the mechanism behind the Android `VpnService` port) and accepts a custom peer authenticator. If you are evaluating Atlas as a component rather than a daemon, raise it in your [access request](https://nexusatlas.dev/request-access) — the integration surface is broader than the config file.