swarm-house/docs/04-swarm-sync.md

# 04 — Swarm sync

What crosses the air, what never does, and how it moves.

## The golden rule

> **Sync results, not raw data — and only when they are actually needed.**

| Data | Synced in flight? | Why |
| --- | --- | --- |
| `state` (pose) | **Yes — broadcast** | Every drone needs every peer's position to coordinate and avoid collisions |
| `detections` | **Yes — published, pulled on demand** | Peers re-task around detections; but not every peer needs every frame of context |
| `telemetry` (raw) | **No** | Nobody in the air needs a peer's raw IMU; it offloads at the base ([03](03-data-platform.md)) |

Bandwidth is the scarcest resource in the system. Every message class gets an explicit budget; anything unbudgeted stays local.

## Broadcast payload: small on the wire, precise at rest

The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
what the prototype uses for bandwidth estimates.

| Field | Wire type (PoC) | Notes |
| --- | --- | --- |
| `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` |
| `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap |
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
| `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision |
| `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model |
| `vel_x/y/z` | 3 × int16 | cm/s |
| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |

45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.

An earlier sketch used relative coordinates and a bounding sphere (no attitude).
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
costs almost nothing at this frame size. Relative localization remains a
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).

`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.

## Transport: event-driven pub/sub over an ad-hoc mesh

Requirements: peer discovery without infrastructure, pub/sub with late-joiner catch-up, graceful behavior under partitions, tiny footprint, ARM64 support.

```mermaid
graph LR
    subgraph droneA [Drone A]
        HA["event hook"] --> PA["publisher"]
    end
    subgraph droneB [Drone B]
        SB["subscriber"] --> LB["local state cache + DuckDB"]
    end
    subgraph droneC [Drone C — out of direct range]
        SC["subscriber"]
    end
    PA -->|"pose @5Hz, detections on event"| SB
    SB -->|"store-and-forward relay"| SC
```

### Recommended (proposal): Zenoh

- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
- Tiny footprint, ARM64-native.

**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.

### Alternatives considered

| Option | Verdict |
| --- | --- |
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
| **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode |
| **MinIO bucket replication** | Keep as a **team option** for bulk derived catch-up if ops already standardise on it; wrong tool for the 5 Hz pose path. Default sketch: MinIO primary on the **ground warehouse**, not on the drone |

## Two sync mechanisms, deliberately separate

1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.

Why SSH-based bulk sync over object-store replication:

- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
- **Zero extra services** on the flight-critical node by default. **MinIO stays first-class on the ground (T3).** On-board MinIO is an exception only if Compose already depends on an S3 API. In-flight bulk path defaults to SSH/rsync; switch to MinIO replication if that is already how the team moves objects.

Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).

Two deliberate non-features of the SSH channel:

- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.

## Design decision: SQL-over-SSH as the peer query channel

**Decision: the in-flight peer query API is read-only DuckDB SQL executed through a dedicated SSH forced command.** This section records how the design got there, because the reasoning is as important as the result.

### The path to the decision

1. **A GraphQL service on every drone** was the first candidate: one flexible, introspectable, schema-first contract. Rejected for the flight side — it means a resolver service, an HTTP stack, and a port on every flight-critical node, all to re-express what is already expressible in SQL against the local store. (GraphQL survives as the *ground* integration standard, where those costs are irrelevant.)
2. **Arrow Flight** — efficient columnar RPC, but again a server process and a protocol surface on board; its sweet spot is heavy serving from the ground warehouse, not grams-and-milliwatts peer queries.
3. **"Just give drones SSH to each other"** — the counter-idea: the transport already exists (ed25519, pinned known_hosts, one multiplexed session for bulk sync), so why not direct CLI access? Objection: free-form shell means one compromised drone owns the fleet; and remote mounts hang on lossy links. So — not raw SSH.
4. **The synthesis: SSH forced commands.** Keep the zero-extra-infrastructure transport, remove the general-purpose shell. Each `authorized_keys` entry binds a key to exactly one command; the query key runs exactly one thing: a read-only query wrapper. The peer's "API surface" is the set of keys it has provisioned — versioned, auditable, revocable per operation ([05](05-network-security.md), [11](11-cicd-delivery.md)).

What made the final option win: **the query language already lives on both ends.** Every drone runs DuckDB over the same versioned schemas; SQL *is* the contract. No IDL, no resolver layer, no serialization format to invent — the response is a Parquet/Arrow stream that the requesting side lands directly into its own store.

### Read-only, enforced in layers

SQL access must not become a write channel. A single "read-only connection" flag is not enough — DuckDB can materialize files via `COPY TO`, attach databases, or read outside the data root. Defense in depth, any single layer failing safe:

| Layer | Mechanism | What it stops |
| --- | --- | --- |
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
| 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering |
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |

### The contract

- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
- **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it.

What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.

## Event-driven end to end

No component polls. The chain from sensor to swarm reaction:

```mermaid
sequenceDiagram
    participant V as video-analytics
    participant W as parquet-writer
    participant H as event hook
    participant Z as pub/sub mesh
    participant P as peer drone
    participant M as peer mission logic

    V->>W: detection row
    W->>H: derived-data event
    H->>Z: publish detection
    Z->>P: deliver (direct or relayed)
    P->>M: local hook fires
    M->>M: re-task decision (out of scope)
```

The same hook mechanism that feeds the mesh also feeds local mission logic — one eventing model on board and across the swarm.