swarm-house/docs/04-swarm-sync.md

04 — Swarm sync

What crosses the air, what never does, and how it moves.

The golden rule

Sync results, not raw data — and only when they are actually needed.

Data	Synced in flight?	Why
state (pose)	Yes — broadcast	Every drone needs every peer's position to coordinate and avoid collisions
detections	Yes — published, pulled on demand	Peers re-task around detections; but not every peer needs every frame of context
telemetry (raw)	No	Nobody in the air needs a peer's raw IMU; it offloads at the base (03)

Bandwidth is the scarcest resource in the system. Every message class gets an explicit budget; anything unbudgeted stays local.

Broadcast payload: small on the wire, precise at rest

The state broadcast is a fixed compact frame. The runnable PoC encodes it as 45 bytes little-endian (simulator/virtual_drone/broadcast.py); that size is what the prototype uses for bandwidth estimates.

Field	Wire type (PoC)	Notes
magic + version	2s + uint8	b"SH", version 1
drone_id	8s ascii	Zero-padded; a fleet uint16 registry id is a natural production swap
ts_ns	int64	Epoch nanoseconds, same clock domain as storage
pos_x/y/z	3 × int32	Millimeters in the mission frame — quantized on the wire only; storage keeps full float precision
att_roll/pitch/yaw	3 × int16	Centi-degrees — attitude stays on the wire so peers need no local shape model
vel_x/y/z	3 × int16	cm/s
frame_ref	uint8	Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit)
flags	uint8	Battery-low, returning, degraded-sensors, …

45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.

An earlier sketch used relative coordinates and a bounding sphere (no attitude). It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and costs almost nothing at this frame size. Relative localization remains a consumer concern when frame_ref differs across peers (09).

detections events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.

Transport: event-driven pub/sub over an ad-hoc mesh

Requirements: peer discovery without infrastructure, pub/sub with late-joiner catch-up, graceful behavior under partitions, tiny footprint, ARM64 support.

graph LR
    subgraph droneA [Drone A]
        HA["event hook"] --> PA["publisher"]
    end
    subgraph droneB [Drone B]
        SB["subscriber"] --> LB["local state cache + DuckDB"]
    end
    subgraph droneC [Drone C — out of direct range]
        SC["subscriber"]
    end
    PA -->|"pose @5Hz, detections on event"| SB
    SB -->|"store-and-forward relay"| SC

Recommended (proposal): Zenoh

• Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
• First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
• Tiny footprint, ARM64-native.

PoC today: the simulator and the 2D prototype exercise the raw UDP pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.

Alternatives considered

Option	Verdict
DDS multicast (ROS 2 default)	Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively
MQTT	Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh
Raw UDP multicast	Implemented in the PoC for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode
MinIO bucket replication	Keep as a team option for bulk derived catch-up if ops already standardise on it; wrong tool for the 5 Hz pose path. Default sketch: MinIO primary on the ground warehouse, not on the drone

Two sync mechanisms, deliberately separate

1. Fast path — pub/sub (Zenoh proposed; UDP in the PoC): pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. Bulk path — rsync over persistent SSH (proposed): sealed detections/state Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.

Why SSH-based bulk sync over object-store replication:

• Identity is already there. Every drone holds pre-provisioned ed25519 keys and a fixed known_hosts/authorized_keys set from ground provisioning (05) — the trust model needs no new machinery.
• One persistent multiplexed session (ControlMaster) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
• rsync delta transfer is resumable across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
• Zero extra services on the flight-critical node by default. MinIO stays first-class on the ground (T3). On-board MinIO is an exception only if Compose already depends on an S3 API. In-flight bulk path defaults to SSH/rsync; switch to MinIO replication if that is already how the team moves objects.

Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own drone= subtree — no write conflicts by construction).

Two deliberate non-features of the SSH channel:

• No remote filesystem mounts in flight. FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
• No free-form remote execution. Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an SSH forced command (command="…" in authorized_keys, one key pair per operation): the key is the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.

Design decision: SQL-over-SSH as the peer query channel

Decision: the in-flight peer query API is read-only DuckDB SQL executed through a dedicated SSH forced command. This section records how the design got there, because the reasoning is as important as the result.

The path to the decision

1. A GraphQL service on every drone was the first candidate: one flexible, introspectable, schema-first contract. Rejected for the flight side — it means a resolver service, an HTTP stack, and a port on every flight-critical node, all to re-express what is already expressible in SQL against the local store. (GraphQL survives as the ground integration standard, where those costs are irrelevant.)
2. Arrow Flight — efficient columnar RPC, but again a server process and a protocol surface on board; its sweet spot is heavy serving from the ground warehouse, not grams-and-milliwatts peer queries.
3. "Just give drones SSH to each other" — the counter-idea: the transport already exists (ed25519, pinned known_hosts, one multiplexed session for bulk sync), so why not direct CLI access? Objection: free-form shell means one compromised drone owns the fleet; and remote mounts hang on lossy links. So — not raw SSH.
4. The synthesis: SSH forced commands. Keep the zero-extra-infrastructure transport, remove the general-purpose shell. Each authorized_keys entry binds a key to exactly one command; the query key runs exactly one thing: a read-only query wrapper. The peer's "API surface" is the set of keys it has provisioned — versioned, auditable, revocable per operation (05, 11).

What made the final option win: the query language already lives on both ends. Every drone runs DuckDB over the same versioned schemas; SQL is the contract. No IDL, no resolver layer, no serialization format to invent — the response is a Parquet/Arrow stream that the requesting side lands directly into its own store.

Read-only, enforced in layers

SQL access must not become a write channel. A single "read-only connection" flag is not enough — DuckDB can materialize files via COPY TO, attach databases, or read outside the data root. Defense in depth, any single layer failing safe:

Layer	Mechanism	What it stops
1. Key = operation	Forced command: the query key can only invoke the query wrapper, nothing else	Arbitrary exec, lateral movement
2. Statement gate	Wrapper accepts a single statement, rejects anything but SELECT/WITH/… (no COPY, ATTACH, INSTALL, SET, multi-statements); production should prefer a real parser + bound parameters, not keywords alone	SQL-as-a-write-channel, config tampering
3. OS permissions	Wrapper runs as a dedicated user with read-only filesystem access to the data root and write access to nothing	Any write that slips past layer 2
4. Engine hardening	:memory: database, external access disabled except the data-root glob, extension loading off	Reaching outside the store
5. Resource caps	Timeout, memory cap, niced CPU (flight software always wins), response size budget	Denial of service via expensive queries

The contract

• Schema-first: all datasets share the versioned schemas from 03; a query addresses dataset + partition predicates + column projection + time range. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
• Read-only SELECT over the local Parquet store, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
• On the ground — GraphQL as the integration standard: the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. Arrow Flight is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it.

What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.

Event-driven end to end

No component polls. The chain from sensor to swarm reaction:

sequenceDiagram
    participant V as video-analytics
    participant W as parquet-writer
    participant H as event hook
    participant Z as pub/sub mesh
    participant P as peer drone
    participant M as peer mission logic

    V->>W: detection row
    W->>H: derived-data event
    H->>Z: publish detection
    Z->>P: deliver (direct or relayed)
    P->>M: local hook fires
    M->>M: re-task decision (out of scope)

The same hook mechanism that feeds the mesh also feeds local mission logic — one eventing model on board and across the swarm.