swarm-house/docs/04-swarm-sync.md

04 — Swarm sync

What crosses the air, what never does, and how it moves.

The golden rule

Sync results, not raw data — and only when they are actually needed.

Data	Synced in flight?	Why
state (pose)	Yes — broadcast	Every drone needs every peer's position to coordinate and avoid collisions
detections	Yes — published, pulled on demand	Peers re-task around detections; but not every peer needs every frame of context
telemetry (raw)	No	Nobody in the air needs a peer's raw IMU; it offloads at the base (03)

Bandwidth is the scarcest resource in the system. Every message class gets an explicit budget; anything unbudgeted stays local.

Broadcast payload: small on the wire, precise at rest

The state broadcast is a fixed compact frame:

Field	Type	Notes
drone_id	uint16	Fleet-scoped registry
ts_ns	int64	Epoch nanoseconds, same clock domain as storage
pos_x/y/z	int32	Millimeters in the mission frame — quantized only here, storage keeps full float precision
att_roll/pitch/yaw	int16	Centi-degrees
vel_x/y/z	int16	cm/s
frame_ref	uint8	Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit)
flags	uint8	Battery-low, returning, degraded-sensors, …

~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.

detections events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.

Transport: event-driven pub/sub over an ad-hoc mesh

Requirements: peer discovery without infrastructure, pub/sub with late-joiner catch-up, graceful behavior under partitions, tiny footprint, ARM64 support.

graph LR
    subgraph droneA [Drone A]
        HA["event hook"] --> PA["publisher"]
    end
    subgraph droneB [Drone B]
        SB["subscriber"] --> LB["local state cache + DuckDB"]
    end
    subgraph droneC [Drone C — out of direct range]
        SC["subscriber"]
    end
    PA -->|"pose @5Hz, detections on event"| SB
    SB -->|"store-and-forward relay"| SC

Recommended: Zenoh

• Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
• First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
• Tiny footprint, ARM64-native.

Alternatives considered

Option	Verdict
DDS multicast (ROS 2 default)	Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively
MQTT	Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh
Raw UDP multicast	Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode
MinIO bucket replication	Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below

Two sync mechanisms, deliberately separate

1. Fast path — pub/sub (Zenoh): pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. Bulk path — rsync over persistent SSH: sealed detections/state Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.

Why SSH-based bulk sync over object-store replication:

• Identity is already there. Every drone holds pre-provisioned ed25519 keys and a fixed known_hosts/authorized_keys set from ground provisioning (05) — the trust model needs no new machinery.
• One persistent multiplexed session (ControlMaster) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
• rsync delta transfer is resumable across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
• Zero extra services on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.

Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own drone= subtree — no write conflicts by construction).

Two deliberate non-features of the SSH channel:

• No remote filesystem mounts in flight. FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
• No free-form remote execution. Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an SSH forced command (command="…" in authorized_keys, one key pair per operation): the key is the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.

Design decision: SQL-over-SSH as the peer query channel

Decision: the in-flight peer query API is read-only DuckDB SQL executed through a dedicated SSH forced command. This section records how the design got there, because the reasoning is as important as the result.

The path to the decision

1. A GraphQL service on every drone was the first candidate: one flexible, introspectable, schema-first contract. Rejected for the flight side — it means a resolver service, an HTTP stack, and a port on every flight-critical node, all to re-express what is already expressible in SQL against the local store. (GraphQL survives as the ground integration standard, where those costs are irrelevant.)
2. Arrow Flight — efficient columnar RPC, but again a server process and a protocol surface on board; its sweet spot is heavy serving from the ground warehouse, not grams-and-milliwatts peer queries.
3. "Just give drones SSH to each other" — the counter-idea: the transport already exists (ed25519, pinned known_hosts, one multiplexed session for bulk sync), so why not direct CLI access? Objection: free-form shell means one compromised drone owns the fleet; and remote mounts hang on lossy links. So — not raw SSH.
4. The synthesis: SSH forced commands. Keep the zero-extra-infrastructure transport, remove the general-purpose shell. Each authorized_keys entry binds a key to exactly one command; the query key runs exactly one thing: a read-only query wrapper. The peer's "API surface" is the set of keys it has provisioned — versioned, auditable, revocable per operation (05, 11).

What made the final option win: the query language already lives on both ends. Every drone runs DuckDB over the same versioned schemas; SQL is the contract. No IDL, no resolver layer, no serialization format to invent — the response is a Parquet/Arrow stream that the requesting side lands directly into its own store.

Read-only, enforced in layers

SQL access must not become a write channel. A single "read-only connection" flag is not enough — DuckDB can materialize files via COPY TO, attach databases, or read outside the data root. Defense in depth, any single layer failing safe:

Layer	Mechanism	What it stops
1. Key = operation	Forced command: the query key can only invoke the query wrapper, nothing else	Arbitrary exec, lateral movement
2. Statement gate	Wrapper accepts a single statement, parses it, rejects anything but SELECT (no COPY, ATTACH, INSTALL, SET, multi-statements); parameters bound, not interpolated	SQL-as-a-write-channel, config tampering
3. OS permissions	Wrapper runs as a dedicated user with read-only filesystem access to the data root and write access to nothing	Any write that slips past layer 2
4. Engine hardening	:memory: database, external access disabled except the data-root glob, extension loading off	Reaching outside the store
5. Resource caps	Timeout, memory cap, niced CPU (flight software always wins), response size budget	Denial of service via expensive queries

The contract

• Schema-first: all datasets share the versioned schemas from 03; a query addresses dataset + partition predicates + column projection + time range. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
• Read-only SELECT over the local Parquet store, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
• On the ground — GraphQL as the integration standard: the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. Arrow Flight is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it.

What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.

Event-driven end to end

No component polls. The chain from sensor to swarm reaction:

sequenceDiagram
    participant V as video-analytics
    participant W as parquet-writer
    participant H as event hook
    participant Z as pub/sub mesh
    participant P as peer drone
    participant M as peer mission logic

    V->>W: detection row
    W->>H: derived-data event
    H->>Z: publish detection
    Z->>P: deliver (direct or relayed)
    P->>M: local hook fires
    M->>M: re-task decision (out of scope)

The same hook mechanism that feeds the mesh also feeds local mission logic — one eventing model on board and across the swarm.