swarm-house/docs/04-swarm-sync.md

04 — Swarm sync

What crosses the air, what never does, and how it moves.

The golden rule

Sync results, not raw data — and only when they are actually needed.

Data	Synced in flight?	Why
state (pose)	Yes — broadcast	Every drone needs every peer's position to coordinate and avoid collisions
detections	Yes — published, pulled on demand	Peers re-task around detections; but not every peer needs every frame of context
telemetry (raw)	No	Nobody in the air needs a peer's raw IMU; it offloads at the base (03)

Bandwidth is the scarcest resource in the system. Every message class gets an explicit budget; anything unbudgeted stays local.

Broadcast payload: small on the wire, precise at rest

The state broadcast is a fixed compact frame:

Field	Type	Notes
drone_id	uint16	Fleet-scoped registry
ts_ns	int64	Epoch nanoseconds, same clock domain as storage
pos_x/y/z	int32	Millimeters in the mission frame — quantized only here, storage keeps full float precision
att_roll/pitch/yaw	int16	Centi-degrees
vel_x/y/z	int16	cm/s
frame_ref	uint8	Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit)
flags	uint8	Battery-low, returning, degraded-sensors, …

~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.

detections events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.

Transport: event-driven pub/sub over an ad-hoc mesh

Requirements: peer discovery without infrastructure, pub/sub with late-joiner catch-up, graceful behavior under partitions, tiny footprint, ARM64 support.

graph LR
    subgraph droneA [Drone A]
        HA["event hook"] --> PA["publisher"]
    end
    subgraph droneB [Drone B]
        SB["subscriber"] --> LB["local state cache + DuckDB"]
    end
    subgraph droneC [Drone C — out of direct range]
        SC["subscriber"]
    end
    PA -->|"pose @5Hz, detections on event"| SB
    SB -->|"store-and-forward relay"| SC

Recommended: Zenoh

• Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
• First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
• Tiny footprint, ARM64-native.

Alternatives considered

Option	Verdict
DDS multicast (ROS 2 default)	Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively
MQTT	Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh
Raw UDP multicast	Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode
MinIO bucket replication	Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below

Two sync mechanisms, deliberately separate

1. Fast path — pub/sub (Zenoh): pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. Bulk path — MinIO replication: sealed detections/state Parquet partitions replicate opportunistically between on-board MinIO instances when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.

Partition healing is automatic: replication is pull-based, versioned by partition path, and idempotent (same layout = same keys; last-writer-wins is safe because each drone only ever writes its own drone= subtree — no write conflicts by construction).

Peer query standard

For everything that is not broadcast, drones (and mission services) query each other explicitly. The contract:

• Schema-first: all datasets share the versioned schemas from 03; a query addresses dataset + partition predicates + column projection + time range.
• Recommended interface: GraphQL over the mesh transport — one flexible, introspectable contract for "give me detections of class X since T from your flight" without inventing endpoints per use case. It also becomes the natural standard for the future multi-team integration surface.
• Alternative: Arrow Flight — far more efficient for bulk columnar transfer, worth adopting on the ground (T3 warehouse serving) where result sets are large; overkill as the in-flight peer protocol.
• Responses stream Parquet/Arrow batches, never JSON blobs, so the receiving side lands data straight into its own store.

Event-driven end to end

No component polls. The chain from sensor to swarm reaction:

sequenceDiagram
    participant V as video-analytics
    participant W as parquet-writer
    participant H as event hook
    participant Z as pub/sub mesh
    participant P as peer drone
    participant M as peer mission logic

    V->>W: detection row
    W->>H: derived-data event
    H->>Z: publish detection
    Z->>P: deliver (direct or relayed)
    P->>M: local hook fires
    M->>M: re-task decision (out of scope)

The same hook mechanism that feeds the mesh also feeds local mission logic — one eventing model on board and across the swarm.