docs: on-prem data platform proposal for autonomous drone swarms

Problem statement, on-board architecture, Parquet/DuckDB storage design,
swarm sync strategy, zero-trust networking, environments, observability,
CI/CD delivery with fleet release manifests, roadmap and open questions.

docs/04-swarm-sync.md

@@ -0,0 +1,106 @@
+# 04 — Swarm sync
+
+What crosses the air, what never does, and how it moves.
+
+## The golden rule
+
+> **Sync results, not raw data — and only when they are actually needed.**
+
+| Data | Synced in flight? | Why |
+| --- | --- | --- |
+| `state` (pose) | **Yes — broadcast** | Every drone needs every peer's position to coordinate and avoid collisions |
+| `detections` | **Yes — published, pulled on demand** | Peers re-task around detections; but not every peer needs every frame of context |
+| `telemetry` (raw) | **No** | Nobody in the air needs a peer's raw IMU; it offloads at the base ([03](03-data-platform.md)) |
+
+Bandwidth is the scarcest resource in the system. Every message class gets an explicit budget; anything unbudgeted stays local.
+
+## Broadcast payload: small on the wire, precise at rest
+
+The `state` broadcast is a fixed compact frame:
+
+| Field | Type | Notes |
+| --- | --- | --- |
+| `drone_id` | uint16 | Fleet-scoped registry |
+| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
+| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision |
+| `att_roll/pitch/yaw` | int16 | Centi-degrees |
+| `vel_x/y/z` | int16 | cm/s |
+| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
+| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
+
+~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.
+
+`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
+
+## Transport: event-driven pub/sub over an ad-hoc mesh
+
+Requirements: peer discovery without infrastructure, pub/sub with late-joiner catch-up, graceful behavior under partitions, tiny footprint, ARM64 support.
+
+```mermaid
+graph LR
+    subgraph droneA [Drone A]
+        HA["event hook"] --> PA["publisher"]
+    end
+    subgraph droneB [Drone B]
+        SB["subscriber"] --> LB["local state cache + DuckDB"]
+    end
+    subgraph droneC [Drone C — out of direct range]
+        SC["subscriber"]
+    end
+    PA -->|"pose @5Hz, detections on event"| SB
+    SB -->|"store-and-forward relay"| SC
+```
+
+### Recommended: Zenoh
+
+- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
+- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
+- Tiny footprint, ARM64-native.
+
+### Alternatives considered
+
+| Option | Verdict |
+| --- | --- |
+| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
+| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
+| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode |
+| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below |
+
+## Two sync mechanisms, deliberately separate
+
+1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
+2. **Bulk path — MinIO replication:** sealed `detections`/`state` Parquet partitions replicate opportunistically between on-board MinIO instances when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
+
+Partition healing is automatic: replication is pull-based, versioned by partition path, and idempotent (same layout = same keys; last-writer-wins is safe because each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
+
+## Peer query standard
+
+For everything that is not broadcast, drones (and mission services) query each other explicitly. The contract:
+
+- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`.
+- **Recommended interface: GraphQL** over the mesh transport — one flexible, introspectable contract for "give me detections of class X since T from your flight" without inventing endpoints per use case. It also becomes the natural standard for the future multi-team integration surface.
+- **Alternative: Arrow Flight** — far more efficient for bulk columnar transfer, worth adopting on the ground (T3 warehouse serving) where result sets are large; overkill as the in-flight peer protocol.
+- Responses stream Parquet/Arrow batches, never JSON blobs, so the receiving side lands data straight into its own store.
+
+## Event-driven end to end
+
+No component polls. The chain from sensor to swarm reaction:
+
+```mermaid
+sequenceDiagram
+    participant V as video-analytics
+    participant W as parquet-writer
+    participant H as event hook
+    participant Z as pub/sub mesh
+    participant P as peer drone
+    participant M as peer mission logic
+
+    V->>W: detection row
+    W->>H: derived-data event
+    H->>Z: publish detection
+    Z->>P: deliver (direct or relayed)
+    P->>M: local hook fires
+    M->>M: re-task decision (out of scope)
+```
+
+The same hook mechanism that feeds the mesh also feeds local mission logic — one eventing model on board and across the swarm.