docs: on-prem data platform proposal for autonomous drone swarms

Problem statement, on-board architecture, Parquet/DuckDB storage design,
swarm sync strategy, zero-trust networking, environments, observability,
CI/CD delivery with fleet release manifests, roadmap and open questions.

docs/01-problem-statement.md

@@ -0,0 +1,59 @@
+# 01 — Problem statement
+
+## Goal
+
+Design the **data and delivery platform** for a fully autonomous drone swarm:
+
+1. **Accumulate** — ingest high-rate sensor and detection data locally on every drone, in flight, in real time.
+2. **Transform & store** — convert raw streams into a compressed, queryable local store that survives the whole mission on on-board NVMe.
+3. **Serve & sync** — expose derived results (position, attitude, detections) to the rest of the swarm with minimal bandwidth, and offload complete flight data to a ground warehouse after landing.
+
+Plus everything a platform needs around that: reproducible builds and deployments, a simulation environment, observability, and a zero-trust network — all fully on-prem and air-gapped.
+
+## Constraints
+
+| Constraint | Consequence |
+| --- | --- |
+| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
+| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention |
+| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
+| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
+| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
+| **Data never leaves the system** | Raw data stays on the drone until physical offload at the base; only derived state crosses the air, encrypted |
+| **Small on-board footprint** | The data plane must not compete with flight-critical software for CPU/RAM |
+
+## What is known (given)
+
+- Drones run **Linux** on ARM64 (and AMD64 variants); on-board compute is substantial (NVMe, many sensors, GPU-class accelerators for vision).
+- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
+- **DuckDB** is already in use for local data handling.
+- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
+- **MinIO** is available and considered as the inter-drone sync mechanism.
+- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
+- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training.
+- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight.
+
+## What is assumed (explicitly marked as assumptions)
+
+| # | Assumption | Basis |
+| --- | --- | --- |
+| A1 | The shared state each drone must publish is small: 3D position + attitude + timestamp + drone id, plus detection events | Coordination and collision avoidance need exactly this; anything more wastes air time |
+| A2 | Raw sensor data is never needed by peer drones in flight | Offload happens at base; peers act on derived state only |
+| A3 | Sensors emit at rates from single Hz (barometer) to hundreds of Hz (IMU); video detections are event-shaped | Typical for this vehicle class; drives nanosecond timestamps and partition sizing |
+| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
+| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
+| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
+
+## Out of scope
+
+- Flight-control algorithms, trajectory planning, collision-avoidance logic — **consumers** of this platform.
+- Model training pipelines — they read from the ground warehouse; the warehouse contract is in scope, training itself is not.
+- Radio hardware selection — the design assumes an IP-capable ad-hoc wireless link and treats bandwidth as scarce.
+
+## Success criteria
+
+1. A single drone can ingest, compress, and locally query its full sensor load for an entire flight without operator action.
+2. Any drone can obtain any peer's latest derived state within a bounded delay while links are up, and reconcile automatically after partitions heal.
+3. Flight offload to the ground warehouse is a plain mirror operation with an integrity audit — no bespoke migration step.
+4. The whole fleet can be built, scanned, versioned, and rolled back as one release artifact.
+5. An engineer can spin up a virtual swarm on a laptop and run the identical data path end to end.