swarm-house/docs/01-problem-statement.md

01 — Problem statement

Goal

Design the data and delivery platform for a fully autonomous drone swarm:

1. Accumulate — ingest high-rate sensor and detection data locally on every drone, in flight, in real time.
2. Transform & store — convert raw streams into a compressed, queryable local store that survives the whole mission on on-board NVMe.
3. Serve & sync — expose derived results (position, attitude, detections) to the rest of the swarm with minimal bandwidth, and offload complete flight data to a ground warehouse after landing.

Plus everything a platform needs around that: reproducible builds and deployments, a simulation environment, observability, and a zero-trust network — all fully on-prem and air-gapped.

Constraints

Constraint	Consequence
No internet, no external access, ever	All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind
Autonomous operation	No human in the control loop during flight; the platform must not require operator intervention
Intermittent mesh connectivity	No component may assume a stable link between any two drones; no swarm-wide orchestrator
Mixed hardware	ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch
Bounded on-board resources	Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight
Data never leaves the system	Raw data stays on the drone until physical offload at the base; only derived state crosses the air, encrypted
Small on-board footprint	The data plane must not compete with flight-critical software for CPU/RAM

What is known (given)

• Drones run Linux on ARM64 (and AMD64 variants); on-board compute is substantial (NVMe, many sensors, GPU-class accelerators for vision).
• The current on-board software is Docker Compose with two services: sensor ingestion and video-stream object detection (a YOLO-like model; weights and approaches vary).
• DuckDB is already in use for local data handling.
• Parquet adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
• MinIO is already in the stack. Exact placement (ground vs on-board vs sync path) was not fully specified in the brief; this proposal treats MinIO as first-class ground-warehouse infrastructure by default and keeps on-board MinIO as an open option — see 00 — Executive and 09 — Open questions.
• A lightweight Kubernetes exists in the ecosystem; this proposal scopes it to ground infrastructure only (see 02 — Architecture).
• After landing, each drone's data is offloaded to an on-prem warehouse for replay and iterative model training.
• Mission intent (declarative goals) reaches the swarm over a narrow C2 channel; there is no continuous ground link in flight.

What is assumed (explicitly marked as assumptions)

#	Assumption	Basis
A1	The shared state each drone must publish is small: 3D position + attitude + timestamp + drone id, plus detection events	Coordination and collision avoidance need exactly this; anything more wastes air time
A2	Raw sensor data is never needed by peer drones in flight	Offload happens at base; peers act on derived state only
A3	Sensors emit at rates from single Hz (barometer) to hundreds of Hz (IMU); video detections are event-shaped	Typical for this vehicle class; drives nanosecond timestamps and partition sizing
A4	Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle	Return-to-base on low energy implies discrete flight sessions
A5	Mission logic consumes the data platform as a service and is out of scope here	Flight control, planning, and model training are separate concerns
A6	A development-only telemetry channel exists on the bench and is absent from production builds	Standard practice; production radio profile carries C2 + swarm data plane only
A7	MinIO's primary home is the ground warehouse (T3), not the drone	On-board object store competes with flight-critical CPU/RAM; local Parquet + DuckDB already cover the hot path. On-board MinIO is not excluded if ops already depend on an S3 API in Compose

Out of scope

• Flight-control algorithms, trajectory planning, collision-avoidance logic — consumers of this platform.
• Model training pipelines — they read from the ground warehouse; the warehouse contract is in scope, training itself is not.
• Radio hardware selection — the design assumes an IP-capable ad-hoc wireless link and treats bandwidth as scarce.

Success criteria

1. A single drone can ingest, compress, and locally query its full sensor load for an entire flight without operator action.
2. Any drone can obtain any peer's latest derived state within a bounded delay while links are up, and reconcile automatically after partitions heal.
3. Flight offload to the ground warehouse is a plain mirror operation with an integrity audit — no bespoke migration step.
4. The whole fleet can be built, scanned, versioned, and rolled back as one release artifact.
5. An engineer can spin up a virtual swarm on a laptop and run the identical data path end to end.