swarm-house/docs/01-problem-statement.md

# 01 — Problem statement

## Goal

Design the **data and delivery platform** for a fully autonomous drone swarm:

1. **Accumulate** — ingest high-rate sensor and detection data locally on every drone, in flight, in real time.
2. **Transform & store** — convert raw streams into a compressed, queryable local store that survives the whole mission on on-board NVMe.
3. **Serve & sync** — expose derived results (position, attitude, detections) to the rest of the swarm with minimal bandwidth, and offload complete flight data to a ground warehouse after landing.

Plus everything a platform needs around that: reproducible builds and deployments, a simulation environment, observability, and a zero-trust network — all fully on-prem and air-gapped.

## Constraints

| Constraint | Consequence |
| --- | --- |
| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention |
| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
| **Data never leaves the system** | Raw data stays on the drone until physical offload at the base; only derived state crosses the air, encrypted |
| **Small on-board footprint** | The data plane must not compete with flight-critical software for CPU/RAM |

## What is known (given)

- Drones run **Linux** on ARM64 (and AMD64 variants); on-board compute is substantial (NVMe, many sensors, GPU-class accelerators for vision).
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
- **DuckDB** is already in use for local data handling.
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
- **MinIO** is already in the stack. Exact placement (ground vs on-board vs sync path) was not fully specified in the brief; this proposal treats MinIO as **first-class ground-warehouse infrastructure** by default and keeps on-board MinIO as an open option — see [00 — Executive](00-executive.md) and [09 — Open questions](09-open-questions.md).
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training.
- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight.

## What is assumed (explicitly marked as assumptions)

| # | Assumption | Basis |
| --- | --- | --- |
| A1 | The shared state each drone must publish is small: 3D position + attitude + timestamp + drone id, plus detection events | Coordination and collision avoidance need exactly this; anything more wastes air time |
| A2 | Raw sensor data is never needed by peer drones in flight | Offload happens at base; peers act on derived state only |
| A3 | Sensors emit at rates from single Hz (barometer) to hundreds of Hz (IMU); video detections are event-shaped | Typical for this vehicle class; drives nanosecond timestamps and partition sizing |
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
| A7 | **MinIO's primary home is the ground warehouse (T3), not the drone** | On-board object store competes with flight-critical CPU/RAM; local Parquet + DuckDB already cover the hot path. On-board MinIO is not excluded if ops already depend on an S3 API in Compose |

## Out of scope

- Flight-control algorithms, trajectory planning, collision-avoidance logic — **consumers** of this platform.
- Model training pipelines — they read from the ground warehouse; the warehouse contract is in scope, training itself is not.
- Radio hardware selection — the design assumes an IP-capable ad-hoc wireless link and treats bandwidth as scarce.

## Success criteria

1. A single drone can ingest, compress, and locally query its full sensor load for an entire flight without operator action.
2. Any drone can obtain any peer's latest derived state within a bounded delay while links are up, and reconcile automatically after partitions heal.
3. Flight offload to the ground warehouse is a plain mirror operation with an integrity audit — no bespoke migration step.
4. The whole fleet can be built, scanned, versioned, and rolled back as one release artifact.
5. An engineer can spin up a virtual swarm on a laptop and run the identical data path end to end.