eSlider
87551fc1c1
Treat MinIO as existing stack infrastructure (default T3 warehouse, not on the drone), add a one-page executive map, and record placement as an open team question rather than a rip-and-replace.
3.0 KiB
00 — Executive overview
A one-page map of the proposal. Details and trade-offs live in the linked docs. Nothing here is a mandate: it is a from-scratch platform sketch the team can reshape once real constraints are on the table.
Goal
Build an on-prem, air-gapped data platform for an autonomous drone swarm: accumulate sensor and detection data on each unit, share only what peers need in flight, and offload complete flights to a ground warehouse for replay and training.
Constraints (given)
- No internet / no external access in flight; data stays inside the system.
- Each drone is autonomous; mesh links are intermittent — no swarm-wide orchestrator.
- On-board today: Docker Compose, DuckDB, YOLO-like video analytics; Parquet under evaluation.
- MinIO is already in the stack — exact placement not fully known yet (see below).
- Ground side can run lightweight Kubernetes; this proposal keeps k3s on the ground only.
Data contract (proposed)
| What | Where | Crosses the air? |
|---|---|---|
| Raw telemetry | Local NVMe (Parquet / Hive layout) | No — offloads after landing |
| Pose / detections (derived) | Local store + peer pub/sub | Yes — small, budgeted |
| Full flight archive | Ground warehouse (Parquet + object store) | After landing only |
One layout on every floor so offload is a mirror, not a migration (03 — Data platform).
Sync rule
- Fast path: compact pose frames (~45 B @ 5 Hz) + event-shaped detections.
- Bulk path (peers): sealed derived partitions when the link allows (rsync/SSH proposed; alternatives open).
- Never push raw high-rate telemetry peer-to-peer in flight.
MinIO (first-class, placement TBD)
MinIO is treated as already chosen infrastructure, not something to rip out.
| Role | Stance in this sketch |
|---|---|
| Ground warehouse (T3) | Primary home — S3 API under the Parquet lake / offload target |
| On-board (drone) | Default: no — competes with flight-critical CPU/RAM; local Parquet + DuckDB is enough for the hot path |
| On-board exception | Not excluded if a team already relies on an S3 API in Compose; then derived-only, never the 5 Hz pose path |
| 5 Hz pose / collision state | Not MinIO — pub/sub (UDP in the PoC; Zenoh proposed) |
| Inter-drone history catch-up | Prefer pull of sealed partitions (SSH/rsync); MinIO replication only if ops already standardised on it |
Open item for the team: document how MinIO is used today, then lock T2/T3 roles (09 — Open questions).
What to read next
- 12 — Design journey — narrative + deep links into code
- Implemented vs proposed — PoC vs production intent
- 09 — Open questions — including MinIO placement
Live visual PoC: swarm.produktor.io (same access as this repository).