eSlider
5f6fd8462c
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
60 lines
5.0 KiB
Markdown
60 lines
5.0 KiB
Markdown
# 01 — Problem statement
|
|
|
|
## Goal
|
|
|
|
Design the **data and delivery platform** for a fully autonomous drone swarm:
|
|
|
|
1. **Accumulate** — ingest high-rate sensor and detection data locally on every drone, in flight, in real time.
|
|
2. **Transform & store** — convert raw streams into a compressed, queryable local store that survives the whole mission on on-board NVMe.
|
|
3. **Serve & sync** — expose derived results (position, attitude, detections) to the rest of the swarm with minimal bandwidth, and offload complete flight data to a ground warehouse after landing.
|
|
|
|
Plus everything a platform needs around that: reproducible builds and deployments, a simulation environment, observability, and a zero-trust network — all fully on-prem and air-gapped.
|
|
|
|
## Constraints
|
|
|
|
| Constraint | Consequence |
|
|
| --- | --- |
|
|
| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
|
|
| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention. A [supervising operator](00-glossary.md#swarm--robotics) may set mission intent over [C2](00-glossary.md#swarm--robotics) before and during flight, but does not steer individual vehicles |
|
|
| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
|
|
| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
|
|
| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
|
|
| **Data never leaves the system** | Raw data stays on the drone until physical offload at the base; only derived state crosses the air, encrypted |
|
|
| **Small on-board footprint** | The data plane must not compete with flight-critical software for CPU/RAM |
|
|
|
|
## What is known (given)
|
|
|
|
- Drones run **Linux** on ARM64 (and AMD64 variants); on-board compute is substantial (NVMe, many sensors, GPU-class accelerators for vision).
|
|
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
|
|
- **DuckDB** is already in use for local data handling.
|
|
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
|
|
- **MinIO** is available and considered as the inter-drone sync mechanism.
|
|
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
|
|
- After landing, each drone's data is **offloaded to the on-prem [source of truth](00-glossary.md#source-of-truth)** for replay and iterative model training.
|
|
- Mission intent (declarative goals) reaches the swarm over a narrow **[C2](00-glossary.md#swarm--robotics) channel**; there is no continuous ground link in flight.
|
|
|
|
## What is assumed (explicitly marked as assumptions)
|
|
|
|
| # | Assumption | Basis |
|
|
| --- | --- | --- |
|
|
| A1 | The shared state each drone must publish is small: 3D position + attitude + timestamp + drone id, plus detection events | Coordination and collision avoidance need exactly this; anything more wastes air time |
|
|
| A2 | Raw sensor data is never needed by peer drones in flight | Offload happens at base; peers act on derived state only |
|
|
| A3 | Sensors emit at rates from single Hz (barometer) to hundreds of Hz (IMU); video detections are event-shaped | Typical for this vehicle class; drives nanosecond timestamps and partition sizing |
|
|
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
|
|
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
|
|
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
|
|
|
|
## Out of scope
|
|
|
|
- Flight-control algorithms, trajectory planning, collision-avoidance logic — **consumers** of this platform.
|
|
- Model training pipelines — they read from the ground warehouse; the warehouse contract is in scope, training itself is not.
|
|
- Radio hardware selection — the design assumes an IP-capable ad-hoc wireless link and treats bandwidth as scarce.
|
|
|
|
## Success criteria
|
|
|
|
1. A single drone can ingest, compress, and locally query its full sensor load for an entire flight without operator action.
|
|
2. Any drone can obtain any peer's latest derived state within a bounded delay while links are up, and reconcile automatically after partitions heal.
|
|
3. Flight offload to the ground warehouse is a plain mirror operation with an integrity audit — no bespoke migration step.
|
|
4. The whole fleet can be built, scanned, versioned, and rolled back as one release artifact.
|
|
5. An engineer can spin up a virtual swarm on a laptop and run the identical data path end to end.
|