docs: on-prem data platform proposal for autonomous drone swarms
Problem statement, on-board architecture, Parquet/DuckDB storage design, swarm sync strategy, zero-trust networking, environments, observability, CI/CD delivery with fleet release manifests, roadmap and open questions.
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
# 00 — Glossary
|
||||
|
||||
Terms and abbreviations used throughout this proposal.
|
||||
|
||||
## Data & storage
|
||||
|
||||
| Term | Meaning |
|
||||
| --- | --- |
|
||||
| **Parquet** | Columnar file format; compresses time series extremely well (ZSTD + delta encoding), supports predicate pushdown |
|
||||
| **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) |
|
||||
| **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate |
|
||||
| **ZSTD** | Zstandard compression; the default codec for all Parquet files here |
|
||||
| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown |
|
||||
| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse |
|
||||
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
|
||||
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
|
||||
| **Schema registry** | Versioned catalog of dataset schemas so producers and consumers can evolve independently |
|
||||
| **ETL** | Extract, transform, load — the pipelines that move and reshape data between tiers |
|
||||
|
||||
## Swarm & robotics
|
||||
|
||||
| Term | Meaning |
|
||||
| --- | --- |
|
||||
| **UAV / UGV / USV / UUV** | Unmanned aerial / ground / surface / underwater vehicle |
|
||||
| **Swarm** | Group of autonomous vehicles coordinating without a central controller |
|
||||
| **Sustainable pulsing** | Swarm ability to repeatedly engage a target area from multiple directions and self-reorganize as conditions evolve |
|
||||
| **Ubiquitous sensing** | The swarm acting as one distributed sensor, fusing local observations into shared situational awareness ("top sight") |
|
||||
| **Man-in-the-loop** | A human supervises intent and boundaries but does not steer individual vehicles |
|
||||
| **GPS-denied** | Operating without satellite positioning; position comes from visual odometry and relative localization |
|
||||
| **Visual odometry** | Estimating a vehicle's motion from its own camera stream |
|
||||
| **ROS 2** | Robot Operating System 2 — the de-facto middleware for robotics software; nodes exchange typed messages over topics |
|
||||
| **DDS** | Data Distribution Service — the default ROS 2 transport middleware |
|
||||
| **Zenoh** | Pub/sub + query protocol designed for constrained, dynamic networks; official ROS 2 RMW implementation exists |
|
||||
| **YOLO-like detector** | Family of single-pass object-detection models; the on-board video service runs one of them, weights are swappable |
|
||||
| **IMU** | Inertial measurement unit — accelerometer + gyroscope (+ magnetometer) |
|
||||
| **LiDAR** | Laser-based distance sensing |
|
||||
| **RSSI** | Received signal strength indicator; Wi-Fi signal strength, usable as a coarse relative-positioning signal |
|
||||
| **C2** | Command and control — the narrow channel that delivers mission intent to the swarm |
|
||||
|
||||
## Infrastructure & delivery
|
||||
|
||||
| Term | Meaning |
|
||||
| --- | --- |
|
||||
| **Air-gapped** | No route to the internet, physically or logically |
|
||||
| **On-prem** | Runs on owned hardware; no public cloud involvement |
|
||||
| **Zero trust** | No implicit trust from network position; every peer authenticates with pre-provisioned credentials |
|
||||
| **mTLS** | Mutual TLS — both sides of a connection present certificates |
|
||||
| **PKI** | Public key infrastructure — the certificate authority hierarchy behind mTLS |
|
||||
| **WireGuard** | Lightweight encrypted tunnel protocol; basis for the swarm mesh overlay |
|
||||
| **Ad-hoc mesh** | Wireless network formed directly between peers, no fixed infrastructure |
|
||||
| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone |
|
||||
| **k3s** | Lightweight Kubernetes distribution; used on the ground only |
|
||||
| **Dev Container** | Reproducible containerized development environment definition |
|
||||
| **MinIO** | S3-compatible object store; runs on-board for derived data and on the ground as the warehouse backend |
|
||||
| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it |
|
||||
| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight |
|
||||
| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version |
|
||||
| **SemVer** | Semantic versioning (`MAJOR.MINOR.PATCH`) applied to every artifact and to the fleet manifest itself |
|
||||
| **Trivy** | Open-source scanner for container images and dependencies |
|
||||
| **SonarQube** | Static code-quality and security analysis platform |
|
||||
| **nvidia-container-toolkit** | Runtime hook exposing GPUs to containers; needed for on-board inference |
|
||||
| **HIL** | Hardware-in-the-loop testing — real flight hardware wired into a simulated environment |
|
||||
| **CRDT** | Conflict-free replicated data type; merges concurrent updates without coordination |
|
||||
Reference in New Issue
Block a user