eSlider/swarm-house
1
0
Fork 0
Code Issues 2 Pull Requests 3 Actions Packages Projects 1 Releases 13 Wiki Activity

docs: on-prem data platform proposal for autonomous drone swarms

Browse Source 
Problem statement, on-board architecture, Parquet/DuckDB storage design,
swarm sync strategy, zero-trust networking, environments, observability,
CI/CD delivery with fleet release manifests, roadmap and open questions.
This commit is contained in:
eSlider
2026-07-08 13:03:23 +01:00
commit 57e48e0b55
15 changed files with 958 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/08-roadmap.md
+59
View File
@@ -0,0 +1,59 @@
# 08 — Roadmap
From simple to complex, ordered by dependency — each stage is independently useful and validates the next. The guiding principle: **build the environment that makes everything else testable first.**
## Dependency graph
```mermaid
graph TD
S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"]
S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"]
S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"]
S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"]
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, MinIO replication"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"]
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"]
S1 --> S2
S1 --> S3
S2 --> S5
S3 --> S4
S4 --> S5
S5 --> S7
S6 --> S7
S3 --> S8
S7 --> S9
S8 --> S9
S2 --> S10
S5 --> S10
S8 --> S11
S7 --> S11
```
## Stage detail
| Stage | Deliverable | Proves | Depends on |
| --- | --- | --- | --- |
| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — |
| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 |
| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 |
| **4 — Serving layer** | Event hook + query API + on-board MinIO | Event-driven consumption; nothing polls | 3 |
| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 |
| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 |
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk replication over the mesh | Staleness budgets met under simulated loss/partitions | 5, 6 |
| **8 — Ground warehouse** | Dock offload (mirror + audit), T3 store, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 |
| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 |
| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 |
## Sequencing rationale
- **Stages 1–2 before everything:** unglamorous, but every later stage is only as fast as its build/test loop.
- **Stage 3 is the heart** — and it needs no network, no security, no swarm: a single machine and real (or simulated) sensor rates. Risk is retired early where iteration is cheapest.
- **Stage 5 before sync (7):** building a distributed protocol without a reproducible multi-node bench means debugging it in the field. The virtual swarm makes stage 7 a CI problem instead.
- **Stage 6 in parallel:** the trust fabric has no dependency on the data path and can mature alongside it.
- **Stages 8–11 stack on a proven core** — by then, every layer under them is regression-tested.