docs: on-prem data platform proposal for autonomous drone swarms

Problem statement, on-board architecture, Parquet/DuckDB storage design, swarm sync strategy, zero-trust networking, environments, observability, CI/CD delivery with fleet release manifests, roadmap and open questions.
2026-07-08 13:03:23 +01:00
commit 57e48e0b55
15 changed files with 958 additions and 0 deletions
@@ -0,0 +1,71 @@
+# 06 — Environments
+
+Three infrastructures, one data contract. The layout, schemas, and pipelines are identical everywhere; only scale and lifetime differ.
+
+```mermaid
+graph LR
+    subgraph dev [3 — Dev and simulation]
+        SIM["virtual drone fleet<br/>(simulator, Compose)"]
+        LAP["engineer laptop<br/>Dev Container"]
+        SLICE["T4: data slices"]
+    end
+    subgraph fleet [1 — Fleet, in flight]
+        D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 MinIO"]
+        D2["drone …"]
+    end
+    subgraph ground [2 — Ground, on-prem]
+        DOCK["base station docks"]
+        DWH["T3 warehouse<br/>object store + Parquet/DuckDB"]
+        K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
+        TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
+    end
+
+    D1 <-.->|mesh| D2
+    D1 -->|"offload: mirror + audit"| DOCK
+    DOCK --> DWH
+    DWH -->|"slices for development"| SLICE
+    SIM -->|"same pipeline, synthetic flights"| SLICE
+    DWH --> TRAIN
+    TRAIN -->|"versioned weights"| K3S
+```
+
+## 1 — Fleet infrastructure (on the drones)
+
+- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md).
+- Storage floors T0–T2. Nothing here requires ground contact during a mission.
+- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.
+
+## 2 — Ground infrastructure (on-prem)
+
+The permanent installation. This **is** allowed to be a cluster — links are wired and stable here, so k3s earns its keep:
+
+| Component | Runs on | Role |
+| --- | --- | --- |
+| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory |
+| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
+| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
+| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
+| **Simulation farm** | k3s | Scaled-up virtual swarm runs for regression and pre-mission validation |
+| **Dashboards** | k3s | Observability stack ([07](07-observability.md)) + flight replay tooling |
+
+Replay is the warehouse's flagship feature: because every broadcast, detection, and re-tasking trigger is preserved with nanosecond timestamps in one layout, any mission can be reconstructed step by step — for engineering analysis, for audit/traceability, and as training input.
+
+## 3 — Dev & simulation infrastructure
+
+The environment engineers live in daily — and deliberately the first thing to build (see [08 — Roadmap](08-roadmap.md)), because everything else is validated inside it.
+
+- **Dev Containers** give every engineer the same toolchain in minutes; no snowflake laptops.
+- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop.
+- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
+- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
+- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).
+
+## Why identical layouts matter (the payoff)
+
+| Operation | What it costs with one layout everywhere |
+| --- | --- |
+| Flight offload | `mc mirror` / `rsync` + audit — no transform step |
+| Debugging a field issue | Pull the flight slice, replay locally, same queries |
+| Validating a pipeline change | Run simulator, diff Parquet outputs |
+| Training data prep | Read T3 directly, no export pipeline |
+| Onboarding an engineer | One Dev Container, one `docker compose up` |