3 Commits
Author SHA1 Message Date
eSlider 87551fc1c1 docs: put MinIO first-class on ground, add executive overview
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
Treat MinIO as existing stack infrastructure (default T3 warehouse, not
on the drone), add a one-page executive map, and record placement as an
open team question rather than a rip-and-replace.
2026-07-17 12:22:21 +01:00
eSlider 2003ff69fb chore: ignore local tooling and prototype-3d WIP
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 17s
CI & Release / Semantic Release (push) Successful in 6s
Keep .cursor, .grok, and prototype-3d on disk without polluting git status.
2026-07-17 12:17:06 +01:00
eSlider 2a6b50929c docs: align wire-spec and mark implemented vs proposed
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 18s
CI & Release / Semantic Release (push) Successful in 6s
Keep the 45-byte pose frame consistent across docs, simulator, and
prototype, clarify MinIO is not peer sync, and state clearly that the
repo is a from-scratch platform sketch with options rather than mandates.
2026-07-17 12:16:39 +01:00
17 changed files with 197 additions and 63 deletions
+5
View File
@@ -13,3 +13,8 @@ dist/
.terraform/
*.tfstate
*.tfstate.*
# Local tooling / WIP (keep on disk, never commit)
.cursor/
.grok/
prototype-3d/
+25 -3
View File
@@ -6,6 +6,8 @@ A fleet of drones flies fully autonomously: no internet uplink, no external acce
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
**New here?** Start with the one-pager: [00 — Executive overview](docs/00-executive.md) (includes how **MinIO** fits). Then the [design journey](docs/12-design-journey.md) if you want the narrative.
## Design principles
1. **Every drone is autonomous.** No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
@@ -17,10 +19,30 @@ This repository describes how to build, deliver, test, and operate that platform
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
## Implemented now vs proposed next
Honest map so a reader knows what runs today versus what is design intent.
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
| --- | --- | --- |
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, **MinIO as T3** | k3s warehouse with MinIO, GitOps overlays, post-flight mirror |
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
## Documentation
| Document | Contents |
| --- | --- |
| [00 — Executive](docs/00-executive.md) | **Start here** — one-page goal, sync rule, MinIO stance |
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
@@ -30,13 +52,13 @@ This repository describes how to build, deliver, test, and operate that platform
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers |
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers (incl. MinIO placement) |
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code |
| [12 — Design journey](docs/12-design-journey.md) | Narrative walk-through linking into the code |
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
New here? Read the [design journey](docs/12-design-journey.md) first: it tells the story chronologically and links straight into the code and decisions. The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record.
New here? Read the [executive overview](docs/00-executive.md) first, then the [design journey](docs/12-design-journey.md) if you want the story. The design principles above are the *what*; the [ADRs](docs/adr/) are the *why and when*.
## Runnable parts
+61
View File
@@ -0,0 +1,61 @@
# 00 — Executive overview
A one-page map of the proposal. Details and trade-offs live in the linked docs.
Nothing here is a mandate: it is a from-scratch platform sketch the team can
reshape once real constraints are on the table.
## Goal
Build an **on-prem, air-gapped data platform** for an autonomous drone swarm:
accumulate sensor and detection data on each unit, share only what peers need in
flight, and offload complete flights to a ground warehouse for replay and
training.
## Constraints (given)
- No internet / no external access in flight; data stays inside the system.
- Each drone is autonomous; mesh links are intermittent — **no swarm-wide orchestrator**.
- On-board today: **Docker Compose**, **DuckDB**, YOLO-like video analytics; **Parquet** under evaluation.
- **MinIO is already in the stack** — exact placement not fully known yet (see below).
- Ground side can run lightweight Kubernetes; this proposal keeps k3s **on the ground only**.
## Data contract (proposed)
| What | Where | Crosses the air? |
| --- | --- | --- |
| Raw telemetry | Local NVMe (Parquet / Hive layout) | **No** — offloads after landing |
| Pose / detections (derived) | Local store + peer pub/sub | **Yes** — small, budgeted |
| Full flight archive | Ground warehouse (Parquet + object store) | After landing only |
One layout on every floor so offload is a **mirror**, not a migration
([03 — Data platform](03-data-platform.md)).
## Sync rule
1. **Fast path:** compact pose frames (~45 B @ 5 Hz) + event-shaped detections.
2. **Bulk path (peers):** sealed derived partitions when the link allows
(rsync/SSH proposed; alternatives open).
3. **Never** push raw high-rate telemetry peer-to-peer in flight.
## MinIO (first-class, placement TBD)
MinIO is treated as **already chosen infrastructure**, not something to rip out.
| Role | Stance in this sketch |
| --- | --- |
| **Ground warehouse (T3)** | **Primary home** — S3 API under the Parquet lake / offload target |
| **On-board (drone)** | **Default: no** — competes with flight-critical CPU/RAM; local Parquet + DuckDB is enough for the hot path |
| **On-board exception** | Not excluded if a team already relies on an S3 API in Compose; then derived-only, never the 5 Hz pose path |
| **5 Hz pose / collision state** | **Not MinIO** — pub/sub (UDP in the PoC; Zenoh proposed) |
| **Inter-drone history catch-up** | Prefer pull of sealed partitions (SSH/rsync); MinIO replication only if ops already standardised on it |
Open item for the team: document how MinIO is used today, then lock T2/T3 roles
([09 — Open questions](09-open-questions.md)).
## What to read next
1. [12 — Design journey](12-design-journey.md) — narrative + deep links into code
2. [Implemented vs proposed](../README.md#implemented-now-vs-proposed-next) — PoC vs production intent
3. [09 — Open questions](09-open-questions.md) — including MinIO placement
Live visual PoC: [swarm.produktor.io](https://swarm.produktor.io/) (same access as this repository).
+1 -1
View File
@@ -55,7 +55,7 @@ Terms and abbreviations used throughout this proposal.
| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone |
| **k3s** | Lightweight Kubernetes distribution; used on the ground only |
| **Dev Container** | Reproducible containerized development environment definition |
| **MinIO** | S3-compatible object store; runs on-board for derived data and on the ground as the warehouse backend |
| **MinIO** | S3-compatible object store **already in the stack**; default home is the **ground warehouse (T3)**. On-board use is an exception, not the baseline |
| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it |
| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight |
| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version |
+2 -1
View File
@@ -28,7 +28,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
- **DuckDB** is already in use for local data handling.
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
- **MinIO** is available and considered as the inter-drone sync mechanism.
- **MinIO** is already in the stack. Exact placement (ground vs on-board vs sync path) was not fully specified in the brief; this proposal treats MinIO as **first-class ground-warehouse infrastructure** by default and keeps on-board MinIO as an open option — see [00 — Executive](00-executive.md) and [09 — Open questions](09-open-questions.md).
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training.
- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight.
@@ -43,6 +43,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
| A7 | **MinIO's primary home is the ground warehouse (T3), not the drone** | On-board object store competes with flight-critical CPU/RAM; local Parquet + DuckDB already cover the hot path. On-board MinIO is not excluded if ops already depend on an S3 API in Compose |
## Out of scope
+15 -9
View File
@@ -20,11 +20,13 @@ graph TB
subgraph serving [Layer 3 — Serving and sync]
HOOK["event hook<br/>fires on new derived data"]
PUB["state publisher<br/>pub/sub broadcast"]
MINIO["MinIO<br/>derived datasets bucket"]
QAPI["query API<br/>peer data requests"]
BULK["bulk sync<br/>rsync over SSH"]
QAPI["query API<br/>SQL-over-SSH"]
end
end
MINIO[("MinIO — ground warehouse T3<br/>already in the stack")]
SENSORS --> WRITER
VIDEO -->|detections| WRITER
WRITER --> NVME
@@ -32,13 +34,16 @@ graph TB
DUCK --> NVME
WRITER -->|derived rows| HOOK
HOOK --> PUB
HOOK --> MINIO
HOOK --> BULK
QAPI --> DUCK
PUB -.->|mesh| PEERS["peer drones"]
MINIO -.->|replication| PEERS
PUB -.->|mesh pose| PEERS["peer drones"]
BULK -.->|sealed partitions| PEERS
QAPI -.->|on demand| PEERS
NVME -->|post-flight offload| MINIO
```
**MinIO** is already in the stack and is the **default ground warehouse (T3)** behind offload — keep it, do not rip it out. Default sketch: **not on the drone** (CPU/RAM budget); on-board MinIO remains an allowed exception if Compose already depends on an S3 API. **In-flight peer bulk sync is SSH/rsync** (or team-standard MinIO replication if that is already how ops works) — never the 5 Hz pose path. See [00 — Executive](00-executive.md) and [04 — Swarm sync](04-swarm-sync.md).
### Layer 1 — Ingestion
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
@@ -53,10 +58,11 @@ graph TB
### Layer 3 — Serving and sync
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer); MinIO remains optional where an S3 API is wanted.
- **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)).
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, bulk-sync hint, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (UDP in the PoC; Zenoh proposed — [04](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer). If the team already operates MinIO replication for that job, reuse it — do not invent a second path. Pose traffic never goes through MinIO.
- **Peer queries** are read-only DuckDB SQL — HTTP explorer gate in the PoC; **SSH forced commands** proposed for flight (SELECT-only gate, read-only OS user, columnar responses [04](04-swarm-sync.md)).
- **MinIO** sits on the **ground** as T3 by default ([00 — Executive](00-executive.md)); on-board MinIO is an explicit exception, not the baseline.
## Communication planes
+5 -3
View File
@@ -68,11 +68,13 @@ Same format on every floor; only volume, retention, and location change:
| Floor | Where | Contents | Retention |
| --- | --- | --- | --- |
| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
| **T1 — warm** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
| **T2 — shared** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission |
| **T3 — warehouse** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; replay, analytics, model training | Years |
| **T1 — warm** | Drone NVMe (Parquet) | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
| **T2 — shared** | Peer-derived cache on NVMe (Parquet) | Derived data only (`detections`, `state`) pulled from peers when the link allows | Current mission |
| **T3 — warehouse** | Ground **MinIO** (or equivalent) + Parquet lakehouse | Every flight of every drone; replay, analytics, model training**primary MinIO home** | Years |
| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
**MinIO note:** default sketch keeps MinIO on the **ground (T3)**. An on-board MinIO sidecar is a possible exception if the team already standardises on an S3 API in Compose — derived data only, never the 5 Hz pose path. Placement is an open question ([09](09-open-questions.md), [00 — Executive](00-executive.md)).
## Flight offload: a mirror, not a migration
Because T1 and T3 share the identical layout, offload after landing is:
+24 -14
View File
@@ -16,19 +16,27 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex
## Broadcast payload: small on the wire, precise at rest
The `state` broadcast is a fixed compact frame:
The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
what the prototype uses for bandwidth estimates.
| Field | Type | Notes |
| Field | Wire type (PoC) | Notes |
| --- | --- | --- |
| `drone_id` | uint16 | Fleet-scoped registry |
| `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` |
| `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap |
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision |
| `att_roll/pitch/yaw` | int16 | Centi-degrees |
| `vel_x/y/z` | int16 | cm/s |
| `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision |
| `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model |
| `vel_x/y/z` | 3 × int16 | cm/s |
| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.
45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.
An earlier sketch used relative coordinates and a bounding sphere (no attitude).
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
costs almost nothing at this frame size. Relative localization remains a
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
@@ -51,32 +59,34 @@ graph LR
SB -->|"store-and-forward relay"| SC
```
### Recommended: Zenoh
### Recommended (proposal): Zenoh
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
- Tiny footprint, ARM64-native.
**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.
### Alternatives considered
| Option | Verdict |
| --- | --- |
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode |
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below |
| **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode |
| **MinIO bucket replication** | Keep as a **team option** for bulk derived catch-up if ops already standardise on it; wrong tool for the 5 Hz pose path. Default sketch: MinIO primary on the **ground warehouse**, not on the drone |
## Two sync mechanisms, deliberately separate
1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.
Why SSH-based bulk sync over object-store replication:
- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.
- **Zero extra services** on the flight-critical node by default. **MinIO stays first-class on the ground (T3).** On-board MinIO is an exception only if Compose already depends on an S3 API. In-flight bulk path defaults to SSH/rsync; switch to MinIO replication if that is already how the team moves objects.
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
@@ -105,7 +115,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
| Layer | Mechanism | What it stops |
| --- | --- | --- |
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
| 2. Statement gate | Wrapper accepts a single statement, parses it, rejects anything but `SELECT` (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); parameters bound, not interpolated | SQL-as-a-write-channel, config tampering |
| 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering |
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
+3 -3
View File
@@ -12,12 +12,12 @@ graph LR
SLICE["T4: data slices"]
end
subgraph fleet [1 — Fleet, in flight]
D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 MinIO"]
D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 peer cache"]
D2["drone …"]
end
subgraph ground [2 — Ground, on-prem]
DOCK["base station docks"]
DWH["T3 warehouse<br/>object store + Parquet/DuckDB"]
DWH["T3 warehouse<br/>MinIO + Parquet/DuckDB"]
K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
end
@@ -43,7 +43,7 @@ The permanent installation. This **is** allowed to be a cluster — links are wi
| Component | Runs on | Role |
| --- | --- | --- |
| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory |
| **Warehouse (T3)** | **MinIO** (already in the stack) + Parquet | Every flight of every drone; the system's long-term memory |
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
+6 -6
View File
@@ -9,11 +9,11 @@ graph TD
S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"]
S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"]
S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"]
S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"]
S4["Stage 4<br/>Serving layer<br/>event hook, query API"]
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, MinIO replication"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk pull"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, MinIO T3, replay tooling"]
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"]
@@ -41,11 +41,11 @@ graph TD
| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — |
| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 |
| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 |
| **4 — Serving layer** | Event hook + query API + on-board MinIO | Event-driven consumption; nothing polls | 3 |
| **4 — Serving layer** | Event hook + query API (SQL gate) | Event-driven consumption; nothing polls | 3 |
| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 |
| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 |
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk replication over the mesh | Staleness budgets met under simulated loss/partitions | 5, 6 |
| **8 — Ground warehouse** | Dock offload (mirror + audit), T3 store, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk partition pull | Staleness budgets met under simulated loss/partitions | 5, 6 |
| **8 — Ground warehouse** | Dock offload (mirror + audit), **MinIO as T3**, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 |
| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 |
| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 |
+13
View File
@@ -67,3 +67,16 @@ Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage).
The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on T1 disk after offload to T3.
*Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job.
## 12 — MinIO placement (already in the stack)
Exact how MinIO is used today was not fully specified. It should not be ripped out.
*Proposal (default):*
- **Ground warehouse (T3):** primary MinIO home — offload target and long-term object API.
- **On the drone:** default **off** — local Parquet + DuckDB cover the hot path; an object store competes with flight-critical CPU/RAM.
- **Exception:** on-board MinIO allowed if Compose already depends on an S3 API — derived datasets only, never 5 Hz pose.
- **Peer bulk catch-up:** prefer sealed-partition pull (SSH/rsync); use MinIO replication only if that is already the ops standard.
*Settle with the team:* one short diagram of today's MinIO topology (buckets, clients, what syncs). Until then, treat A7 in [01](01-problem-statement.md) as the working assumption. Summary table: [00 — Executive](00-executive.md).
+20 -12
View File
@@ -1,9 +1,12 @@
# 12 — Design journey
How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report. The formal decisions, with options and
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
story behind them, and each chapter links straight into the code it produced.
It is a walk-through, not a report — and **not a mandate**. The goal is a
from-scratch platform sketch that shows how the pieces fit; every concrete
choice is an option with trade-offs recorded as
[Architecture Decision Records](adr/README.md). Replace any piece if a better
fit shows up. This is the story behind the decisions, and each chapter links
straight into the code it produced.
---
@@ -70,15 +73,17 @@ stored and in which structure, not to step into that work.
## 5. What actually needs to sync
The most time-critical item is where each peer is, so every unit has time to
react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and
time) instead of absolute coordinates. Relative is cheaper and enough for
coordination. To avoid sending orientation, a unit is modelled as a **sphere**
that bounds its extent. That trades a little compute for much less data, and no
per-shape encoding. If the units are identical, their 3D model can be provisioned
ahead instead of transmitted.
react. The wire frame carries **mission-frame position** (millimetres on the
wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags —
45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the
fusion pain. An earlier sketch used relative coordinates and a bounding sphere
with no orientation; it was dropped. When peers disagree on frames,
`frame_ref` makes the mismatch explicit for consumers
([09 — Open questions](09-open-questions.md)).
> **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
## 6. Security as nested layers, not one wall
@@ -139,10 +144,13 @@ gives redundancy and a way to cross-check later. On the ground, when the fleet
returns, the same path unifies every unit's data into a
[local warehouse](06-environments.md). The warehouse is read-mostly for
analytics, so transaction contention is not a concern, which opens a
DuckDB / DuckLake approach with MinIO underneath, the
DuckDB / DuckLake approach with **MinIO underneath on the ground (T3)** — MinIO
is already in the stack, so the warehouse leans on it rather than inventing a
second object store — the
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
uniform structure is what lets an analyst trust a single picture even when a
unit's data has a gap.
unit's data has a gap. On-board MinIO stays an open exception, not the default
([00 — Executive](00-executive.md)).
> **Read the code**
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
+10 -5
View File
@@ -32,10 +32,15 @@ wrapper, permissions, and output format a peer drone would use.
## Consequences
- One access path is built, secured, and tested — "what you test is what
- One access path is designed for flight and bench — "what you test is what
flies".
- The SQL gate is safety-critical and is covered by unit tests
(`simulator/tests/test_sql_gate.py`).
- The explorer and the prototype's live mode are *just another read-only
consumer* of this same contract ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- The statement gate in the PoC (`simulator/explorer/server.py`, covered by
`simulator/tests/test_sql_gate.py`) is a **first layer**: keyword allow/deny
over HTTP for the explorer. Production still needs the remaining layers in
[04 — Swarm sync](../04-swarm-sync.md) (forced-command key, read-only OS user,
engine hardening, resource caps) and a real SQL parser rather than keywords
alone.
- The explorer and the prototype's live mode are *stand-in consumers* of the
same read-only contract over HTTP today; the proposed flight path is
SQL-over-SSH ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
+1 -1
View File
@@ -268,7 +268,7 @@ export default function App(): JSX.Element {
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
</Section>
<Section title="Legend">
<div style={styles.dim}>blue line pose broadcast (5 Hz, ~46 B)</div>
<div style={styles.dim}>blue line pose broadcast (5 Hz, 45 B)</div>
<div style={styles.dim}>green line bulk sync (sealed partitions)</div>
<div style={styles.dim}>flash broadcast event delivered</div>
<div style={styles.dim}>red circle transit object crossing the area</div>
+1 -1
View File
@@ -57,7 +57,7 @@ export function areaWidth(): number {
export const LINK_RANGE = 320;
const CRUISE = 28;
const SEPARATION = 55;
const POSE_BYTES = 46;
const POSE_BYTES = 45;
const POSE_HZ = 5;
// Steady per-link broadcast throughput (both directions), bytes/s
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
+1 -1
View File
@@ -9,7 +9,7 @@ from virtual_drone.flight import Pose
def test_frame_size_matches_wire_spec() -> None:
# Documented as 46 B in docs/04; struct packs to FRAME.size (45 B on this layout).
# Keep docs/04, the journey, and prototype/src/sim.ts POSE_BYTES in lockstep.
assert FRAME.size == 45
+4 -3
View File
@@ -1,12 +1,13 @@
"""State broadcast over UDP: the compact pose frame from the sync design.
Frame layout (little-endian, 46 bytes):
Frame layout (little-endian, 45 bytes) — source of truth for docs and the
prototype bandwidth estimate:
magic 2s b"SH"
version B
drone_id 8s zero-padded ascii
drone_id 8s zero-padded ascii (PoC; production may switch to uint16 registry)
ts_ns q epoch nanoseconds
pos_mm 3i position, millimeters (quantized on the wire only)
pos_mm 3i position in the mission frame, millimeters (wire quantization only)
att_cdeg 3h roll/pitch/yaw, centi-degrees
vel_cms 3h velocity, cm/s
frame_ref B