8 Commits
Author SHA1 Message Date
eSlider 87551fc1c1 docs: put MinIO first-class on ground, add executive overview
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
Treat MinIO as existing stack infrastructure (default T3 warehouse, not
on the drone), add a one-page executive map, and record placement as an
open team question rather than a rip-and-replace.
2026-07-17 12:22:21 +01:00
eSlider 2003ff69fb chore: ignore local tooling and prototype-3d WIP
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 17s
CI & Release / Semantic Release (push) Successful in 6s
Keep .cursor, .grok, and prototype-3d on disk without polluting git status.
2026-07-17 12:17:06 +01:00
eSlider 2a6b50929c docs: align wire-spec and mark implemented vs proposed
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 18s
CI & Release / Semantic Release (push) Successful in 6s
Keep the 45-byte pose frame consistent across docs, simulator, and
prototype, clarify MinIO is not peer sync, and state clearly that the
repo is a from-scratch platform sketch with options rather than mandates.
2026-07-17 12:16:39 +01:00
eSlider ed531f9fb6 Link license to readme.md
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
2026-07-09 15:55:04 +01:00
eSlider 336f9c0428 Add license
CI & Release / Verify simulator (push) Successful in 11s
CI & Release / Trivy scan (push) Successful in 8s
CI & Release / Semantic Release (push) Successful in 5s
2026-07-09 15:52:16 +01:00
eSlider 3d13f744e6 docs(journey): link the live visual prototype at swarm.produktor.io
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
2026-07-09 12:47:12 +01:00
eSlider dfb0556eda docs: add design journey, a narrative walk-through linking into the code
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
A chronological companion to the ADRs: how the design came together, from
the first mental model and 2D prototype through the three data tiers, the
relative-pose broadcast, the layered WireGuard/SSH trust model, SQL as the
peer contract, offload and replication, and the end-to-end proof of concept.
Each chapter links straight into the relevant source and decision records.
Flagged as the recommended first read in the README.
2026-07-09 12:30:40 +01:00
eSlider d3ccd6f94e docs: record architecture decisions (ADR-0001..0009) and operations knowledge
CI & Release / Verify simulator (push) Successful in 11s
CI & Release / Trivy scan (push) Successful in 8s
CI & Release / Semantic Release (push) Successful in 5s
Add docs/adr/ with a chronological ADR set reconstructed from the project
history (no orchestrator, one storage format, derived-state sync,
SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root
image, runtime data paths, bounded lake scans), an index README, and a
template. Distinguish repo-local ADRs from ecosystem-wide ASRs.

Expand AGENTS.md with a decision-records section and a runtime/operations
guide (service URLs, rebuild workflow, CPU budget knobs). Reference the
ADRs from README, CONTRIBUTING, and the affected design docs, and add an
open question for T1 partition pruning.
2026-07-09 11:41:05 +01:00
33 changed files with 882 additions and 44 deletions
+5
View File
@@ -13,3 +13,8 @@ dist/
.terraform/
*.tfstate
*.tfstate.*
# Local tooling / WIP (keep on disk, never commit)
.cursor/
.grok/
prototype-3d/
+49
View File
@@ -29,3 +29,52 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development workflow and TDD re
- TypeScript: strict mode, functional components and hooks, no extra UI libraries.
- YAML: 2-space indent.
- Diagrams: Mermaid inside Markdown.
## Decision records
Significant architectural decisions are recorded chronologically as **ADRs** in
[`docs/adr/`](docs/adr/) — one immutable file per decision, in the
Status / Context / Options / Decision / Consequences format. Start from
[`docs/adr/README.md`](docs/adr/README.md).
- **ADR** = a point-in-time decision for *this* repo → `docs/adr/`.
- **ASR** = a standing, ecosystem-wide standard (Go-first, secrets, layout) →
lives in the `inventar` repo, **not** here.
When a change alters a boundary, contract, or trade-off, add an ADR (copy
[`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the
index) and cross-link it from the affected design doc. A later reversal gets a
new ADR that supersedes the old one — never edit an Accepted record.
## Runtime & operations (sim environment)
The k3d ground miniature is `swarm-sim`. Services are exposed as NodePorts:
| Service | URL |
| --- | --- |
| Explorer (T1 live lake) | `http://localhost:30088` |
| Warehouse explorer (T3) | `http://localhost:30089` |
| Grafana (dashboards `swarm-fleet`, `swarm-platform`) | `http://localhost:30300` |
| Prometheus | `http://localhost:30990` |
| MinIO console | `http://localhost:30901` |
| Prototype dev server (`npm run dev`) | `http://localhost:5173` |
Rebuild the simulator image and roll it out:
```bash
cd simulator && docker build -t swarm-house/simulator:dev .
k3d image import swarm-house/simulator:dev -c swarm-sim
kubectl rollout restart -n swarm statefulset/drone deployment/explorer deployment/exporter
cd infra/terraform/sim-env && terraform apply
```
**CPU budget knobs** (see [ADR-0009](docs/adr/ADR-0009-bounded-lake-scans.md)):
`METRIC_FLIGHT_WINDOW` (scan only recent flights), `SCAN_INTERVAL_S`,
`VIEW_REFRESH_S`, `TREE_CACHE_S`, `KEEP_ALIVE=1` (drones idle after seal instead
of restarting), `IMU_HZ`. Never scan the full lake on a hot path — it grows with
pod churn. Long-horizon analysis is a T3 warehouse query, not an exporter job.
Runtime data lives under `var/t1` / `var/t3`, owned by uid **10001**
([ADR-0007](docs/adr/ADR-0007-non-root-image.md),
[ADR-0008](docs/adr/ADR-0008-runtime-data-paths.md)); both are gitignored.
Terraform state is local and never committed.
+9
View File
@@ -81,9 +81,18 @@ identifying names of people, companies, or locations.
- [ ] Tests added or updated; `pytest tests/ -q` passes locally
- [ ] `docker build` in `simulator/` succeeds (test stage included)
- [ ] Documentation updated if behaviour or boundaries changed
- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/))
- [ ] English only; no identifying references
- [ ] Conventional commit message
## Decisions
Architectural decisions are recorded chronologically as ADRs in
[`docs/adr/`](docs/adr/). Read [`docs/adr/README.md`](docs/adr/README.md) before
changing a boundary or contract; add a new ADR (from
[`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md)) rather than editing an Accepted
one when you reverse course.
## Questions
Open design questions live in [`docs/09-open-questions.md`](docs/09-open-questions.md).
+19
View File
@@ -0,0 +1,19 @@
PROPRIETARY LICENSE
Copyright (c) 2026 Andriy Oblivantsev. All rights reserved.
This material is proprietary and confidential.
No person or organization may, without prior written permission from the copyright holder:
- Publish or republish this material, in whole or in part.
- Copy, reproduce, distribute, transmit, forward, or share this material by any means.
- Use, modify, adapt, translate, create derivative works from, or incorporate this material into any other work.
- Make this material available to any third party, whether publicly or privately.
- Store this material in any repository, archive, or knowledge base accessible to others.
Permission must be obtained in writing before any of the above activities are undertaken.
Unauthorized use, publication, forwarding, reproduction, distribution, or disclosure of this material is strictly prohibited and may result in civil and criminal penalties under applicable copyright and intellectual property laws.
This license grants no rights except the right to view the material for its intended purpose by an authorized recipient.
+29 -1
View File
@@ -6,6 +6,8 @@ A fleet of drones flies fully autonomously: no internet uplink, no external acce
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
**New here?** Start with the one-pager: [00 — Executive overview](docs/00-executive.md) (includes how **MinIO** fits). Then the [design journey](docs/12-design-journey.md) if you want the narrative.
## Design principles
1. **Every drone is autonomous.** No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
@@ -17,10 +19,30 @@ This repository describes how to build, deliver, test, and operate that platform
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
## Implemented now vs proposed next
Honest map so a reader knows what runs today versus what is design intent.
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
| --- | --- | --- |
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, **MinIO as T3** | k3s warehouse with MinIO, GitOps overlays, post-flight mirror |
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
## Documentation
| Document | Contents |
| --- | --- |
| [00 — Executive](docs/00-executive.md) | **Start here** — one-page goal, sync rule, MinIO stance |
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
@@ -30,9 +52,13 @@ This repository describes how to build, deliver, test, and operate that platform
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers |
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers (incl. MinIO placement) |
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
| [12 — Design journey](docs/12-design-journey.md) | Narrative walk-through linking into the code |
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
New here? Read the [executive overview](docs/00-executive.md) first, then the [design journey](docs/12-design-journey.md) if you want the story. The design principles above are the *what*; the [ADRs](docs/adr/) are the *why and when*.
## Runnable parts
@@ -73,3 +99,5 @@ Contributing (TDD-first workflow): [`CONTRIBUTING.md`](CONTRIBUTING.md).
## CI & releases
Every push runs the pipeline in [`.github/workflows/release.yaml`](.github/workflows/release.yaml): **pytest unit tests**, a compile check, a 10-second virtual smoke flight verified with DuckDB, and a Trivy vulnerability/misconfiguration scan. The simulator Docker build also runs `pytest` — a failing test blocks the image. Pushes to `main` then cut a semantic version from the conventional-commit type (`feat:` → minor, `!`/`BREAKING CHANGE` → major, else patch), tag the commit, and publish a release with a **fleet release manifest** (versioned artifact list with checksums — the delivery unit described in [11 — CI/CD & delivery](docs/11-cicd-delivery.md)) plus a source bundle.
[LICENSE](LICENSE)
+61
View File
@@ -0,0 +1,61 @@
# 00 — Executive overview
A one-page map of the proposal. Details and trade-offs live in the linked docs.
Nothing here is a mandate: it is a from-scratch platform sketch the team can
reshape once real constraints are on the table.
## Goal
Build an **on-prem, air-gapped data platform** for an autonomous drone swarm:
accumulate sensor and detection data on each unit, share only what peers need in
flight, and offload complete flights to a ground warehouse for replay and
training.
## Constraints (given)
- No internet / no external access in flight; data stays inside the system.
- Each drone is autonomous; mesh links are intermittent — **no swarm-wide orchestrator**.
- On-board today: **Docker Compose**, **DuckDB**, YOLO-like video analytics; **Parquet** under evaluation.
- **MinIO is already in the stack** — exact placement not fully known yet (see below).
- Ground side can run lightweight Kubernetes; this proposal keeps k3s **on the ground only**.
## Data contract (proposed)
| What | Where | Crosses the air? |
| --- | --- | --- |
| Raw telemetry | Local NVMe (Parquet / Hive layout) | **No** — offloads after landing |
| Pose / detections (derived) | Local store + peer pub/sub | **Yes** — small, budgeted |
| Full flight archive | Ground warehouse (Parquet + object store) | After landing only |
One layout on every floor so offload is a **mirror**, not a migration
([03 — Data platform](03-data-platform.md)).
## Sync rule
1. **Fast path:** compact pose frames (~45 B @ 5 Hz) + event-shaped detections.
2. **Bulk path (peers):** sealed derived partitions when the link allows
(rsync/SSH proposed; alternatives open).
3. **Never** push raw high-rate telemetry peer-to-peer in flight.
## MinIO (first-class, placement TBD)
MinIO is treated as **already chosen infrastructure**, not something to rip out.
| Role | Stance in this sketch |
| --- | --- |
| **Ground warehouse (T3)** | **Primary home** — S3 API under the Parquet lake / offload target |
| **On-board (drone)** | **Default: no** — competes with flight-critical CPU/RAM; local Parquet + DuckDB is enough for the hot path |
| **On-board exception** | Not excluded if a team already relies on an S3 API in Compose; then derived-only, never the 5 Hz pose path |
| **5 Hz pose / collision state** | **Not MinIO** — pub/sub (UDP in the PoC; Zenoh proposed) |
| **Inter-drone history catch-up** | Prefer pull of sealed partitions (SSH/rsync); MinIO replication only if ops already standardised on it |
Open item for the team: document how MinIO is used today, then lock T2/T3 roles
([09 — Open questions](09-open-questions.md)).
## What to read next
1. [12 — Design journey](12-design-journey.md) — narrative + deep links into code
2. [Implemented vs proposed](../README.md#implemented-now-vs-proposed-next) — PoC vs production intent
3. [09 — Open questions](09-open-questions.md) — including MinIO placement
Live visual PoC: [swarm.produktor.io](https://swarm.produktor.io/) (same access as this repository).
+1 -1
View File
@@ -55,7 +55,7 @@ Terms and abbreviations used throughout this proposal.
| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone |
| **k3s** | Lightweight Kubernetes distribution; used on the ground only |
| **Dev Container** | Reproducible containerized development environment definition |
| **MinIO** | S3-compatible object store; runs on-board for derived data and on the ground as the warehouse backend |
| **MinIO** | S3-compatible object store **already in the stack**; default home is the **ground warehouse (T3)**. On-board use is an exception, not the baseline |
| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it |
| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight |
| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version |
+2 -1
View File
@@ -28,7 +28,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
- **DuckDB** is already in use for local data handling.
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
- **MinIO** is available and considered as the inter-drone sync mechanism.
- **MinIO** is already in the stack. Exact placement (ground vs on-board vs sync path) was not fully specified in the brief; this proposal treats MinIO as **first-class ground-warehouse infrastructure** by default and keeps on-board MinIO as an open option — see [00 — Executive](00-executive.md) and [09 — Open questions](09-open-questions.md).
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training.
- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight.
@@ -43,6 +43,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
| A7 | **MinIO's primary home is the ground warehouse (T3), not the drone** | On-board object store competes with flight-critical CPU/RAM; local Parquet + DuckDB already cover the hot path. On-board MinIO is not excluded if ops already depend on an S3 API in Compose |
## Out of scope
+15 -9
View File
@@ -20,11 +20,13 @@ graph TB
subgraph serving [Layer 3 — Serving and sync]
HOOK["event hook<br/>fires on new derived data"]
PUB["state publisher<br/>pub/sub broadcast"]
MINIO["MinIO<br/>derived datasets bucket"]
QAPI["query API<br/>peer data requests"]
BULK["bulk sync<br/>rsync over SSH"]
QAPI["query API<br/>SQL-over-SSH"]
end
end
MINIO[("MinIO — ground warehouse T3<br/>already in the stack")]
SENSORS --> WRITER
VIDEO -->|detections| WRITER
WRITER --> NVME
@@ -32,13 +34,16 @@ graph TB
DUCK --> NVME
WRITER -->|derived rows| HOOK
HOOK --> PUB
HOOK --> MINIO
HOOK --> BULK
QAPI --> DUCK
PUB -.->|mesh| PEERS["peer drones"]
MINIO -.->|replication| PEERS
PUB -.->|mesh pose| PEERS["peer drones"]
BULK -.->|sealed partitions| PEERS
QAPI -.->|on demand| PEERS
NVME -->|post-flight offload| MINIO
```
**MinIO** is already in the stack and is the **default ground warehouse (T3)** behind offload — keep it, do not rip it out. Default sketch: **not on the drone** (CPU/RAM budget); on-board MinIO remains an allowed exception if Compose already depends on an S3 API. **In-flight peer bulk sync is SSH/rsync** (or team-standard MinIO replication if that is already how ops works) — never the 5 Hz pose path. See [00 — Executive](00-executive.md) and [04 — Swarm sync](04-swarm-sync.md).
### Layer 1 — Ingestion
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
@@ -53,10 +58,11 @@ graph TB
### Layer 3 — Serving and sync
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer); MinIO remains optional where an S3 API is wanted.
- **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)).
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, bulk-sync hint, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (UDP in the PoC; Zenoh proposed — [04](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer). If the team already operates MinIO replication for that job, reuse it — do not invent a second path. Pose traffic never goes through MinIO.
- **Peer queries** are read-only DuckDB SQL — HTTP explorer gate in the PoC; **SSH forced commands** proposed for flight (SELECT-only gate, read-only OS user, columnar responses [04](04-swarm-sync.md)).
- **MinIO** sits on the **ground** as T3 by default ([00 — Executive](00-executive.md)); on-board MinIO is an explicit exception, not the baseline.
## Communication planes
+5 -3
View File
@@ -68,11 +68,13 @@ Same format on every floor; only volume, retention, and location change:
| Floor | Where | Contents | Retention |
| --- | --- | --- | --- |
| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
| **T1 — warm** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
| **T2 — shared** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission |
| **T3 — warehouse** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; replay, analytics, model training | Years |
| **T1 — warm** | Drone NVMe (Parquet) | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
| **T2 — shared** | Peer-derived cache on NVMe (Parquet) | Derived data only (`detections`, `state`) pulled from peers when the link allows | Current mission |
| **T3 — warehouse** | Ground **MinIO** (or equivalent) + Parquet lakehouse | Every flight of every drone; replay, analytics, model training**primary MinIO home** | Years |
| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
**MinIO note:** default sketch keeps MinIO on the **ground (T3)**. An on-board MinIO sidecar is a possible exception if the team already standardises on an S3 API in Compose — derived data only, never the 5 Hz pose path. Placement is an open question ([09](09-open-questions.md), [00 — Executive](00-executive.md)).
## Flight offload: a mirror, not a migration
Because T1 and T3 share the identical layout, offload after landing is:
+24 -14
View File
@@ -16,19 +16,27 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex
## Broadcast payload: small on the wire, precise at rest
The `state` broadcast is a fixed compact frame:
The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
what the prototype uses for bandwidth estimates.
| Field | Type | Notes |
| Field | Wire type (PoC) | Notes |
| --- | --- | --- |
| `drone_id` | uint16 | Fleet-scoped registry |
| `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` |
| `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap |
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision |
| `att_roll/pitch/yaw` | int16 | Centi-degrees |
| `vel_x/y/z` | int16 | cm/s |
| `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision |
| `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model |
| `vel_x/y/z` | 3 × int16 | cm/s |
| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.
45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.
An earlier sketch used relative coordinates and a bounding sphere (no attitude).
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
costs almost nothing at this frame size. Relative localization remains a
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
@@ -51,32 +59,34 @@ graph LR
SB -->|"store-and-forward relay"| SC
```
### Recommended: Zenoh
### Recommended (proposal): Zenoh
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
- Tiny footprint, ARM64-native.
**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.
### Alternatives considered
| Option | Verdict |
| --- | --- |
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode |
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below |
| **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode |
| **MinIO bucket replication** | Keep as a **team option** for bulk derived catch-up if ops already standardise on it; wrong tool for the 5 Hz pose path. Default sketch: MinIO primary on the **ground warehouse**, not on the drone |
## Two sync mechanisms, deliberately separate
1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.
Why SSH-based bulk sync over object-store replication:
- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.
- **Zero extra services** on the flight-critical node by default. **MinIO stays first-class on the ground (T3).** On-board MinIO is an exception only if Compose already depends on an S3 API. In-flight bulk path defaults to SSH/rsync; switch to MinIO replication if that is already how the team moves objects.
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
@@ -105,7 +115,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
| Layer | Mechanism | What it stops |
| --- | --- | --- |
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
| 2. Statement gate | Wrapper accepts a single statement, parses it, rejects anything but `SELECT` (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); parameters bound, not interpolated | SQL-as-a-write-channel, config tampering |
| 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering |
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
+2
View File
@@ -2,6 +2,8 @@
Zero trust, fully air-gapped, everything provisioned before wheels-up.
> **Decision:** [ADR-0005 — WireGuard beneath SSH](adr/ADR-0005-wireguard-beneath-ssh.md).
## Trust model
- **Nothing joins the swarm at runtime.** Every device receives its identity (key pair + certificate) during ground provisioning, signed by the fleet's offline CA. There is no trust-on-first-use, no runtime enrollment endpoint, no exception path.
+5 -3
View File
@@ -2,6 +2,8 @@
Three infrastructures, one data contract. The layout, schemas, and pipelines are identical everywhere; only scale and lifetime differ.
> **Decisions:** [ADR-0006 — IaC boundaries](adr/ADR-0006-iac-boundaries.md), [ADR-0008 — runtime data paths](adr/ADR-0008-runtime-data-paths.md).
```mermaid
graph LR
subgraph dev [3 — Dev and simulation]
@@ -10,12 +12,12 @@ graph LR
SLICE["T4: data slices"]
end
subgraph fleet [1 — Fleet, in flight]
D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 MinIO"]
D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 peer cache"]
D2["drone …"]
end
subgraph ground [2 — Ground, on-prem]
DOCK["base station docks"]
DWH["T3 warehouse<br/>object store + Parquet/DuckDB"]
DWH["T3 warehouse<br/>MinIO + Parquet/DuckDB"]
K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
end
@@ -41,7 +43,7 @@ The permanent installation. This **is** allowed to be a cluster — links are wi
| Component | Runs on | Role |
| --- | --- | --- |
| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory |
| **Warehouse (T3)** | **MinIO** (already in the stack) + Parquet | Every flight of every drone; the system's long-term memory |
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
+2
View File
@@ -2,6 +2,8 @@
The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.
> **Decision:** [ADR-0009 — bound observability lake scans to a flight window and cache](adr/ADR-0009-bounded-lake-scans.md).
## Two kinds of signals, one storage
| Signal | Examples | Where it goes |
+6 -6
View File
@@ -9,11 +9,11 @@ graph TD
S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"]
S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"]
S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"]
S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"]
S4["Stage 4<br/>Serving layer<br/>event hook, query API"]
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, MinIO replication"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk pull"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, MinIO T3, replay tooling"]
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"]
@@ -41,11 +41,11 @@ graph TD
| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — |
| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 |
| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 |
| **4 — Serving layer** | Event hook + query API + on-board MinIO | Event-driven consumption; nothing polls | 3 |
| **4 — Serving layer** | Event hook + query API (SQL gate) | Event-driven consumption; nothing polls | 3 |
| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 |
| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 |
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk replication over the mesh | Staleness budgets met under simulated loss/partitions | 5, 6 |
| **8 — Ground warehouse** | Dock offload (mirror + audit), T3 store, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk partition pull | Staleness budgets met under simulated loss/partitions | 5, 6 |
| **8 — Ground warehouse** | Dock offload (mirror + audit), **MinIO as T3**, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 |
| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 |
| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 |
+19
View File
@@ -61,3 +61,22 @@ Who owns the device registry (drone ids, key issuance, revocation) organizationa
Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage).
*Proposal:* define a minimal profile early — state publisher + UDP-multicast pose broadcast, no local Parquet store, no video — so the swarm protocol never assumes full-stack peers.
## 11 — Lake retention and partition pruning
The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on T1 disk after offload to T3.
*Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job.
## 12 — MinIO placement (already in the stack)
Exact how MinIO is used today was not fully specified. It should not be ripped out.
*Proposal (default):*
- **Ground warehouse (T3):** primary MinIO home — offload target and long-term object API.
- **On the drone:** default **off** — local Parquet + DuckDB cover the hot path; an object store competes with flight-critical CPU/RAM.
- **Exception:** on-board MinIO allowed if Compose already depends on an S3 API — derived datasets only, never 5 Hz pose.
- **Peer bulk catch-up:** prefer sealed-partition pull (SSH/rsync); use MinIO replication only if that is already the ops standard.
*Settle with the team:* one short diagram of today's MinIO topology (buckets, clients, what syncs). Until then, treat A7 in [01](01-problem-statement.md) as the working assumption. Summary table: [00 — Executive](00-executive.md).
+2
View File
@@ -2,6 +2,8 @@
How software, models, and configuration reach the fleet — reproducibly, scanned, versioned, and atomically rollback-able. Everything lives inside the air gap.
> **Decisions:** [ADR-0006 — IaC boundaries & fleet manifest](adr/ADR-0006-iac-boundaries.md), [ADR-0007 — non-root image](adr/ADR-0007-non-root-image.md).
## Source and pipelines: GitLab
GitLab (self-managed, on-prem) is the backbone: repositories, CI/CD, and the container registry in one system.
+202
View File
@@ -0,0 +1,202 @@
# 12 — Design journey
How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report — and **not a mandate**. The goal is a
from-scratch platform sketch that shows how the pieces fit; every concrete
choice is an option with trade-offs recorded as
[Architecture Decision Records](adr/README.md). Replace any piece if a better
fit shows up. This is the story behind the decisions, and each chapter links
straight into the code it produced.
---
## 1. Starting point
At the start there was only a rough brief and a few adjacent interests to lean
on. Key rotation matters in every pipeline: a cipher that is safe today may not
be safe next year, so the infrastructure has to leave room to react, without
spending too much latency or bandwidth to do it. That balance, security against
speed and volume, is a recurring trade-off. Geospatial data is older ground:
routing, indexing, fast spatial lookups at scale. So the first instinct was that
the value might be less about new storage and more about optimizing access to
data that already exists.
## 2. From words to a model, then to a picture
Once the shape was clear, an autonomous swarm where each unit talks to any peer
in range and [no controller spans the swarm](adr/ADR-0001-no-swarm-orchestrator.md),
the next step was to make the model visible. A small 2D prototype was enough to
check it: units patrolling, static and moving obstacles, links forming and
breaking, and a rough count of how much data crosses the air. Speed was not the
question yet, only volume, because volume is what you size the infrastructure
for. Starting from a picture gives an anchor before building anything.
> **Read the code**
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L219-L287) — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L57-L63) — the pose-frame size and rate that drive the volume estimate
The picture is running, not just described: the visual prototype is live at
[swarm.produktor.io](https://swarm.produktor.io/) (same access as this
repository). It draws units patrolling, links forming and breaking by range, and
a running estimate of how much data crosses the air.
## 3. Naming the limits
Before any code, the limits had to be named. Maximum distance between units in
the air. The fidelity data is stored at. The fidelity it can be transmitted at.
Those drive volume, and volume drives the whole shape of the system. The
conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need.
> **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
## 4. Three tiers inside each unit
That gave [three layers](03-data-platform.md) on every drone:
1. **Raw capture**, never transformed.
2. **ETL / reduction**, where data is cut down to what has to be shared:
millimetres to centimetres, thinning time series where that is enough,
dropping what nobody downstream reads.
3. **A bidirectional interface**: on one side it broadcasts position into the
shared channel, on the other it lets peers pull data.
The ETL is deliberately not over-specified. How sensor data is transformed is a
data-engineering decision. The platform's job is the contract, meaning what gets
stored and in which structure, not to step into that work.
> **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
## 5. What actually needs to sync
The most time-critical item is where each peer is, so every unit has time to
react. The wire frame carries **mission-frame position** (millimetres on the
wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags —
45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the
fusion pain. An earlier sketch used relative coordinates and a bounding sphere
with no orientation; it was dropped. When peers disagree on frames,
`frame_ref` makes the mismatch explicit for consumers
([09 — Open questions](09-open-questions.md)).
> **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
## 6. Security as nested layers, not one wall
The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear,
even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct
trajectories over time, and wireless encryption has been broken by ordinary
people for two decades. Relying on that layer alone is risky, and pushing heavy
encryption there is wasteful. So the trust model is [layered](05-network-security.md):
1. Wireless transport (WPA class) as the outer shell.
2. **WireGuard over UDP** as the mesh overlay: proven, peer-authenticated by
static keys, cheap.
3. **SSH inside the tunnel** as the application layer, instead of HTTP/HTTPS. SSH
is one of the most tested remote-access technologies we have. HTTPS would pull
in a certificate authority and extra parts, and it is the first surface a
security researcher probes. [Fewer moving parts, on purpose](adr/ADR-0005-wireguard-beneath-ssh.md).
Identity and encryption solve together at provisioning time: a per-flight,
per-unit key pair recorded in a flight registry. Each unit holds its own key pair
plus the public keys of its peers. A separate key registry for SSH means that if
one WireGuard key leaks, SSH access does not follow automatically.
> **Read the code**
> - [`infra/ansible/drone-provision.yml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/ansible/drone-provision.yml#L20-L70) — ed25519 identity, peers authorized only through a forced command, WireGuard brought up
> - [05 — Network & security](05-network-security.md) and [ADR-0005](adr/ADR-0005-wireguard-beneath-ssh.md)
## 7. SQL as the contract
Beyond the broadcast, people and peers want richer access. The whole progression
is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier
schemes. Each time the same pain: hard to debug in the moment, and a lot of
infrastructure to support, test, and agree on. The simpler conclusion:
[**SQL is the contract**](adr/ADR-0004-sql-over-ssh-contract.md). Everyone who
touches data already speaks SQL, and we do not know in advance which slice each
of them needs, so give them a declarative, extensible way in rather than guessing
endpoints ahead of time.
Concretely: an SSH connection whose forced command runs a read-only **DuckDB**
query directly. No extra auth handshake between the engine and the data, the way
a long-running database needs. No new port or protocol. When access control is
needed, it rides on Linux file permissions, because the data is discrete
Hive-partitioned files and permissions can be set per file. Not new invention:
coming back to foundations DevOps and sysadmins already trust.
> **Read the code**
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L43-L53) — the read-only gate that rejects anything that writes
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first
## 8. Replication and offload
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
channel. It transfers diffs of the Hive tree efficiently. This matters because a
unit can be lost, and if it is, we want its data to already exist elsewhere.
When peers have a stable link the partition tree is pulled between them, which
gives redundancy and a way to cross-check later. On the ground, when the fleet
returns, the same path unifies every unit's data into a
[local warehouse](06-environments.md). The warehouse is read-mostly for
analytics, so transaction contention is not a concern, which opens a
DuckDB / DuckLake approach with **MinIO underneath on the ground (T3)** — MinIO
is already in the stack, so the warehouse leans on it rather than inventing a
second object store — the
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
uniform structure is what lets an analyst trust a single picture even when a
unit's data has a gap. On-board MinIO stays an open exception, not the default
([00 — Executive](00-executive.md)).
> **Read the code**
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
> - [06 — Environments](06-environments.md) and [ADR-0002](adr/ADR-0002-one-storage-format.md)
## 9. A proof of concept, not slides
A concept can be handed over and be wrong. Instead there is a generator: a
virtual drone producing real sensor data into the Hive layout, a small frontend
to watch it fill, and a **Grafana** dashboard on top, not to get ahead of the
analysts, but to see the infrastructure work end to end. Closing the loop
(Terraform, connector, exporter, dashboard) is how a misconfiguration shows up
early. A panel that renders wrong points at exactly which layer broke, before an
analyst has to report that the infra is up but nothing shows. Better to exercise
every layer now and drop any layer or technology later if there is a sound
argument against it.
Watching that loop is also where the [CPU cost surfaced](adr/ADR-0009-bounded-lake-scans.md):
scanning the whole lake on every query does not scale, so the scan is
[bounded to a flight window and cached](07-observability.md).
> **Read the code**
> - [`simulator/monitoring/exporter.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/exporter.py#L38-L107) — the DuckDB scan that turns Parquet into Prometheus metrics
> - [`simulator/monitoring/lake.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/lake.py#L13-L33) — the flight-window scan that keeps CPU bounded as the lake grows
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release
## 10. Working method
A note on how this is meant to be built, not just what. Working together needs a
shared language, and when the working language is nobody's first language,
answering live is expensive. The form that scales is written-first: lay the idea
down carefully, let others read and answer in their own time, keep the history. A
new colleague, or an agent, can then load the full context of why a decision was
made by reading the discussion. That is exactly why the decisions here are
[ADRs](adr/README.md) and why the [contribution flow](../CONTRIBUTING.md) is
test-first: the reasoning is captured where the next person will look for it.
The mental image is a city planner. Draw perfectly straight roads and people
still cross the grass where it is faster. The job is not to decide for them, but
to understand the real paths, the pain points and limits of each team, and pave
those. So constraints get checked with the other DevOps, data engineers, and
analysts before infrastructure is laid, instead of building roads nobody uses.
---
The platform is air-gapped and on-prem by design: nothing leaves the swarm, and
nothing joins it at runtime. Start from the [README](../README.md) for the map,
or the [ADR index](adr/README.md) for the decisions in full.
@@ -0,0 +1,38 @@
# ADR-0001: No swarm-wide orchestrator; coordinate through data
## Status
Accepted (2026-07-08)
## Context
The fleet flies fully autonomously with no internet uplink and only
intermittent mesh connectivity between drones. A conventional control plane
(Kubernetes across the swarm, a leader election, a central scheduler) assumes
stable membership and low-latency links — exactly what a swarm does not have.
Partitions are the normal case, not the exception.
## Options
| Option | Pros | Cons |
| --- | --- | --- |
| A — Cluster orchestrator spanning the swarm | Familiar tooling | Assumes stable membership; a partition stalls coordination; single point of failure in the air |
| B — Each drone autonomous, coordinating through exchanged data | Survives partitions; no air-side control plane to fail | No global view; behaviour must be derivable from local state |
## Decision
Every drone is an autonomous node. There is **no orchestrator spanning the
swarm**. Coordination happens by exchanging derived state (see
[ADR-0003](ADR-0003-sync-derived-state-only.md)), and each drone makes its own
in-flight decisions against its local data window. Orchestration tooling
(Kubernetes, Flux) is confined to the **ground** segment
([ADR-0006](ADR-0006-iac-boundaries.md)).
## Consequences
- The platform degrades gracefully under partition: a drone that loses peers
keeps flying and recording.
- There is no single "fleet state" to query in the air; health questions are
answered from each drone's own store and reconciled on the ground.
- Design principle 1 in [`../../README.md`](../../README.md) and the on-board
model in [`../02-architecture.md`](../02-architecture.md) follow from this.
+32
View File
@@ -0,0 +1,32 @@
# ADR-0002: One storage format on every tier — Parquet + DuckDB, Hive layout
## Status
Accepted (2026-07-08)
## Context
Data lives in three places: on the drone's NVMe in flight (T1), in the ground
warehouse after landing (T3), and in the dev/simulation environment. If each
tier used a different format or layout, offload would need transformation code,
schemas would drift, and a query proven on the bench would not run unchanged on
flight data.
## Decision
Use **one storage format on every floor**: columnar **Parquet** files in an
identical **Hive-partitioned** layout (`dataset=…/flight=…/drone=…/…`), queried
with **DuckDB**. Flight offload T1 → T3 is therefore a plain **mirror**
(copy/rsync of partition directories), not an ETL step.
## Consequences
- Offload is a file operation, testable with a checksum; no format converters
to maintain or version.
- The same SQL runs against T1, T3, and simulated data — the simulator
([`../../simulator/`](../../simulator/)) generates the *real* layout, so tests
exercise the production format.
- Partitioning choices are load-bearing; changing them is a schema migration.
Detailed in [`../03-data-platform.md`](../03-data-platform.md).
- DuckDB's single-file, no-server model fits an air-gapped drone with no room
for a database daemon.
@@ -0,0 +1,33 @@
# ADR-0003: Sync derived state only; raw telemetry stays local
## Status
Accepted (2026-07-08)
## Context
Each drone produces high-rate sensor telemetry and on-board video detections —
far more than the mesh can carry. But peers only need enough to coordinate
flight: where everyone is, where they are heading, what was detected. Pushing
raw feeds across the air would saturate the radio and drain batteries for data
nobody reads in flight.
## Decision
Only **derived state** crosses the air — position, attitude, detections — as
small pub/sub broadcasts (~5 Hz). **Raw telemetry stays on local NVMe** until
the drone lands, then travels to the warehouse during offload
([ADR-0002](ADR-0002-one-storage-format.md)). The design is **event-driven**:
new derived data triggers downstream action through hooks; nothing polls asking
"anything new yet?".
## Consequences
- Air bandwidth scales with the number of peers and the broadcast rate, not
with sensor resolution.
- The warehouse is the only place with the full picture; in-flight decisions use
the local window plus peer broadcasts.
- Broadcast staleness per peer becomes a first-class health signal
([`../07-observability.md`](../07-observability.md)).
- What syncs and what does not is specified in
[`../04-swarm-sync.md`](../04-swarm-sync.md).
@@ -0,0 +1,46 @@
# ADR-0004: Read-only SQL-over-SSH as the peer query contract; no debug API
## Status
Accepted (2026-07-08). Supersedes the earlier sketch of a bespoke query service.
## Context
Beyond the 5 Hz broadcast, a peer (or an engineer on the bench) sometimes needs
to *ask* a drone a richer question: "show me your detections in the last minute
near this position". Inventing a service for this means a new port, a new
protocol, an auth layer, and a second code path that only runs during
debugging — the classic "works on the bench, untested in flight" trap.
## Options
| Option | Pros | Cons |
| --- | --- | --- |
| A — Custom query microservice + REST API | Flexible | New port/protocol/auth to secure; a debug-only path that never flies |
| B — Read-only DuckDB SQL over an SSH forced command | Reuses SSH trust + the query language already on both ends; identical path on bench and in flight | SQL must be gated to read-only |
## Decision
Peer data access is **read-only DuckDB SQL executed through an SSH forced
command**. SQL is already the query language on both ends
([ADR-0002](ADR-0002-one-storage-format.md)), and SSH already carries the trust
model ([ADR-0005](ADR-0005-wireguard-beneath-ssh.md)), so no new service, port,
or protocol is invented. A **SQL gate** rejects anything that writes
(`COPY`, `INSERT`, `ATTACH`, pragmas). There is **no separate debug API**: an
engineer debugging on the ground runs the identical query through the identical
wrapper, permissions, and output format a peer drone would use.
## Consequences
- One access path is designed for flight and bench — "what you test is what
flies".
- The statement gate in the PoC (`simulator/explorer/server.py`, covered by
`simulator/tests/test_sql_gate.py`) is a **first layer**: keyword allow/deny
over HTTP for the explorer. Production still needs the remaining layers in
[04 — Swarm sync](../04-swarm-sync.md) (forced-command key, read-only OS user,
engine hardening, resource caps) and a real SQL parser rather than keywords
alone.
- The explorer and the prototype's live mode are *stand-in consumers* of the
same read-only contract over HTTP today; the proposed flight path is
SQL-over-SSH ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
@@ -0,0 +1,40 @@
# ADR-0005: WireGuard beneath SSH for the mesh transport
## Status
Accepted (2026-07-08)
## Context
Peer bulk sync and SQL-over-SSH ([ADR-0004](ADR-0004-sql-over-ssh-contract.md))
need an encrypted, authenticated channel across an ad-hoc radio mesh. SSH alone
would work at the application layer, but it leaves the drones' listening
surface (sshd, any other service port) exposed on the raw mesh network, and it
gives no cheap, uniform way to authenticate *the peer* before the application
handshake.
## Options
| Option | Pros | Cons |
| --- | --- | --- |
| A — SSH only | One daemon, familiar | Service ports exposed on the raw mesh; per-service auth; no network-layer peer identity |
| B — WireGuard tunnel, SSH inside it | Network-layer peer auth via static keys; only the WG port is exposed; SSH speaks over a private, encrypted subnet | Two layers to provision |
## Decision
Run **WireGuard as the network layer** and **SSH on top of it**. WireGuard
authenticates peers by pre-shared static public keys and presents a private
encrypted subnet; SSH forced commands then carry the SQL and bulk-sync contract
over that subnet. Keys are issued **per device before deployment** — nothing
joins the mesh at runtime (design principle 5).
## Consequences
- The only thing exposed on the raw radio is the WireGuard port; sshd and the
data services listen only on the WG interface.
- Peer identity is established once, at the network layer, and reused by every
application above it.
- Provisioning must place both WG and SSH keys during host prep — handled by
`infra/ansible/drone-provision.yml` ([ADR-0006](ADR-0006-iac-boundaries.md)).
- Rationale and the "why not SSH alone" argument live in
[`../05-network-security.md`](../05-network-security.md).
+41
View File
@@ -0,0 +1,41 @@
# ADR-0006: IaC split — Ansible hosts, Terraform ground, Flux ground-only, fleet manifest for drones
## Status
Accepted (2026-07-08)
## Context
Three very different things need provisioning: (1) drone hosts, which are
air-gapped and never reachable by a controller in flight; (2) the ground
segment (warehouse, observability, offload), a normal k3s cluster; (3) the
simulation environment that mirrors the ground segment locally. Using one tool
for all three would force a GitOps controller or a Terraform apply loop onto the
drone — impossible for an autonomous, disconnected node
([ADR-0001](ADR-0001-no-swarm-orchestrator.md)).
## Decision
Split infrastructure ownership by lifecycle:
| Layer | Tool | Scope |
| --- | --- | --- |
| Host preparation | **Ansible** | Drone bench prep (keys, forced commands, WireGuard, Compose bundle) and the local k3d sim cluster |
| Ground workloads | **Terraform** | k3s/k3d resources: virtual fleet + observability (`sim-env`), warehouse + T1→T3 offload (`ground`) |
| Ground continuous delivery | **Flux (GitOps)** | Ground segment **only** — policy labels, dashboard bundles |
| Drone delivery | **Fleet release manifest** | A versioned, digest-pinned artifact list; no controller pulls to the drone |
Drones are delivered by the **fleet release manifest** (one version for the
whole fleet, atomic rollback), never by a controller reaching into the air.
## Consequences
- No GitOps agent or `terraform apply` ever targets a drone; the air side stays
controller-free.
- Terraform state is environment-local and kept out of Git
([ADR-0008](ADR-0008-runtime-data-paths.md)).
- The sim cluster is a faithful miniature: the same Terraform describes it and
the ground segment.
- Boundaries are documented in
[`../06-environments.md`](../06-environments.md#iac-boundaries); delivery in
[`../11-cicd-delivery.md`](../11-cicd-delivery.md).
+27
View File
@@ -0,0 +1,27 @@
# ADR-0007: Run the simulator image as a non-root user
## Status
Accepted (2026-07-08)
## Context
The Trivy scan in CI flagged **DS-0002**: the simulator image ran as `root`
because no `USER` was set. A container writing the data lake as root also
produces root-owned Parquet on bind mounts, which then blocks a later non-root
process from the same files.
## Decision
Create a dedicated unprivileged user **`swarm` (uid/gid 10001)** in the
`simulator/Dockerfile` and switch to it with `USER swarm` before the entrypoint.
The uid is fixed (not auto-assigned) so host-side ownership of the shared data
paths ([ADR-0008](ADR-0008-runtime-data-paths.md)) is deterministic.
## Consequences
- Trivy DS-0002 clears; the image follows least-privilege.
- Bind-mounted data is owned by `10001:10001`; host prep (`sim-cluster.yml`) and
Ansible `chown` the `var/t1` / `var/t3` paths to this uid so k3d pods can write.
- Any future service image reusing this data must run as the same uid or share
the group.
+40
View File
@@ -0,0 +1,40 @@
# ADR-0008: Shared `var/t1` / `var/t3` runtime paths; state out of Git
## Status
Accepted (2026-07-08)
## Context
Compose, the k3d fleet, and the ground offload job all need to read and write
the same data lake, so that "go live" in the prototype shows the data the
simulator just generated. Early on, Terraform state and the provider cache were
accidentally committed (~6 MB of `*.tfstate` and a vendored provider binary),
polluting history and risking stale/secret data in Git.
## Decision
Standardise two **gitignored** runtime directories at the repo root:
| Path | Tier | Contents |
| --- | --- | --- |
| `var/t1/` | T1 live lake | In-flight Parquet from Compose or the k3d fleet |
| `var/t3/` | T3 warehouse | Offloaded historical partitions |
They are created automatically and overridable via
`SWARM_T1_DIR` / `SWARM_WAREHOUSE_DATA` / `SWARM_SIM_DATA`. Treat `var/` as
**runtime-only** per the ecosystem canonical layout. `.gitignore` excludes
`*.tfstate`, `*.tfstate.backup`, and `.terraform/` while **keeping**
`.terraform.lock.hcl` (the lock is source, the state is not).
## Consequences
- One lake feeds every component; the live prototype reflects real generated
data.
- Git history carries no machine state or large binaries; `terraform` treats
its state as local/remote-backend concern, not versioned source.
- Ownership of these paths is fixed to uid 10001
([ADR-0007](ADR-0007-non-root-image.md)).
- Recovery from earlier state drift is documented in
[`../../infra/terraform/README.md`](../../infra/terraform/README.md)
(`terraform import`).
+52
View File
@@ -0,0 +1,52 @@
# ADR-0009: Bound observability lake scans to a flight window and cache
## Status
Accepted (2026-07-08)
## Context
The k3d sim node was running at ~245% CPU. Three causes compounded:
1. **Explorer** rebuilt a DuckDB view over the *entire* lake on every HTTP
request (~2000 Parquet files, 200 MB+) — roughly 1.5 cores.
2. **Exporter** scanned the full lake every 5 s — roughly 0.7 cores.
3. **Drones** exited after sealing a flight; Kubernetes restarted each pod,
and every restart appended a fresh `flight=` partition tree, so the lake grew
without bound and every scan got more expensive.
The cost is inherent to scanning an append-only lake whose size grows with pod
churn, not to any single slow query. Metrics and the live view only ever need
recent flights.
## Decision
Bound and cache the scans instead of reading the whole lake:
- **Flight window.** The exporter and explorer scan only the most recent
`METRIC_FLIGHT_WINDOW` flights (default 5) via a shared helper
(`simulator/monitoring/lake.py`). Older partitions stay on disk but out of the
hot path.
- **Caching.** Explorer caches DuckDB views (`VIEW_REFRESH_S=30`) and the
partition tree (`TREE_CACHE_S=15`); the exporter scans every
`SCAN_INTERVAL_S=30` instead of 5 s.
- **Stop restart churn.** Drones set `KEEP_ALIVE=1` and idle after sealing
instead of exiting, so no new flight tree is created per restart. In k3d,
`IMU_HZ=20` and resource limits cap per-drone load; the live prototype polls
every 2 s.
- **Self-observability.** Add node-exporter, a
`swarm_exporter_scan_duration_seconds` metric, and a **Swarm Platform**
Grafana dashboard (node CPU/memory + scan health) alongside the existing fleet
dashboard.
## Consequences
- Steady state dropped to explorer ~29 m, exporter ~16 m CPU; the k3d node to
~56% (from ~245%).
- Metrics reflect recent flights only; long-horizon analysis is a warehouse
(T3) query, not an exporter concern — consistent with
[ADR-0003](ADR-0003-sync-derived-state-only.md).
- Old flight partitions accumulate on disk; pruning them is a follow-up
(tracked in [`../09-open-questions.md`](../09-open-questions.md)).
- Observability design is documented in
[`../07-observability.md`](../07-observability.md).
+40
View File
@@ -0,0 +1,40 @@
# Architecture Decision Records
This directory records the significant decisions behind Swarm House, in the
order they were made. Each record is immutable once **Accepted** — a later
decision that changes course gets a new number and supersedes the old one
rather than editing it.
## ADR vs ASR
| Kind | What it captures | Where it lives |
| --- | --- | --- |
| **ADR** (Architecture Decision Record) | A point-in-time decision for *this* repository, with the options considered and their consequences | Here, in `docs/adr/` |
| **ASR** (Architecture Standard Record) | A standing, cross-repository policy every project must obey (Go-first, secrets, layout) | The ecosystem `inventar` repo, not here |
The rules in [`../../AGENTS.md`](../../AGENTS.md) are this repo's local standing
constraints; anything ecosystem-wide belongs in `inventar` as an ASR. Design
decisions specific to the platform are ADRs and belong here.
## Index
| ADR | Decision | Status |
| --- | --- | --- |
| [ADR-0001](ADR-0001-no-swarm-orchestrator.md) | No swarm-wide orchestrator; coordinate through data | Accepted |
| [ADR-0002](ADR-0002-one-storage-format.md) | One storage format on every tier — Parquet + DuckDB, Hive layout | Accepted |
| [ADR-0003](ADR-0003-sync-derived-state-only.md) | Sync derived state only; raw telemetry stays local | Accepted |
| [ADR-0004](ADR-0004-sql-over-ssh-contract.md) | Read-only SQL-over-SSH as the peer query contract; no debug API | Accepted |
| [ADR-0005](ADR-0005-wireguard-beneath-ssh.md) | WireGuard beneath SSH for the mesh transport | Accepted |
| [ADR-0006](ADR-0006-iac-boundaries.md) | IaC split: Ansible hosts, Terraform ground, Flux ground-only, fleet manifest for drones | Accepted |
| [ADR-0007](ADR-0007-non-root-image.md) | Run the simulator image as a non-root user | Accepted |
| [ADR-0008](ADR-0008-runtime-data-paths.md) | Shared `var/t1` / `var/t3` runtime paths; state out of Git | Accepted |
| [ADR-0009](ADR-0009-bounded-lake-scans.md) | Bound observability lake scans to a flight window and cache | Accepted |
## Writing a new ADR
1. Copy [`TEMPLATE.md`](TEMPLATE.md) to `ADR-NNNN-short-slug.md` (next free number).
2. Fill in Context, Options (if more than one was real), Decision, Consequences.
3. Add a row to the index above.
4. Cross-link the ADR from the design doc it affects (and vice versa).
Keep the tone laconic: state the decision and why, skip the narrative.
+29
View File
@@ -0,0 +1,29 @@
# ADR-NNNN: Short decision title
## Status
Proposed | Accepted | Superseded by ADR-XXXX
## Context
What forces are in play? What problem or constraint triggered the decision?
Keep it to the facts that actually shaped the choice.
## Options
| Option | Pros | Cons |
| --- | --- | --- |
| A — … | … | … |
| B — … | … | … |
(Drop this section when only one option was ever real.)
## Decision
The choice, stated plainly. One paragraph.
## Consequences
- What becomes easier.
- What becomes harder or is now forbidden.
- Follow-ups, links to affected docs.
+1 -1
View File
@@ -268,7 +268,7 @@ export default function App(): JSX.Element {
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
</Section>
<Section title="Legend">
<div style={styles.dim}>blue line pose broadcast (5 Hz, ~46 B)</div>
<div style={styles.dim}>blue line pose broadcast (5 Hz, 45 B)</div>
<div style={styles.dim}>green line bulk sync (sealed partitions)</div>
<div style={styles.dim}>flash broadcast event delivered</div>
<div style={styles.dim}>red circle transit object crossing the area</div>
+1 -1
View File
@@ -57,7 +57,7 @@ export function areaWidth(): number {
export const LINK_RANGE = 320;
const CRUISE = 28;
const SEPARATION = 55;
const POSE_BYTES = 46;
const POSE_BYTES = 45;
const POSE_HZ = 5;
// Steady per-link broadcast throughput (both directions), bytes/s
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
+1 -1
View File
@@ -9,7 +9,7 @@ from virtual_drone.flight import Pose
def test_frame_size_matches_wire_spec() -> None:
# Documented as 46 B in docs/04; struct packs to FRAME.size (45 B on this layout).
# Keep docs/04, the journey, and prototype/src/sim.ts POSE_BYTES in lockstep.
assert FRAME.size == 45
+4 -3
View File
@@ -1,12 +1,13 @@
"""State broadcast over UDP: the compact pose frame from the sync design.
Frame layout (little-endian, 46 bytes):
Frame layout (little-endian, 45 bytes) — source of truth for docs and the
prototype bandwidth estimate:
magic 2s b"SH"
version B
drone_id 8s zero-padded ascii
drone_id 8s zero-padded ascii (PoC; production may switch to uint16 registry)
ts_ns q epoch nanoseconds
pos_mm 3i position, millimeters (quantized on the wire only)
pos_mm 3i position in the mission frame, millimeters (wire quantization only)
att_cdeg 3h roll/pitch/yaw, centi-degrees
vel_cms 3h velocity, cm/s
frame_ref B