eSlider
d3ccd6f94e
Add docs/adr/ with a chronological ADR set reconstructed from the project history (no orchestrator, one storage format, derived-state sync, SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root image, runtime data paths, bounded lake scans), an index README, and a template. Distinguish repo-local ADRs from ecosystem-wide ASRs. Expand AGENTS.md with a decision-records section and a runtime/operations guide (service URLs, rebuild workflow, CPU budget knobs). Reference the ADRs from README, CONTRIBUTING, and the affected design docs, and add an open question for T1 partition pruning.
79 lines
6.5 KiB
Markdown
79 lines
6.5 KiB
Markdown
# Swarm House
|
|
|
|
**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective.
|
|
|
|
A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale.
|
|
|
|
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
|
|
|
|
## Design principles
|
|
|
|
1. **Every drone is autonomous.** No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
|
|
2. **One storage format on every floor.** Parquet + DuckDB with an identical Hive-partitioned layout on the drone, in the ground warehouse, and in the dev environment. Flight offload is a plain mirror operation.
|
|
3. **Sync results, not raw data.** Only derived state (position, attitude, detections) crosses the air. Raw telemetry stays on local NVMe until the drone lands.
|
|
4. **Event-driven, not polled.** New derived data triggers downstream actions through hooks; nothing burns cycles asking "anything new yet?".
|
|
5. **Zero trust, provisioned ahead.** Keys are issued per device before deployment; nothing joins the swarm at runtime. Dev channels are physically absent from production builds.
|
|
6. **The whole fleet has one version.** A release manifest pins every image digest, model weight, schema, and config. Rollback is atomic.
|
|
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
|
|
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
|
|
|
|
## Documentation
|
|
|
|
| Document | Contents |
|
|
| --- | --- |
|
|
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
|
|
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
|
|
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
|
|
| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path |
|
|
| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard |
|
|
| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport |
|
|
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
|
|
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
|
|
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
|
|
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers |
|
|
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
|
|
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
|
|
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
|
|
|
|
The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record.
|
|
|
|
## Runnable parts
|
|
|
|
| Component | Purpose | Stack |
|
|
| --- | --- | --- |
|
|
| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
|
|
| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG |
|
|
| [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible |
|
|
| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + T1→T3 offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider |
|
|
| [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize |
|
|
|
|
Shared runtime data (gitignored): `var/t1/` (live lake), `var/t3/` (warehouse). See [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
# Spin up a virtual swarm of 10 drones (Compose, no cluster needed)
|
|
cd simulator && DRONE_COUNT=10 docker compose up
|
|
|
|
# Run the visualization
|
|
cd prototype && npm install && npm run dev
|
|
```
|
|
|
|
Or run the full miniature of the ground segment — k3d cluster, fleet as
|
|
StatefulSet, Grafana, explorer, warehouse with post-flight offload:
|
|
|
|
```bash
|
|
ansible-playbook infra/ansible/sim-cluster.yml
|
|
cd infra/terraform/sim-env && terraform init && terraform apply
|
|
cd ../ground && terraform init && terraform apply
|
|
# then press "go live" in the prototype header
|
|
```
|
|
|
|
See [06 — Environments → IaC boundaries](docs/06-environments.md#iac-boundaries) for why Ansible owns hosts, Terraform owns the ground segment, and neither flies on the drone.
|
|
|
|
Contributing (TDD-first workflow): [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
|
|
|
## CI & releases
|
|
|
|
Every push runs the pipeline in [`.github/workflows/release.yaml`](.github/workflows/release.yaml): **pytest unit tests**, a compile check, a 10-second virtual smoke flight verified with DuckDB, and a Trivy vulnerability/misconfiguration scan. The simulator Docker build also runs `pytest` — a failing test blocks the image. Pushes to `main` then cut a semantic version from the conventional-commit type (`feat:` → minor, `!`/`BREAKING CHANGE` → major, else patch), tag the commit, and publish a release with a **fleet release manifest** (versioned artifact list with checksums — the delivery unit described in [11 — CI/CD & delivery](docs/11-cicd-delivery.md)) plus a source bundle.
|