eSlider
87551fc1c1
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
Treat MinIO as existing stack infrastructure (default T3 warehouse, not on the drone), add a one-page executive map, and record placement as an open team question rather than a rip-and-replace.
104 lines
8.6 KiB
Markdown
104 lines
8.6 KiB
Markdown
# Swarm House
|
|
|
|
**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective.
|
|
|
|
A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale.
|
|
|
|
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
|
|
|
|
**New here?** Start with the one-pager: [00 — Executive overview](docs/00-executive.md) (includes how **MinIO** fits). Then the [design journey](docs/12-design-journey.md) if you want the narrative.
|
|
|
|
## Design principles
|
|
|
|
1. **Every drone is autonomous.** No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
|
|
2. **One storage format on every floor.** Parquet + DuckDB with an identical Hive-partitioned layout on the drone, in the ground warehouse, and in the dev environment. Flight offload is a plain mirror operation.
|
|
3. **Sync results, not raw data.** Only derived state (position, attitude, detections) crosses the air. Raw telemetry stays on local NVMe until the drone lands.
|
|
4. **Event-driven, not polled.** New derived data triggers downstream actions through hooks; nothing burns cycles asking "anything new yet?".
|
|
5. **Zero trust, provisioned ahead.** Keys are issued per device before deployment; nothing joins the swarm at runtime. Dev channels are physically absent from production builds.
|
|
6. **The whole fleet has one version.** A release manifest pins every image digest, model weight, schema, and config. Rollback is atomic.
|
|
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
|
|
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
|
|
|
|
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
|
|
|
|
## Implemented now vs proposed next
|
|
|
|
Honest map so a reader knows what runs today versus what is design intent.
|
|
|
|
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
|
|
| --- | --- | --- |
|
|
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
|
|
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
|
|
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
|
|
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
|
|
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
|
|
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, **MinIO as T3** | k3s warehouse with MinIO, GitOps overlays, post-flight mirror |
|
|
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
|
|
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
|
|
|
|
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
|
|
|
|
## Documentation
|
|
|
|
| Document | Contents |
|
|
| --- | --- |
|
|
| [00 — Executive](docs/00-executive.md) | **Start here** — one-page goal, sync rule, MinIO stance |
|
|
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
|
|
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
|
|
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
|
|
| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path |
|
|
| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard |
|
|
| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport |
|
|
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
|
|
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
|
|
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
|
|
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers (incl. MinIO placement) |
|
|
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
|
|
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
|
|
| [12 — Design journey](docs/12-design-journey.md) | Narrative walk-through linking into the code |
|
|
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
|
|
|
|
New here? Read the [executive overview](docs/00-executive.md) first, then the [design journey](docs/12-design-journey.md) if you want the story. The design principles above are the *what*; the [ADRs](docs/adr/) are the *why and when*.
|
|
|
|
## Runnable parts
|
|
|
|
| Component | Purpose | Stack |
|
|
| --- | --- | --- |
|
|
| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
|
|
| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG |
|
|
| [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible |
|
|
| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + T1→T3 offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider |
|
|
| [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize |
|
|
|
|
Shared runtime data (gitignored): `var/t1/` (live lake), `var/t3/` (warehouse). See [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
# Spin up a virtual swarm of 10 drones (Compose, no cluster needed)
|
|
cd simulator && DRONE_COUNT=10 docker compose up
|
|
|
|
# Run the visualization
|
|
cd prototype && npm install && npm run dev
|
|
```
|
|
|
|
Or run the full miniature of the ground segment — k3d cluster, fleet as
|
|
StatefulSet, Grafana, explorer, warehouse with post-flight offload:
|
|
|
|
```bash
|
|
ansible-playbook infra/ansible/sim-cluster.yml
|
|
cd infra/terraform/sim-env && terraform init && terraform apply
|
|
cd ../ground && terraform init && terraform apply
|
|
# then press "go live" in the prototype header
|
|
```
|
|
|
|
See [06 — Environments → IaC boundaries](docs/06-environments.md#iac-boundaries) for why Ansible owns hosts, Terraform owns the ground segment, and neither flies on the drone.
|
|
|
|
Contributing (TDD-first workflow): [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
|
|
|
## CI & releases
|
|
|
|
Every push runs the pipeline in [`.github/workflows/release.yaml`](.github/workflows/release.yaml): **pytest unit tests**, a compile check, a 10-second virtual smoke flight verified with DuckDB, and a Trivy vulnerability/misconfiguration scan. The simulator Docker build also runs `pytest` — a failing test blocks the image. Pushes to `main` then cut a semantic version from the conventional-commit type (`feat:` → minor, `!`/`BREAKING CHANGE` → major, else patch), tag the commit, and publish a release with a **fleet release manifest** (versioned artifact list with checksums — the delivery unit described in [11 — CI/CD & delivery](docs/11-cicd-delivery.md)) plus a source bundle.
|
|
|
|
[LICENSE](LICENSE)
|