swarm-house/README.md

Swarm House

An on-prem data platform for autonomous drone swarms — a design proposal from a DevOps perspective.

Fleet flies autonomous. No internet uplink, no external access. Data stays in the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. Swarm exchanges just enough state to coordinate; everything else stays local until a ground warehouse receives it after landing — reproducibly, securely, at scale.

This repository covers storage layout, sync strategy, network trust model, CI/CD chain, and the simulation environment that tests it without real hardware.

Restricted use. Third parties may not use, copy, modify, or distribute this source code, documentation, or related materials without explicit written permission from the copyright holder. No license is granted by default. See LICENSE for the full terms.

Design principles

1. Every drone is autonomous. No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
2. One storage format on every floor. Parquet + DuckDB with an identical Hive-partitioned layout on the drone, in the ground warehouse, and in the dev environment. Flight offload is a plain mirror operation.
3. Sync results, not raw data. Only derived state (position, attitude, detections) crosses the air. Raw telemetry stays on local NVMe until the drone lands.
4. Event-driven, not polled. New derived data triggers downstream actions through hooks; nothing burns cycles asking "anything new yet?".
5. Zero trust, provisioned ahead. Keys are issued per device before deployment; nothing joins the swarm at runtime. Dev channels are physically absent from production builds.
6. The whole fleet has one version. A release manifest pins every image digest, model weight, schema, and config. Rollback is atomic.
7. SQL is the contract. Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. No debug API. The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.

Documentation

Document	Contents
00 — Glossary	Terms and abbreviations used throughout
01 — Problem statement	Goal, constraints, knowns vs assumptions
02 — Architecture	On-board layers, service composition, communication planes
03 — Data platform	Parquet/DuckDB storage floors (T0–T4), partitioning, hot-write path
04 — Swarm sync	What syncs, what does not, pub/sub transport, query standard
05 — Network & security	Zero-trust mesh, key provisioning, encrypted transport
06 — Environments	Drone / ground warehouse / dev-simulation infrastructures
07 — Observability	Logs, service metrics, hardware telemetry
08 — Roadmap	Dependency graph, simple to complex
09 — Open questions	Known unknowns and proposed answers
10 — Domain context	Swarm autonomy principles this design builds on
11 — CI/CD & delivery	Pipelines, registry, dev containers, fleet releases
12 — Design journey	Start here — a narrative walk-through of how the design came together, linking into the code
ADRs	Architecture Decision Records — the decisions behind the above, in the order they were made

New? Start with the design journey — story plus code links. Principles = what; ADRs = why and when.

Runnable parts

Component	Purpose	Stack
simulator/	Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose	Python, pyarrow, DuckDB
prototype/	2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; live mode renders real positions from the k3d fleet	React, TypeScript, Vite, SVG
infra/ansible/	Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster	Ansible
infra/terraform/	Declared workloads on the sim cluster: virtual fleet + observability (sim-env), warehouse + T1→T3 offload (ground)	Terraform, kubernetes provider
infra/gitops/	Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest	Flux, Kustomize

Shared runtime data (gitignored): var/t1/ (live lake, T1), var/t3/ (warehouse, T3). See CONTRIBUTING.md.

Quick start

# Spin up a virtual swarm of 10 drones (Compose, no cluster needed)
cd simulator && DRONE_COUNT=10 docker compose up

# Run the visualization
cd prototype && npm install && npm run dev

Or run the full miniature of the ground segment — k3d cluster, fleet as StatefulSet, Grafana, explorer, warehouse with post-flight offload:

ansible-playbook infra/ansible/sim-cluster.yml
cd infra/terraform/sim-env && terraform init && terraform apply
cd ../ground && terraform init && terraform apply
# then press "go live" in the prototype header

See 06 — Environments → IaC boundaries for why Ansible owns hosts, Terraform owns the ground segment, and neither flies on the drone.

Contributing (TDD-first workflow): CONTRIBUTING.md.

CI & releases

Every push runs the pipeline in .github/workflows/release.yaml: pytest unit tests, a compile check, a 10-second virtual smoke flight verified with DuckDB, and a Trivy vulnerability/misconfiguration scan. The simulator Docker build also runs pytest — a failing test blocks the image. Pushes to main then cut a semantic version from the conventional-commit type (feat: → minor, !/BREAKING CHANGE → major, else patch), tag the commit, and publish a release with a fleet release manifest (versioned artifact list with checksums — the delivery unit described in 11 — CI/CD & delivery) plus a source bundle.

LICENSE