Swarm House

An on-prem data platform for autonomous drone swarms — a design proposal from a DevOps perspective.

A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale.

This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.

Design principles

1. Every drone is autonomous. No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
2. One storage format on every floor. Parquet + DuckDB with an identical Hive-partitioned layout on the drone, in the ground warehouse, and in the dev environment. Flight offload is a plain mirror operation.
3. Sync results, not raw data. Only derived state (position, attitude, detections) crosses the air. Raw telemetry stays on local NVMe until the drone lands.
4. Event-driven, not polled. New derived data triggers downstream actions through hooks; nothing burns cycles asking "anything new yet?".
5. Zero trust, provisioned ahead. Keys are issued per device before deployment; nothing joins the swarm at runtime. Dev channels are physically absent from production builds.
6. The whole fleet has one version. A release manifest pins every image digest, model weight, schema, and config. Rollback is atomic.
7. SQL is the contract. Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. No debug API. The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.

Documentation

Document	Contents
00 — Glossary	Terms and abbreviations used throughout
01 — Problem statement	Goal, constraints, knowns vs assumptions
02 — Architecture	On-board layers, service composition, communication planes
03 — Data platform	Parquet/DuckDB storage tiers, partitioning, hot-write path
04 — Swarm sync	What syncs, what does not, pub/sub transport, query standard
05 — Network & security	Zero-trust mesh, key provisioning, encrypted transport
06 — Environments	Drone / ground warehouse / dev-simulation infrastructures
07 — Observability	Logs, service metrics, hardware telemetry
08 — Roadmap	Dependency graph, simple to complex
09 — Open questions	Known unknowns and proposed answers
10 — Domain context	Swarm autonomy principles this design builds on
11 — CI/CD & delivery	Pipelines, registry, dev containers, fleet releases

Runnable parts

Component	Purpose	Stack
simulator/	Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose	Python, pyarrow, DuckDB
prototype/	2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics	React, TypeScript, Vite, SVG

Quick start

# Spin up a virtual swarm of 10 drones
cd simulator && DRONE_COUNT=10 docker compose up

# Run the visualization
cd prototype && npm install && npm run dev