eSlider
523e15f490
Record the full reasoning path (GraphQL service, Arrow Flight, raw SSH, forced-command synthesis) and specify read-only enforcement as five fail-safe layers: key-per-operation, SELECT-only statement gate, read-only OS user, engine hardening, resource caps. Promote 'SQL is the contract' to a top-level design principle.
3.9 KiB
Swarm House
An on-prem data platform for autonomous drone swarms — a design proposal from a DevOps perspective.
A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale.
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
Design principles
- Every drone is autonomous. No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane.
- One storage format on every floor. Parquet + DuckDB with an identical Hive-partitioned layout on the drone, in the ground warehouse, and in the dev environment. Flight offload is a plain mirror operation.
- Sync results, not raw data. Only derived state (position, attitude, detections) crosses the air. Raw telemetry stays on local NVMe until the drone lands.
- Event-driven, not polled. New derived data triggers downstream actions through hooks; nothing burns cycles asking "anything new yet?".
- Zero trust, provisioned ahead. Keys are issued per device before deployment; nothing joins the swarm at runtime. Dev channels are physically absent from production builds.
- The whole fleet has one version. A release manifest pins every image digest, model weight, schema, and config. Rollback is atomic.
- SQL is the contract. Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
Documentation
| Document | Contents |
|---|---|
| 00 — Glossary | Terms and abbreviations used throughout |
| 01 — Problem statement | Goal, constraints, knowns vs assumptions |
| 02 — Architecture | On-board layers, service composition, communication planes |
| 03 — Data platform | Parquet/DuckDB storage tiers, partitioning, hot-write path |
| 04 — Swarm sync | What syncs, what does not, pub/sub transport, query standard |
| 05 — Network & security | Zero-trust mesh, key provisioning, encrypted transport |
| 06 — Environments | Drone / ground warehouse / dev-simulation infrastructures |
| 07 — Observability | Logs, service metrics, hardware telemetry |
| 08 — Roadmap | Dependency graph, simple to complex |
| 09 — Open questions | Known unknowns and proposed answers |
| 10 — Domain context | Swarm autonomy principles this design builds on |
| 11 — CI/CD & delivery | Pipelines, registry, dev containers, fleet releases |
Runnable parts
| Component | Purpose | Stack |
|---|---|---|
simulator/ |
Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
prototype/ |
2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics | React, TypeScript, Vite, SVG |
Quick start
# Spin up a virtual swarm of 10 drones
cd simulator && DRONE_COUNT=10 docker compose up
# Run the visualization
cd prototype && npm install && npm run dev