swarm-house/docs/12-design-journey.md

12 — Design journey

How this design came together, told in the order the thinking actually happened. It is a walk-through, not a report. The formal decisions, with options and trade-offs, live as Architecture Decision Records. This is the story behind them, and each chapter links straight into the code it produced.

---

1. Starting point

At the start there was only a rough brief and a few adjacent interests to lean on. Key rotation matters in every pipeline: a cipher that is safe today may not be safe next year, so the infrastructure has to leave room to react, without spending too much latency or bandwidth to do it. That balance, security against speed and volume, is a recurring trade-off. Geospatial data is older ground: routing, indexing, fast spatial lookups at scale. So the first instinct was that the value might be less about new storage and more about optimizing access to data that already exists.

2. From words to a model, then to a picture

Once the shape was clear, an autonomous swarm where each unit talks to any peer in range and no controller spans the swarm, the next step was to make the model visible. A small 2D prototype was enough to check it: units patrolling, static and moving obstacles, links forming and breaking, and a rough count of how much data crosses the air. Speed was not the question yet, only volume, because volume is what you size the infrastructure for. Starting from a picture gives an anchor before building anything.

Read the code

• prototype/src/sim.ts — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes
• prototype/src/sim.ts — the pose-frame size and rate that drive the volume estimate

3. Naming the limits

Before any code, the limits had to be named. Maximum distance between units in the air. The fidelity data is stored at. The fidelity it can be transmitted at. Those drive volume, and volume drives the whole shape of the system. The conclusion: constrain the transport, but do not constrain on-board capture. Keep everything locally. Send only what peers truly need.

Read more

• 01 — Problem statement and 03 — Data platform — the constraints and the storage tiers

4. Three tiers inside each unit

That gave three layers on every drone:

1. Raw capture, never transformed.
2. ETL / reduction, where data is cut down to what has to be shared: millimetres to centimetres, thinning time series where that is enough, dropping what nobody downstream reads.
3. A bidirectional interface: on one side it broadcasts position into the shared channel, on the other it lets peers pull data.

The ETL is deliberately not over-specified. How sensor data is transformed is a data-engineering decision. The platform's job is the contract, meaning what gets stored and in which structure, not to step into that work.

Read the code

• simulator/virtual_drone/writer.py — the Hive-partition layout and the seal step that compacts a flight
• simulator/virtual_drone/sensors.py — the raw capture that feeds tier one

5. What actually needs to sync

The most time-critical item is where each peer is, so every unit has time to react. The choice was to broadcast relative pose (x, y, z and time) instead of absolute coordinates. Relative is cheaper and enough for coordination. To avoid sending orientation, a unit is modelled as a sphere that bounds its extent. That trades a little compute for much less data, and no per-shape encoding. If the units are identical, their 3D model can be provisioned ahead instead of transmitted.

Read the code

• simulator/virtual_drone/broadcast.py — the 46-byte pose frame, position quantized on the wire only
• prototype/src/sim.ts — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible

6. Security as nested layers, not one wall

The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear, even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct trajectories over time, and wireless encryption has been broken by ordinary people for two decades. Relying on that layer alone is risky, and pushing heavy encryption there is wasteful. So the trust model is layered:

1. Wireless transport (WPA class) as the outer shell.
2. WireGuard over UDP as the mesh overlay: proven, peer-authenticated by static keys, cheap.
3. SSH inside the tunnel as the application layer, instead of HTTP/HTTPS. SSH is one of the most tested remote-access technologies we have. HTTPS would pull in a certificate authority and extra parts, and it is the first surface a security researcher probes. Fewer moving parts, on purpose.

Identity and encryption solve together at provisioning time: a per-flight, per-unit key pair recorded in a flight registry. Each unit holds its own key pair plus the public keys of its peers. A separate key registry for SSH means that if one WireGuard key leaks, SSH access does not follow automatically.

Read the code

• infra/ansible/drone-provision.yml — ed25519 identity, peers authorized only through a forced command, WireGuard brought up
• 05 — Network & security and ADR-0005

7. SQL as the contract

Beyond the broadcast, people and peers want richer access. The whole progression is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier schemes. Each time the same pain: hard to debug in the moment, and a lot of infrastructure to support, test, and agree on. The simpler conclusion: SQL is the contract. Everyone who touches data already speaks SQL, and we do not know in advance which slice each of them needs, so give them a declarative, extensible way in rather than guessing endpoints ahead of time.

Concretely: an SSH connection whose forced command runs a read-only DuckDB query directly. No extra auth handshake between the engine and the data, the way a long-running database needs. No new port or protocol. When access control is needed, it rides on Linux file permissions, because the data is discrete Hive-partitioned files and permissions can be set per file. Not new invention: coming back to foundations DevOps and sysadmins already trust.

Read the code

• simulator/explorer/server.py — the read-only gate that rejects anything that writes
• simulator/explorer/server.py — the query path a peer or an engineer hits identically
• simulator/tests/test_sql_gate.py — the gate is safety-critical, so it is tested first

8. Replication and offload

Moving accumulated data uses rsync-class tooling (rclone) over the same SSH channel. It transfers diffs of the Hive tree efficiently. This matters because a unit can be lost, and if it is, we want its data to already exist elsewhere. When peers have a stable link the partition tree is pulled between them, which gives redundancy and a way to cross-check later. On the ground, when the fleet returns, the same path unifies every unit's data into a local warehouse. The warehouse is read-mostly for analytics, so transaction contention is not a concern, which opens a DuckDB / DuckLake approach with MinIO underneath, the same storage model on both ends. One uniform structure is what lets an analyst trust a single picture even when a unit's data has a gap.

Read the code

• infra/terraform/ground/main.tf — the post-flight T1 to T3 offload CronJob
• 06 — Environments and ADR-0002

9. A proof of concept, not slides

A concept can be handed over and be wrong. Instead there is a generator: a virtual drone producing real sensor data into the Hive layout, a small frontend to watch it fill, and a Grafana dashboard on top, not to get ahead of the analysts, but to see the infrastructure work end to end. Closing the loop (Terraform, connector, exporter, dashboard) is how a misconfiguration shows up early. A panel that renders wrong points at exactly which layer broke, before an analyst has to report that the infra is up but nothing shows. Better to exercise every layer now and drop any layer or technology later if there is a sound argument against it.

Watching that loop is also where the CPU cost surfaced: scanning the whole lake on every query does not scale, so the scan is bounded to a flight window and cached.

Read the code

• simulator/monitoring/exporter.py — the DuckDB scan that turns Parquet into Prometheus metrics
• simulator/monitoring/lake.py — the flight-window scan that keeps CPU bounded as the lake grows
• simulator/monitoring/grafana/dashboards/ — the fleet and platform dashboards
• .github/workflows/release.yaml — tests, smoke flight, Trivy, semantic release

10. Working method

A note on how this is meant to be built, not just what. Working together needs a shared language, and when the working language is nobody's first language, answering live is expensive. The form that scales is written-first: lay the idea down carefully, let others read and answer in their own time, keep the history. A new colleague, or an agent, can then load the full context of why a decision was made by reading the discussion. That is exactly why the decisions here are ADRs and why the contribution flow is test-first: the reasoning is captured where the next person will look for it.

The mental image is a city planner. Draw perfectly straight roads and people still cross the grass where it is faster. The job is not to decide for them, but to understand the real paths, the pain points and limits of each team, and pave those. So constraints get checked with the other DevOps, data engineers, and analysts before infrastructure is laid, instead of building roads nobody uses.

---

The platform is air-gapped and on-prem by design: nothing leaves the swarm, and nothing joins it at runtime. Start from the README for the map, or the ADR index for the decisions in full.