swarm-house/docs/12-design-journey.md

# 12 — Design journey

How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report — and **not a mandate**. The goal is a
from-scratch platform sketch that shows how the pieces fit; every concrete
choice is an option with trade-offs recorded as
[Architecture Decision Records](adr/README.md). Replace any piece if a better
fit shows up. This is the story behind the decisions, and each chapter links
straight into the code it produced.

---

## 1. Starting point

At the start there was only a rough brief and a few adjacent interests to lean
on. Key rotation matters in every pipeline: a cipher that is safe today may not
be safe next year, so the infrastructure has to leave room to react, without
spending too much latency or bandwidth to do it. That balance, security against
speed and volume, is a recurring trade-off. Geospatial data is older ground:
routing, indexing, fast spatial lookups at scale. So the first instinct was that
the value might be less about new storage and more about optimizing access to
data that already exists.

## 2. From words to a model, then to a picture

Once the shape was clear, an autonomous swarm where each unit talks to any peer
in range and [no controller spans the swarm](adr/ADR-0001-no-swarm-orchestrator.md),
the next step was to make the model visible. A small 2D prototype was enough to
check it: units patrolling, static and moving obstacles, links forming and
breaking, and a rough count of how much data crosses the air. Speed was not the
question yet, only volume, because volume is what you size the infrastructure
for. Starting from a picture gives an anchor before building anything.

> **Read the code**
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L219-L287) — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L57-L63) — the pose-frame size and rate that drive the volume estimate

The picture is running, not just described: the visual prototype is live at
[swarm.produktor.io](https://swarm.produktor.io/) (same access as this
repository). It draws units patrolling, links forming and breaking by range, and
a running estimate of how much data crosses the air.

## 3. Naming the limits

Before any code, the limits had to be named. Maximum distance between units in
the air. The fidelity data is stored at. The fidelity it can be transmitted at.
Those drive volume, and volume drives the whole shape of the system. The
conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need.

> **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers

## 4. Three tiers inside each unit

That gave [three layers](03-data-platform.md) on every drone:

1. **Raw capture**, never transformed.
2. **ETL / reduction**, where data is cut down to what has to be shared:
   millimetres to centimetres, thinning time series where that is enough,
   dropping what nobody downstream reads.
3. **A bidirectional interface**: on one side it broadcasts position into the
   shared channel, on the other it lets peers pull data.

The ETL is deliberately not over-specified. How sensor data is transformed is a
data-engineering decision. The platform's job is the contract, meaning what gets
stored and in which structure, not to step into that work.

> **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one

## 5. What actually needs to sync

The most time-critical item is where each peer is, so every unit has time to
react. The wire frame carries **mission-frame position** (millimetres on the
wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags —
45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the
fusion pain. An earlier sketch used relative coordinates and a bounding sphere
with no orientation; it was dropped. When peers disagree on frames,
`frame_ref` makes the mismatch explicit for consumers
([09 — Open questions](09-open-questions.md)).

> **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible

## 6. Security as nested layers, not one wall

The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear,
even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct
trajectories over time, and wireless encryption has been broken by ordinary
people for two decades. Relying on that layer alone is risky, and pushing heavy
encryption there is wasteful. So the trust model is [layered](05-network-security.md):

1. Wireless transport (WPA class) as the outer shell.
2. **WireGuard over UDP** as the mesh overlay: proven, peer-authenticated by
   static keys, cheap.
3. **SSH inside the tunnel** as the application layer, instead of HTTP/HTTPS. SSH
   is one of the most tested remote-access technologies we have. HTTPS would pull
   in a certificate authority and extra parts, and it is the first surface a
   security researcher probes. [Fewer moving parts, on purpose](adr/ADR-0005-wireguard-beneath-ssh.md).

Identity and encryption solve together at provisioning time: a per-flight,
per-unit key pair recorded in a flight registry. Each unit holds its own key pair
plus the public keys of its peers. A separate key registry for SSH means that if
one WireGuard key leaks, SSH access does not follow automatically.

> **Read the code**
> - [`infra/ansible/drone-provision.yml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/ansible/drone-provision.yml#L20-L70) — ed25519 identity, peers authorized only through a forced command, WireGuard brought up
> - [05 — Network & security](05-network-security.md) and [ADR-0005](adr/ADR-0005-wireguard-beneath-ssh.md)

## 7. SQL as the contract

Beyond the broadcast, people and peers want richer access. The whole progression
is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier
schemes. Each time the same pain: hard to debug in the moment, and a lot of
infrastructure to support, test, and agree on. The simpler conclusion:
[**SQL is the contract**](adr/ADR-0004-sql-over-ssh-contract.md). Everyone who
touches data already speaks SQL, and we do not know in advance which slice each
of them needs, so give them a declarative, extensible way in rather than guessing
endpoints ahead of time.

Concretely: an SSH connection whose forced command runs a read-only **DuckDB**
query directly. No extra auth handshake between the engine and the data, the way
a long-running database needs. No new port or protocol. When access control is
needed, it rides on Linux file permissions, because the data is discrete
Hive-partitioned files and permissions can be set per file. Not new invention:
coming back to foundations DevOps and sysadmins already trust.

> **Read the code**
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L43-L53) — the read-only gate that rejects anything that writes
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first

## 8. Replication and offload

Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
channel. It transfers diffs of the Hive tree efficiently. This matters because a
unit can be lost, and if it is, we want its data to already exist elsewhere.
When peers have a stable link the partition tree is pulled between them, which
gives redundancy and a way to cross-check later. On the ground, when the fleet
returns, the same path unifies every unit's data into a
[local warehouse](06-environments.md). The warehouse is read-mostly for
analytics, so transaction contention is not a concern, which opens a
DuckDB / DuckLake approach with MinIO underneath, the
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
uniform structure is what lets an analyst trust a single picture even when a
unit's data has a gap.

> **Read the code**
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
> - [06 — Environments](06-environments.md) and [ADR-0002](adr/ADR-0002-one-storage-format.md)

## 9. A proof of concept, not slides

A concept can be handed over and be wrong. Instead there is a generator: a
virtual drone producing real sensor data into the Hive layout, a small frontend
to watch it fill, and a **Grafana** dashboard on top, not to get ahead of the
analysts, but to see the infrastructure work end to end. Closing the loop
(Terraform, connector, exporter, dashboard) is how a misconfiguration shows up
early. A panel that renders wrong points at exactly which layer broke, before an
analyst has to report that the infra is up but nothing shows. Better to exercise
every layer now and drop any layer or technology later if there is a sound
argument against it.

Watching that loop is also where the [CPU cost surfaced](adr/ADR-0009-bounded-lake-scans.md):
scanning the whole lake on every query does not scale, so the scan is
[bounded to a flight window and cached](07-observability.md).

> **Read the code**
> - [`simulator/monitoring/exporter.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/exporter.py#L38-L107) — the DuckDB scan that turns Parquet into Prometheus metrics
> - [`simulator/monitoring/lake.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/lake.py#L13-L33) — the flight-window scan that keeps CPU bounded as the lake grows
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release

## 10. Working method

A note on how this is meant to be built, not just what. Working together needs a
shared language, and when the working language is nobody's first language,
answering live is expensive. The form that scales is written-first: lay the idea
down carefully, let others read and answer in their own time, keep the history. A
new colleague, or an agent, can then load the full context of why a decision was
made by reading the discussion. That is exactly why the decisions here are
[ADRs](adr/README.md) and why the [contribution flow](../CONTRIBUTING.md) is
test-first: the reasoning is captured where the next person will look for it.

The mental image is a city planner. Draw perfectly straight roads and people
still cross the grass where it is faster. The job is not to decide for them, but
to understand the real paths, the pain points and limits of each team, and pave
those. So constraints get checked with the other DevOps, data engineers, and
analysts before infrastructure is laid, instead of building roads nobody uses.

---

The platform is air-gapped and on-prem by design: nothing leaves the swarm, and
nothing joins it at runtime. Start from the [README](../README.md) for the map,
or the [ADR index](adr/README.md) for the decisions in full.