diff --git a/README.md b/README.md index 8d919d5..8cfa7d3 100644 --- a/README.md +++ b/README.md @@ -33,9 +33,10 @@ This repository describes how to build, deliver, test, and operate that platform | [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers | | [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on | | [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases | +| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code | | [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made | -The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record. +New here? Read the [design journey](docs/12-design-journey.md) first: it tells the story chronologically and links straight into the code and decisions. The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record. ## Runnable parts diff --git a/docs/12-design-journey.md b/docs/12-design-journey.md new file mode 100644 index 0000000..749df8b --- /dev/null +++ b/docs/12-design-journey.md @@ -0,0 +1,189 @@ +# 12 — Design journey + +How this design came together, told in the order the thinking actually happened. +It is a walk-through, not a report. The formal decisions, with options and +trade-offs, live as [Architecture Decision Records](adr/README.md). This is the +story behind them, and each chapter links straight into the code it produced. + +--- + +## 1. Starting point + +At the start there was only a rough brief and a few adjacent interests to lean +on. Key rotation matters in every pipeline: a cipher that is safe today may not +be safe next year, so the infrastructure has to leave room to react, without +spending too much latency or bandwidth to do it. That balance, security against +speed and volume, is a recurring trade-off. Geospatial data is older ground: +routing, indexing, fast spatial lookups at scale. So the first instinct was that +the value might be less about new storage and more about optimizing access to +data that already exists. + +## 2. From words to a model, then to a picture + +Once the shape was clear, an autonomous swarm where each unit talks to any peer +in range and [no controller spans the swarm](adr/ADR-0001-no-swarm-orchestrator.md), +the next step was to make the model visible. A small 2D prototype was enough to +check it: units patrolling, static and moving obstacles, links forming and +breaking, and a rough count of how much data crosses the air. Speed was not the +question yet, only volume, because volume is what you size the infrastructure +for. Starting from a picture gives an anchor before building anything. + +> **Read the code** +> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L219-L287) — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes +> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L57-L63) — the pose-frame size and rate that drive the volume estimate + +## 3. Naming the limits + +Before any code, the limits had to be named. Maximum distance between units in +the air. The fidelity data is stored at. The fidelity it can be transmitted at. +Those drive volume, and volume drives the whole shape of the system. The +conclusion: constrain the transport, but do not constrain on-board capture. Keep +everything locally. Send only what peers truly need. + +> **Read more** +> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers + +## 4. Three tiers inside each unit + +That gave [three layers](03-data-platform.md) on every drone: + +1. **Raw capture**, never transformed. +2. **ETL / reduction**, where data is cut down to what has to be shared: + millimetres to centimetres, thinning time series where that is enough, + dropping what nobody downstream reads. +3. **A bidirectional interface**: on one side it broadcasts position into the + shared channel, on the other it lets peers pull data. + +The ETL is deliberately not over-specified. How sensor data is transformed is a +data-engineering decision. The platform's job is the contract, meaning what gets +stored and in which structure, not to step into that work. + +> **Read the code** +> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight +> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one + +## 5. What actually needs to sync + +The most time-critical item is where each peer is, so every unit has time to +react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and +time) instead of absolute coordinates. Relative is cheaper and enough for +coordination. To avoid sending orientation, a unit is modelled as a **sphere** +that bounds its extent. That trades a little compute for much less data, and no +per-shape encoding. If the units are identical, their 3D model can be provisioned +ahead instead of transmitted. + +> **Read the code** +> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only +> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible + +## 6. Security as nested layers, not one wall + +The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear, +even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct +trajectories over time, and wireless encryption has been broken by ordinary +people for two decades. Relying on that layer alone is risky, and pushing heavy +encryption there is wasteful. So the trust model is [layered](05-network-security.md): + +1. Wireless transport (WPA class) as the outer shell. +2. **WireGuard over UDP** as the mesh overlay: proven, peer-authenticated by + static keys, cheap. +3. **SSH inside the tunnel** as the application layer, instead of HTTP/HTTPS. SSH + is one of the most tested remote-access technologies we have. HTTPS would pull + in a certificate authority and extra parts, and it is the first surface a + security researcher probes. [Fewer moving parts, on purpose](adr/ADR-0005-wireguard-beneath-ssh.md). + +Identity and encryption solve together at provisioning time: a per-flight, +per-unit key pair recorded in a flight registry. Each unit holds its own key pair +plus the public keys of its peers. A separate key registry for SSH means that if +one WireGuard key leaks, SSH access does not follow automatically. + +> **Read the code** +> - [`infra/ansible/drone-provision.yml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/ansible/drone-provision.yml#L20-L70) — ed25519 identity, peers authorized only through a forced command, WireGuard brought up +> - [05 — Network & security](05-network-security.md) and [ADR-0005](adr/ADR-0005-wireguard-beneath-ssh.md) + +## 7. SQL as the contract + +Beyond the broadcast, people and peers want richer access. The whole progression +is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier +schemes. Each time the same pain: hard to debug in the moment, and a lot of +infrastructure to support, test, and agree on. The simpler conclusion: +[**SQL is the contract**](adr/ADR-0004-sql-over-ssh-contract.md). Everyone who +touches data already speaks SQL, and we do not know in advance which slice each +of them needs, so give them a declarative, extensible way in rather than guessing +endpoints ahead of time. + +Concretely: an SSH connection whose forced command runs a read-only **DuckDB** +query directly. No extra auth handshake between the engine and the data, the way +a long-running database needs. No new port or protocol. When access control is +needed, it rides on Linux file permissions, because the data is discrete +Hive-partitioned files and permissions can be set per file. Not new invention: +coming back to foundations DevOps and sysadmins already trust. + +> **Read the code** +> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L43-L53) — the read-only gate that rejects anything that writes +> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically +> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first + +## 8. Replication and offload + +Moving accumulated data uses rsync-class tooling (rclone) over the same SSH +channel. It transfers diffs of the Hive tree efficiently. This matters because a +unit can be lost, and if it is, we want its data to already exist elsewhere. +When peers have a stable link the partition tree is pulled between them, which +gives redundancy and a way to cross-check later. On the ground, when the fleet +returns, the same path unifies every unit's data into a +[local warehouse](06-environments.md). The warehouse is read-mostly for +analytics, so transaction contention is not a concern, which opens a +DuckDB / DuckLake approach with MinIO underneath, the +[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One +uniform structure is what lets an analyst trust a single picture even when a +unit's data has a gap. + +> **Read the code** +> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob +> - [06 — Environments](06-environments.md) and [ADR-0002](adr/ADR-0002-one-storage-format.md) + +## 9. A proof of concept, not slides + +A concept can be handed over and be wrong. Instead there is a generator: a +virtual drone producing real sensor data into the Hive layout, a small frontend +to watch it fill, and a **Grafana** dashboard on top, not to get ahead of the +analysts, but to see the infrastructure work end to end. Closing the loop +(Terraform, connector, exporter, dashboard) is how a misconfiguration shows up +early. A panel that renders wrong points at exactly which layer broke, before an +analyst has to report that the infra is up but nothing shows. Better to exercise +every layer now and drop any layer or technology later if there is a sound +argument against it. + +Watching that loop is also where the [CPU cost surfaced](adr/ADR-0009-bounded-lake-scans.md): +scanning the whole lake on every query does not scale, so the scan is +[bounded to a flight window and cached](07-observability.md). + +> **Read the code** +> - [`simulator/monitoring/exporter.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/exporter.py#L38-L107) — the DuckDB scan that turns Parquet into Prometheus metrics +> - [`simulator/monitoring/lake.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/lake.py#L13-L33) — the flight-window scan that keeps CPU bounded as the lake grows +> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards +> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release + +## 10. Working method + +A note on how this is meant to be built, not just what. Working together needs a +shared language, and when the working language is nobody's first language, +answering live is expensive. The form that scales is written-first: lay the idea +down carefully, let others read and answer in their own time, keep the history. A +new colleague, or an agent, can then load the full context of why a decision was +made by reading the discussion. That is exactly why the decisions here are +[ADRs](adr/README.md) and why the [contribution flow](../CONTRIBUTING.md) is +test-first: the reasoning is captured where the next person will look for it. + +The mental image is a city planner. Draw perfectly straight roads and people +still cross the grass where it is faster. The job is not to decide for them, but +to understand the real paths, the pain points and limits of each team, and pave +those. So constraints get checked with the other DevOps, data engineers, and +analysts before infrastructure is laid, instead of building roads nobody uses. + +--- + +The platform is air-gapped and on-prem by design: nothing leaves the swarm, and +nothing joins it at runtime. Start from the [README](../README.md) for the map, +or the [ADR index](adr/README.md) for the decisions in full.