eSlider
5f6fd8462c
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
213 lines
13 KiB
Markdown
213 lines
13 KiB
Markdown
# 12 — Design journey
|
||
|
||
How this design came together, told in the order the thinking actually happened.
|
||
It is a walk-through, not a report. The formal decisions, with options and
|
||
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
|
||
story behind them, and each chapter links straight into the code it produced.
|
||
Terms: [Glossary](00-glossary.md).
|
||
|
||
---
|
||
|
||
## 1. Starting point
|
||
|
||
At the start there was only a rough brief and a few adjacent interests to lean
|
||
on. Key rotation matters in every pipeline: a cipher that is safe today may not
|
||
be safe next year, so the infrastructure has to leave room to react, without
|
||
spending too much latency or bandwidth to do it. That balance, security against
|
||
speed and volume, is a recurring trade-off. Geospatial data is older ground:
|
||
routing, indexing, fast spatial lookups at scale. So the first instinct was that
|
||
the value might be less about new storage and more about optimizing access to
|
||
data that already exists.
|
||
|
||
## 2. From words to a model, then to a picture
|
||
|
||
Once the shape was clear, an autonomous swarm where each unit talks to any peer
|
||
in range and [no controller spans the swarm](adr/ADR-0001-no-swarm-orchestrator.md),
|
||
the next step was to make the model visible. A small 2D prototype was enough to
|
||
check it: units patrolling, static and moving obstacles, links forming and
|
||
breaking, and a rough count of how much data crosses the air. Speed was not the
|
||
question yet, only volume, because volume is what you size the infrastructure
|
||
for. Starting from a picture gives an anchor before building anything.
|
||
|
||
> **Read the code**
|
||
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L219-L287) — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes
|
||
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L57-L63) — the pose-frame size and rate that drive the volume estimate
|
||
|
||
The picture is running, not just described: the visual prototype is live at
|
||
[swarm.produktor.io](https://swarm.produktor.io/) (same access as this
|
||
repository). It draws units patrolling, links forming and breaking by range, and
|
||
a running estimate of how much data crosses the air.
|
||
|
||
## 3. Naming the limits
|
||
|
||
Before any code, the limits had to be named. Maximum distance between units in
|
||
the air. The fidelity data is stored at. The fidelity it can be transmitted at.
|
||
Those drive volume, and volume drives the whole shape of the system. The
|
||
conclusion: constrain the transport, but do not constrain on-board capture. Keep
|
||
everything locally. Send only what peers truly need.
|
||
|
||
> **Read more**
|
||
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0–T4)
|
||
|
||
## 4. Three pipeline stages inside each unit
|
||
|
||
That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0–T4:
|
||
|
||
1. **Raw capture** (pipeline stage 1), never transformed.
|
||
2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared:
|
||
millimeters to centimeters, thinning time series where that is enough,
|
||
dropping what nobody downstream reads.
|
||
3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the
|
||
shared channel, on the other it lets peers pull data.
|
||
|
||
The ETL is deliberately not over-specified. How sensor data is transformed is a
|
||
data-engineering decision. The platform's job is the contract, meaning what gets
|
||
stored and in which structure, not to step into that work.
|
||
|
||
> **Read the code**
|
||
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
|
||
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1
|
||
|
||
## 5. What actually needs to sync
|
||
|
||
The most time-critical item is where each peer is, so every unit has time to
|
||
react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and
|
||
time) instead of absolute coordinates. Relative is cheaper and enough for
|
||
coordination. To avoid sending orientation, a unit is modelled as a **sphere**
|
||
that bounds its extent. That trades a little compute for much less data, and no
|
||
per-shape encoding. If the units are identical, their 3D model can be provisioned
|
||
ahead instead of transmitted.
|
||
|
||
> **Read the code**
|
||
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only
|
||
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
|
||
|
||
## 6. Security as nested layers, not one wall
|
||
|
||
The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear,
|
||
even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct
|
||
trajectories over time, and wireless encryption has been broken by ordinary
|
||
people for two decades. Relying on that layer alone is risky, and pushing heavy
|
||
encryption there is wasteful. So the trust model is [layered](05-network-security.md):
|
||
|
||
1. Wireless transport (WPA class) as the outer shell.
|
||
2. **WireGuard over UDP** as the mesh overlay: proven, peer-authenticated by
|
||
static keys, cheap.
|
||
3. **SSH inside the tunnel** as the application layer, instead of HTTP/HTTPS. SSH
|
||
is one of the most tested remote-access technologies we have. HTTPS would pull
|
||
in a certificate authority and extra parts, and it is the first surface a
|
||
security researcher probes. [Fewer moving parts, on purpose](adr/ADR-0005-wireguard-beneath-ssh.md).
|
||
|
||
Identity and encryption solve together at provisioning time: a per-flight,
|
||
per-unit key pair recorded in a flight registry. Each unit holds its own key pair
|
||
plus the public keys of its peers. A separate key registry for SSH means that if
|
||
one WireGuard key leaks, SSH access does not follow automatically.
|
||
|
||
> **Read the code**
|
||
> - [`infra/ansible/drone-provision.yml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/ansible/drone-provision.yml#L20-L70) — ed25519 identity, peers authorized only through a forced command, WireGuard brought up
|
||
> - [05 — Network & security](05-network-security.md) and [ADR-0005](adr/ADR-0005-wireguard-beneath-ssh.md)
|
||
|
||
## 7. SQL as the contract
|
||
|
||
Beyond the broadcast, people and peers want richer access. The whole progression
|
||
is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier
|
||
schemes. Each time the same pain: hard to debug in the moment, and a lot of
|
||
infrastructure to support, test, and agree on. The simpler conclusion:
|
||
[**SQL is the contract**](adr/ADR-0004-sql-over-ssh-contract.md). Everyone who
|
||
touches data already speaks SQL, and we do not know in advance which slice each
|
||
of them needs, so give them a declarative, extensible way in rather than guessing
|
||
endpoints ahead of time.
|
||
|
||
Concretely: an SSH connection whose forced command runs a read-only **DuckDB**
|
||
query directly. No extra auth handshake between the engine and the data, the way
|
||
a long-running database needs. No new port or protocol. When access control is
|
||
needed, it rides on Linux file permissions, because the data is discrete
|
||
Hive-partitioned files and permissions can be set per file. Not new invention:
|
||
coming back to foundations DevOps and sysadmins already trust.
|
||
|
||
> **Read the code**
|
||
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L43-L53) — the read-only gate that rejects anything that writes
|
||
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
|
||
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first
|
||
|
||
The explorer is running, not just described: the data-plane view (partition tree
|
||
and read-only SQL console) is live at
|
||
[swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)
|
||
(same access as this repository and [swarm.produktor.io](https://swarm.produktor.io/)).
|
||
Browse the Hive layout, run `SELECT` queries against the live lake, and see the
|
||
same statement gate a peer drone would hit in flight.
|
||
|
||
## 8. Replication and offload
|
||
|
||
Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH
|
||
channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock
|
||
[offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently.
|
||
This matters because a unit can be lost, and if it is, we want its data to
|
||
already exist elsewhere. When peers have a stable link the partition tree is
|
||
pulled between them, which gives redundancy and a way to cross-check later. On
|
||
the ground, when the fleet returns, the same path unifies every unit's data into
|
||
a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)).
|
||
The warehouse is read-mostly for analytics, so transaction contention is not a
|
||
concern — plain DuckDB over Parquet with MinIO underneath is the baseline;
|
||
DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md)
|
||
§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md).
|
||
One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a
|
||
unit's data has a gap.
|
||
|
||
> **Read the code**
|
||
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
|
||
> - [06 — Environments](06-environments.md) and [ADR-0002](adr/ADR-0002-one-storage-format.md)
|
||
|
||
## 9. A proof of concept, not slides
|
||
|
||
A concept can be handed over and be wrong. Instead there is a generator: a
|
||
virtual drone producing real sensor data into the Hive layout, a small frontend
|
||
to watch it fill, and a **Grafana** dashboard on top, not to get ahead of the
|
||
analysts, but to see the infrastructure work end to end. Closing the loop
|
||
(Terraform, connector, exporter, dashboard) is how a misconfiguration shows up
|
||
early. A panel that renders wrong points at exactly which layer broke, before an
|
||
analyst has to report that the infra is up but nothing shows. Better to exercise
|
||
every layer now and drop any layer or technology later if there is a sound
|
||
argument against it.
|
||
|
||
Watching that loop is also where the [CPU cost surfaced](adr/ADR-0009-bounded-lake-scans.md):
|
||
scanning the whole lake on every query does not scale, so the scan is
|
||
[bounded to a flight window and cached](07-observability.md).
|
||
|
||
> **Read the code**
|
||
> - [`simulator/monitoring/exporter.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/exporter.py#L38-L107) — the DuckDB scan that turns Parquet into Prometheus metrics
|
||
> - [`simulator/monitoring/lake.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/lake.py#L13-L33) — the flight-window scan that keeps CPU bounded as the lake grows
|
||
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
|
||
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release
|
||
|
||
The dashboard is running, not just described: **Swarm Fleet — generated data** is
|
||
live at
|
||
[swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s)
|
||
(same access as this repository, [swarm.produktor.io](https://swarm.produktor.io/),
|
||
and [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)).
|
||
Telemetry rows, detections, pose frames, battery, and link quality — all derived
|
||
from the Parquet lake via SQL, not from agents on the drones.
|
||
|
||
## 10. Working method
|
||
|
||
A note on how this is meant to be built, not just what. Working together needs a
|
||
shared language, and when the working language is nobody's first language,
|
||
answering live is expensive. The form that scales is written-first: lay the idea
|
||
down carefully, let others read and answer in their own time, keep the history. A
|
||
new colleague, or an agent, can then load the full context of why a decision was
|
||
made by reading the discussion. That is exactly why the decisions here are
|
||
[ADRs](adr/README.md) and why the [contribution flow](../CONTRIBUTING.md) is
|
||
test-first: the reasoning is captured where the next person will look for it.
|
||
|
||
The mental image is a city planner. Draw perfectly straight roads and people
|
||
still cross the grass where it is faster. The job is not to decide for them, but
|
||
to understand the real paths, the pain points and limits of each team, and pave
|
||
those. So constraints get checked with the other DevOps, data engineers, and
|
||
analysts before infrastructure is laid, instead of building roads nobody uses.
|
||
|
||
---
|
||
|
||
The platform is air-gapped and on-prem by design: nothing leaves the swarm, and
|
||
nothing joins it at runtime. Start from the [README](../README.md) for the map,
|
||
or the [ADR index](adr/README.md) for the decisions in full.
|