3 Commits
Author SHA1 Message Date
eSlider 336f9c0428 Add license
CI & Release / Verify simulator (push) Successful in 11s
CI & Release / Trivy scan (push) Successful in 8s
CI & Release / Semantic Release (push) Successful in 5s
2026-07-09 15:52:16 +01:00
eSlider 3d13f744e6 docs(journey): link the live visual prototype at swarm.produktor.io
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
2026-07-09 12:47:12 +01:00
eSlider dfb0556eda docs: add design journey, a narrative walk-through linking into the code
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
A chronological companion to the ADRs: how the design came together, from
the first mental model and 2D prototype through the three data tiers, the
relative-pose broadcast, the layered WireGuard/SSH trust model, SQL as the
peer contract, offload and replication, and the end-to-end proof of concept.
Each chapter links straight into the relevant source and decision records.
Flagged as the recommended first read in the README.
2026-07-09 12:30:40 +01:00
3 changed files with 215 additions and 1 deletions
+19
View File
@@ -0,0 +1,19 @@
PROPRIETARY LICENSE
Copyright (c) 2026 Andriy Oblivantsev. All rights reserved.
This material is proprietary and confidential.
No person or organization may, without prior written permission from the copyright holder:
- Publish or republish this material, in whole or in part.
- Copy, reproduce, distribute, transmit, forward, or share this material by any means.
- Use, modify, adapt, translate, create derivative works from, or incorporate this material into any other work.
- Make this material available to any third party, whether publicly or privately.
- Store this material in any repository, archive, or knowledge base accessible to others.
Permission must be obtained in writing before any of the above activities are undertaken.
Unauthorized use, publication, forwarding, reproduction, distribution, or disclosure of this material is strictly prohibited and may result in civil and criminal penalties under applicable copyright and intellectual property laws.
This license grants no rights except the right to view the material for its intended purpose by an authorized recipient.
+2 -1
View File
@@ -33,9 +33,10 @@ This repository describes how to build, deliver, test, and operate that platform
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers | | [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers |
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on | | [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases | | [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code |
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made | | [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record. New here? Read the [design journey](docs/12-design-journey.md) first: it tells the story chronologically and links straight into the code and decisions. The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record.
## Runnable parts ## Runnable parts
+194
View File
@@ -0,0 +1,194 @@
# 12 — Design journey
How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report. The formal decisions, with options and
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
story behind them, and each chapter links straight into the code it produced.
---
## 1. Starting point
At the start there was only a rough brief and a few adjacent interests to lean
on. Key rotation matters in every pipeline: a cipher that is safe today may not
be safe next year, so the infrastructure has to leave room to react, without
spending too much latency or bandwidth to do it. That balance, security against
speed and volume, is a recurring trade-off. Geospatial data is older ground:
routing, indexing, fast spatial lookups at scale. So the first instinct was that
the value might be less about new storage and more about optimizing access to
data that already exists.
## 2. From words to a model, then to a picture
Once the shape was clear, an autonomous swarm where each unit talks to any peer
in range and [no controller spans the swarm](adr/ADR-0001-no-swarm-orchestrator.md),
the next step was to make the model visible. A small 2D prototype was enough to
check it: units patrolling, static and moving obstacles, links forming and
breaking, and a rough count of how much data crosses the air. Speed was not the
question yet, only volume, because volume is what you size the infrastructure
for. Starting from a picture gives an anchor before building anything.
> **Read the code**
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L219-L287) — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L57-L63) — the pose-frame size and rate that drive the volume estimate
The picture is running, not just described: the visual prototype is live at
[swarm.produktor.io](https://swarm.produktor.io/) (same access as this
repository). It draws units patrolling, links forming and breaking by range, and
a running estimate of how much data crosses the air.
## 3. Naming the limits
Before any code, the limits had to be named. Maximum distance between units in
the air. The fidelity data is stored at. The fidelity it can be transmitted at.
Those drive volume, and volume drives the whole shape of the system. The
conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need.
> **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
## 4. Three tiers inside each unit
That gave [three layers](03-data-platform.md) on every drone:
1. **Raw capture**, never transformed.
2. **ETL / reduction**, where data is cut down to what has to be shared:
millimetres to centimetres, thinning time series where that is enough,
dropping what nobody downstream reads.
3. **A bidirectional interface**: on one side it broadcasts position into the
shared channel, on the other it lets peers pull data.
The ETL is deliberately not over-specified. How sensor data is transformed is a
data-engineering decision. The platform's job is the contract, meaning what gets
stored and in which structure, not to step into that work.
> **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
## 5. What actually needs to sync
The most time-critical item is where each peer is, so every unit has time to
react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and
time) instead of absolute coordinates. Relative is cheaper and enough for
coordination. To avoid sending orientation, a unit is modelled as a **sphere**
that bounds its extent. That trades a little compute for much less data, and no
per-shape encoding. If the units are identical, their 3D model can be provisioned
ahead instead of transmitted.
> **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
## 6. Security as nested layers, not one wall
The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear,
even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct
trajectories over time, and wireless encryption has been broken by ordinary
people for two decades. Relying on that layer alone is risky, and pushing heavy
encryption there is wasteful. So the trust model is [layered](05-network-security.md):
1. Wireless transport (WPA class) as the outer shell.
2. **WireGuard over UDP** as the mesh overlay: proven, peer-authenticated by
static keys, cheap.
3. **SSH inside the tunnel** as the application layer, instead of HTTP/HTTPS. SSH
is one of the most tested remote-access technologies we have. HTTPS would pull
in a certificate authority and extra parts, and it is the first surface a
security researcher probes. [Fewer moving parts, on purpose](adr/ADR-0005-wireguard-beneath-ssh.md).
Identity and encryption solve together at provisioning time: a per-flight,
per-unit key pair recorded in a flight registry. Each unit holds its own key pair
plus the public keys of its peers. A separate key registry for SSH means that if
one WireGuard key leaks, SSH access does not follow automatically.
> **Read the code**
> - [`infra/ansible/drone-provision.yml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/ansible/drone-provision.yml#L20-L70) — ed25519 identity, peers authorized only through a forced command, WireGuard brought up
> - [05 — Network & security](05-network-security.md) and [ADR-0005](adr/ADR-0005-wireguard-beneath-ssh.md)
## 7. SQL as the contract
Beyond the broadcast, people and peers want richer access. The whole progression
is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier
schemes. Each time the same pain: hard to debug in the moment, and a lot of
infrastructure to support, test, and agree on. The simpler conclusion:
[**SQL is the contract**](adr/ADR-0004-sql-over-ssh-contract.md). Everyone who
touches data already speaks SQL, and we do not know in advance which slice each
of them needs, so give them a declarative, extensible way in rather than guessing
endpoints ahead of time.
Concretely: an SSH connection whose forced command runs a read-only **DuckDB**
query directly. No extra auth handshake between the engine and the data, the way
a long-running database needs. No new port or protocol. When access control is
needed, it rides on Linux file permissions, because the data is discrete
Hive-partitioned files and permissions can be set per file. Not new invention:
coming back to foundations DevOps and sysadmins already trust.
> **Read the code**
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L43-L53) — the read-only gate that rejects anything that writes
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first
## 8. Replication and offload
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
channel. It transfers diffs of the Hive tree efficiently. This matters because a
unit can be lost, and if it is, we want its data to already exist elsewhere.
When peers have a stable link the partition tree is pulled between them, which
gives redundancy and a way to cross-check later. On the ground, when the fleet
returns, the same path unifies every unit's data into a
[local warehouse](06-environments.md). The warehouse is read-mostly for
analytics, so transaction contention is not a concern, which opens a
DuckDB / DuckLake approach with MinIO underneath, the
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
uniform structure is what lets an analyst trust a single picture even when a
unit's data has a gap.
> **Read the code**
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
> - [06 — Environments](06-environments.md) and [ADR-0002](adr/ADR-0002-one-storage-format.md)
## 9. A proof of concept, not slides
A concept can be handed over and be wrong. Instead there is a generator: a
virtual drone producing real sensor data into the Hive layout, a small frontend
to watch it fill, and a **Grafana** dashboard on top, not to get ahead of the
analysts, but to see the infrastructure work end to end. Closing the loop
(Terraform, connector, exporter, dashboard) is how a misconfiguration shows up
early. A panel that renders wrong points at exactly which layer broke, before an
analyst has to report that the infra is up but nothing shows. Better to exercise
every layer now and drop any layer or technology later if there is a sound
argument against it.
Watching that loop is also where the [CPU cost surfaced](adr/ADR-0009-bounded-lake-scans.md):
scanning the whole lake on every query does not scale, so the scan is
[bounded to a flight window and cached](07-observability.md).
> **Read the code**
> - [`simulator/monitoring/exporter.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/exporter.py#L38-L107) — the DuckDB scan that turns Parquet into Prometheus metrics
> - [`simulator/monitoring/lake.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/lake.py#L13-L33) — the flight-window scan that keeps CPU bounded as the lake grows
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release
## 10. Working method
A note on how this is meant to be built, not just what. Working together needs a
shared language, and when the working language is nobody's first language,
answering live is expensive. The form that scales is written-first: lay the idea
down carefully, let others read and answer in their own time, keep the history. A
new colleague, or an agent, can then load the full context of why a decision was
made by reading the discussion. That is exactly why the decisions here are
[ADRs](adr/README.md) and why the [contribution flow](../CONTRIBUTING.md) is
test-first: the reasoning is captured where the next person will look for it.
The mental image is a city planner. Draw perfectly straight roads and people
still cross the grass where it is faster. The job is not to decide for them, but
to understand the real paths, the pain points and limits of each team, and pave
those. So constraints get checked with the other DevOps, data engineers, and
analysts before infrastructure is laid, instead of building roads nobody uses.
---
The platform is air-gapped and on-prem by design: nothing leaves the swarm, and
nothing joins it at runtime. Start from the [README](../README.md) for the map,
or the [ADR index](adr/README.md) for the decisions in full.