docs: audit links, glossary T0-T4, readability pass
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped

Extend glossary with storage floors, source of truth, and linkable anchors.
Fix broken links, pose frame drift, rsync/offload wording, and README tone.
Add docs/improvement audit records and live demo links in design journey.
This commit is contained in:
2026-07-10 10:46:59 +01:00
parent ed531f9fb6
commit 5f6fd8462c
27 changed files with 376 additions and 193 deletions
+37 -19
View File
@@ -4,6 +4,7 @@ How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report. The formal decisions, with options and
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
story behind them, and each chapter links straight into the code it produced.
Terms: [Glossary](00-glossary.md).
---
@@ -46,17 +47,17 @@ conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need.
> **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0T4)
## 4. Three tiers inside each unit
## 4. Three pipeline stages inside each unit
That gave [three layers](03-data-platform.md) on every drone:
That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0T4:
1. **Raw capture**, never transformed.
2. **ETL / reduction**, where data is cut down to what has to be shared:
millimetres to centimetres, thinning time series where that is enough,
1. **Raw capture** (pipeline stage 1), never transformed.
2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared:
millimeters to centimeters, thinning time series where that is enough,
dropping what nobody downstream reads.
3. **A bidirectional interface**: on one side it broadcasts position into the
3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the
shared channel, on the other it lets peers pull data.
The ETL is deliberately not over-specified. How sensor data is transformed is a
@@ -65,7 +66,7 @@ stored and in which structure, not to step into that work.
> **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1
## 5. What actually needs to sync
@@ -129,19 +130,28 @@ coming back to foundations DevOps and sysadmins already trust.
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first
The explorer is running, not just described: the data-plane view (partition tree
and read-only SQL console) is live at
[swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)
(same access as this repository and [swarm.produktor.io](https://swarm.produktor.io/)).
Browse the Hive layout, run `SELECT` queries against the live lake, and see the
same statement gate a peer drone would hit in flight.
## 8. Replication and offload
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
channel. It transfers diffs of the Hive tree efficiently. This matters because a
unit can be lost, and if it is, we want its data to already exist elsewhere.
When peers have a stable link the partition tree is pulled between them, which
gives redundancy and a way to cross-check later. On the ground, when the fleet
returns, the same path unifies every unit's data into a
[local warehouse](06-environments.md). The warehouse is read-mostly for
analytics, so transaction contention is not a concern, which opens a
DuckDB / DuckLake approach with MinIO underneath, the
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
uniform structure is what lets an analyst trust a single picture even when a
Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH
channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock
[offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently.
This matters because a unit can be lost, and if it is, we want its data to
already exist elsewhere. When peers have a stable link the partition tree is
pulled between them, which gives redundancy and a way to cross-check later. On
the ground, when the fleet returns, the same path unifies every unit's data into
a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)).
The warehouse is read-mostly for analytics, so transaction contention is not a
concern — plain DuckDB over Parquet with MinIO underneath is the baseline;
DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md)
§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md).
One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a
unit's data has a gap.
> **Read the code**
@@ -170,6 +180,14 @@ scanning the whole lake on every query does not scale, so the scan is
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release
The dashboard is running, not just described: **Swarm Fleet — generated data** is
live at
[swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s)
(same access as this repository, [swarm.produktor.io](https://swarm.produktor.io/),
and [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)).
Telemetry rows, detections, pose frames, battery, and link quality — all derived
from the Parquet lake via SQL, not from agents on the drones.
## 10. Working method
A note on how this is meant to be built, not just what. Working together needs a