docs: audit links, glossary T0-T4, readability pass
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
This commit is contained in:
+37
-19
@@ -4,6 +4,7 @@ How this design came together, told in the order the thinking actually happened.
|
||||
It is a walk-through, not a report. The formal decisions, with options and
|
||||
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
|
||||
story behind them, and each chapter links straight into the code it produced.
|
||||
Terms: [Glossary](00-glossary.md).
|
||||
|
||||
---
|
||||
|
||||
@@ -46,17 +47,17 @@ conclusion: constrain the transport, but do not constrain on-board capture. Keep
|
||||
everything locally. Send only what peers truly need.
|
||||
|
||||
> **Read more**
|
||||
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
|
||||
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0–T4)
|
||||
|
||||
## 4. Three tiers inside each unit
|
||||
## 4. Three pipeline stages inside each unit
|
||||
|
||||
That gave [three layers](03-data-platform.md) on every drone:
|
||||
That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0–T4:
|
||||
|
||||
1. **Raw capture**, never transformed.
|
||||
2. **ETL / reduction**, where data is cut down to what has to be shared:
|
||||
millimetres to centimetres, thinning time series where that is enough,
|
||||
1. **Raw capture** (pipeline stage 1), never transformed.
|
||||
2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared:
|
||||
millimeters to centimeters, thinning time series where that is enough,
|
||||
dropping what nobody downstream reads.
|
||||
3. **A bidirectional interface**: on one side it broadcasts position into the
|
||||
3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the
|
||||
shared channel, on the other it lets peers pull data.
|
||||
|
||||
The ETL is deliberately not over-specified. How sensor data is transformed is a
|
||||
@@ -65,7 +66,7 @@ stored and in which structure, not to step into that work.
|
||||
|
||||
> **Read the code**
|
||||
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
|
||||
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
|
||||
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1
|
||||
|
||||
## 5. What actually needs to sync
|
||||
|
||||
@@ -129,19 +130,28 @@ coming back to foundations DevOps and sysadmins already trust.
|
||||
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
|
||||
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first
|
||||
|
||||
The explorer is running, not just described: the data-plane view (partition tree
|
||||
and read-only SQL console) is live at
|
||||
[swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)
|
||||
(same access as this repository and [swarm.produktor.io](https://swarm.produktor.io/)).
|
||||
Browse the Hive layout, run `SELECT` queries against the live lake, and see the
|
||||
same statement gate a peer drone would hit in flight.
|
||||
|
||||
## 8. Replication and offload
|
||||
|
||||
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
|
||||
channel. It transfers diffs of the Hive tree efficiently. This matters because a
|
||||
unit can be lost, and if it is, we want its data to already exist elsewhere.
|
||||
When peers have a stable link the partition tree is pulled between them, which
|
||||
gives redundancy and a way to cross-check later. On the ground, when the fleet
|
||||
returns, the same path unifies every unit's data into a
|
||||
[local warehouse](06-environments.md). The warehouse is read-mostly for
|
||||
analytics, so transaction contention is not a concern, which opens a
|
||||
DuckDB / DuckLake approach with MinIO underneath, the
|
||||
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
|
||||
uniform structure is what lets an analyst trust a single picture even when a
|
||||
Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH
|
||||
channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock
|
||||
[offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently.
|
||||
This matters because a unit can be lost, and if it is, we want its data to
|
||||
already exist elsewhere. When peers have a stable link the partition tree is
|
||||
pulled between them, which gives redundancy and a way to cross-check later. On
|
||||
the ground, when the fleet returns, the same path unifies every unit's data into
|
||||
a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)).
|
||||
The warehouse is read-mostly for analytics, so transaction contention is not a
|
||||
concern — plain DuckDB over Parquet with MinIO underneath is the baseline;
|
||||
DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md)
|
||||
§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md).
|
||||
One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a
|
||||
unit's data has a gap.
|
||||
|
||||
> **Read the code**
|
||||
@@ -170,6 +180,14 @@ scanning the whole lake on every query does not scale, so the scan is
|
||||
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
|
||||
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release
|
||||
|
||||
The dashboard is running, not just described: **Swarm Fleet — generated data** is
|
||||
live at
|
||||
[swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s)
|
||||
(same access as this repository, [swarm.produktor.io](https://swarm.produktor.io/),
|
||||
and [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)).
|
||||
Telemetry rows, detections, pose frames, battery, and link quality — all derived
|
||||
from the Parquet lake via SQL, not from agents on the drones.
|
||||
|
||||
## 10. Working method
|
||||
|
||||
A note on how this is meant to be built, not just what. Working together needs a
|
||||
|
||||
Reference in New Issue
Block a user