5 Commits
Author SHA1 Message Date
eSlider 2a6b50929c docs: align wire-spec and mark implemented vs proposed
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 18s
CI & Release / Semantic Release (push) Successful in 6s
Keep the 45-byte pose frame consistent across docs, simulator, and
prototype, clarify MinIO is not peer sync, and state clearly that the
repo is a from-scratch platform sketch with options rather than mandates.
2026-07-17 12:16:39 +01:00
eSlider ed531f9fb6 Link license to readme.md
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
2026-07-09 15:55:04 +01:00
eSlider 336f9c0428 Add license
CI & Release / Verify simulator (push) Successful in 11s
CI & Release / Trivy scan (push) Successful in 8s
CI & Release / Semantic Release (push) Successful in 5s
2026-07-09 15:52:16 +01:00
eSlider 3d13f744e6 docs(journey): link the live visual prototype at swarm.produktor.io
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
2026-07-09 12:47:12 +01:00
eSlider dfb0556eda docs: add design journey, a narrative walk-through linking into the code
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
A chronological companion to the ADRs: how the design came together, from
the first mental model and 2D prototype through the three data tiers, the
relative-pose broadcast, the layered WireGuard/SSH trust model, SQL as the
peer contract, offload and replication, and the end-to-end proof of concept.
Each chapter links straight into the relevant source and decision records.
Flagged as the recommended first read in the README.
2026-07-09 12:30:40 +01:00
10 changed files with 292 additions and 33 deletions
+19
View File
@@ -0,0 +1,19 @@
PROPRIETARY LICENSE
Copyright (c) 2026 Andriy Oblivantsev. All rights reserved.
This material is proprietary and confidential.
No person or organization may, without prior written permission from the copyright holder:
- Publish or republish this material, in whole or in part.
- Copy, reproduce, distribute, transmit, forward, or share this material by any means.
- Use, modify, adapt, translate, create derivative works from, or incorporate this material into any other work.
- Make this material available to any third party, whether publicly or privately.
- Store this material in any repository, archive, or knowledge base accessible to others.
Permission must be obtained in writing before any of the above activities are undertaken.
Unauthorized use, publication, forwarding, reproduction, distribution, or disclosure of this material is strictly prohibited and may result in civil and criminal penalties under applicable copyright and intellectual property laws.
This license grants no rights except the right to view the material for its intended purpose by an authorized recipient.
+23 -1
View File
@@ -17,6 +17,25 @@ This repository describes how to build, deliver, test, and operate that platform
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
## Implemented now vs proposed next
Honest map so a reader knows what runs today versus what is design intent.
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
| --- | --- | --- |
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, optional MinIO | k3s warehouse, GitOps overlays, post-flight mirror |
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
## Documentation
| Document | Contents |
@@ -33,9 +52,10 @@ This repository describes how to build, deliver, test, and operate that platform
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers |
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code |
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record.
New here? Read the [design journey](docs/12-design-journey.md) first: it tells the story chronologically and links straight into the code and decisions. The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record.
## Runnable parts
@@ -76,3 +96,5 @@ Contributing (TDD-first workflow): [`CONTRIBUTING.md`](CONTRIBUTING.md).
## CI & releases
Every push runs the pipeline in [`.github/workflows/release.yaml`](.github/workflows/release.yaml): **pytest unit tests**, a compile check, a 10-second virtual smoke flight verified with DuckDB, and a Trivy vulnerability/misconfiguration scan. The simulator Docker build also runs `pytest` — a failing test blocks the image. Pushes to `main` then cut a semantic version from the conventional-commit type (`feat:` → minor, `!`/`BREAKING CHANGE` → major, else patch), tag the commit, and publish a release with a **fleet release manifest** (versioned artifact list with checksums — the delivery unit described in [11 — CI/CD & delivery](docs/11-cicd-delivery.md)) plus a source bundle.
[LICENSE](LICENSE)
+11 -8
View File
@@ -20,8 +20,9 @@ graph TB
subgraph serving [Layer 3 — Serving and sync]
HOOK["event hook<br/>fires on new derived data"]
PUB["state publisher<br/>pub/sub broadcast"]
MINIO["MinIO<br/>derived datasets bucket"]
QAPI["query API<br/>peer data requests"]
BULK["bulk sync<br/>rsync over SSH"]
MINIO["MinIO<br/>optional derived bucket"]
QAPI["query API<br/>SQL-over-SSH"]
end
end
@@ -33,12 +34,14 @@ graph TB
WRITER -->|derived rows| HOOK
HOOK --> PUB
HOOK --> MINIO
HOOK --> BULK
QAPI --> DUCK
PUB -.->|mesh| PEERS["peer drones"]
MINIO -.->|replication| PEERS
PUB -.->|mesh pose| PEERS["peer drones"]
BULK -.->|sealed partitions| PEERS
QAPI -.->|on demand| PEERS
```
MinIO stays available where an S3 API helps (on-board derived datasets, ground warehouse). **In-flight peer bulk sync is SSH/rsync**, not object-store replication — see [04 — Swarm sync](04-swarm-sync.md).
### Layer 1 — Ingestion
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
@@ -53,10 +56,10 @@ graph TB
### Layer 3 — Serving and sync
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer); MinIO remains optional where an S3 API is wanted.
- **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)).
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, optional MinIO put, bulk-sync hint, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (UDP in the PoC; Zenoh proposed — [04](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer). MinIO is optional where an S3 API is wanted; it is **not** the in-flight peer replication path.
- **Peer queries** are read-only DuckDB SQL — HTTP explorer gate in the PoC; **SSH forced commands** proposed for flight (SELECT-only gate, read-only OS user, columnar responses [04](04-swarm-sync.md)).
## Communication planes
+23 -13
View File
@@ -16,19 +16,27 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex
## Broadcast payload: small on the wire, precise at rest
The `state` broadcast is a fixed compact frame:
The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
what the prototype uses for bandwidth estimates.
| Field | Type | Notes |
| Field | Wire type (PoC) | Notes |
| --- | --- | --- |
| `drone_id` | uint16 | Fleet-scoped registry |
| `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` |
| `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap |
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision |
| `att_roll/pitch/yaw` | int16 | Centi-degrees |
| `vel_x/y/z` | int16 | cm/s |
| `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision |
| `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model |
| `vel_x/y/z` | 3 × int16 | cm/s |
| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.
45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.
An earlier sketch used relative coordinates and a bounding sphere (no attitude).
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
costs almost nothing at this frame size. Relative localization remains a
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
@@ -51,25 +59,27 @@ graph LR
SB -->|"store-and-forward relay"| SC
```
### Recommended: Zenoh
### Recommended (proposal): Zenoh
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
- Tiny footprint, ARM64-native.
**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.
### Alternatives considered
| Option | Verdict |
| --- | --- |
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode |
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below |
| **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode |
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path; optional on board for derived datasets and primary on the ground warehouse — not the in-flight bulk path |
## Two sync mechanisms, deliberately separate
1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.
Why SSH-based bulk sync over object-store replication:
@@ -105,7 +115,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
| Layer | Mechanism | What it stops |
| --- | --- | --- |
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
| 2. Statement gate | Wrapper accepts a single statement, parses it, rejects anything but `SELECT` (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); parameters bound, not interpolated | SQL-as-a-write-channel, config tampering |
| 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering |
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
+199
View File
@@ -0,0 +1,199 @@
# 12 — Design journey
How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report — and **not a mandate**. The goal is a
from-scratch platform sketch that shows how the pieces fit; every concrete
choice is an option with trade-offs recorded as
[Architecture Decision Records](adr/README.md). Replace any piece if a better
fit shows up. This is the story behind the decisions, and each chapter links
straight into the code it produced.
---
## 1. Starting point
At the start there was only a rough brief and a few adjacent interests to lean
on. Key rotation matters in every pipeline: a cipher that is safe today may not
be safe next year, so the infrastructure has to leave room to react, without
spending too much latency or bandwidth to do it. That balance, security against
speed and volume, is a recurring trade-off. Geospatial data is older ground:
routing, indexing, fast spatial lookups at scale. So the first instinct was that
the value might be less about new storage and more about optimizing access to
data that already exists.
## 2. From words to a model, then to a picture
Once the shape was clear, an autonomous swarm where each unit talks to any peer
in range and [no controller spans the swarm](adr/ADR-0001-no-swarm-orchestrator.md),
the next step was to make the model visible. A small 2D prototype was enough to
check it: units patrolling, static and moving obstacles, links forming and
breaking, and a rough count of how much data crosses the air. Speed was not the
question yet, only volume, because volume is what you size the infrastructure
for. Starting from a picture gives an anchor before building anything.
> **Read the code**
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L219-L287) — the tick: links form and break by range, pose broadcasts and bulk sync accumulate bytes
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L57-L63) — the pose-frame size and rate that drive the volume estimate
The picture is running, not just described: the visual prototype is live at
[swarm.produktor.io](https://swarm.produktor.io/) (same access as this
repository). It draws units patrolling, links forming and breaking by range, and
a running estimate of how much data crosses the air.
## 3. Naming the limits
Before any code, the limits had to be named. Maximum distance between units in
the air. The fidelity data is stored at. The fidelity it can be transmitted at.
Those drive volume, and volume drives the whole shape of the system. The
conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need.
> **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
## 4. Three tiers inside each unit
That gave [three layers](03-data-platform.md) on every drone:
1. **Raw capture**, never transformed.
2. **ETL / reduction**, where data is cut down to what has to be shared:
millimetres to centimetres, thinning time series where that is enough,
dropping what nobody downstream reads.
3. **A bidirectional interface**: on one side it broadcasts position into the
shared channel, on the other it lets peers pull data.
The ETL is deliberately not over-specified. How sensor data is transformed is a
data-engineering decision. The platform's job is the contract, meaning what gets
stored and in which structure, not to step into that work.
> **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
## 5. What actually needs to sync
The most time-critical item is where each peer is, so every unit has time to
react. The wire frame carries **mission-frame position** (millimetres on the
wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags —
45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the
fusion pain. An earlier sketch used relative coordinates and a bounding sphere
with no orientation; it was dropped. When peers disagree on frames,
`frame_ref` makes the mismatch explicit for consumers
([09 — Open questions](09-open-questions.md)).
> **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
## 6. Security as nested layers, not one wall
The assumption was an ad-hoc wireless mesh. Broadcasting positions in the clear,
even over encrypted Wi-Fi, is not acceptable: anyone who joins can reconstruct
trajectories over time, and wireless encryption has been broken by ordinary
people for two decades. Relying on that layer alone is risky, and pushing heavy
encryption there is wasteful. So the trust model is [layered](05-network-security.md):
1. Wireless transport (WPA class) as the outer shell.
2. **WireGuard over UDP** as the mesh overlay: proven, peer-authenticated by
static keys, cheap.
3. **SSH inside the tunnel** as the application layer, instead of HTTP/HTTPS. SSH
is one of the most tested remote-access technologies we have. HTTPS would pull
in a certificate authority and extra parts, and it is the first surface a
security researcher probes. [Fewer moving parts, on purpose](adr/ADR-0005-wireguard-beneath-ssh.md).
Identity and encryption solve together at provisioning time: a per-flight,
per-unit key pair recorded in a flight registry. Each unit holds its own key pair
plus the public keys of its peers. A separate key registry for SSH means that if
one WireGuard key leaks, SSH access does not follow automatically.
> **Read the code**
> - [`infra/ansible/drone-provision.yml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/ansible/drone-provision.yml#L20-L70) — ed25519 identity, peers authorized only through a forced command, WireGuard brought up
> - [05 — Network & security](05-network-security.md) and [ADR-0005](adr/ADR-0005-wireguard-beneath-ssh.md)
## 7. SQL as the contract
Beyond the broadcast, people and peers want richer access. The whole progression
is familiar: fully dynamic APIs, then JSON, OpenAPI/Swagger, GraphQL, then heavier
schemes. Each time the same pain: hard to debug in the moment, and a lot of
infrastructure to support, test, and agree on. The simpler conclusion:
[**SQL is the contract**](adr/ADR-0004-sql-over-ssh-contract.md). Everyone who
touches data already speaks SQL, and we do not know in advance which slice each
of them needs, so give them a declarative, extensible way in rather than guessing
endpoints ahead of time.
Concretely: an SSH connection whose forced command runs a read-only **DuckDB**
query directly. No extra auth handshake between the engine and the data, the way
a long-running database needs. No new port or protocol. When access control is
needed, it rides on Linux file permissions, because the data is discrete
Hive-partitioned files and permissions can be set per file. Not new invention:
coming back to foundations DevOps and sysadmins already trust.
> **Read the code**
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L43-L53) — the read-only gate that rejects anything that writes
> - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically
> - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first
## 8. Replication and offload
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
channel. It transfers diffs of the Hive tree efficiently. This matters because a
unit can be lost, and if it is, we want its data to already exist elsewhere.
When peers have a stable link the partition tree is pulled between them, which
gives redundancy and a way to cross-check later. On the ground, when the fleet
returns, the same path unifies every unit's data into a
[local warehouse](06-environments.md). The warehouse is read-mostly for
analytics, so transaction contention is not a concern, which opens a
DuckDB / DuckLake approach with MinIO underneath, the
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
uniform structure is what lets an analyst trust a single picture even when a
unit's data has a gap.
> **Read the code**
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
> - [06 — Environments](06-environments.md) and [ADR-0002](adr/ADR-0002-one-storage-format.md)
## 9. A proof of concept, not slides
A concept can be handed over and be wrong. Instead there is a generator: a
virtual drone producing real sensor data into the Hive layout, a small frontend
to watch it fill, and a **Grafana** dashboard on top, not to get ahead of the
analysts, but to see the infrastructure work end to end. Closing the loop
(Terraform, connector, exporter, dashboard) is how a misconfiguration shows up
early. A panel that renders wrong points at exactly which layer broke, before an
analyst has to report that the infra is up but nothing shows. Better to exercise
every layer now and drop any layer or technology later if there is a sound
argument against it.
Watching that loop is also where the [CPU cost surfaced](adr/ADR-0009-bounded-lake-scans.md):
scanning the whole lake on every query does not scale, so the scan is
[bounded to a flight window and cached](07-observability.md).
> **Read the code**
> - [`simulator/monitoring/exporter.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/exporter.py#L38-L107) — the DuckDB scan that turns Parquet into Prometheus metrics
> - [`simulator/monitoring/lake.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/lake.py#L13-L33) — the flight-window scan that keeps CPU bounded as the lake grows
> - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards
> - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release
## 10. Working method
A note on how this is meant to be built, not just what. Working together needs a
shared language, and when the working language is nobody's first language,
answering live is expensive. The form that scales is written-first: lay the idea
down carefully, let others read and answer in their own time, keep the history. A
new colleague, or an agent, can then load the full context of why a decision was
made by reading the discussion. That is exactly why the decisions here are
[ADRs](adr/README.md) and why the [contribution flow](../CONTRIBUTING.md) is
test-first: the reasoning is captured where the next person will look for it.
The mental image is a city planner. Draw perfectly straight roads and people
still cross the grass where it is faster. The job is not to decide for them, but
to understand the real paths, the pain points and limits of each team, and pave
those. So constraints get checked with the other DevOps, data engineers, and
analysts before infrastructure is laid, instead of building roads nobody uses.
---
The platform is air-gapped and on-prem by design: nothing leaves the swarm, and
nothing joins it at runtime. Start from the [README](../README.md) for the map,
or the [ADR index](adr/README.md) for the decisions in full.
+10 -5
View File
@@ -32,10 +32,15 @@ wrapper, permissions, and output format a peer drone would use.
## Consequences
- One access path is built, secured, and tested — "what you test is what
- One access path is designed for flight and bench — "what you test is what
flies".
- The SQL gate is safety-critical and is covered by unit tests
(`simulator/tests/test_sql_gate.py`).
- The explorer and the prototype's live mode are *just another read-only
consumer* of this same contract ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- The statement gate in the PoC (`simulator/explorer/server.py`, covered by
`simulator/tests/test_sql_gate.py`) is a **first layer**: keyword allow/deny
over HTTP for the explorer. Production still needs the remaining layers in
[04 — Swarm sync](../04-swarm-sync.md) (forced-command key, read-only OS user,
engine hardening, resource caps) and a real SQL parser rather than keywords
alone.
- The explorer and the prototype's live mode are *stand-in consumers* of the
same read-only contract over HTTP today; the proposed flight path is
SQL-over-SSH ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
+1 -1
View File
@@ -268,7 +268,7 @@ export default function App(): JSX.Element {
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
</Section>
<Section title="Legend">
<div style={styles.dim}>blue line pose broadcast (5 Hz, ~46 B)</div>
<div style={styles.dim}>blue line pose broadcast (5 Hz, 45 B)</div>
<div style={styles.dim}>green line bulk sync (sealed partitions)</div>
<div style={styles.dim}>flash broadcast event delivered</div>
<div style={styles.dim}>red circle transit object crossing the area</div>
+1 -1
View File
@@ -57,7 +57,7 @@ export function areaWidth(): number {
export const LINK_RANGE = 320;
const CRUISE = 28;
const SEPARATION = 55;
const POSE_BYTES = 46;
const POSE_BYTES = 45;
const POSE_HZ = 5;
// Steady per-link broadcast throughput (both directions), bytes/s
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
+1 -1
View File
@@ -9,7 +9,7 @@ from virtual_drone.flight import Pose
def test_frame_size_matches_wire_spec() -> None:
# Documented as 46 B in docs/04; struct packs to FRAME.size (45 B on this layout).
# Keep docs/04, the journey, and prototype/src/sim.ts POSE_BYTES in lockstep.
assert FRAME.size == 45
+4 -3
View File
@@ -1,12 +1,13 @@
"""State broadcast over UDP: the compact pose frame from the sync design.
Frame layout (little-endian, 46 bytes):
Frame layout (little-endian, 45 bytes) — source of truth for docs and the
prototype bandwidth estimate:
magic 2s b"SH"
version B
drone_id 8s zero-padded ascii
drone_id 8s zero-padded ascii (PoC; production may switch to uint16 registry)
ts_ns q epoch nanoseconds
pos_mm 3i position, millimeters (quantized on the wire only)
pos_mm 3i position in the mission frame, millimeters (wire quantization only)
att_cdeg 3h roll/pitch/yaw, centi-degrees
vel_cms 3h velocity, cm/s
frame_ref B