From 5f6fd8462cc9486a074e2a850de5d40e4398165f Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:46:59 +0100 Subject: [PATCH 1/9] docs: audit links, glossary T0-T4, readability pass Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey. --- README.md | 17 +++-- docs/00-glossary.md | 68 +++++++++++++++++- docs/01-problem-statement.md | 6 +- docs/02-architecture.md | 12 ++-- docs/03-data-platform.md | 28 ++++---- docs/04-swarm-sync.md | 24 ++++--- docs/05-network-security.md | 8 +-- docs/06-environments.md | 18 ++--- docs/07-observability.md | 4 +- docs/08-roadmap.md | 2 +- docs/09-open-questions.md | 24 ++++++- docs/10-domain-context.md | 12 ++-- docs/11-cicd-delivery.md | 2 +- docs/12-design-journey.md | 56 ++++++++++----- docs/adr/ADR-0001-no-swarm-orchestrator.md | 2 +- docs/adr/ADR-0002-one-storage-format.md | 10 ++- docs/adr/ADR-0003-sync-derived-state-only.md | 2 +- docs/adr/README.md | 6 +- docs/improvement/2026-07-10-audit.md | 74 ++++++++++++++++++++ docs/improvement/README.md | 18 +++++ infra/ansible/README.md | 27 +++---- infra/gitops/README.md | 28 +++----- infra/terraform/README.md | 13 +++- infra/terraform/ground/README.md | 30 +++----- infra/terraform/sim-env/README.md | 30 +++----- prototype/README.md | 19 +++-- simulator/README.md | 29 ++++---- 27 files changed, 376 insertions(+), 193 deletions(-) create mode 100644 docs/improvement/2026-07-10-audit.md create mode 100644 docs/improvement/README.md diff --git a/README.md b/README.md index 7ca6989..8db51b1 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,14 @@ **An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective. -A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale. +Fleet flies autonomous. No internet uplink, no external access. Data stays in the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. Swarm exchanges just enough state to coordinate; everything else stays local until a ground warehouse receives it after landing — reproducibly, securely, at scale. -This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware. +This repository covers storage layout, sync strategy, network trust model, CI/CD chain, and the simulation environment that tests it without real hardware. + +> **Restricted use.** Third parties may not use, copy, modify, or distribute this +> source code, documentation, or related materials without **explicit written +> permission** from the copyright holder. No license is granted by default. See +> [LICENSE](LICENSE) for the full terms. ## Design principles @@ -24,7 +29,7 @@ This repository describes how to build, deliver, test, and operate that platform | [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout | | [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions | | [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes | -| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path | +| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage floors (T0–T4), partitioning, hot-write path | | [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard | | [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport | | [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures | @@ -36,7 +41,7 @@ This repository describes how to build, deliver, test, and operate that platform | [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code | | [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made | -New here? Read the [design journey](docs/12-design-journey.md) first: it tells the story chronologically and links straight into the code and decisions. The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record. +New? Start with the [design journey](docs/12-design-journey.md) — story plus code links. Principles = *what*; [ADRs](docs/adr/) = *why and when*. ## Runnable parts @@ -45,10 +50,10 @@ New here? Read the [design journey](docs/12-design-journey.md) first: it tells t | [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB | | [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG | | [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible | -| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + T1→T3 offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider | +| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + [T1→T3](docs/03-data-platform.md#storage-floors-tiers) offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider | | [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize | -Shared runtime data (gitignored): `var/t1/` (live lake), `var/t3/` (warehouse). See [`CONTRIBUTING.md`](CONTRIBUTING.md). +Shared runtime data (gitignored): [`var/t1/`](docs/00-glossary.md#t1--warm) (live lake, [T1](docs/00-glossary.md#t1--warm)), [`var/t3/`](docs/00-glossary.md#t3--warehouse) (warehouse, [T3](docs/00-glossary.md#t3--warehouse)). See [`CONTRIBUTING.md`](CONTRIBUTING.md). ## Quick start diff --git a/docs/00-glossary.md b/docs/00-glossary.md index e7fc14b..25cbdaa 100644 --- a/docs/00-glossary.md +++ b/docs/00-glossary.md @@ -2,6 +2,70 @@ Terms and abbreviations used throughout this proposal. +**Vocabulary guide:** **architecture layers** (1–3) = on-board software stack; **storage floors** (T0–T4) = retention/latency ladder; **pipeline stages** = narrative capture → reduction → sync only ([12 — Design journey](12-design-journey.md) §4). Do not mix these three. + +## Project vocabulary + +### T0 — hot + +RAM / DuckDB in-process sliding window of the last minutes; what mission logic queries in flight. Retention: minutes. Maps to **hot** in the classic three-tier model. + +### T1 — warm + +Drone NVMe: full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight. Retention: current flight plus quota-based headroom. Maps to **warm**. Runtime path: [`var/t1/`](../var/t1/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)). + +### T2 — shared + +On-board MinIO bucket: derived data only (`detections`, `state`), replicated opportunistically across the swarm. Retention: current mission. + +### T3 — warehouse + +Ground on-prem object store + Parquet lakehouse: every flight of every drone, forever; replay, analytics, model training. Retention: years. Maps to **cold** / **DWH**. Runtime path: [`var/t3/`](../var/t3/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)). After offload and integrity audit, T3 is the [source of truth](#source-of-truth) for historical flight data. + +### T4 — dev + +Engineer laptop or sim farm: slices pulled from T3, or synthetic data from the simulator. Retention: ephemeral. + +### Source of truth + +The authoritative copy of flight data after [offload](#offload) and integrity audit: the [T3](#t3--warehouse) warehouse. Each drone holds its own partial record in flight ([T1](#t1--warm)); peers exchange derived state only ([04 — Swarm sync](04-swarm-sync.md)). Replay, post-flight fusion, training, and audit read from T3 — not from any single drone's local store. See [06 — Environments](06-environments.md), [ADR-0003](adr/ADR-0003-sync-derived-state-only.md). + +### Architecture layer 1 — ingestion + +On-board services that capture sensor streams and video analytics ([02 — Architecture](02-architecture.md)): `sensor-ingest`, `video-analytics`. + +### Architecture layer 2 — storage and transform + +On-board Parquet write path, sealer, DuckDB, NVMe ([02 — Architecture](02-architecture.md)). + +### Architecture layer 3 — serving and sync + +On-board event hook, pub/sub broadcast, MinIO derived bucket, peer query API ([02 — Architecture](02-architecture.md)). + +### Pipeline stage + +Narrative inside each drone: (1) raw capture, (2) ETL/reduction, (3) bidirectional sync interface. Not the same as storage floors T0–T4 ([12 — Design journey](12-design-journey.md) §4). + +### Bulk sync + +In-flight pull of sealed `detections`/`state` Parquet partitions between peers via **rsync** delta over persistent SSH forced commands ([04 — Swarm sync](04-swarm-sync.md)). + +### Offload + +Post-flight mirror of sealed T1 partitions into T3 at the dock: **rsync** or MinIO **`mc mirror`**, then integrity audit before local prune ([03 — Data platform](03-data-platform.md)). + +### rclone + +Optional rsync-class tool when the far end is object storage and rsync is awkward; not the default in-flight mesh path ([09 — Open questions](09-open-questions.md) §12). + +### ADR + +Architecture Decision Record — a point-in-time decision for this repository ([adr/README.md](adr/README.md)). + +### ASR + +Architecture Standard Record — a standing cross-repository policy (lives in the ecosystem `inventar` repo, not here). See [adr/README.md](adr/README.md). + ## Data & storage | Term | Meaning | @@ -10,8 +74,8 @@ Terms and abbreviations used throughout this proposal. | **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) | | **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate | | **ZSTD** | Zstandard compression; the default codec for all Parquet files here | -| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown | -| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse | +| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown; same role as [source of truth](#source-of-truth) | +| **Hot / warm / cold tiers** | Classic three-tier model: in-memory window → local NVMe → ground warehouse. Here mapped to [T0](#t0--hot), [T1](#t1--warm), and [T3](#t3--warehouse); [T2](#t2--shared) and [T4](#t4--dev) are named explicitly | | **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes | | **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier | | **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame | diff --git a/docs/01-problem-statement.md b/docs/01-problem-statement.md index 4f3e809..f2001d4 100644 --- a/docs/01-problem-statement.md +++ b/docs/01-problem-statement.md @@ -15,7 +15,7 @@ Plus everything a platform needs around that: reproducible builds and deployment | Constraint | Consequence | | --- | --- | | **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind | -| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention | +| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention. A [supervising operator](00-glossary.md#swarm--robotics) may set mission intent over [C2](00-glossary.md#swarm--robotics) before and during flight, but does not steer individual vehicles | | **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator | | **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch | | **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight | @@ -30,8 +30,8 @@ Plus everything a platform needs around that: reproducible builds and deployment - **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked. - **MinIO** is available and considered as the inter-drone sync mechanism. - A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)). -- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training. -- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight. +- After landing, each drone's data is **offloaded to the on-prem [source of truth](00-glossary.md#source-of-truth)** for replay and iterative model training. +- Mission intent (declarative goals) reaches the swarm over a narrow **[C2](00-glossary.md#swarm--robotics) channel**; there is no continuous ground link in flight. ## What is assumed (explicitly marked as assumptions) diff --git a/docs/02-architecture.md b/docs/02-architecture.md index c2aff81..ddd6c3b 100644 --- a/docs/02-architecture.md +++ b/docs/02-architecture.md @@ -2,7 +2,7 @@ ## On-board: three layers, one Compose file -Every drone runs the same three-layer data plane. **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)). +Every drone runs the same [three-layer](00-glossary.md#architecture-layer-1--ingestion) data plane ([storage floors](00-glossary.md#t0--hot) T0–T4 are a separate concept — see [03 — Data platform](03-data-platform.md#storage-floors-tiers)). **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)). ```mermaid graph TB @@ -42,7 +42,7 @@ graph TB ### Layer 1 — Ingestion - `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on. -- `video-analytics` runs a **YOLO-like object detector** on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images. +- `video-analytics` runs a [**YOLO-like**](00-glossary.md#swarm--robotics) object detector on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images. - Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient). ### Layer 2 — Storage and transform @@ -55,7 +55,7 @@ graph TB - The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls. - The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)). -- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer); MinIO remains optional where an S3 API is wanted. +- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH ([rsync](00-glossary.md#bulk-sync) delta transfer); MinIO remains optional where an S3 API is wanted. - **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)). ## Communication planes @@ -65,11 +65,11 @@ Three isolated planes with different lifecycles: ```mermaid graph LR subgraph planes [Communication planes] - C2["C2 control plane
mission intent, narrow, in production"] + C2["[C2](00-glossary.md#swarm--robotics) control plane
mission intent, narrow, in production"] DATA["swarm data plane
state broadcast + sync, in production"] DEV["dev/debug plane
live telemetry, bench only"] end - OPERATOR["supervising operator
(man-in-the-loop)"] --> C2 + OPERATOR["supervising operator
([man-in-the-loop](00-glossary.md#swarm--robotics))"] --> C2 C2 --> SWARM["swarm"] SWARM <--> DATA DEV -.->|absent from production builds| SWARM @@ -79,7 +79,7 @@ graph LR | --- | --- | --- | --- | | **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes | | **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes | -| **Dev/debug plane** | Full live telemetry for bench development and HIL tests | High | **No — physically absent from production builds** | +| **Dev/debug plane** | Full live telemetry for bench development and [HIL](00-glossary.md#infrastructure--delivery) tests | High | **No — physically absent from production builds** | The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it. diff --git a/docs/03-data-platform.md b/docs/03-data-platform.md index 1002bab..c2ab0a3 100644 --- a/docs/03-data-platform.md +++ b/docs/03-data-platform.md @@ -1,6 +1,6 @@ # 03 — Data platform -The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every floor** of the system. +The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every [storage floor](00-glossary.md#t0--hot)** of the system. ## Datasets @@ -14,7 +14,7 @@ Three logical datasets, distinct because their consumers and sync policies diffe ## Partitioning layout -Identity dimensions first, calendar time last, everything Hive-style `key=value`: +Identity dimensions first, calendar time last, everything [Hive-style](00-glossary.md#data--storage) `key=value`: ``` dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/ @@ -29,7 +29,7 @@ Design decisions and their reasons: - **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.) - **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both. -- **Flight identifiers.** The canonical flight id is a **UUIDv7** (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest. +- **Flight identifiers.** The canonical flight id is a [**UUIDv7**](00-glossary.md#data--storage) (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest. - **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time. - **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)). @@ -50,7 +50,7 @@ sequenceDiagram S->>W: rows (int64 ns, typed) W->>C: flush min_NN.parquet (small, ZSTD-fast) end - Note over Z: window closes (hour sealed
or flight ends) + Note over Z: window closes (hour [sealed](00-glossary.md#data--storage)
or flight ends) Z->>C: read all min_*.parquet Z->>P: write one compact file (ZSTD-high) Z->>C: remove blocks after verify @@ -63,26 +63,28 @@ sequenceDiagram ## Storage floors (tiers) -Same format on every floor; only volume, retention, and location change: +Same [Parquet](00-glossary.md#data--storage) + [DuckDB](00-glossary.md#data--storage) format on every floor; only volume, retention, and location change. See [glossary](00-glossary.md#t0--hot) for the full T0–T4 definitions. | Floor | Where | Contents | Retention | | --- | --- | --- | --- | -| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes | -| **T1 — warm** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) | -| **T2 — shared** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission | -| **T3 — warehouse** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; replay, analytics, model training | Years | -| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral | +| **[T0 — hot](00-glossary.md#t0--hot)** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes | +| **[T1 — warm](00-glossary.md#t1--warm)** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) | +| **[T2 — shared](00-glossary.md#t2--shared)** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission | +| **[T3 — warehouse](00-glossary.md#t3--warehouse)** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; [source of truth](00-glossary.md#source-of-truth) for replay, analytics, model training | Years | +| **[T4 — dev](00-glossary.md#t4--dev)** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral | ## Flight offload: a mirror, not a migration -Because T1 and T3 share the identical layout, offload after landing is: +Because [T1](00-glossary.md#t1--warm) and [T3](00-glossary.md#t3--warehouse) share the identical layout, [offload](00-glossary.md#offload) after landing is: ```bash # per drone, at the base station -mc mirror drone-nvme/flights/ warehouse/flights/ # or rsync over the wired dock +rsync -a drone-nvme/flights/ warehouse/flights/ # or mc mirror to object store duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…" ``` +For S3-only backends where rsync is awkward, [rclone](00-glossary.md#rclone) is an optional rsync-class alternative — not the default mesh path ([09 — Open questions](09-open-questions.md) §12). + The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job. ## Retention and quotas on board @@ -138,6 +140,6 @@ On the ground this powers replay overlays, coverage analysis (did ubiquitous sen ## Schema management -- Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)). +- Every dataset schema is versioned in-repo (planned `schemas/` directory — [roadmap stage 2](08-roadmap.md), [09 — Open questions](09-open-questions.md) §14; SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)). - Parquet files carry the schema version in file metadata; readers select the matching deserializer. - Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release. diff --git a/docs/04-swarm-sync.md b/docs/04-swarm-sync.md index c7c4f38..888a8f8 100644 --- a/docs/04-swarm-sync.md +++ b/docs/04-swarm-sync.md @@ -16,19 +16,21 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex ## Broadcast payload: small on the wire, precise at rest -The `state` broadcast is a fixed compact frame: +The `state` broadcast is a fixed compact frame (46 bytes, little-endian): | Field | Type | Notes | | --- | --- | --- | -| `drone_id` | uint16 | Fleet-scoped registry | +| `magic` | 2 bytes | `b"SH"` | +| `version` | uint8 | Protocol version | +| `drone_id` | 8 bytes | Zero-padded ASCII fleet id | | `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage | | `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision | | `att_roll/pitch/yaw` | int16 | Centi-degrees | | `vel_x/y/z` | int16 | cm/s | -| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) | +| `frame_ref` | uint8 | Frame of reference id ([GPS-denied](00-glossary.md#swarm--robotics): local/visual-odometry frames must be explicit) | | `flags` | uint8 | Battery-low, returning, degraded-sensors, … | -~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh. +46 bytes per frame → a 50-drone swarm at 5 Hz is ~11.5 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh. `detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison. @@ -51,7 +53,7 @@ graph LR SB -->|"store-and-forward relay"| SC ``` -### Recommended: Zenoh +### Recommended: [Zenoh](00-glossary.md#swarm--robotics) - Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub. - First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware. @@ -68,13 +70,13 @@ graph LR ## Two sync mechanisms, deliberately separate -1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache. -2. **Bulk path — rsync over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. +1. **Fast path — pub/sub ([Zenoh](00-glossary.md#swarm--robotics)):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache. +2. **Bulk path — [rsync](00-glossary.md#bulk-sync) over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. Why SSH-based bulk sync over object-store replication: -- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery. -- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes. +- **Identity is already there.** Every drone holds pre-provisioned [ed25519](00-glossary.md#infrastructure--delivery) keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery. +- **One persistent multiplexed session** ([`ControlMaster`](00-glossary.md#infrastructure--delivery)) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes. - **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing. - **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it. @@ -83,7 +85,7 @@ Partition healing is automatic: replication is pull-based, addressed by partitio Two deliberate non-features of the SSH channel: - **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount. -- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction. +- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH [forced command](00-glossary.md#infrastructure--delivery)** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction. ## Design decision: SQL-over-SSH as the peer query channel @@ -114,7 +116,7 @@ SQL access must not become a write channel. A single "read-only connection" flag - **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction. - **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs. -- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it. +- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from [T3](00-glossary.md#t3--warehouse) if result sizes outgrow it. What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one. diff --git a/docs/05-network-security.md b/docs/05-network-security.md index 10b6ff2..94fda4c 100644 --- a/docs/05-network-security.md +++ b/docs/05-network-security.md @@ -23,9 +23,9 @@ graph TB A["drone A
wg key + certs"] B["drone B
wg key + certs"] C["drone C
wg key + certs"] - A <-->|"WireGuard + mTLS"| B - B <-->|"WireGuard + mTLS"| C - A <-->|"WireGuard + mTLS"| C + A <-->|"WireGuard tunnel"| B + B <-->|"WireGuard tunnel"| C + A <-->|"WireGuard tunnel"| C end PROV -->|"per-device identity, peer list"| A PROV --> B @@ -65,7 +65,7 @@ Keep both layers: WireGuard for *who is on the network and whether the wire is r ## Data at rest - On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable. -- The ground warehouse applies standard at-rest encryption plus role-based access; it is the single most valuable asset in the system (every flight ever flown). +- The ground warehouse applies standard at-rest encryption plus role-based access; it is the [source of truth](00-glossary.md#source-of-truth) and the single most valuable asset in the system (every flight ever flown). ## Attack surface, deliberately shortened diff --git a/docs/06-environments.md b/docs/06-environments.md index 2d6831f..89be9af 100644 --- a/docs/06-environments.md +++ b/docs/06-environments.md @@ -9,15 +9,15 @@ graph LR subgraph dev [3 — Dev and simulation] SIM["virtual drone fleet
(simulator, Compose)"] LAP["engineer laptop
Dev Container"] - SLICE["T4: data slices"] + SLICE["[T4](00-glossary.md#t4--dev): data slices"] end subgraph fleet [1 — Fleet, in flight] - D1["drone: Compose data plane
T0 hot + T1 NVMe + T2 MinIO"] + D1["drone: Compose data plane
[T0](00-glossary.md#t0--hot) hot + [T1](00-glossary.md#t1--warm) NVMe + [T2](00-glossary.md#t2--shared) MinIO"] D2["drone …"] end subgraph ground [2 — Ground, on-prem] DOCK["base station docks"] - DWH["T3 warehouse
object store + Parquet/DuckDB"] + DWH["[T3](00-glossary.md#t3--warehouse) warehouse
object store + Parquet/DuckDB"] K3S["k3s: CI runners, registry mirror,
sim farm, dashboards"] TRAIN["model training (out of scope)
reads T3, ships weights"] end @@ -34,7 +34,7 @@ graph LR ## 1 — Fleet infrastructure (on the drones) - Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md). -- Storage floors T0–T2. Nothing here requires ground contact during a mission. +- Storage floors [T0–T2](00-glossary.md#t0--hot). Nothing here requires ground contact during a mission. - Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest. ## 2 — Ground infrastructure (on-prem) @@ -43,7 +43,7 @@ The permanent installation. This **is** allowed to be a cluster — links are wi | Component | Runs on | Role | | --- | --- | --- | -| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory | +| **Warehouse ([T3](00-glossary.md#t3--warehouse))** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's [source of truth](00-glossary.md#source-of-truth) and long-term memory | | **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) | | **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see | | **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) | @@ -60,7 +60,7 @@ The environment engineers live in daily — and deliberately the first thing to - **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free. - **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop. - **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story. -- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified. +- **[T4](00-glossary.md#t4--dev) data slices**: a thin tool pulls partition subsets from [T3](00-glossary.md#t3--warehouse) (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified. - The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets). ## IaC boundaries @@ -74,13 +74,13 @@ Infrastructure as code follows the same discipline as the data plane: each tool | Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. | | On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). | -Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, T1) and **`var/t3/`** (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree. +Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, [T1](00-glossary.md#t1--warm)) and **`var/t3/`** (warehouse, [T3](00-glossary.md#t3--warehouse)). Compose, k3d, and DuckDB on the host all read the same Parquet tree. Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle. ### k3d: the ground segment as an executable miniature -`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API. +`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the [T1](00-glossary.md#t1--warm)→[T3](00-glossary.md#t3--warehouse) offload CronJob. The sim CronJob uses `cp -ru` plus `mc mirror` as a shortcut; production dock [offload](00-glossary.md#offload) uses rsync or `mc mirror` with checksum audit ([03](03-data-platform.md)). The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API. ### Next fidelity step: microVMs @@ -90,7 +90,7 @@ The k3d fleet shares one kernel and one network namespace tree — good enough t | Operation | What it costs with one layout everywhere | | --- | --- | -| Flight offload | `mc mirror` / `rsync` + audit — no transform step | +| Flight offload | `rsync` / `mc mirror` + audit — no transform step ([03](03-data-platform.md)) | | Debugging a field issue | Pull the flight slice, replay locally, same queries | | Validating a pipeline change | Run simulator, diff Parquet outputs | | Training data prep | Read T3 directly, no export pipeline | diff --git a/docs/07-observability.md b/docs/07-observability.md index 71a0061..0a16d70 100644 --- a/docs/07-observability.md +++ b/docs/07-observability.md @@ -29,7 +29,7 @@ graph LR LG["structured logs
= Parquet partitions"] end subgraph ground [Ground k3s] - DWH["T3 warehouse"] + DWH["[T3](00-glossary.md#t3--warehouse) warehouse"] PR["Prometheus
(live: docks, CI, sim farm)"] LK["Loki
(live logs, dev plane)"] GF["Grafana
dashboards + replay"] @@ -44,7 +44,7 @@ graph LR ``` - **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane. -- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute. +- **Flight post-mortems query the [source of truth](00-glossary.md#source-of-truth) warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute. ## Health questions this design answers cheaply diff --git a/docs/08-roadmap.md b/docs/08-roadmap.md index 0a8420b..6be60f5 100644 --- a/docs/08-roadmap.md +++ b/docs/08-roadmap.md @@ -12,7 +12,7 @@ graph TD S4["Stage 4
Serving layer
event hook, query API, local MinIO"] S5["Stage 5
Virtual swarm simulation
N drones in Compose, seeded scenarios, CI regression"] S6["Stage 6
Mesh network and security
WireGuard overlay, PKI provisioning, mTLS"] - S7["Stage 7
Swarm sync
pose broadcast, detections pub/sub, MinIO replication"] + S7["Stage 7
Swarm sync
pose broadcast, detections pub/sub, bulk replication (rsync over SSH)"] S8["Stage 8
Ground warehouse
offload + audit, T3 store, replay tooling"] S9["Stage 9
Observability
platform metrics as sensor stream, ground dashboards"] S10["Stage 10
Fleet releases
manifest, mirror, dock delivery, atomic rollback"] diff --git a/docs/09-open-questions.md b/docs/09-open-questions.md index 3066c66..3041c10 100644 --- a/docs/09-open-questions.md +++ b/docs/09-open-questions.md @@ -18,7 +18,7 @@ Visual-odometry positions are relative; different drones may hold different loca By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging. -*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from ground truth in the warehouse. If a persistent shared structure becomes unavoidable, use a CRDT type for it rather than inventing reconciliation. +*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from the [source of truth](00-glossary.md#source-of-truth) in the warehouse. If a persistent shared structure becomes unavoidable, use a [CRDT](00-glossary.md#infrastructure--delivery) type for it rather than inventing reconciliation. ## 4 — Degradation policy under sustained link loss @@ -42,7 +42,7 @@ The degradation ladder ([03](03-data-platform.md)) needs a domain-approved order The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration. -*Proposal:* a small HIL rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI. +*Proposal:* a small [HIL](00-glossary.md#infrastructure--delivery) rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI. ## 8 — Video retention policy @@ -64,6 +64,24 @@ Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage). ## 11 — Lake retention and partition pruning -The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on T1 disk after offload to T3. +The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on [T1](00-glossary.md#t1--warm) disk after offload to [T3](00-glossary.md#t3--warehouse). *Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job. + +## 12 — Bulk transfer tool choice + +In-flight peer bulk sync, dock offload, and sim shortcuts have used different wording (rsync, rclone, `mc mirror`, `cp -ru`). + +*Proposal:* **rsync over SSH** is the default for peer [bulk sync](00-glossary.md#bulk-sync) and wired dock [offload](00-glossary.md#offload) of the Hive partition tree; **MinIO `mc mirror`** when the far end is the object-store facade; **[rclone](00-glossary.md#rclone)** only when the backend is S3-compatible and rsync is awkward; sim CronJob keeps **`cp -ru` + `mc mirror`** as a documented shortcut. Pick one primary tool per path in provisioning, not all at once. + +## 13 — DuckLake vs plain DuckDB + Parquet on T3 + +The design journey mentions DuckLake; no ADR or implementation commits to it yet. + +*Proposal:* baseline remains DuckDB `read_parquet` globs over the [T3](00-glossary.md#t3--warehouse) Hive tree with MinIO underneath; evaluate DuckLake only if catalog/versioning pain appears at warehouse scale — decision gets its own ADR before any dependency lands. + +## 14 — `schemas/` registry layout + +[03 — Data platform](03-data-platform.md) references an in-repo `schemas/` directory that does not exist yet (roadmap stage 2). + +*Proposal:* add `schemas/` as part of stage 2 with one SemVer file per dataset version, referenced from the fleet release manifest; until then treat schema versions as manifest-only fields and document the gap here rather than implying the directory already exists. diff --git a/docs/10-domain-context.md b/docs/10-domain-context.md index 476cb3d..99f6ded 100644 --- a/docs/10-domain-context.md +++ b/docs/10-domain-context.md @@ -9,14 +9,14 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera | **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) | | **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) | | **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) | -| **Sustainable pulsing** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) | -| **Ubiquitous sensing** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) | +| **[Sustainable pulsing](00-glossary.md#swarm--robotics)** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) | +| **[Ubiquitous sensing](00-glossary.md#swarm--robotics)** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) | | **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) | | **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) | -| **GPS-denied operation** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) | -| **Man-in-the-loop** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) | +| **[GPS-denied operation](00-glossary.md#swarm--robotics)** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) | +| **[Man-in-the-loop](00-glossary.md#swarm--robotics)** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) | | **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) | -| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) | +| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the [source of truth](00-glossary.md#source-of-truth) warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) | | **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) | | **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) | @@ -25,7 +25,7 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera Every one of these principles quietly assumes a working data layer underneath: - *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget. -- *Ubiquitous sensing* assumes observations are **fused later** — that is a warehouse with aligned timestamps and schemas. +- *Ubiquitous sensing* assumes observations are **fused later** — that is the [source of truth](00-glossary.md#source-of-truth) warehouse with aligned timestamps and schemas. - *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop. - *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data. - *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format. diff --git a/docs/11-cicd-delivery.md b/docs/11-cicd-delivery.md index c84d457..6ffba00 100644 --- a/docs/11-cicd-delivery.md +++ b/docs/11-cicd-delivery.md @@ -85,7 +85,7 @@ schemas: config: compose-bundle: 3.4.0 radio-profile: production-2 - peer-registry: fleet-42-r7 # provisioning + revocations, see 05 + peer-registry: fleet-42-r7 # provisioning + revocations, see [05 — Network & security](05-network-security.md) ``` - Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm. diff --git a/docs/12-design-journey.md b/docs/12-design-journey.md index 679eda1..5ce2a3e 100644 --- a/docs/12-design-journey.md +++ b/docs/12-design-journey.md @@ -4,6 +4,7 @@ How this design came together, told in the order the thinking actually happened. It is a walk-through, not a report. The formal decisions, with options and trade-offs, live as [Architecture Decision Records](adr/README.md). This is the story behind them, and each chapter links straight into the code it produced. +Terms: [Glossary](00-glossary.md). --- @@ -46,17 +47,17 @@ conclusion: constrain the transport, but do not constrain on-board capture. Keep everything locally. Send only what peers truly need. > **Read more** -> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers +> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0–T4) -## 4. Three tiers inside each unit +## 4. Three pipeline stages inside each unit -That gave [three layers](03-data-platform.md) on every drone: +That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0–T4: -1. **Raw capture**, never transformed. -2. **ETL / reduction**, where data is cut down to what has to be shared: - millimetres to centimetres, thinning time series where that is enough, +1. **Raw capture** (pipeline stage 1), never transformed. +2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared: + millimeters to centimeters, thinning time series where that is enough, dropping what nobody downstream reads. -3. **A bidirectional interface**: on one side it broadcasts position into the +3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the shared channel, on the other it lets peers pull data. The ETL is deliberately not over-specified. How sensor data is transformed is a @@ -65,7 +66,7 @@ stored and in which structure, not to step into that work. > **Read the code** > - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight -> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one +> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1 ## 5. What actually needs to sync @@ -129,19 +130,28 @@ coming back to foundations DevOps and sysadmins already trust. > - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically > - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first +The explorer is running, not just described: the data-plane view (partition tree +and read-only SQL console) is live at +[swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/) +(same access as this repository and [swarm.produktor.io](https://swarm.produktor.io/)). +Browse the Hive layout, run `SELECT` queries against the live lake, and see the +same statement gate a peer drone would hit in flight. + ## 8. Replication and offload -Moving accumulated data uses rsync-class tooling (rclone) over the same SSH -channel. It transfers diffs of the Hive tree efficiently. This matters because a -unit can be lost, and if it is, we want its data to already exist elsewhere. -When peers have a stable link the partition tree is pulled between them, which -gives redundancy and a way to cross-check later. On the ground, when the fleet -returns, the same path unifies every unit's data into a -[local warehouse](06-environments.md). The warehouse is read-mostly for -analytics, so transaction contention is not a concern, which opens a -DuckDB / DuckLake approach with MinIO underneath, the -[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One -uniform structure is what lets an analyst trust a single picture even when a +Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH +channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock +[offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently. +This matters because a unit can be lost, and if it is, we want its data to +already exist elsewhere. When peers have a stable link the partition tree is +pulled between them, which gives redundancy and a way to cross-check later. On +the ground, when the fleet returns, the same path unifies every unit's data into +a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)). +The warehouse is read-mostly for analytics, so transaction contention is not a +concern — plain DuckDB over Parquet with MinIO underneath is the baseline; +DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md) +§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md). +One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a unit's data has a gap. > **Read the code** @@ -170,6 +180,14 @@ scanning the whole lake on every query does not scale, so the scan is > - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards > - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release +The dashboard is running, not just described: **Swarm Fleet — generated data** is +live at +[swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s) +(same access as this repository, [swarm.produktor.io](https://swarm.produktor.io/), +and [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)). +Telemetry rows, detections, pose frames, battery, and link quality — all derived +from the Parquet lake via SQL, not from agents on the drones. + ## 10. Working method A note on how this is meant to be built, not just what. Working together needs a diff --git a/docs/adr/ADR-0001-no-swarm-orchestrator.md b/docs/adr/ADR-0001-no-swarm-orchestrator.md index cb0a249..4b0a471 100644 --- a/docs/adr/ADR-0001-no-swarm-orchestrator.md +++ b/docs/adr/ADR-0001-no-swarm-orchestrator.md @@ -17,7 +17,7 @@ Partitions are the normal case, not the exception. | Option | Pros | Cons | | --- | --- | --- | | A — Cluster orchestrator spanning the swarm | Familiar tooling | Assumes stable membership; a partition stalls coordination; single point of failure in the air | -| B — Each drone autonomous, coordinating through exchanged data | Survives partitions; no air-side control plane to fail | No global view; behaviour must be derivable from local state | +| B — Each drone autonomous, coordinating through exchanged data | Survives partitions; no air-side control plane to fail | No global view; behavior must be derivable from local state | ## Decision diff --git a/docs/adr/ADR-0002-one-storage-format.md b/docs/adr/ADR-0002-one-storage-format.md index 2c01c16..a583455 100644 --- a/docs/adr/ADR-0002-one-storage-format.md +++ b/docs/adr/ADR-0002-one-storage-format.md @@ -6,9 +6,13 @@ Accepted (2026-07-08) ## Context -Data lives in three places: on the drone's NVMe in flight (T1), in the ground -warehouse after landing (T3), and in the dev/simulation environment. If each -tier used a different format or layout, offload would need transformation code, +Data lives across [storage floors](../00-glossary.md#t0--hot) T0–T4. Primary persistent +stores are [T1](../00-glossary.md#t1--warm) (drone NVMe in flight) and +[T3](../00-glossary.md#t3--warehouse) (ground warehouse after landing); +[T0](../00-glossary.md#t0--hot) is the in-memory hot window, +[T2](../00-glossary.md#t2--shared) optional on-board derived replication, +[T4](../00-glossary.md#t4--dev) dev/sim slices. If each floor used a different +format or layout, offload would need transformation code, schemas would drift, and a query proven on the bench would not run unchanged on flight data. diff --git a/docs/adr/ADR-0003-sync-derived-state-only.md b/docs/adr/ADR-0003-sync-derived-state-only.md index b473cf5..5ce09ee 100644 --- a/docs/adr/ADR-0003-sync-derived-state-only.md +++ b/docs/adr/ADR-0003-sync-derived-state-only.md @@ -25,7 +25,7 @@ new derived data triggers downstream action through hooks; nothing polls asking - Air bandwidth scales with the number of peers and the broadcast rate, not with sensor resolution. -- The warehouse is the only place with the full picture; in-flight decisions use +- The [source of truth](../00-glossary.md#source-of-truth) is the only place with the full picture; in-flight decisions use the local window plus peer broadcasts. - Broadcast staleness per peer becomes a first-class health signal ([`../07-observability.md`](../07-observability.md)). diff --git a/docs/adr/README.md b/docs/adr/README.md index c4d8e8b..8985ee6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -27,14 +27,14 @@ decisions specific to the platform are ADRs and belong here. | [ADR-0005](ADR-0005-wireguard-beneath-ssh.md) | WireGuard beneath SSH for the mesh transport | Accepted | | [ADR-0006](ADR-0006-iac-boundaries.md) | IaC split: Ansible hosts, Terraform ground, Flux ground-only, fleet manifest for drones | Accepted | | [ADR-0007](ADR-0007-non-root-image.md) | Run the simulator image as a non-root user | Accepted | -| [ADR-0008](ADR-0008-runtime-data-paths.md) | Shared `var/t1` / `var/t3` runtime paths; state out of Git | Accepted | +| [ADR-0008](ADR-0008-runtime-data-paths.md) | Shared [`var/t1`](../00-glossary.md#t1--warm) / [`var/t3`](../00-glossary.md#t3--warehouse) runtime paths; state out of Git | Accepted | | [ADR-0009](ADR-0009-bounded-lake-scans.md) | Bound observability lake scans to a flight window and cache | Accepted | ## Writing a new ADR 1. Copy [`TEMPLATE.md`](TEMPLATE.md) to `ADR-NNNN-short-slug.md` (next free number). -2. Fill in Context, Options (if more than one was real), Decision, Consequences. +2. Context, Options, Decision, Consequences. 3. Add a row to the index above. 4. Cross-link the ADR from the design doc it affects (and vice versa). -Keep the tone laconic: state the decision and why, skip the narrative. +Accepted = immutable. Reversal → new ADR supersedes. Never edit Accepted. diff --git a/docs/improvement/2026-07-10-audit.md b/docs/improvement/2026-07-10-audit.md new file mode 100644 index 0000000..7bc2550 --- /dev/null +++ b/docs/improvement/2026-07-10-audit.md @@ -0,0 +1,74 @@ +# Documentation audit — 2026-07-10 + +Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify T0–T4 vocabulary, align rsync/offload wording with code, improve readability. + +## Summary + +| Metric | Result | +| --- | --- | +| Markdown files checked | 33 | +| Broken relative links (after fixes) | 0 | +| Glossary `###` anchors added | 15 (T0–T4, layers, bulk sync, source of truth, …) | +| Open questions added | §12–§14 (tooling, DuckLake, `schemas/`) | + +## Findings fixed + +### Links + +- `infra/terraform/ground/README.md` pointed at non-existent `03-storage-design.md` → `03-data-platform.md`. +- `11-cicd-delivery.md` bare `see 05` → proper markdown link. + +### Factual drift (code vs docs) + +| Item | Was | Now (matches `broadcast.py`, `sim.ts`) | +| --- | --- | --- | +| Pose frame | ~40 bytes, uint16 `drone_id` | 46 bytes, 8-byte ASCII `drone_id` | + +### Terminology + +- Extended [00 — Glossary](../00-glossary.md) with T0–T4, architecture layers, pipeline stages, source of truth. +- Disambiguated design journey §4: pipeline stages ≠ storage floors. +- Roadmap Stage 7: rsync bulk path, not “MinIO replication”. + +### rsync / rclone / offload + +| Path | Canonical tool | +| --- | --- | +| Peer bulk sync | rsync over SSH | +| Dock offload | rsync or `mc mirror` | +| S3 backends (optional) | rclone | +| Sim CronJob | `cp -ru` + `mc mirror` (documented shortcut) | + +### Live demos (design journey) + +- [swarm.produktor.io](https://swarm.produktor.io/) — visual prototype +- [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/) — SQL explorer +- [swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s) — Grafana fleet dashboard + +### README / LICENSE + +- Restricted-use notice with link to [LICENSE](../../LICENSE). +- Component READMEs tightened (caveman style). + +## Remaining (documented, not bugs) + +| Item | Where | +| --- | --- | +| `schemas/` directory absent | OQ §14, roadmap stage 2 | +| DuckLake undecided | OQ §13 | +| rsync bulk sync not in sim code | Design only | +| Glossary table terms link to section headers, not row anchors | Optional polish | +| T4 after “Source of truth” in glossary order | Cosmetic | +| GitOps “source of truth” vs data SSOT | Different meanings in `11-cicd-delivery` | + +## Suggested CI checks (for issue #1) + +```bash +# Relative .md link resolver (Python one-liner in audit script) +# Glossary anchor: every 00-glossary.md#... from docs/ must resolve +# Optional: markdown-spellcheck with US locale +``` + +## Files touched + +`docs/00`–`12`, `docs/adr/*`, nine READMEs, root `README.md`. diff --git a/docs/improvement/README.md b/docs/improvement/README.md new file mode 100644 index 0000000..5744f51 --- /dev/null +++ b/docs/improvement/README.md @@ -0,0 +1,18 @@ +# Documentation improvement records + +Point-in-time audits and follow-up work for links, terminology, readability, and factual alignment with code. + +| Report | Date | Scope | +| --- | --- | --- | +| [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0–T4; live demo links | + +## Future automation + +Tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1): CI link checker, glossary anchor validation, terminology lint. + +## How to add a new report + +1. Copy the structure from the latest audit file. +2. Name it `YYYY-MM-DD-short-topic.md`. +3. Add a row to the table above. +4. Cross-link from [09 — Open questions](../09-open-questions.md) if new ambiguities appear. diff --git a/infra/ansible/README.md b/infra/ansible/README.md index 5e00f2b..2051447 100644 --- a/infra/ansible/README.md +++ b/infra/ansible/README.md @@ -1,14 +1,11 @@ # Ansible — host preparation -Ansible owns everything that happens **on a host before workloads run**. -Terraform ([`../terraform/`](../terraform/)) owns the workloads themselves. -The boundary is deliberate and documented in -[06 — Environments](../../docs/06-environments.md#iac-boundaries). +Ansible = host prep before workloads run. Terraform ([`../terraform/`](../terraform/)) owns workloads. Boundary: [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries). | Playbook | Target | What it does | | --- | --- | --- | -| `drone-provision.yml` | Drone (on the bench, before a mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from the fleet release manifest | -| `sim-cluster.yml` | Local workstation / CI host | k3d cluster that miniatures the ground segment; shared data volume; simulator image import | +| `drone-provision.yml` | Drone (bench, before mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from fleet manifest | +| `sim-cluster.yml` | Local workstation / CI host | k3d cluster miniaturing ground segment; shared data volume; simulator image import | ## Provision a drone @@ -16,16 +13,12 @@ The boundary is deliberate and documented in ansible-playbook -i inventory.example.yml drone-provision.yml ``` -The playbook is idempotent and runs only over the bench network — drones -are never provisioned in flight (see [05 — Network & security](../../docs/05-network-security.md)): +Idempotent. Bench network only. Never provision in flight ([05 — Network & security](../../docs/05-network-security.md)): -1. **Identity**: generate the drone's ed25519 keypair if absent. -2. **Peer trust**: install every fleet member's public key into - `authorized_keys`, each pinned to the read-only SQL forced command — - the only thing a peer can execute. -3. **WireGuard**: render `wg0.conf` with the drone's address and peers. -4. **Data plane**: lay down the Compose bundle referenced by the fleet - release manifest and enable it as a systemd unit. +1. **Identity**: generate drone ed25519 keypair if absent. +2. **Peer trust**: fleet public keys in `authorized_keys`, each pinned to read-only SQL forced command. +3. **WireGuard**: render `wg0.conf` with drone address and peers. +4. **Data plane**: Compose bundle from fleet manifest, systemd unit enabled. ## Create the simulation cluster @@ -34,6 +27,4 @@ ansible-playbook sim-cluster.yml # creates k3d cluster 'swarm-sim' cd ../terraform/sim-env && terraform init && terraform apply ``` -The cluster mounts a shared host directory as `/data` on the (single) -node, so drone pods, the exporter, and the explorer see one Parquet lake — -the same contract as the on-board NVMe layout. +Shared host directory mounted as `/data` — drone pods, exporter, explorer see one Parquet lake (same contract as on-board NVMe). diff --git a/infra/gitops/README.md b/infra/gitops/README.md index dbabcf6..a8d8f28 100644 --- a/infra/gitops/README.md +++ b/infra/gitops/README.md @@ -1,20 +1,16 @@ # GitOps — ground segment (Flux) -Flux reconciles **long-lived ground configuration** from this repository. It does -not run on drones and does not replace the fleet release manifest — those are -different lifecycles by design ([11 — CI/CD](../../docs/11-cicd-delivery.md)). +Flux = long-lived ground config from git. Not on drones. Not fleet manifest. Different lifecycles ([11 — CI/CD](../../docs/11-cicd-delivery.md)). ## Boundary: Terraform vs Flux vs fleet manifest | Layer | Tool | Owns | | --- | --- | --- | -| Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts — the shape of the simulation | -| Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs — things that change without reprovisioning PVCs | -| Drones | **Fleet release manifest** | Compose bundle digests, model weights, peer registry — atomic, dock-only delivery | +| Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts | +| Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs | +| Drones | **[Fleet release manifest](../../docs/11-cicd-delivery.md)** | Compose digests, model weights, peer registry — dock-only | -Drones are **outside GitOps**: there is no reconciler in flight. A dock applies -the pinned manifest once; mid-mission drift is impossible because the update -endpoint does not exist in the radio profile. +Drones outside GitOps. No reconciler in flight. Dock applies pinned manifest once. No mid-mission drift — no update endpoint in radio profile. ## Layout @@ -26,9 +22,7 @@ infra/gitops/ ## Bootstrap on the k3d simulation cluster -`infra/ansible/sim-cluster.yml` installs Flux controllers and applies the CRs -below. After the first `git push`, Flux polls `origin` and reconciles -`infra/gitops/ground/` into the `ground` namespace. +`infra/ansible/sim-cluster.yml` installs Flux and applies CRs. After first `git push`, Flux reconciles `infra/gitops/ground/`. ```bash # Manual bootstrap (if you skipped Ansible): @@ -36,17 +30,13 @@ flux install --namespace=flux-system kubectl apply -k infra/gitops/flux ``` -To reconcile immediately without waiting for the poll interval: +Reconcile immediately: ```bash flux reconcile source git swarm-house -n flux-system flux reconcile kustomization ground-segment -n flux-system ``` -## What Flux manages here (example) +## What Flux manages -The [`ground/`](ground/) overlay currently carries a **GitOps-managed ConfigMap** -that tags the warehouse segment with fleet policy metadata. In production this -pattern extends to Grafana dashboard bundles, Prometheus rule files, and -offload-schedule ConfigMaps — all versioned in git, all auditable, none of -them requiring a `terraform apply` to tweak a label. +[`ground/`](ground/) overlay: GitOps-managed ConfigMap with fleet policy metadata. Production extends to Grafana dashboards, Prometheus rules, offload-schedule ConfigMaps — git-versioned, no `terraform apply` for label tweaks. diff --git a/infra/terraform/README.md b/infra/terraform/README.md index d508982..e184c29 100644 --- a/infra/terraform/README.md +++ b/infra/terraform/README.md @@ -1,7 +1,18 @@ +# Terraform — ground workloads + +Declared workloads on the k3d simulation cluster. See [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries). + +| Module | Tier | Purpose | +| --- | --- | --- | +| [`sim-env/`](sim-env/) | [T4](../../docs/00-glossary.md#t4--dev) dev/sim | Virtual fleet + observability on k3d | +| [`ground/`](ground/) | [T3](../../docs/00-glossary.md#t3--warehouse) | Warehouse + [T1→T3](../../docs/03-data-platform.md#storage-floors-tiers) offload | + +Two modules, two states. Fleet disposable; warehouse survives `destroy` of sim-env. + ## Recovering from partial state If resources were created outside Terraform (or state was lost), import them -before `terraform apply`: +before `terraform apply`. Run from the target module directory (`sim-env/` or `ground/`): ```bash # sim-env (namespace: swarm) diff --git a/infra/terraform/ground/README.md b/infra/terraform/ground/README.md index e6fc8c6..7b21911 100644 --- a/infra/terraform/ground/README.md +++ b/infra/terraform/ground/README.md @@ -1,19 +1,14 @@ -# Terraform — ground warehouse (T3) +# Terraform — ground warehouse ([T3](../../../docs/00-glossary.md#t3--warehouse)) -The stationary half of the system: the historical warehouse that receives -sealed partitions after every flight ([03 — Storage tiers](../../../docs/03-storage-design.md)). -It is the longest-lived, most conventional infrastructure in the design — -and therefore the most natural Terraform territory. +Ground warehouse. Receives sealed partitions post-flight ([storage floors](../../../docs/03-data-platform.md#storage-floors-tiers)). Long-lived infra → Terraform. -Runs in the same k3d cluster as [`../sim-env`](../sim-env/) but in its own -namespace with its own state: the fleet is disposable, the warehouse is not, -and the two lifecycles must never share a `terraform destroy`. +Runs in same k3d cluster as [`../sim-env`](../sim-env/) but separate namespace and state: fleet disposable, warehouse not — never share `terraform destroy`. | Resource | Purpose | | --- | --- | -| Deployment `minio` + PVC + NodePort 30901 | Object-store facade of the warehouse for downstream consumers (training pipelines, replay) | -| CronJob `offload` | Post-flight offload: copies Hive partitions verbatim from the fleet lake (T1) into the warehouse (T3), then mirrors into MinIO | -| Deployment `warehouse-explorer` + NodePort 30089 | Second explorer instance over the warehouse path — historical read-only SQL across **all** flights | +| Deployment `minio` + PVC + NodePort 30901 | Object-store facade for downstream consumers | +| CronJob `offload` | [T1](../../../docs/00-glossary.md#t1--warm) → [T3](../../../docs/00-glossary.md#t3--warehouse) offload: Hive partitions verbatim, then `mc mirror` into MinIO | +| Deployment `warehouse-explorer` + NodePort 30089 | Read-only SQL across **all** offloaded flights | ## Usage @@ -23,16 +18,9 @@ terraform apply terraform output ``` -After a couple of offload runs, the warehouse explorer shows the same -partition tree as the live lake but accumulated across flights — the T1 -layout **is** the T3 layout, which is the whole point: no transform step, -no schema drift, DuckDB queries work identically on both ends. +After offload runs, warehouse explorer shows same partition tree as live lake, accumulated across flights. [T1](../../../docs/00-glossary.md#t1--warm) layout **is** [T3](../../../docs/00-glossary.md#t3--warehouse) layout. No transform. No schema drift. Same DuckDB queries both ends. ## Simulation vs production -- The CronJob compresses "drone lands, docks, offloads, prunes" into a - periodic rsync-style copy. Production offload is event-driven per docking - and verifies checksums from the flight's partition manifest before pruning - the on-board lake. -- MinIO credentials here are throwaway defaults; production credentials are - sealed per site and never in state or VCS. +- Sim CronJob: `cp -ru` + `mc mirror` on a schedule — shortcut for "lands, offloads, prunes". Production: event-driven per dock, rsync or `mc mirror`, checksum audit before prune ([03 — Data platform](../../../docs/03-data-platform.md)). +- MinIO credentials here are throwaway defaults; production credentials sealed per site, never in state or VCS. diff --git a/infra/terraform/sim-env/README.md b/infra/terraform/sim-env/README.md index b305735..f118f15 100644 --- a/infra/terraform/sim-env/README.md +++ b/infra/terraform/sim-env/README.md @@ -1,17 +1,14 @@ -# Terraform — simulation environment (T4) +# Terraform — simulation environment ([T4](../../../docs/00-glossary.md#t4--dev)) -Declares the simulated fleet and its observability stack on the k3d cluster -created by [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml). -This module is the executable miniature of the dev/sim environment from -[06 — Environments](../../../docs/06-environments.md). +Sim fleet + observability on k3d. Cluster from [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml). Executable [T4](../../../docs/00-glossary.md#t4--dev) miniature ([06 — Environments](../../../docs/06-environments.md)). | Resource | Purpose | | --- | --- | -| StatefulSet `drone` × `drone_count` | Virtual drones; stable pod names become `DRONE_ID`s; all write Hive-partitioned Parquet into the shared `/data` lake | -| Deployment `exporter` | Prometheus exporter reading the lake with DuckDB | -| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL console (also feeds the prototype's live mode) | -| Deployment `prometheus` + NodePort 30990 | Scrapes the exporter | -| Deployment `grafana` + NodePort 30300 | Same dashboard JSON as the Compose profile, provisioned from a ConfigMap | +| StatefulSet `drone` × `drone_count` | Virtual drones; pod names → `DRONE_ID`; shared `/data` lake | +| Deployment `exporter` | Prometheus exporter reading lake with DuckDB | +| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL (prototype live mode) | +| Deployment `prometheus` + NodePort 30990 | Scrapes exporter | +| Deployment `grafana` + NodePort 30300 | Dashboard JSON from ConfigMap | ## Usage @@ -22,7 +19,7 @@ terraform apply terraform output # URLs ``` -Scale the fleet without touching YAML: +Scale without YAML edits: ```bash terraform apply -var drone_count=9 -var speedup=4 @@ -30,11 +27,6 @@ terraform apply -var drone_count=9 -var speedup=4 ## Notes -- `image_pull_policy = "Never"` — the simulator image is imported by the - Ansible playbook (`k3d image import`); the cluster never pulls from a - registry, mirroring the air-gap doctrine. -- Pods discover their identity from the pod name (StatefulSet ordinal), so - a flight survives `kubectl delete pod` the same way a drone survives a - power cycle: new process, same identity, new flight ID. -- State is local (`terraform.tfstate` in this directory) — the simulation - cluster is disposable; nothing here is shared infrastructure. +- `image_pull_policy = "Never"` — image imported by Ansible (`k3d image import`); no registry pull. Air-gap mirror. +- Pod name → identity (StatefulSet ordinal). `kubectl delete pod` = power cycle: new process, same identity, new flight ID. +- Local state only — cluster disposable. diff --git a/prototype/README.md b/prototype/README.md index dae6e30..b708c59 100644 --- a/prototype/README.md +++ b/prototype/README.md @@ -1,8 +1,8 @@ # Swarm visualization prototype -A 2D top-down view of the swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol an area with static and mobile obstacles, exchange 5 Hz pose broadcasts (blue link flashes), and run opportunistic bulk sync of sealed partitions (green links with live rate and cumulative up/down counters). +2D top-down view of swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol with obstacles, 5 Hz pose broadcasts (blue flashes), opportunistic bulk sync (green links with rate and byte counters). -No backend — a pure client-side simulation of the same behavioral model the Python simulator implements over real Parquet and UDP. +No backend. Client-side sim. Same behavioral model as Python simulator (Parquet + UDP). ## Run @@ -11,9 +11,16 @@ npm install npm run dev # http://localhost:5173 ``` +## Live mode + +With k3d fleet + explorer running (`ansible-playbook` + `terraform apply` — see [sim-env README](../infra/terraform/sim-env/README.md)): + +- Explorer at `http://localhost:30088` +- Header → **go live** — polls read-only SQL API for real drone positions + ## What to look at -- **Links appear and disappear** as drones move in and out of radio range; persistent links render semi-transparent, broadcast deliveries flash them briefly. -- **Busy links turn green** and show current rate plus total transferred (`↑` / `↓`) — the bandwidth budget from the sync design made visible. -- **Per-drone label**: id, current Wi-Fi channel (hopping), battery. -- Controls: drone count, simulation speed, pause. +- Links form and break by radio range; broadcasts flash active links +- Busy links turn green — rate plus `↑` / `↓` totals +- Per-drone label: id, Wi-Fi channel, battery +- Controls: drone count, speed, pause diff --git a/simulator/README.md b/simulator/README.md index 380754c..ed157e7 100644 --- a/simulator/README.md +++ b/simulator/README.md @@ -1,8 +1,8 @@ # Virtual drone fleet -A data generator that exercises the **exact on-board pipeline** described in [03 — Data platform](../docs/03-data-platform.md): seeded kinematics along a patrol route, sensor streams at realistic rates, detection events, the hot `current/` → sealed Parquet write path, and the compact UDP state broadcast between drones. +Data generator. Same on-board pipeline as [03 — Data platform](../docs/03-data-platform.md): seeded kinematics, sensor streams, detection events, hot `current/` → sealed Parquet, compact UDP state broadcast. -One process = one drone. Scaling the swarm is a Compose flag. +One process = one drone. Scale swarm with Compose flag. ## Run a swarm @@ -14,7 +14,7 @@ FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --build --scale drone=5 DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10 ``` -Output lands in [`../var/t1/`](../var/t1/) (the shared T1 lake — same path the k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed: +Output lands in [`../var/t1/`](../var/t1/) ([T1](../docs/00-glossary.md#t1--warm) live lake — same path k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed: ``` var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet @@ -22,7 +22,7 @@ var/t1/dataset=detections/flight=…/drone=…/… var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts ``` -Historical flights after offload live under [`../var/t3/`](../var/t3/) (T3 warehouse). +Historical flights after offload: [`../var/t3/`](../var/t3/) ([T3](../docs/00-glossary.md#t3--warehouse) warehouse). ## Run a single drone without Docker @@ -52,33 +52,32 @@ print(con.sql(""" EOF ``` -## Monitoring: Grafana over the generated data +## Monitoring -The `monitoring` profile spins up a small metrics chain — a DuckDB-based exporter that scans the generated Parquet every few seconds, Prometheus, and a pre-provisioned Grafana dashboard: +`monitoring` profile: DuckDB exporter → Prometheus → Grafana ([07 — Observability](../docs/07-observability.md)): ```bash -docker compose --profile monitoring up -d # exporter + prometheus + grafana -# generate some flights in parallel or beforehand: +docker compose --profile monitoring up -d FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5 ``` -- Grafana: `http://localhost:3000` (anonymous admin — demo only) → dashboard **Swarm Fleet — generated data** +- Grafana: `http://localhost:3000` → **Swarm Fleet — generated data** - Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics` -Panels: telemetry rows by drone/sensor, detections by class, pose frames sent/received, battery per drone, RSSI and estimated distance per link, Parquet bytes/files on disk. The exporter is deliberately a demonstration of the observability doctrine from [07 — Observability](../docs/07-observability.md): fleet statistics are *derived from the data platform itself* — no agent on the drone, just SQL over the same Parquet everyone else reads. +Fleet stats derived from Parquet via SQL — no on-drone agent. ## Data-plane explorer -The `monitoring` profile also starts a **data-plane explorer** at `http://localhost:8088` — a single-page web view over the lake itself, complementing Grafana (which shows aggregates, not the structure): +`monitoring` profile also starts explorer at `http://localhost:8088` (k3d: `:30088`): -- **Partition tree**, live: `dataset → flight → drone → sensor → year/…/hour → file`, with file counts and bytes rolled up at every level. You can watch `current/` files appear and get sealed while a swarm is flying. -- **Read-only SQL console** with the datasets pre-registered as views (`telemetry`, `detections`, `state`) and one-click sample queries (fleet overview, battery timeline, detections by class, peer link quality, schema). +- Partition tree live: `dataset → flight → drone → sensor → hour → file` +- Read-only SQL console — same [statement gate](../docs/04-swarm-sync.md) as peer query channel: `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only -The console enforces the **same statement gate as the peer query channel** from [04 — Swarm sync](../docs/04-swarm-sync.md): a single statement, `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only, write/config/extension keywords rejected. That is the point: exploring the data plane on the bench uses the same read-only SQL contract a peer drone uses in flight — the explorer is the "no debug API" principle made visible. +Point: bench exploration uses same read-only SQL contract as flight peers. ## Reproducibility -Every run is deterministic per `(SEED, DRONE_ID)`: same route jitter, same sensor noise, same detection sequence. A bug report is a seed and a config, not a description. +Deterministic per `(SEED, DRONE_ID)`. Bug report = seed + config. ## Knobs -- 2.54.0 From 9bec50e8f7a4c5a66462ca0ae3200eda6d735440 Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:48:27 +0100 Subject: [PATCH 2/9] fix(ci): fetch PR ref on checkout instead of shallow main clone Shallow clone of default branch did not contain commits on feature branches, causing pull_request verify jobs to fail at checkout. --- .github/workflows/release.yaml | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index ba20cd3..e704cb2 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -24,8 +24,10 @@ jobs: steps: - name: Checkout run: | - git clone --depth=50 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git . - git checkout ${{ gitea.sha }} + git init . + git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git + git fetch --depth=50 origin "${{ gitea.ref }}" + git checkout FETCH_HEAD - name: Install dependencies run: pip install --quiet -r simulator/requirements.txt @@ -61,8 +63,10 @@ jobs: steps: - name: Checkout run: | - git clone --depth=1 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git . - git checkout ${{ gitea.sha }} + git init . + git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git + git fetch --depth=1 origin "${{ gitea.ref }}" + git checkout FETCH_HEAD - name: Install Trivy run: | -- 2.54.0 From e6e57843cbba6563004f90da0e92650ee27d74d2 Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:54:28 +0100 Subject: [PATCH 3/9] feat: add CI documentation rules checker and docs-verify job Enforce link resolution, glossary anchors, and readability rules on every push and PR; docs-only pushes to main still run docs-verify while skipping the simulator gate. --- .github/workflows/release.yaml | 21 ++- CONTRIBUTING.md | 1 + docs/improvement/2026-07-10-audit.md | 8 +- docs/improvement/README.md | 6 +- docs/improvement/ci-rules.md | 56 +++++++ scripts/check_docs.py | 219 +++++++++++++++++++++++++++ scripts/docs_rules.yaml | 47 ++++++ simulator/tests/test_check_docs.py | 73 +++++++++ 8 files changed, 422 insertions(+), 9 deletions(-) create mode 100644 docs/improvement/ci-rules.md create mode 100644 scripts/check_docs.py create mode 100644 scripts/docs_rules.yaml create mode 100644 simulator/tests/test_check_docs.py diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index e704cb2..620f186 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -1,4 +1,5 @@ # Swarm House - Gitea Actions +# docs-verify: markdown links, glossary anchors, readability rules (every push/PR) # verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR) # scan: Trivy vulnerability + misconfiguration scan # release: semantic version bump (conventional commits), tag, Gitea release @@ -17,6 +18,24 @@ on: branches: [main] jobs: + docs-verify: + name: Verify documentation + runs-on: ubuntu-latest + container: python:3.12-bookworm + steps: + - name: Checkout + run: | + git init . + git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git + git fetch --depth=50 origin "${{ gitea.ref }}" + git checkout FETCH_HEAD + + - name: Install checker dependencies + run: pip install --quiet pyyaml + + - name: Documentation rules (links, glossary, readability) + run: python scripts/check_docs.py + verify: name: Verify simulator runs-on: ubuntu-latest @@ -85,7 +104,7 @@ jobs: name: Semantic Release runs-on: ubuntu-latest container: python:3.12-bookworm - needs: [verify, scan] + needs: [docs-verify, verify, scan] if: gitea.event_name == 'push' permissions: contents: write diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index db66de9..dd12c7b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,6 +17,7 @@ The CI pipeline enforces this: | Gate | When it runs | | --- | --- | +| `python scripts/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) | | `pytest tests/` | Every push/PR (`verify` job) | | `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image | | Smoke flight + DuckDB assertion | Every push/PR | diff --git a/docs/improvement/2026-07-10-audit.md b/docs/improvement/2026-07-10-audit.md index 7bc2550..9b888dc 100644 --- a/docs/improvement/2026-07-10-audit.md +++ b/docs/improvement/2026-07-10-audit.md @@ -16,7 +16,7 @@ Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify ### Links - `infra/terraform/ground/README.md` pointed at non-existent `03-storage-design.md` → `03-data-platform.md`. -- `11-cicd-delivery.md` bare `see 05` → proper markdown link. +- `11-cicd-delivery.md` bare cross-reference to doc 05 → proper markdown link. ### Factual drift (code vs docs) @@ -63,11 +63,7 @@ Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify ## Suggested CI checks (for issue #1) -```bash -# Relative .md link resolver (Python one-liner in audit script) -# Glossary anchor: every 00-glossary.md#... from docs/ must resolve -# Optional: markdown-spellcheck with US locale -``` +Implemented in [`scripts/check_docs.py`](../../scripts/check_docs.py) — see [ci-rules.md](ci-rules.md). ## Files touched diff --git a/docs/improvement/README.md b/docs/improvement/README.md index 5744f51..69596cc 100644 --- a/docs/improvement/README.md +++ b/docs/improvement/README.md @@ -6,9 +6,11 @@ Point-in-time audits and follow-up work for links, terminology, readability, and | --- | --- | --- | | [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0–T4; live demo links | -## Future automation +## CI automation -Tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1): CI link checker, glossary anchor validation, terminology lint. +Enforced in Gitea Actions (`docs-verify` job). Rules live in [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml); see [ci-rules.md](ci-rules.md). + +Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1). ## How to add a new report diff --git a/docs/improvement/ci-rules.md b/docs/improvement/ci-rules.md new file mode 100644 index 0000000..2c22722 --- /dev/null +++ b/docs/improvement/ci-rules.md @@ -0,0 +1,56 @@ +# CI documentation rules + +Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate). + +Implementation: [`scripts/check_docs.py`](../../scripts/check_docs.py) reads [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml). + +Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**. + +Tracked from [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1) and the [2026-07-10 audit](2026-07-10-audit.md). + +## Gates + +| Rule | Severity | What it checks | +| --- | --- | --- | +| Relative links | error | Every `[text](path)` and `[text](path#anchor)` to a local `.md` file resolves to an existing file | +| Fragment anchors | error | `#anchor` fragments match a `##` or `###` heading in the target file (GitHub-style slug) | +| Banned targets | error | Known-bad paths (e.g. deleted `03-storage-design.md`) are not linked | +| Glossary fragments | error | All `00-glossary.md#…` links use anchors that exist in the glossary | +| Section vs term links | warning | Links to glossary *section* headers (`#swarm--robotics`, …) are listed; prefer `###` term anchors where they exist | +| Bare cross-refs | error | Prose cross-references to numbered docs without a proper markdown link | + +Warnings are printed in the job log but do not fail CI. Errors exit non-zero and block merge. + +## PR workflow + +```mermaid +flowchart LR + PR[Pull request] --> DV[docs-verify] + PR --> V[verify simulator] + V --> S[Trivy scan] + DV --> OK{All required green?} + V --> OK + S --> OK + OK -->|yes| Merge[Mergeable] +``` + +| Event | `docs-verify` | `verify` + `scan` | +| --- | --- | --- | +| PR → `main` | always | always | +| Push → `main` (code paths) | always | runs | +| Push → `main` (docs/README/AGENTS only) | always | skipped (`paths-ignore`) | + +Docs-only changes therefore still get link and glossary validation on `main`; they do not trigger a semantic release. + +## Local run + +```bash +python scripts/check_docs.py +python scripts/check_docs.py --warnings-as-errors # treat section-link warnings as failures +``` + +## Changing rules + +1. Edit `scripts/docs_rules.yaml` (paths, banned targets, preferred anchors). +2. Adjust `scripts/check_docs.py` only when adding a new rule *type*. +3. Add a row to the table above and mention it in the audit/improvement record if the change is significant. diff --git a/scripts/check_docs.py b/scripts/check_docs.py new file mode 100644 index 0000000..b0f2c4a --- /dev/null +++ b/scripts/check_docs.py @@ -0,0 +1,219 @@ +#!/usr/bin/env python3 +"""Validate markdown links, glossary anchors, and readability rules.""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover - stdlib fallback for minimal envs + yaml = None # type: ignore[assignment] + +ROOT = Path(__file__).resolve().parents[1] +RULES_PATH = Path(__file__).with_name("docs_rules.yaml") + +LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)") +BARE_SEE_RE = re.compile(r"(? dict: + text = path.read_text(encoding="utf-8") + if yaml is not None: + return yaml.safe_load(text) + raise RuntimeError("PyYAML is required to load docs_rules.yaml") + + +def slug_heading(text: str) -> str: + """Anchor slug used in this repository's glossary links.""" + text = text.strip() + text = re.sub(r"\s&\s", " — ", text) + if " — " in text: + left, right = text.split(" — ", 1) + return f"{_slug_part(left)}--{_slug_part(right)}" + return _slug_part(text) + + +def _slug_part(text: str) -> str: + slug = text.strip().lower() + slug = re.sub(r"[^\w\s-]", "", slug) + slug = re.sub(r"\s+", "-", slug) + return slug + + +def collect_anchors(md_path: Path) -> set[str]: + anchors: set[str] = set() + for line in md_path.read_text(encoding="utf-8").splitlines(): + if line.startswith("##"): + anchors.add(slug_heading(line.lstrip("#").strip())) + return anchors + + +def expand_markdown_paths(patterns: list[str]) -> list[Path]: + files: set[Path] = set() + for pattern in patterns: + for path in ROOT.glob(pattern): + if path.is_file() and path.suffix == ".md": + files.add(path) + return sorted(files) + + +def split_link_target(raw: str) -> tuple[str, str | None]: + target = raw.strip() + if target.startswith("<") and target.endswith(">"): + target = target[1:-1] + if "#" in target: + path_part, fragment = target.split("#", 1) + return path_part, fragment or None + return target, None + + +def resolve_path(source: Path, link_path: str) -> Path | None: + if not link_path or link_path.startswith(SKIP_SCHEMES): + return None + candidate = (source.parent / link_path).resolve() + try: + candidate.relative_to(ROOT.resolve()) + except ValueError: + return None + return candidate + + +def is_glossary_link(link_path: str, rules: dict) -> bool: + glossary = rules["glossary"]["file"].replace("\\", "/") + normalized = link_path.replace("\\", "/") + return ( + normalized == glossary + or normalized.endswith("/" + glossary) + or normalized.endswith(glossary.split("/")[-1]) + ) + + +def check_file( + md_path: Path, + rules: dict, + anchor_cache: dict[Path, set[str]], +) -> tuple[list[str], list[str]]: + errors: list[str] = [] + warnings: list[str] = [] + rel = md_path.relative_to(ROOT) + text = md_path.read_text(encoding="utf-8") + + if rules.get("readability", {}).get("forbid_bare_see_refs"): + prose = re.sub(r"`[^`]*`", "", text) + for match in BARE_SEE_RE.finditer(prose): + errors.append( + f"{rel}:{_line_no(text, match.start())}: bare cross-ref " + f"'see {match.group(1)}' — use a markdown link" + ) + + for match in LINK_RE.finditer(text): + raw = match.group(1) + if any(raw.startswith(s) for s in SKIP_SCHEMES): + continue + + link_path, fragment = split_link_target(raw) + if not link_path.endswith(".md"): + continue + + for banned in rules.get("banned_link_targets", []): + if banned in link_path.replace("\\", "/"): + errors.append( + f"{rel}:{_line_no(text, match.start())}: banned link target " + f"'{link_path}'" + ) + + target = resolve_path(md_path, link_path) + if target is None: + errors.append( + f"{rel}:{_line_no(text, match.start())}: link escapes repo or " + f"is invalid: {raw}" + ) + continue + if not target.is_file(): + errors.append( + f"{rel}:{_line_no(text, match.start())}: broken link " + f"{link_path} -> {target.relative_to(ROOT)}" + ) + continue + + if fragment: + if target not in anchor_cache: + anchor_cache[target] = collect_anchors(target) + if fragment not in anchor_cache[target]: + errors.append( + f"{rel}:{_line_no(text, match.start())}: unknown anchor " + f"#{fragment} in {target.relative_to(ROOT)}" + ) + + if fragment and is_glossary_link(link_path, rules): + section_anchors = set(rules.get("glossary_section_anchors", [])) + if fragment in section_anchors: + warnings.append( + f"{rel}:{_line_no(text, match.start())}: glossary section " + f"link #{fragment} — prefer a ### term anchor where one exists" + ) + + return errors, warnings + + +def _line_no(text: str, index: int) -> int: + return text.count("\n", 0, index) + 1 + + +def run_checks(warnings_as_errors: bool = False) -> int: + rules = load_rules(ROOT / "scripts/docs_rules.yaml") + anchor_cache: dict[Path, set[str]] = {} + all_errors: list[str] = [] + all_warnings: list[str] = [] + + glossary_path = ROOT / rules["glossary"]["file"] + anchor_cache[glossary_path] = collect_anchors(glossary_path) + + for md_path in expand_markdown_paths(rules["markdown_paths"]): + errors, warnings = check_file(md_path, rules, anchor_cache) + all_errors.extend(errors) + all_warnings.extend(warnings) + + if all_warnings: + print("Documentation warnings:") + for msg in sorted(all_warnings): + print(f" warning: {msg}") + print() + + if all_errors: + print("Documentation errors:") + for msg in sorted(all_errors): + print(f" error: {msg}") + print(f"\n{len(all_errors)} error(s), {len(all_warnings)} warning(s)") + return 1 + + if warnings_as_errors and all_warnings: + print(f"\n{len(all_warnings)} warning(s) treated as errors") + return 1 + + print( + f"Documentation checks passed " + f"({len(expand_markdown_paths(rules['markdown_paths']))} files, " + f"{len(all_warnings)} warning(s))" + ) + return 0 + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--warnings-as-errors", + action="store_true", + help="Exit non-zero when section-link warnings are present", + ) + args = parser.parse_args() + sys.exit(run_checks(warnings_as_errors=args.warnings_as_errors)) + + +if __name__ == "__main__": + main() diff --git a/scripts/docs_rules.yaml b/scripts/docs_rules.yaml new file mode 100644 index 0000000..63c4233 --- /dev/null +++ b/scripts/docs_rules.yaml @@ -0,0 +1,47 @@ +# Documentation rules enforced by CI (scripts/check_docs.py). +# See docs/improvement/ci-rules.md for rationale and PR workflow. + +markdown_paths: + - docs/**/*.md + - README.md + - CONTRIBUTING.md + - AGENTS.md + - simulator/README.md + - prototype/README.md + - infra/**/README.md + +glossary: + file: docs/00-glossary.md + +# Relative link targets that must never reappear (known regressions). +banned_link_targets: + - 03-storage-design.md + +# Glossary links to these section headers are allowed but reported as warnings +# when a term-specific ### anchor exists (see preferred_term_anchors). +glossary_section_anchors: + - data--storage + - swarm--robotics + - infrastructure--delivery + +# First-use vocabulary should link to these ### anchors, not section headers. +preferred_term_anchors: + - t0--hot + - t1--warm + - t2--shared + - t3--warehouse + - t4--dev + - source-of-truth + - architecture-layer-1--ingestion + - architecture-layer-2--storage-and-transform + - architecture-layer-3--serving-and-sync + - pipeline-stage + - bulk-sync + - offload + - rclone + - adr + - asr + +readability: + # Fail when prose uses bare "see 05" instead of a markdown link. + forbid_bare_see_refs: true diff --git a/simulator/tests/test_check_docs.py b/simulator/tests/test_check_docs.py new file mode 100644 index 0000000..f079da3 --- /dev/null +++ b/simulator/tests/test_check_docs.py @@ -0,0 +1,73 @@ +"""Tests for scripts/check_docs.py — documentation CI rules.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +SCRIPTS = Path(__file__).resolve().parents[2] / "scripts" +sys.path.insert(0, str(SCRIPTS)) + +from check_docs import ( # noqa: E402 + collect_anchors, + run_checks, + slug_heading, + split_link_target, +) + +ROOT = Path(__file__).resolve().parents[2] +GLOSSARY = ROOT / "docs/00-glossary.md" + + +def test_slug_heading_matches_glossary_anchors() -> None: + assert slug_heading("T0 — hot") == "t0--hot" + assert slug_heading("T3 — warehouse") == "t3--warehouse" + assert slug_heading("Source of truth") == "source-of-truth" + assert slug_heading("Data & storage") == "data--storage" + + +def test_glossary_term_anchors_exist() -> None: + anchors = collect_anchors(GLOSSARY) + for term in ( + "t0--hot", + "t1--warm", + "t2--shared", + "t3--warehouse", + "t4--dev", + "source-of-truth", + "bulk-sync", + "offload", + ): + assert term in anchors, f"missing glossary anchor #{term}" + + +def test_split_link_target() -> None: + assert split_link_target("foo.md#bar") == ("foo.md", "bar") + assert split_link_target("foo.md") == ("foo.md", None) + + +def test_repo_docs_pass() -> None: + assert run_checks() == 0 + + +def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + docs = tmp_path / "docs" + docs.mkdir() + (docs / "00-glossary.md").write_text("# Glossary\n\n## Data & storage\n") + (docs / "bad.md").write_text("See [missing](missing.md).\n") + rules = tmp_path / "scripts" + rules.mkdir() + (rules / "docs_rules.yaml").write_text( + "markdown_paths:\n - docs/**/*.md\n" + "glossary:\n file: docs/00-glossary.md\n" + "banned_link_targets: []\n" + "glossary_section_anchors: []\n" + "preferred_term_anchors: []\n" + "readability:\n forbid_bare_see_refs: false\n" + ) + checker = SCRIPTS / "check_docs.py" + monkeypatch.setattr("check_docs.ROOT", tmp_path) + monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.yaml") + assert run_checks() == 1 -- 2.54.0 From bed6f93626024b57b3e2d122c057a9cc458a1957 Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:56:13 +0100 Subject: [PATCH 4/9] docs: default caveman style for issues, PRs, and reviews Document caveman-full as the collaboration voice in AGENTS and CONTRIBUTING, and add Gitea issue/PR templates that pre-fill terse structure. --- .github/ISSUE_TEMPLATE/config.yml | 8 +++++++ .github/ISSUE_TEMPLATE/default.md | 24 +++++++++++++++++++++ .github/pull_request_template.md | 19 +++++++++++++++++ AGENTS.md | 17 +++++++++++++++ CONTRIBUTING.md | 35 +++++++++++++++++++++++++++++++ docs/improvement/README.md | 7 +++++++ 6 files changed, 110 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/default.md create mode 100644 .github/pull_request_template.md diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..aeca6a8 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: + - name: Design docs + url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/12-design-journey.md + about: Start with the design journey, then open an issue here. + - name: Open questions + url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/09-open-questions.md + about: Known unknowns — check before duplicating. diff --git a/.github/ISSUE_TEMPLATE/default.md b/.github/ISSUE_TEMPLATE/default.md new file mode 100644 index 0000000..0b6b8fe --- /dev/null +++ b/.github/ISSUE_TEMPLATE/default.md @@ -0,0 +1,24 @@ +--- +name: Change / task +about: Default issue template — caveman style +title: "" +labels: [] +--- + +Write **caveman-full**: short lines, fragments OK, no filler. See [`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default). + +## What + + + +## Why + + + +## Done when + +- [ ] + +## Notes + + diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..fb2857d --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,19 @@ +Write **caveman-full** by default. See [`CONTRIBUTING.md`](../CONTRIBUTING.md#caveman-style-default). + +## What + + + +## Why + + + +## Test + +- [ ] `pytest tests/ -q` (if code touched) +- [ ] `python scripts/check_docs.py` (if docs touched) +- [ ] ADR added if boundary/contract changed + +## Links + + diff --git a/AGENTS.md b/AGENTS.md index f110b52..2c6e9b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,23 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development workflow and TDD re ## Conventions +### Caveman style (default for collaboration) + +**Issues, pull requests, and code reviews** use **caveman-full** by default: + +- Short sentences or fragments; drop articles and filler when clear +- Lead with fact, verdict, or ask — not preamble +- Bullets over paragraphs; one idea per line +- Keep technical terms, acronyms, paths, and code exact +- English only; no identifying references (same as all repo prose) + +README tiers: root [`README.md`](README.md) = **caveman-lite** (full sentences, less filler); +component READMEs = **caveman-full**. Design docs in `docs/` stay readable prose unless +editing for consistency. + +Templates: [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/), +[`.github/pull_request_template.md`](.github/pull_request_template.md). + - Markdown: ATX headings, fenced code blocks with language tags, pipe tables. - Python: PEP 8, type hints, pure functions where possible, no unnecessary classes. - TypeScript: strict mode, functional components and hooks, no extra UI libraries. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dd12c7b..6a5b35b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -77,6 +77,41 @@ test: cover broadcast frame roundtrip Never include AI tool references, `Co-authored-by` trailers for assistants, or identifying names of people, companies, or locations. +## Caveman style (default) + +Use **caveman-full** in **issues**, **pull requests**, and **reviews** unless the +author explicitly asks otherwise. + +| Do | Skip | +| --- | --- | +| Short lines, fragments OK | Long preamble ("I wanted to start by saying…") | +| Lead with what / why / verdict | Filler ("just", "basically", "kind of") | +| Bullets, checklists | Repeated context already in diff or issue | +| Exact terms, paths, code | Invented abbreviations or vague "the thing" | + +**Examples** + +| Fluffy | Caveman | +| --- | --- | +| I think we should probably consider adding a link checker to CI because links might break. | Add CI link checker. Broken relative links fail merge. | +| This pull request implements the documentation improvements that were discussed in the audit. | Docs audit: glossary T0–T4, fix ground README link, rsync wording. | +| LGTM but maybe we could think about whether the anchor format is consistent? | Merge OK. Fix glossary anchors in follow-up (#1). | + +README intensity: root = caveman-lite; `simulator/`, `prototype/`, `infra/`, `docs/adr/` READMEs = +caveman-full. Numbered design docs in `docs/` stay full prose. + +Gitea pre-fills [issue](.github/ISSUE_TEMPLATE/default.md) and +[PR](.github/pull_request_template.md) templates with this style. + +### Reviews + +Default review voice = caveman-full: + +1. **Verdict first** — merge / wait / no, or approve / request changes +2. **Findings** — bullet per blocker or nit; file + line when useful +3. **Scope** — say if issue AC met, partial, or out of scope +4. No essay; link ADR, issue, or doc section instead of restating design + ## Pull request checklist - [ ] Tests added or updated; `pytest tests/ -q` passes locally diff --git a/docs/improvement/README.md b/docs/improvement/README.md index 69596cc..cfef87f 100644 --- a/docs/improvement/README.md +++ b/docs/improvement/README.md @@ -6,6 +6,13 @@ Point-in-time audits and follow-up work for links, terminology, readability, and | --- | --- | --- | | [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0–T4; live demo links | +## Collaboration style + +Issues, PRs, and reviews default to **caveman-full** ([`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default), +[`AGENTS.md`](../../AGENTS.md#caveman-style-default-for-collaboration)). Gitea templates live under +[`.github/ISSUE_TEMPLATE/`](../../.github/ISSUE_TEMPLATE/) and +[`.github/pull_request_template.md`](../../.github/pull_request_template.md). + ## CI automation Enforced in Gitea Actions (`docs-verify` job). Rules live in [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml); see [ci-rules.md](ci-rules.md). -- 2.54.0 From 10a18f13219d536304dbe8c3615475c81ecf541c Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:57:39 +0100 Subject: [PATCH 5/9] fix: revert ADR edits and enforce immutable accepted records MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restore docs/adr to main; add check_adrs.py CI gate so existing ADR-*.md files cannot be modified — supersede with a new numbered ADR instead. --- .github/workflows/release.yaml | 6 ++ AGENTS.md | 7 +- CONTRIBUTING.md | 2 +- docs/adr/ADR-0001-no-swarm-orchestrator.md | 2 +- docs/adr/ADR-0002-one-storage-format.md | 10 +-- docs/adr/ADR-0003-sync-derived-state-only.md | 2 +- docs/adr/README.md | 6 +- docs/improvement/ci-rules.md | 2 + scripts/check_adrs.py | 75 ++++++++++++++++++++ simulator/tests/test_check_adrs.py | 70 ++++++++++++++++++ 10 files changed, 168 insertions(+), 14 deletions(-) create mode 100644 scripts/check_adrs.py create mode 100644 simulator/tests/test_check_adrs.py diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 620f186..a786b9e 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -30,9 +30,15 @@ jobs: git fetch --depth=50 origin "${{ gitea.ref }}" git checkout FETCH_HEAD + - name: Fetch base branch for ADR immutability + run: git fetch --depth=1 origin main + - name: Install checker dependencies run: pip install --quiet pyyaml + - name: ADR immutability (no edits to accepted records) + run: python scripts/check_adrs.py --base origin/main + - name: Documentation rules (links, glossary, readability) run: python scripts/check_docs.py diff --git a/AGENTS.md b/AGENTS.md index 2c6e9b3..1036ef2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,12 @@ Status / Context / Options / Decision / Consequences format. Start from When a change alters a boundary, contract, or trade-off, add an ADR (copy [`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the index) and cross-link it from the affected design doc. A later reversal gets a -new ADR that supersedes the old one — never edit an Accepted record. +new ADR that supersedes the old one — **never edit an Accepted record**. + +**Immutable:** every `docs/adr/ADR-*.md` file is frozen once on `main`. CI rejects +any change that modifies an existing ADR. Updates go in a new numbered ADR only. +The index ([`docs/adr/README.md`](docs/adr/README.md)) and [`TEMPLATE.md`](docs/adr/TEMPLATE.md) +may still change; decision bodies do not. ## Runtime & operations (sim environment) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6a5b35b..88164d9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -117,7 +117,7 @@ Default review voice = caveman-full: - [ ] Tests added or updated; `pytest tests/ -q` passes locally - [ ] `docker build` in `simulator/` succeeds (test stage included) - [ ] Documentation updated if behaviour or boundaries changed -- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/)) +- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/)) — never edit an existing `ADR-*.md`; supersede with a new number - [ ] English only; no identifying references - [ ] Conventional commit message diff --git a/docs/adr/ADR-0001-no-swarm-orchestrator.md b/docs/adr/ADR-0001-no-swarm-orchestrator.md index 4b0a471..cb0a249 100644 --- a/docs/adr/ADR-0001-no-swarm-orchestrator.md +++ b/docs/adr/ADR-0001-no-swarm-orchestrator.md @@ -17,7 +17,7 @@ Partitions are the normal case, not the exception. | Option | Pros | Cons | | --- | --- | --- | | A — Cluster orchestrator spanning the swarm | Familiar tooling | Assumes stable membership; a partition stalls coordination; single point of failure in the air | -| B — Each drone autonomous, coordinating through exchanged data | Survives partitions; no air-side control plane to fail | No global view; behavior must be derivable from local state | +| B — Each drone autonomous, coordinating through exchanged data | Survives partitions; no air-side control plane to fail | No global view; behaviour must be derivable from local state | ## Decision diff --git a/docs/adr/ADR-0002-one-storage-format.md b/docs/adr/ADR-0002-one-storage-format.md index a583455..2c01c16 100644 --- a/docs/adr/ADR-0002-one-storage-format.md +++ b/docs/adr/ADR-0002-one-storage-format.md @@ -6,13 +6,9 @@ Accepted (2026-07-08) ## Context -Data lives across [storage floors](../00-glossary.md#t0--hot) T0–T4. Primary persistent -stores are [T1](../00-glossary.md#t1--warm) (drone NVMe in flight) and -[T3](../00-glossary.md#t3--warehouse) (ground warehouse after landing); -[T0](../00-glossary.md#t0--hot) is the in-memory hot window, -[T2](../00-glossary.md#t2--shared) optional on-board derived replication, -[T4](../00-glossary.md#t4--dev) dev/sim slices. If each floor used a different -format or layout, offload would need transformation code, +Data lives in three places: on the drone's NVMe in flight (T1), in the ground +warehouse after landing (T3), and in the dev/simulation environment. If each +tier used a different format or layout, offload would need transformation code, schemas would drift, and a query proven on the bench would not run unchanged on flight data. diff --git a/docs/adr/ADR-0003-sync-derived-state-only.md b/docs/adr/ADR-0003-sync-derived-state-only.md index 5ce09ee..b473cf5 100644 --- a/docs/adr/ADR-0003-sync-derived-state-only.md +++ b/docs/adr/ADR-0003-sync-derived-state-only.md @@ -25,7 +25,7 @@ new derived data triggers downstream action through hooks; nothing polls asking - Air bandwidth scales with the number of peers and the broadcast rate, not with sensor resolution. -- The [source of truth](../00-glossary.md#source-of-truth) is the only place with the full picture; in-flight decisions use +- The warehouse is the only place with the full picture; in-flight decisions use the local window plus peer broadcasts. - Broadcast staleness per peer becomes a first-class health signal ([`../07-observability.md`](../07-observability.md)). diff --git a/docs/adr/README.md b/docs/adr/README.md index 8985ee6..c4d8e8b 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -27,14 +27,14 @@ decisions specific to the platform are ADRs and belong here. | [ADR-0005](ADR-0005-wireguard-beneath-ssh.md) | WireGuard beneath SSH for the mesh transport | Accepted | | [ADR-0006](ADR-0006-iac-boundaries.md) | IaC split: Ansible hosts, Terraform ground, Flux ground-only, fleet manifest for drones | Accepted | | [ADR-0007](ADR-0007-non-root-image.md) | Run the simulator image as a non-root user | Accepted | -| [ADR-0008](ADR-0008-runtime-data-paths.md) | Shared [`var/t1`](../00-glossary.md#t1--warm) / [`var/t3`](../00-glossary.md#t3--warehouse) runtime paths; state out of Git | Accepted | +| [ADR-0008](ADR-0008-runtime-data-paths.md) | Shared `var/t1` / `var/t3` runtime paths; state out of Git | Accepted | | [ADR-0009](ADR-0009-bounded-lake-scans.md) | Bound observability lake scans to a flight window and cache | Accepted | ## Writing a new ADR 1. Copy [`TEMPLATE.md`](TEMPLATE.md) to `ADR-NNNN-short-slug.md` (next free number). -2. Context, Options, Decision, Consequences. +2. Fill in Context, Options (if more than one was real), Decision, Consequences. 3. Add a row to the index above. 4. Cross-link the ADR from the design doc it affects (and vice versa). -Accepted = immutable. Reversal → new ADR supersedes. Never edit Accepted. +Keep the tone laconic: state the decision and why, skip the narrative. diff --git a/docs/improvement/ci-rules.md b/docs/improvement/ci-rules.md index 2c22722..13929b1 100644 --- a/docs/improvement/ci-rules.md +++ b/docs/improvement/ci-rules.md @@ -18,6 +18,7 @@ Tracked from [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1) a | Glossary fragments | error | All `00-glossary.md#…` links use anchors that exist in the glossary | | Section vs term links | warning | Links to glossary *section* headers (`#swarm--robotics`, …) are listed; prefer `###` term anchors where they exist | | Bare cross-refs | error | Prose cross-references to numbered docs without a proper markdown link | +| ADR immutability | error | No modifications to existing `docs/adr/ADR-*.md` (compare `origin/main`); add new ADR to supersede | Warnings are printed in the job log but do not fail CI. Errors exit non-zero and block merge. @@ -46,6 +47,7 @@ Docs-only changes therefore still get link and glossary validation on `main`; th ```bash python scripts/check_docs.py +python scripts/check_adrs.py --base origin/main python scripts/check_docs.py --warnings-as-errors # treat section-link warnings as failures ``` diff --git a/scripts/check_adrs.py b/scripts/check_adrs.py new file mode 100644 index 0000000..f27db7f --- /dev/null +++ b/scripts/check_adrs.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Reject modifications to accepted Architecture Decision Records.""" + +from __future__ import annotations + +import argparse +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +ADR_PREFIX = "docs/adr/ADR-" + + +def _git(*args: str) -> str: + return subprocess.check_output( + ["git", *args], + cwd=ROOT, + text=True, + stderr=subprocess.DEVNULL, + ).strip() + + +def _exists_on(base: str, path: str) -> bool: + try: + _git("cat-file", "-e", f"{base}:{path}") + return True + except subprocess.CalledProcessError: + return False + + +def find_modified_adrs(base: str) -> list[str]: + try: + diff_out = _git("diff", "--name-only", f"{base}...HEAD", "--", "docs/adr/") + except subprocess.CalledProcessError: + return [] + + modified: list[str] = [] + for rel in diff_out.splitlines(): + if not rel.startswith(ADR_PREFIX) or not rel.endswith(".md"): + continue + if _exists_on(base, rel): + modified.append(rel) + return sorted(modified) + + +def run_check(base: str = "origin/main") -> int: + modified = find_modified_adrs(base) + if modified: + print("Immutable ADR violation — accepted records cannot be edited:") + for path in modified: + print(f" error: {path}") + print( + "\nReversal or refinement → new ADR-NNNN file. " + "Never edit an existing ADR-*.md." + ) + return 1 + + print(f"ADR immutability OK (base {base})") + return 0 + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--base", + default="origin/main", + help="Git ref to compare against (default: origin/main)", + ) + args = parser.parse_args() + sys.exit(run_check(base=args.base)) + + +if __name__ == "__main__": + main() diff --git a/simulator/tests/test_check_adrs.py b/simulator/tests/test_check_adrs.py new file mode 100644 index 0000000..af90ee4 --- /dev/null +++ b/simulator/tests/test_check_adrs.py @@ -0,0 +1,70 @@ +"""Tests for scripts/check_adrs.py — ADR immutability gate.""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +SCRIPTS = Path(__file__).resolve().parents[2] / "scripts" +sys.path.insert(0, str(SCRIPTS)) + +from check_adrs import find_modified_adrs, run_check # noqa: E402 + + +def _git(cwd: Path, *args: str) -> None: + subprocess.check_call(["git", *args], cwd=cwd, stdout=subprocess.DEVNULL) + + +def test_no_modified_adrs_on_current_branch() -> None: + assert run_check(base="origin/main") == 0 + + +def test_detects_adr_edit_in_sandbox(tmp_path: Path) -> None: + _git(tmp_path, "init") + _git(tmp_path, "config", "user.email", "ci@test") + _git(tmp_path, "config", "user.name", "ci") + adr = tmp_path / "docs/adr" + adr.mkdir(parents=True) + (adr / "ADR-0001-test.md").write_text("v1\n") + _git(tmp_path, "add", ".") + _git(tmp_path, "commit", "-m", "add adr") + _git(tmp_path, "branch", "-M", "main") + (adr / "ADR-0001-test.md").write_text("v2\n") + _git(tmp_path, "checkout", "-b", "feature") + _git(tmp_path, "add", ".") + _git(tmp_path, "commit", "-m", "edit adr") + + monkeypatch = pytest.MonkeyPatch() + monkeypatch.setattr("check_adrs.ROOT", tmp_path) + try: + assert find_modified_adrs("main") == ["docs/adr/ADR-0001-test.md"] + assert run_check(base="main") == 1 + finally: + monkeypatch.undo() + + +def test_new_adr_allowed(tmp_path: Path) -> None: + _git(tmp_path, "init") + _git(tmp_path, "config", "user.email", "ci@test") + _git(tmp_path, "config", "user.name", "ci") + adr = tmp_path / "docs/adr" + adr.mkdir(parents=True) + (adr / "ADR-0001-test.md").write_text("v1\n") + _git(tmp_path, "add", ".") + _git(tmp_path, "commit", "-m", "add adr") + _git(tmp_path, "branch", "-M", "main") + (adr / "ADR-0002-new.md").write_text("new\n") + _git(tmp_path, "checkout", "-b", "feature") + _git(tmp_path, "add", ".") + _git(tmp_path, "commit", "-m", "add adr 2") + + monkeypatch = pytest.MonkeyPatch() + monkeypatch.setattr("check_adrs.ROOT", tmp_path) + try: + assert find_modified_adrs("main") == [] + assert run_check(base="main") == 0 + finally: + monkeypatch.undo() -- 2.54.0 From b4de3ed0769c9b1b10b28d14d9b5606a61ac7a7d Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:59:22 +0100 Subject: [PATCH 6/9] refactor: move scripts/ to bin/ Relocate check_docs, check_adrs, and docs_rules; update CI, tests, and docs. --- .github/pull_request_template.md | 2 +- .github/workflows/release.yaml | 4 ++-- CONTRIBUTING.md | 2 +- {scripts => bin}/check_adrs.py | 0 {scripts => bin}/check_docs.py | 2 +- {scripts => bin}/docs_rules.yaml | 2 +- docs/improvement/2026-07-10-audit.md | 2 +- docs/improvement/README.md | 2 +- docs/improvement/ci-rules.md | 12 ++++++------ prototype/tsconfig.tsbuildinfo | 2 +- simulator/tests/__init__.py | 1 + simulator/tests/test_check_adrs.py | 4 ++-- simulator/tests/test_check_docs.py | 6 +++--- var/t1/.gitkeep | 1 + var/t3/.gitkeep | 1 + 15 files changed, 23 insertions(+), 20 deletions(-) rename {scripts => bin}/check_adrs.py (100%) rename {scripts => bin}/check_docs.py (99%) rename {scripts => bin}/docs_rules.yaml (94%) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index fb2857d..fda7e31 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -11,7 +11,7 @@ Write **caveman-full** by default. See [`CONTRIBUTING.md`](../CONTRIBUTING.md#ca ## Test - [ ] `pytest tests/ -q` (if code touched) -- [ ] `python scripts/check_docs.py` (if docs touched) +- [ ] `python bin/check_docs.py` (if docs touched) - [ ] ADR added if boundary/contract changed ## Links diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index a786b9e..39f7444 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -37,10 +37,10 @@ jobs: run: pip install --quiet pyyaml - name: ADR immutability (no edits to accepted records) - run: python scripts/check_adrs.py --base origin/main + run: python bin/check_adrs.py --base origin/main - name: Documentation rules (links, glossary, readability) - run: python scripts/check_docs.py + run: python bin/check_docs.py verify: name: Verify simulator diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 88164d9..1692b18 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,7 +17,7 @@ The CI pipeline enforces this: | Gate | When it runs | | --- | --- | -| `python scripts/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) | +| `python bin/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) | | `pytest tests/` | Every push/PR (`verify` job) | | `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image | | Smoke flight + DuckDB assertion | Every push/PR | diff --git a/scripts/check_adrs.py b/bin/check_adrs.py similarity index 100% rename from scripts/check_adrs.py rename to bin/check_adrs.py diff --git a/scripts/check_docs.py b/bin/check_docs.py similarity index 99% rename from scripts/check_docs.py rename to bin/check_docs.py index b0f2c4a..90daf29 100644 --- a/scripts/check_docs.py +++ b/bin/check_docs.py @@ -166,7 +166,7 @@ def _line_no(text: str, index: int) -> int: def run_checks(warnings_as_errors: bool = False) -> int: - rules = load_rules(ROOT / "scripts/docs_rules.yaml") + rules = load_rules(RULES_PATH) anchor_cache: dict[Path, set[str]] = {} all_errors: list[str] = [] all_warnings: list[str] = [] diff --git a/scripts/docs_rules.yaml b/bin/docs_rules.yaml similarity index 94% rename from scripts/docs_rules.yaml rename to bin/docs_rules.yaml index 63c4233..938376b 100644 --- a/scripts/docs_rules.yaml +++ b/bin/docs_rules.yaml @@ -1,4 +1,4 @@ -# Documentation rules enforced by CI (scripts/check_docs.py). +# Documentation rules enforced by CI (bin/check_docs.py). # See docs/improvement/ci-rules.md for rationale and PR workflow. markdown_paths: diff --git a/docs/improvement/2026-07-10-audit.md b/docs/improvement/2026-07-10-audit.md index 9b888dc..47c0c8b 100644 --- a/docs/improvement/2026-07-10-audit.md +++ b/docs/improvement/2026-07-10-audit.md @@ -63,7 +63,7 @@ Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify ## Suggested CI checks (for issue #1) -Implemented in [`scripts/check_docs.py`](../../scripts/check_docs.py) — see [ci-rules.md](ci-rules.md). +Implemented in [`bin/check_docs.py`](../../bin/check_docs.py) — see [ci-rules.md](ci-rules.md). ## Files touched diff --git a/docs/improvement/README.md b/docs/improvement/README.md index cfef87f..162ba80 100644 --- a/docs/improvement/README.md +++ b/docs/improvement/README.md @@ -15,7 +15,7 @@ Issues, PRs, and reviews default to **caveman-full** ([`CONTRIBUTING.md`](../../ ## CI automation -Enforced in Gitea Actions (`docs-verify` job). Rules live in [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml); see [ci-rules.md](ci-rules.md). +Enforced in Gitea Actions (`docs-verify` job). Rules live in [`bin/docs_rules.yaml`](../../bin/docs_rules.yaml); see [ci-rules.md](ci-rules.md). Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1). diff --git a/docs/improvement/ci-rules.md b/docs/improvement/ci-rules.md index 13929b1..8887be6 100644 --- a/docs/improvement/ci-rules.md +++ b/docs/improvement/ci-rules.md @@ -2,7 +2,7 @@ Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate). -Implementation: [`scripts/check_docs.py`](../../scripts/check_docs.py) reads [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml). +Implementation: [`bin/check_docs.py`](../../bin/check_docs.py) reads [`bin/docs_rules.yaml`](../../bin/docs_rules.yaml). Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**. @@ -46,13 +46,13 @@ Docs-only changes therefore still get link and glossary validation on `main`; th ## Local run ```bash -python scripts/check_docs.py -python scripts/check_adrs.py --base origin/main -python scripts/check_docs.py --warnings-as-errors # treat section-link warnings as failures +python bin/check_docs.py +python bin/check_adrs.py --base origin/main +python bin/check_docs.py --warnings-as-errors # treat section-link warnings as failures ``` ## Changing rules -1. Edit `scripts/docs_rules.yaml` (paths, banned targets, preferred anchors). -2. Adjust `scripts/check_docs.py` only when adding a new rule *type*. +1. Edit `bin/docs_rules.yaml` (paths, banned targets, preferred anchors). +2. Adjust `bin/check_docs.py` only when adding a new rule *type*. 3. Add a row to the table above and mention it in the audit/improvement record if the change is significant. diff --git a/prototype/tsconfig.tsbuildinfo b/prototype/tsconfig.tsbuildinfo index 6953ed3..6f80c2e 100644 --- a/prototype/tsconfig.tsbuildinfo +++ b/prototype/tsconfig.tsbuildinfo @@ -1 +1 @@ -{"root":["./src/App.tsx","./src/main.tsx","./src/sim.ts"],"version":"5.9.3"} \ No newline at end of file +{"root":["./src/App.tsx","./src/live.ts","./src/main.tsx","./src/sim.ts","./src/vite-env.d.ts"],"version":"5.9.3"} \ No newline at end of file diff --git a/simulator/tests/__init__.py b/simulator/tests/__init__.py index e69de29..8b13789 100644 --- a/simulator/tests/__init__.py +++ b/simulator/tests/__init__.py @@ -0,0 +1 @@ + diff --git a/simulator/tests/test_check_adrs.py b/simulator/tests/test_check_adrs.py index af90ee4..f276388 100644 --- a/simulator/tests/test_check_adrs.py +++ b/simulator/tests/test_check_adrs.py @@ -1,4 +1,4 @@ -"""Tests for scripts/check_adrs.py — ADR immutability gate.""" +"""Tests for bin/check_adrs.py — ADR immutability gate.""" from __future__ import annotations @@ -8,7 +8,7 @@ from pathlib import Path import pytest -SCRIPTS = Path(__file__).resolve().parents[2] / "scripts" +SCRIPTS = Path(__file__).resolve().parents[2] / "bin" sys.path.insert(0, str(SCRIPTS)) from check_adrs import find_modified_adrs, run_check # noqa: E402 diff --git a/simulator/tests/test_check_docs.py b/simulator/tests/test_check_docs.py index f079da3..540f5bd 100644 --- a/simulator/tests/test_check_docs.py +++ b/simulator/tests/test_check_docs.py @@ -1,4 +1,4 @@ -"""Tests for scripts/check_docs.py — documentation CI rules.""" +"""Tests for bin/check_docs.py — documentation CI rules.""" from __future__ import annotations @@ -7,7 +7,7 @@ from pathlib import Path import pytest -SCRIPTS = Path(__file__).resolve().parents[2] / "scripts" +SCRIPTS = Path(__file__).resolve().parents[2] / "bin" sys.path.insert(0, str(SCRIPTS)) from check_docs import ( # noqa: E402 @@ -57,7 +57,7 @@ def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) - docs.mkdir() (docs / "00-glossary.md").write_text("# Glossary\n\n## Data & storage\n") (docs / "bad.md").write_text("See [missing](missing.md).\n") - rules = tmp_path / "scripts" + rules = tmp_path / "bin" rules.mkdir() (rules / "docs_rules.yaml").write_text( "markdown_paths:\n - docs/**/*.md\n" diff --git a/var/t1/.gitkeep b/var/t1/.gitkeep index e69de29..8b13789 100644 --- a/var/t1/.gitkeep +++ b/var/t1/.gitkeep @@ -0,0 +1 @@ + diff --git a/var/t3/.gitkeep b/var/t3/.gitkeep index e69de29..8b13789 100644 --- a/var/t3/.gitkeep +++ b/var/t3/.gitkeep @@ -0,0 +1 @@ + -- 2.54.0 From df04736336f1bfa119749d219a6675afa1c023af Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 10:59:29 +0100 Subject: [PATCH 7/9] chore: drop unrelated files from bin/ move commit --- prototype/tsconfig.tsbuildinfo | 2 +- simulator/tests/__init__.py | 1 - var/t1/.gitkeep | 1 - var/t3/.gitkeep | 1 - 4 files changed, 1 insertion(+), 4 deletions(-) diff --git a/prototype/tsconfig.tsbuildinfo b/prototype/tsconfig.tsbuildinfo index 6f80c2e..6953ed3 100644 --- a/prototype/tsconfig.tsbuildinfo +++ b/prototype/tsconfig.tsbuildinfo @@ -1 +1 @@ -{"root":["./src/App.tsx","./src/live.ts","./src/main.tsx","./src/sim.ts","./src/vite-env.d.ts"],"version":"5.9.3"} \ No newline at end of file +{"root":["./src/App.tsx","./src/main.tsx","./src/sim.ts"],"version":"5.9.3"} \ No newline at end of file diff --git a/simulator/tests/__init__.py b/simulator/tests/__init__.py index 8b13789..e69de29 100644 --- a/simulator/tests/__init__.py +++ b/simulator/tests/__init__.py @@ -1 +0,0 @@ - diff --git a/var/t1/.gitkeep b/var/t1/.gitkeep index 8b13789..e69de29 100644 --- a/var/t1/.gitkeep +++ b/var/t1/.gitkeep @@ -1 +0,0 @@ - diff --git a/var/t3/.gitkeep b/var/t3/.gitkeep index 8b13789..e69de29 100644 --- a/var/t3/.gitkeep +++ b/var/t3/.gitkeep @@ -1 +0,0 @@ - -- 2.54.0 From 826cef0c3eb9f164e8b6378c571432a5caa0a92a Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 11:02:55 +0100 Subject: [PATCH 8/9] fix: drop PyYAML dep so verify job pytest passes Store doc rules in bin/docs_rules.json (stdlib json). test_check_docs no longer fails in the simulator job that only installs simulator/requirements.txt. --- .github/workflows/release.yaml | 3 -- bin/check_docs.py | 13 ++------- bin/docs_rules.json | 42 ++++++++++++++++++++++++++ bin/docs_rules.yaml | 47 ------------------------------ docs/improvement/README.md | 2 +- docs/improvement/ci-rules.md | 4 +-- simulator/tests/test_check_docs.py | 22 ++++++++------ 7 files changed, 61 insertions(+), 72 deletions(-) create mode 100644 bin/docs_rules.json delete mode 100644 bin/docs_rules.yaml diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 39f7444..060406d 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -33,9 +33,6 @@ jobs: - name: Fetch base branch for ADR immutability run: git fetch --depth=1 origin main - - name: Install checker dependencies - run: pip install --quiet pyyaml - - name: ADR immutability (no edits to accepted records) run: python bin/check_adrs.py --base origin/main diff --git a/bin/check_docs.py b/bin/check_docs.py index 90daf29..1cb64b0 100644 --- a/bin/check_docs.py +++ b/bin/check_docs.py @@ -4,17 +4,13 @@ from __future__ import annotations import argparse +import json import re import sys from pathlib import Path -try: - import yaml -except ImportError: # pragma: no cover - stdlib fallback for minimal envs - yaml = None # type: ignore[assignment] - ROOT = Path(__file__).resolve().parents[1] -RULES_PATH = Path(__file__).with_name("docs_rules.yaml") +RULES_PATH = Path(__file__).with_name("docs_rules.json") LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)") BARE_SEE_RE = re.compile(r"(? dict: - text = path.read_text(encoding="utf-8") - if yaml is not None: - return yaml.safe_load(text) - raise RuntimeError("PyYAML is required to load docs_rules.yaml") + return json.loads(path.read_text(encoding="utf-8")) def slug_heading(text: str) -> str: diff --git a/bin/docs_rules.json b/bin/docs_rules.json new file mode 100644 index 0000000..38aa906 --- /dev/null +++ b/bin/docs_rules.json @@ -0,0 +1,42 @@ +{ + "markdown_paths": [ + "docs/**/*.md", + "README.md", + "CONTRIBUTING.md", + "AGENTS.md", + "simulator/README.md", + "prototype/README.md", + "infra/**/README.md" + ], + "glossary": { + "file": "docs/00-glossary.md" + }, + "banned_link_targets": [ + "03-storage-design.md" + ], + "glossary_section_anchors": [ + "data--storage", + "swarm--robotics", + "infrastructure--delivery" + ], + "preferred_term_anchors": [ + "t0--hot", + "t1--warm", + "t2--shared", + "t3--warehouse", + "t4--dev", + "source-of-truth", + "architecture-layer-1--ingestion", + "architecture-layer-2--storage-and-transform", + "architecture-layer-3--serving-and-sync", + "pipeline-stage", + "bulk-sync", + "offload", + "rclone", + "adr", + "asr" + ], + "readability": { + "forbid_bare_see_refs": true + } +} diff --git a/bin/docs_rules.yaml b/bin/docs_rules.yaml deleted file mode 100644 index 938376b..0000000 --- a/bin/docs_rules.yaml +++ /dev/null @@ -1,47 +0,0 @@ -# Documentation rules enforced by CI (bin/check_docs.py). -# See docs/improvement/ci-rules.md for rationale and PR workflow. - -markdown_paths: - - docs/**/*.md - - README.md - - CONTRIBUTING.md - - AGENTS.md - - simulator/README.md - - prototype/README.md - - infra/**/README.md - -glossary: - file: docs/00-glossary.md - -# Relative link targets that must never reappear (known regressions). -banned_link_targets: - - 03-storage-design.md - -# Glossary links to these section headers are allowed but reported as warnings -# when a term-specific ### anchor exists (see preferred_term_anchors). -glossary_section_anchors: - - data--storage - - swarm--robotics - - infrastructure--delivery - -# First-use vocabulary should link to these ### anchors, not section headers. -preferred_term_anchors: - - t0--hot - - t1--warm - - t2--shared - - t3--warehouse - - t4--dev - - source-of-truth - - architecture-layer-1--ingestion - - architecture-layer-2--storage-and-transform - - architecture-layer-3--serving-and-sync - - pipeline-stage - - bulk-sync - - offload - - rclone - - adr - - asr - -readability: - # Fail when prose uses bare "see 05" instead of a markdown link. - forbid_bare_see_refs: true diff --git a/docs/improvement/README.md b/docs/improvement/README.md index 162ba80..3ccac0b 100644 --- a/docs/improvement/README.md +++ b/docs/improvement/README.md @@ -15,7 +15,7 @@ Issues, PRs, and reviews default to **caveman-full** ([`CONTRIBUTING.md`](../../ ## CI automation -Enforced in Gitea Actions (`docs-verify` job). Rules live in [`bin/docs_rules.yaml`](../../bin/docs_rules.yaml); see [ci-rules.md](ci-rules.md). +Enforced in Gitea Actions (`docs-verify` job). Rules live in [`bin/docs_rules.json`](../../bin/docs_rules.json); see [ci-rules.md](ci-rules.md). Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1). diff --git a/docs/improvement/ci-rules.md b/docs/improvement/ci-rules.md index 8887be6..d472712 100644 --- a/docs/improvement/ci-rules.md +++ b/docs/improvement/ci-rules.md @@ -2,7 +2,7 @@ Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate). -Implementation: [`bin/check_docs.py`](../../bin/check_docs.py) reads [`bin/docs_rules.yaml`](../../bin/docs_rules.yaml). +Implementation: [`bin/check_docs.py`](../../bin/check_docs.py) reads [`bin/docs_rules.json`](../../bin/docs_rules.json). Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**. @@ -53,6 +53,6 @@ python bin/check_docs.py --warnings-as-errors # treat section-link warnings as ## Changing rules -1. Edit `bin/docs_rules.yaml` (paths, banned targets, preferred anchors). +1. Edit `bin/docs_rules.json` (paths, banned targets, preferred anchors). 2. Adjust `bin/check_docs.py` only when adding a new rule *type*. 3. Add a row to the table above and mention it in the audit/improvement record if the change is significant. diff --git a/simulator/tests/test_check_docs.py b/simulator/tests/test_check_docs.py index 540f5bd..4460667 100644 --- a/simulator/tests/test_check_docs.py +++ b/simulator/tests/test_check_docs.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import sys from pathlib import Path @@ -59,15 +60,18 @@ def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) - (docs / "bad.md").write_text("See [missing](missing.md).\n") rules = tmp_path / "bin" rules.mkdir() - (rules / "docs_rules.yaml").write_text( - "markdown_paths:\n - docs/**/*.md\n" - "glossary:\n file: docs/00-glossary.md\n" - "banned_link_targets: []\n" - "glossary_section_anchors: []\n" - "preferred_term_anchors: []\n" - "readability:\n forbid_bare_see_refs: false\n" + (rules / "docs_rules.json").write_text( + json.dumps( + { + "markdown_paths": ["docs/**/*.md"], + "glossary": {"file": "docs/00-glossary.md"}, + "banned_link_targets": [], + "glossary_section_anchors": [], + "preferred_term_anchors": [], + "readability": {"forbid_bare_see_refs": False}, + } + ) ) - checker = SCRIPTS / "check_docs.py" monkeypatch.setattr("check_docs.ROOT", tmp_path) - monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.yaml") + monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.json") assert run_checks() == 1 -- 2.54.0 From 302de4024d297322813d7171ae133b28283dc794 Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Fri, 10 Jul 2026 12:18:50 +0100 Subject: [PATCH 9/9] docs: move live demo links to PR #3 Drop duplicate external URLs from design journey; small PR #3 owns them. --- docs/12-design-journey.md | 15 --------------- docs/improvement/2026-07-10-audit.md | 4 +--- 2 files changed, 1 insertion(+), 18 deletions(-) diff --git a/docs/12-design-journey.md b/docs/12-design-journey.md index 5ce2a3e..ee107fe 100644 --- a/docs/12-design-journey.md +++ b/docs/12-design-journey.md @@ -130,13 +130,6 @@ coming back to foundations DevOps and sysadmins already trust. > - [`simulator/explorer/server.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/explorer/server.py#L111-L131) — the query path a peer or an engineer hits identically > - [`simulator/tests/test_sql_gate.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/tests/test_sql_gate.py#L13-L35) — the gate is safety-critical, so it is tested first -The explorer is running, not just described: the data-plane view (partition tree -and read-only SQL console) is live at -[swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/) -(same access as this repository and [swarm.produktor.io](https://swarm.produktor.io/)). -Browse the Hive layout, run `SELECT` queries against the live lake, and see the -same statement gate a peer drone would hit in flight. - ## 8. Replication and offload Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH @@ -180,14 +173,6 @@ scanning the whole lake on every query does not scale, so the scan is > - [`simulator/monitoring/grafana/dashboards/`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/monitoring/grafana/dashboards) — the fleet and platform dashboards > - [`.github/workflows/release.yaml`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/.github/workflows/release.yaml) — tests, smoke flight, Trivy, semantic release -The dashboard is running, not just described: **Swarm Fleet — generated data** is -live at -[swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s) -(same access as this repository, [swarm.produktor.io](https://swarm.produktor.io/), -and [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/)). -Telemetry rows, detections, pose frames, battery, and link quality — all derived -from the Parquet lake via SQL, not from agents on the drones. - ## 10. Working method A note on how this is meant to be built, not just what. Working together needs a diff --git a/docs/improvement/2026-07-10-audit.md b/docs/improvement/2026-07-10-audit.md index 47c0c8b..ead72e9 100644 --- a/docs/improvement/2026-07-10-audit.md +++ b/docs/improvement/2026-07-10-audit.md @@ -41,9 +41,7 @@ Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify ### Live demos (design journey) -- [swarm.produktor.io](https://swarm.produktor.io/) — visual prototype -- [swarm-data-explorer.produktor.io](https://swarm-data-explorer.produktor.io/) — SQL explorer -- [swarm-analalytics.produktor.io](https://swarm-analalytics.produktor.io/d/swarm-fleet/swarm-fleet-e28094-generated-data?orgId=1&refresh=5s) — Grafana fleet dashboard +Tracked in [PR #3](https://git.produktor.io/eSlider/swarm-house/pulls/3) — explorer + Grafana links in `12-design-journey.md` §7/§9. Prototype link already on `main` §2. ### README / LICENSE -- 2.54.0