Author SHA1 Message Date
eSlider 87551fc1c1 docs: put MinIO first-class on ground, add executive overview
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
Treat MinIO as existing stack infrastructure (default T3 warehouse, not
on the drone), add a one-page executive map, and record placement as an
open team question rather than a rip-and-replace.
2026-07-17 12:22:21 +01:00
eSlider 2003ff69fb chore: ignore local tooling and prototype-3d WIP
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 17s
CI & Release / Semantic Release (push) Successful in 6s
Keep .cursor, .grok, and prototype-3d on disk without polluting git status.
2026-07-17 12:17:06 +01:00
eSlider 2a6b50929c docs: align wire-spec and mark implemented vs proposed
CI & Release / Verify simulator (push) Successful in 18s
CI & Release / Trivy scan (push) Successful in 18s
CI & Release / Semantic Release (push) Successful in 6s
Keep the 45-byte pose frame consistent across docs, simulator, and
prototype, clarify MinIO is not peer sync, and state clearly that the
repo is a from-scratch platform sketch with options rather than mandates.
2026-07-17 12:16:39 +01:00
42 changed files with 367 additions and 1069 deletions
-8
View File
@@ -1,8 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: Design docs
url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/12-design-journey.md
about: Start with the design journey, then open an issue here.
- name: Open questions
url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/09-open-questions.md
about: Known unknowns — check before duplicating.
-24
View File
@@ -1,24 +0,0 @@
---
name: Change / task
about: Default issue template — caveman style
title: ""
labels: []
---
Write **caveman-full**: short lines, fragments OK, no filler. See [`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default).
## What
<!-- One line. What need fix or build. -->
## Why
<!-- Problem or goal. Skip if obvious from title. -->
## Done when
- [ ]
## Notes
<!-- Links, ADR, open question §. Optional. -->
-19
View File
@@ -1,19 +0,0 @@
Write **caveman-full** by default. See [`CONTRIBUTING.md`](../CONTRIBUTING.md#caveman-style-default).
## What
<!-- What changed. Bullets OK. -->
## Why
<!-- Problem this fixes or design gap closed. -->
## Test
- [ ] `pytest tests/ -q` (if code touched)
- [ ] `python bin/check_docs.py` (if docs touched)
- [ ] ADR added if boundary/contract changed
## Links
<!-- Closes / Relates to #issue. Optional demo URL. -->
+5 -31
View File
@@ -1,5 +1,4 @@
# Swarm House - Gitea Actions # Swarm House - Gitea Actions
# docs-verify: markdown links, glossary anchors, readability rules (every push/PR)
# verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR) # verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR)
# scan: Trivy vulnerability + misconfiguration scan # scan: Trivy vulnerability + misconfiguration scan
# release: semantic version bump (conventional commits), tag, Gitea release # release: semantic version bump (conventional commits), tag, Gitea release
@@ -18,27 +17,6 @@ on:
branches: [main] branches: [main]
jobs: jobs:
docs-verify:
name: Verify documentation
runs-on: ubuntu-latest
container: python:3.12-bookworm
steps:
- name: Checkout
run: |
git init .
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
git fetch --depth=50 origin "${{ gitea.ref }}"
git checkout FETCH_HEAD
- name: Fetch base branch for ADR immutability
run: git fetch --depth=1 origin main
- name: ADR immutability (no edits to accepted records)
run: python bin/check_adrs.py --base origin/main
- name: Documentation rules (links, glossary, readability)
run: python bin/check_docs.py
verify: verify:
name: Verify simulator name: Verify simulator
runs-on: ubuntu-latest runs-on: ubuntu-latest
@@ -46,10 +24,8 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
run: | run: |
git init . git clone --depth=50 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git .
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git git checkout ${{ gitea.sha }}
git fetch --depth=50 origin "${{ gitea.ref }}"
git checkout FETCH_HEAD
- name: Install dependencies - name: Install dependencies
run: pip install --quiet -r simulator/requirements.txt run: pip install --quiet -r simulator/requirements.txt
@@ -85,10 +61,8 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
run: | run: |
git init . git clone --depth=1 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git .
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git git checkout ${{ gitea.sha }}
git fetch --depth=1 origin "${{ gitea.ref }}"
git checkout FETCH_HEAD
- name: Install Trivy - name: Install Trivy
run: | run: |
@@ -107,7 +81,7 @@ jobs:
name: Semantic Release name: Semantic Release
runs-on: ubuntu-latest runs-on: ubuntu-latest
container: python:3.12-bookworm container: python:3.12-bookworm
needs: [docs-verify, verify, scan] needs: [verify, scan]
if: gitea.event_name == 'push' if: gitea.event_name == 'push'
permissions: permissions:
contents: write contents: write
+5
View File
@@ -13,3 +13,8 @@ dist/
.terraform/ .terraform/
*.tfstate *.tfstate
*.tfstate.* *.tfstate.*
# Local tooling / WIP (keep on disk, never commit)
.cursor/
.grok/
prototype-3d/
+1 -23
View File
@@ -24,23 +24,6 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development workflow and TDD re
## Conventions ## Conventions
### Caveman style (default for collaboration)
**Issues, pull requests, and code reviews** use **caveman-full** by default:
- Short sentences or fragments; drop articles and filler when clear
- Lead with fact, verdict, or ask — not preamble
- Bullets over paragraphs; one idea per line
- Keep technical terms, acronyms, paths, and code exact
- English only; no identifying references (same as all repo prose)
README tiers: root [`README.md`](README.md) = **caveman-lite** (full sentences, less filler);
component READMEs = **caveman-full**. Design docs in `docs/` stay readable prose unless
editing for consistency.
Templates: [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/),
[`.github/pull_request_template.md`](.github/pull_request_template.md).
- Markdown: ATX headings, fenced code blocks with language tags, pipe tables. - Markdown: ATX headings, fenced code blocks with language tags, pipe tables.
- Python: PEP 8, type hints, pure functions where possible, no unnecessary classes. - Python: PEP 8, type hints, pure functions where possible, no unnecessary classes.
- TypeScript: strict mode, functional components and hooks, no extra UI libraries. - TypeScript: strict mode, functional components and hooks, no extra UI libraries.
@@ -61,12 +44,7 @@ Status / Context / Options / Decision / Consequences format. Start from
When a change alters a boundary, contract, or trade-off, add an ADR (copy When a change alters a boundary, contract, or trade-off, add an ADR (copy
[`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the [`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the
index) and cross-link it from the affected design doc. A later reversal gets a index) and cross-link it from the affected design doc. A later reversal gets a
new ADR that supersedes the old one — **never edit an Accepted record**. new ADR that supersedes the old one — never edit an Accepted record.
**Immutable:** every `docs/adr/ADR-*.md` file is frozen once on `main`. CI rejects
any change that modifies an existing ADR. Updates go in a new numbered ADR only.
The index ([`docs/adr/README.md`](docs/adr/README.md)) and [`TEMPLATE.md`](docs/adr/TEMPLATE.md)
may still change; decision bodies do not.
## Runtime & operations (sim environment) ## Runtime & operations (sim environment)
+1 -37
View File
@@ -17,7 +17,6 @@ The CI pipeline enforces this:
| Gate | When it runs | | Gate | When it runs |
| --- | --- | | --- | --- |
| `python bin/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) |
| `pytest tests/` | Every push/PR (`verify` job) | | `pytest tests/` | Every push/PR (`verify` job) |
| `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image | | `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image |
| Smoke flight + DuckDB assertion | Every push/PR | | Smoke flight + DuckDB assertion | Every push/PR |
@@ -77,47 +76,12 @@ test: cover broadcast frame roundtrip
Never include AI tool references, `Co-authored-by` trailers for assistants, or Never include AI tool references, `Co-authored-by` trailers for assistants, or
identifying names of people, companies, or locations. identifying names of people, companies, or locations.
## Caveman style (default)
Use **caveman-full** in **issues**, **pull requests**, and **reviews** unless the
author explicitly asks otherwise.
| Do | Skip |
| --- | --- |
| Short lines, fragments OK | Long preamble ("I wanted to start by saying…") |
| Lead with what / why / verdict | Filler ("just", "basically", "kind of") |
| Bullets, checklists | Repeated context already in diff or issue |
| Exact terms, paths, code | Invented abbreviations or vague "the thing" |
**Examples**
| Fluffy | Caveman |
| --- | --- |
| I think we should probably consider adding a link checker to CI because links might break. | Add CI link checker. Broken relative links fail merge. |
| This pull request implements the documentation improvements that were discussed in the audit. | Docs audit: glossary T0T4, fix ground README link, rsync wording. |
| LGTM but maybe we could think about whether the anchor format is consistent? | Merge OK. Fix glossary anchors in follow-up (#1). |
README intensity: root = caveman-lite; `simulator/`, `prototype/`, `infra/`, `docs/adr/` READMEs =
caveman-full. Numbered design docs in `docs/` stay full prose.
Gitea pre-fills [issue](.github/ISSUE_TEMPLATE/default.md) and
[PR](.github/pull_request_template.md) templates with this style.
### Reviews
Default review voice = caveman-full:
1. **Verdict first** — merge / wait / no, or approve / request changes
2. **Findings** — bullet per blocker or nit; file + line when useful
3. **Scope** — say if issue AC met, partial, or out of scope
4. No essay; link ADR, issue, or doc section instead of restating design
## Pull request checklist ## Pull request checklist
- [ ] Tests added or updated; `pytest tests/ -q` passes locally - [ ] Tests added or updated; `pytest tests/ -q` passes locally
- [ ] `docker build` in `simulator/` succeeds (test stage included) - [ ] `docker build` in `simulator/` succeeds (test stage included)
- [ ] Documentation updated if behaviour or boundaries changed - [ ] Documentation updated if behaviour or boundaries changed
- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/)) — never edit an existing `ADR-*.md`; supersede with a new number - [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/))
- [ ] English only; no identifying references - [ ] English only; no identifying references
- [ ] Conventional commit message - [ ] Conventional commit message
+29 -12
View File
@@ -2,14 +2,11 @@
**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective. **An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective.
Fleet flies autonomous. No internet uplink, no external access. Data stays in the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. Swarm exchanges just enough state to coordinate; everything else stays local until a ground warehouse receives it after landing — reproducibly, securely, at scale. A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale.
This repository covers storage layout, sync strategy, network trust model, CI/CD chain, and the simulation environment that tests it without real hardware. This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
> **Restricted use.** Third parties may not use, copy, modify, or distribute this **New here?** Start with the one-pager: [00 — Executive overview](docs/00-executive.md) (includes how **MinIO** fits). Then the [design journey](docs/12-design-journey.md) if you want the narrative.
> source code, documentation, or related materials without **explicit written
> permission** from the copyright holder. No license is granted by default. See
> [LICENSE](LICENSE) for the full terms.
## Design principles ## Design principles
@@ -22,26 +19,46 @@ This repository covers storage layout, sync strategy, network trust model, CI/CD
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it. 7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies. 8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
## Implemented now vs proposed next
Honest map so a reader knows what runs today versus what is design intent.
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
| --- | --- | --- |
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, **MinIO as T3** | k3s warehouse with MinIO, GitOps overlays, post-flight mirror |
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
## Documentation ## Documentation
| Document | Contents | | Document | Contents |
| --- | --- | | --- | --- |
| [00 — Executive](docs/00-executive.md) | **Start here** — one-page goal, sync rule, MinIO stance |
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout | | [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions | | [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes | | [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage floors (T0T4), partitioning, hot-write path | | [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path |
| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard | | [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard |
| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport | | [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport |
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures | | [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry | | [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex | | [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers | | [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers (incl. MinIO placement) |
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on | | [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases | | [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code | | [12 — Design journey](docs/12-design-journey.md) | Narrative walk-through linking into the code |
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made | | [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
New? Start with the [design journey](docs/12-design-journey.md) — story plus code links. Principles = *what*; [ADRs](docs/adr/) = *why and when*. New here? Read the [executive overview](docs/00-executive.md) first, then the [design journey](docs/12-design-journey.md) if you want the story. The design principles above are the *what*; the [ADRs](docs/adr/) are the *why and when*.
## Runnable parts ## Runnable parts
@@ -50,10 +67,10 @@ New? Start with the [design journey](docs/12-design-journey.md) — story plus c
| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB | | [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG | | [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG |
| [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible | | [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible |
| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + [T1→T3](docs/03-data-platform.md#storage-floors-tiers) offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider | | [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + T1→T3 offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider |
| [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize | | [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize |
Shared runtime data (gitignored): [`var/t1/`](docs/00-glossary.md#t1--warm) (live lake, [T1](docs/00-glossary.md#t1--warm)), [`var/t3/`](docs/00-glossary.md#t3--warehouse) (warehouse, [T3](docs/00-glossary.md#t3--warehouse)). See [`CONTRIBUTING.md`](CONTRIBUTING.md). Shared runtime data (gitignored): `var/t1/` (live lake), `var/t3/` (warehouse). See [`CONTRIBUTING.md`](CONTRIBUTING.md).
## Quick start ## Quick start
-75
View File
@@ -1,75 +0,0 @@
#!/usr/bin/env python3
"""Reject modifications to accepted Architecture Decision Records."""
from __future__ import annotations
import argparse
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
ADR_PREFIX = "docs/adr/ADR-"
def _git(*args: str) -> str:
return subprocess.check_output(
["git", *args],
cwd=ROOT,
text=True,
stderr=subprocess.DEVNULL,
).strip()
def _exists_on(base: str, path: str) -> bool:
try:
_git("cat-file", "-e", f"{base}:{path}")
return True
except subprocess.CalledProcessError:
return False
def find_modified_adrs(base: str) -> list[str]:
try:
diff_out = _git("diff", "--name-only", f"{base}...HEAD", "--", "docs/adr/")
except subprocess.CalledProcessError:
return []
modified: list[str] = []
for rel in diff_out.splitlines():
if not rel.startswith(ADR_PREFIX) or not rel.endswith(".md"):
continue
if _exists_on(base, rel):
modified.append(rel)
return sorted(modified)
def run_check(base: str = "origin/main") -> int:
modified = find_modified_adrs(base)
if modified:
print("Immutable ADR violation — accepted records cannot be edited:")
for path in modified:
print(f" error: {path}")
print(
"\nReversal or refinement → new ADR-NNNN file. "
"Never edit an existing ADR-*.md."
)
return 1
print(f"ADR immutability OK (base {base})")
return 0
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--base",
default="origin/main",
help="Git ref to compare against (default: origin/main)",
)
args = parser.parse_args()
sys.exit(run_check(base=args.base))
if __name__ == "__main__":
main()
-212
View File
@@ -1,212 +0,0 @@
#!/usr/bin/env python3
"""Validate markdown links, glossary anchors, and readability rules."""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
RULES_PATH = Path(__file__).with_name("docs_rules.json")
LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)")
BARE_SEE_RE = re.compile(r"(?<![\[(/])see\s+(\d{2})(?![\w-])", re.IGNORECASE)
SKIP_SCHEMES = ("http://", "https://", "mailto:", "tel:", "#")
def load_rules(path: Path = RULES_PATH) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def slug_heading(text: str) -> str:
"""Anchor slug used in this repository's glossary links."""
text = text.strip()
text = re.sub(r"\s&\s", "", text)
if "" in text:
left, right = text.split("", 1)
return f"{_slug_part(left)}--{_slug_part(right)}"
return _slug_part(text)
def _slug_part(text: str) -> str:
slug = text.strip().lower()
slug = re.sub(r"[^\w\s-]", "", slug)
slug = re.sub(r"\s+", "-", slug)
return slug
def collect_anchors(md_path: Path) -> set[str]:
anchors: set[str] = set()
for line in md_path.read_text(encoding="utf-8").splitlines():
if line.startswith("##"):
anchors.add(slug_heading(line.lstrip("#").strip()))
return anchors
def expand_markdown_paths(patterns: list[str]) -> list[Path]:
files: set[Path] = set()
for pattern in patterns:
for path in ROOT.glob(pattern):
if path.is_file() and path.suffix == ".md":
files.add(path)
return sorted(files)
def split_link_target(raw: str) -> tuple[str, str | None]:
target = raw.strip()
if target.startswith("<") and target.endswith(">"):
target = target[1:-1]
if "#" in target:
path_part, fragment = target.split("#", 1)
return path_part, fragment or None
return target, None
def resolve_path(source: Path, link_path: str) -> Path | None:
if not link_path or link_path.startswith(SKIP_SCHEMES):
return None
candidate = (source.parent / link_path).resolve()
try:
candidate.relative_to(ROOT.resolve())
except ValueError:
return None
return candidate
def is_glossary_link(link_path: str, rules: dict) -> bool:
glossary = rules["glossary"]["file"].replace("\\", "/")
normalized = link_path.replace("\\", "/")
return (
normalized == glossary
or normalized.endswith("/" + glossary)
or normalized.endswith(glossary.split("/")[-1])
)
def check_file(
md_path: Path,
rules: dict,
anchor_cache: dict[Path, set[str]],
) -> tuple[list[str], list[str]]:
errors: list[str] = []
warnings: list[str] = []
rel = md_path.relative_to(ROOT)
text = md_path.read_text(encoding="utf-8")
if rules.get("readability", {}).get("forbid_bare_see_refs"):
prose = re.sub(r"`[^`]*`", "", text)
for match in BARE_SEE_RE.finditer(prose):
errors.append(
f"{rel}:{_line_no(text, match.start())}: bare cross-ref "
f"'see {match.group(1)}' — use a markdown link"
)
for match in LINK_RE.finditer(text):
raw = match.group(1)
if any(raw.startswith(s) for s in SKIP_SCHEMES):
continue
link_path, fragment = split_link_target(raw)
if not link_path.endswith(".md"):
continue
for banned in rules.get("banned_link_targets", []):
if banned in link_path.replace("\\", "/"):
errors.append(
f"{rel}:{_line_no(text, match.start())}: banned link target "
f"'{link_path}'"
)
target = resolve_path(md_path, link_path)
if target is None:
errors.append(
f"{rel}:{_line_no(text, match.start())}: link escapes repo or "
f"is invalid: {raw}"
)
continue
if not target.is_file():
errors.append(
f"{rel}:{_line_no(text, match.start())}: broken link "
f"{link_path} -> {target.relative_to(ROOT)}"
)
continue
if fragment:
if target not in anchor_cache:
anchor_cache[target] = collect_anchors(target)
if fragment not in anchor_cache[target]:
errors.append(
f"{rel}:{_line_no(text, match.start())}: unknown anchor "
f"#{fragment} in {target.relative_to(ROOT)}"
)
if fragment and is_glossary_link(link_path, rules):
section_anchors = set(rules.get("glossary_section_anchors", []))
if fragment in section_anchors:
warnings.append(
f"{rel}:{_line_no(text, match.start())}: glossary section "
f"link #{fragment} — prefer a ### term anchor where one exists"
)
return errors, warnings
def _line_no(text: str, index: int) -> int:
return text.count("\n", 0, index) + 1
def run_checks(warnings_as_errors: bool = False) -> int:
rules = load_rules(RULES_PATH)
anchor_cache: dict[Path, set[str]] = {}
all_errors: list[str] = []
all_warnings: list[str] = []
glossary_path = ROOT / rules["glossary"]["file"]
anchor_cache[glossary_path] = collect_anchors(glossary_path)
for md_path in expand_markdown_paths(rules["markdown_paths"]):
errors, warnings = check_file(md_path, rules, anchor_cache)
all_errors.extend(errors)
all_warnings.extend(warnings)
if all_warnings:
print("Documentation warnings:")
for msg in sorted(all_warnings):
print(f" warning: {msg}")
print()
if all_errors:
print("Documentation errors:")
for msg in sorted(all_errors):
print(f" error: {msg}")
print(f"\n{len(all_errors)} error(s), {len(all_warnings)} warning(s)")
return 1
if warnings_as_errors and all_warnings:
print(f"\n{len(all_warnings)} warning(s) treated as errors")
return 1
print(
f"Documentation checks passed "
f"({len(expand_markdown_paths(rules['markdown_paths']))} files, "
f"{len(all_warnings)} warning(s))"
)
return 0
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--warnings-as-errors",
action="store_true",
help="Exit non-zero when section-link warnings are present",
)
args = parser.parse_args()
sys.exit(run_checks(warnings_as_errors=args.warnings_as_errors))
if __name__ == "__main__":
main()
-42
View File
@@ -1,42 +0,0 @@
{
"markdown_paths": [
"docs/**/*.md",
"README.md",
"CONTRIBUTING.md",
"AGENTS.md",
"simulator/README.md",
"prototype/README.md",
"infra/**/README.md"
],
"glossary": {
"file": "docs/00-glossary.md"
},
"banned_link_targets": [
"03-storage-design.md"
],
"glossary_section_anchors": [
"data--storage",
"swarm--robotics",
"infrastructure--delivery"
],
"preferred_term_anchors": [
"t0--hot",
"t1--warm",
"t2--shared",
"t3--warehouse",
"t4--dev",
"source-of-truth",
"architecture-layer-1--ingestion",
"architecture-layer-2--storage-and-transform",
"architecture-layer-3--serving-and-sync",
"pipeline-stage",
"bulk-sync",
"offload",
"rclone",
"adr",
"asr"
],
"readability": {
"forbid_bare_see_refs": true
}
}
+61
View File
@@ -0,0 +1,61 @@
# 00 — Executive overview
A one-page map of the proposal. Details and trade-offs live in the linked docs.
Nothing here is a mandate: it is a from-scratch platform sketch the team can
reshape once real constraints are on the table.
## Goal
Build an **on-prem, air-gapped data platform** for an autonomous drone swarm:
accumulate sensor and detection data on each unit, share only what peers need in
flight, and offload complete flights to a ground warehouse for replay and
training.
## Constraints (given)
- No internet / no external access in flight; data stays inside the system.
- Each drone is autonomous; mesh links are intermittent — **no swarm-wide orchestrator**.
- On-board today: **Docker Compose**, **DuckDB**, YOLO-like video analytics; **Parquet** under evaluation.
- **MinIO is already in the stack** — exact placement not fully known yet (see below).
- Ground side can run lightweight Kubernetes; this proposal keeps k3s **on the ground only**.
## Data contract (proposed)
| What | Where | Crosses the air? |
| --- | --- | --- |
| Raw telemetry | Local NVMe (Parquet / Hive layout) | **No** — offloads after landing |
| Pose / detections (derived) | Local store + peer pub/sub | **Yes** — small, budgeted |
| Full flight archive | Ground warehouse (Parquet + object store) | After landing only |
One layout on every floor so offload is a **mirror**, not a migration
([03 — Data platform](03-data-platform.md)).
## Sync rule
1. **Fast path:** compact pose frames (~45 B @ 5 Hz) + event-shaped detections.
2. **Bulk path (peers):** sealed derived partitions when the link allows
(rsync/SSH proposed; alternatives open).
3. **Never** push raw high-rate telemetry peer-to-peer in flight.
## MinIO (first-class, placement TBD)
MinIO is treated as **already chosen infrastructure**, not something to rip out.
| Role | Stance in this sketch |
| --- | --- |
| **Ground warehouse (T3)** | **Primary home** — S3 API under the Parquet lake / offload target |
| **On-board (drone)** | **Default: no** — competes with flight-critical CPU/RAM; local Parquet + DuckDB is enough for the hot path |
| **On-board exception** | Not excluded if a team already relies on an S3 API in Compose; then derived-only, never the 5 Hz pose path |
| **5 Hz pose / collision state** | **Not MinIO** — pub/sub (UDP in the PoC; Zenoh proposed) |
| **Inter-drone history catch-up** | Prefer pull of sealed partitions (SSH/rsync); MinIO replication only if ops already standardised on it |
Open item for the team: document how MinIO is used today, then lock T2/T3 roles
([09 — Open questions](09-open-questions.md)).
## What to read next
1. [12 — Design journey](12-design-journey.md) — narrative + deep links into code
2. [Implemented vs proposed](../README.md#implemented-now-vs-proposed-next) — PoC vs production intent
3. [09 — Open questions](09-open-questions.md) — including MinIO placement
Live visual PoC: [swarm.produktor.io](https://swarm.produktor.io/) (same access as this repository).
+3 -67
View File
@@ -2,70 +2,6 @@
Terms and abbreviations used throughout this proposal. Terms and abbreviations used throughout this proposal.
**Vocabulary guide:** **architecture layers** (13) = on-board software stack; **storage floors** (T0T4) = retention/latency ladder; **pipeline stages** = narrative capture → reduction → sync only ([12 — Design journey](12-design-journey.md) §4). Do not mix these three.
## Project vocabulary
### T0 — hot
RAM / DuckDB in-process sliding window of the last minutes; what mission logic queries in flight. Retention: minutes. Maps to **hot** in the classic three-tier model.
### T1 — warm
Drone NVMe: full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight. Retention: current flight plus quota-based headroom. Maps to **warm**. Runtime path: [`var/t1/`](../var/t1/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)).
### T2 — shared
On-board MinIO bucket: derived data only (`detections`, `state`), replicated opportunistically across the swarm. Retention: current mission.
### T3 — warehouse
Ground on-prem object store + Parquet lakehouse: every flight of every drone, forever; replay, analytics, model training. Retention: years. Maps to **cold** / **DWH**. Runtime path: [`var/t3/`](../var/t3/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)). After offload and integrity audit, T3 is the [source of truth](#source-of-truth) for historical flight data.
### T4 — dev
Engineer laptop or sim farm: slices pulled from T3, or synthetic data from the simulator. Retention: ephemeral.
### Source of truth
The authoritative copy of flight data after [offload](#offload) and integrity audit: the [T3](#t3--warehouse) warehouse. Each drone holds its own partial record in flight ([T1](#t1--warm)); peers exchange derived state only ([04 — Swarm sync](04-swarm-sync.md)). Replay, post-flight fusion, training, and audit read from T3 — not from any single drone's local store. See [06 — Environments](06-environments.md), [ADR-0003](adr/ADR-0003-sync-derived-state-only.md).
### Architecture layer 1 — ingestion
On-board services that capture sensor streams and video analytics ([02 — Architecture](02-architecture.md)): `sensor-ingest`, `video-analytics`.
### Architecture layer 2 — storage and transform
On-board Parquet write path, sealer, DuckDB, NVMe ([02 — Architecture](02-architecture.md)).
### Architecture layer 3 — serving and sync
On-board event hook, pub/sub broadcast, MinIO derived bucket, peer query API ([02 — Architecture](02-architecture.md)).
### Pipeline stage
Narrative inside each drone: (1) raw capture, (2) ETL/reduction, (3) bidirectional sync interface. Not the same as storage floors T0T4 ([12 — Design journey](12-design-journey.md) §4).
### Bulk sync
In-flight pull of sealed `detections`/`state` Parquet partitions between peers via **rsync** delta over persistent SSH forced commands ([04 — Swarm sync](04-swarm-sync.md)).
### Offload
Post-flight mirror of sealed T1 partitions into T3 at the dock: **rsync** or MinIO **`mc mirror`**, then integrity audit before local prune ([03 — Data platform](03-data-platform.md)).
### rclone
Optional rsync-class tool when the far end is object storage and rsync is awkward; not the default in-flight mesh path ([09 — Open questions](09-open-questions.md) §12).
### ADR
Architecture Decision Record — a point-in-time decision for this repository ([adr/README.md](adr/README.md)).
### ASR
Architecture Standard Record — a standing cross-repository policy (lives in the ecosystem `inventar` repo, not here). See [adr/README.md](adr/README.md).
## Data & storage ## Data & storage
| Term | Meaning | | Term | Meaning |
@@ -74,8 +10,8 @@ Architecture Standard Record — a standing cross-repository policy (lives in th
| **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) | | **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) |
| **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate | | **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate |
| **ZSTD** | Zstandard compression; the default codec for all Parquet files here | | **ZSTD** | Zstandard compression; the default codec for all Parquet files here |
| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown; same role as [source of truth](#source-of-truth) | | **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown |
| **Hot / warm / cold tiers** | Classic three-tier model: in-memory window → local NVMe → ground warehouse. Here mapped to [T0](#t0--hot), [T1](#t1--warm), and [T3](#t3--warehouse); [T2](#t2--shared) and [T4](#t4--dev) are named explicitly | | **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse |
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes | | **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier | | **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
| **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame | | **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame |
@@ -119,7 +55,7 @@ Architecture Standard Record — a standing cross-repository policy (lives in th
| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone | | **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone |
| **k3s** | Lightweight Kubernetes distribution; used on the ground only | | **k3s** | Lightweight Kubernetes distribution; used on the ground only |
| **Dev Container** | Reproducible containerized development environment definition | | **Dev Container** | Reproducible containerized development environment definition |
| **MinIO** | S3-compatible object store; runs on-board for derived data and on the ground as the warehouse backend | | **MinIO** | S3-compatible object store **already in the stack**; default home is the **ground warehouse (T3)**. On-board use is an exception, not the baseline |
| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it | | **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it |
| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight | | **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight |
| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version | | **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version |
+5 -4
View File
@@ -15,7 +15,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
| Constraint | Consequence | | Constraint | Consequence |
| --- | --- | | --- | --- |
| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind | | **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention. A [supervising operator](00-glossary.md#swarm--robotics) may set mission intent over [C2](00-glossary.md#swarm--robotics) before and during flight, but does not steer individual vehicles | | **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention |
| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator | | **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch | | **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight | | **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
@@ -28,10 +28,10 @@ Plus everything a platform needs around that: reproducible builds and deployment
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary). - The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
- **DuckDB** is already in use for local data handling. - **DuckDB** is already in use for local data handling.
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked. - **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
- **MinIO** is available and considered as the inter-drone sync mechanism. - **MinIO** is already in the stack. Exact placement (ground vs on-board vs sync path) was not fully specified in the brief; this proposal treats MinIO as **first-class ground-warehouse infrastructure** by default and keeps on-board MinIO as an open option — see [00 — Executive](00-executive.md) and [09 — Open questions](09-open-questions.md).
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)). - A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
- After landing, each drone's data is **offloaded to the on-prem [source of truth](00-glossary.md#source-of-truth)** for replay and iterative model training. - After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training.
- Mission intent (declarative goals) reaches the swarm over a narrow **[C2](00-glossary.md#swarm--robotics) channel**; there is no continuous ground link in flight. - Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight.
## What is assumed (explicitly marked as assumptions) ## What is assumed (explicitly marked as assumptions)
@@ -43,6 +43,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions | | A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns | | A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only | | A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
| A7 | **MinIO's primary home is the ground warehouse (T3), not the drone** | On-board object store competes with flight-critical CPU/RAM; local Parquet + DuckDB already cover the hot path. On-board MinIO is not excluded if ops already depend on an S3 API in Compose |
## Out of scope ## Out of scope
+20 -14
View File
@@ -2,7 +2,7 @@
## On-board: three layers, one Compose file ## On-board: three layers, one Compose file
Every drone runs the same [three-layer](00-glossary.md#architecture-layer-1--ingestion) data plane ([storage floors](00-glossary.md#t0--hot) T0T4 are a separate concept — see [03 — Data platform](03-data-platform.md#storage-floors-tiers)). **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)). Every drone runs the same three-layer data plane. **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)).
```mermaid ```mermaid
graph TB graph TB
@@ -20,11 +20,13 @@ graph TB
subgraph serving [Layer 3 — Serving and sync] subgraph serving [Layer 3 — Serving and sync]
HOOK["event hook<br/>fires on new derived data"] HOOK["event hook<br/>fires on new derived data"]
PUB["state publisher<br/>pub/sub broadcast"] PUB["state publisher<br/>pub/sub broadcast"]
MINIO["MinIO<br/>derived datasets bucket"] BULK["bulk sync<br/>rsync over SSH"]
QAPI["query API<br/>peer data requests"] QAPI["query API<br/>SQL-over-SSH"]
end end
end end
MINIO[("MinIO — ground warehouse T3<br/>already in the stack")]
SENSORS --> WRITER SENSORS --> WRITER
VIDEO -->|detections| WRITER VIDEO -->|detections| WRITER
WRITER --> NVME WRITER --> NVME
@@ -32,17 +34,20 @@ graph TB
DUCK --> NVME DUCK --> NVME
WRITER -->|derived rows| HOOK WRITER -->|derived rows| HOOK
HOOK --> PUB HOOK --> PUB
HOOK --> MINIO HOOK --> BULK
QAPI --> DUCK QAPI --> DUCK
PUB -.->|mesh| PEERS["peer drones"] PUB -.->|mesh pose| PEERS["peer drones"]
MINIO -.->|replication| PEERS BULK -.->|sealed partitions| PEERS
QAPI -.->|on demand| PEERS QAPI -.->|on demand| PEERS
NVME -->|post-flight offload| MINIO
``` ```
**MinIO** is already in the stack and is the **default ground warehouse (T3)** behind offload — keep it, do not rip it out. Default sketch: **not on the drone** (CPU/RAM budget); on-board MinIO remains an allowed exception if Compose already depends on an S3 API. **In-flight peer bulk sync is SSH/rsync** (or team-standard MinIO replication if that is already how ops works) — never the 5 Hz pose path. See [00 — Executive](00-executive.md) and [04 — Swarm sync](04-swarm-sync.md).
### Layer 1 — Ingestion ### Layer 1 — Ingestion
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on. - `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
- `video-analytics` runs a [**YOLO-like**](00-glossary.md#swarm--robotics) object detector on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images. - `video-analytics` runs a **YOLO-like object detector** on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images.
- Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient). - Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient).
### Layer 2 — Storage and transform ### Layer 2 — Storage and transform
@@ -53,10 +58,11 @@ graph TB
### Layer 3 — Serving and sync ### Layer 3 — Serving and sync
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls. - The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, bulk-sync hint, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)). - The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (UDP in the PoC; Zenoh proposed — [04](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH ([rsync](00-glossary.md#bulk-sync) delta transfer); MinIO remains optional where an S3 API is wanted. - **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer). If the team already operates MinIO replication for that job, reuse it — do not invent a second path. Pose traffic never goes through MinIO.
- **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)). - **Peer queries** are read-only DuckDB SQL — HTTP explorer gate in the PoC; **SSH forced commands** proposed for flight (SELECT-only gate, read-only OS user, columnar responses [04](04-swarm-sync.md)).
- **MinIO** sits on the **ground** as T3 by default ([00 — Executive](00-executive.md)); on-board MinIO is an explicit exception, not the baseline.
## Communication planes ## Communication planes
@@ -65,11 +71,11 @@ Three isolated planes with different lifecycles:
```mermaid ```mermaid
graph LR graph LR
subgraph planes [Communication planes] subgraph planes [Communication planes]
C2["[C2](00-glossary.md#swarm--robotics) control plane<br/>mission intent, narrow, in production"] C2["C2 control plane<br/>mission intent, narrow, in production"]
DATA["swarm data plane<br/>state broadcast + sync, in production"] DATA["swarm data plane<br/>state broadcast + sync, in production"]
DEV["dev/debug plane<br/>live telemetry, bench only"] DEV["dev/debug plane<br/>live telemetry, bench only"]
end end
OPERATOR["supervising operator<br/>([man-in-the-loop](00-glossary.md#swarm--robotics))"] --> C2 OPERATOR["supervising operator<br/>(man-in-the-loop)"] --> C2
C2 --> SWARM["swarm"] C2 --> SWARM["swarm"]
SWARM <--> DATA SWARM <--> DATA
DEV -.->|absent from production builds| SWARM DEV -.->|absent from production builds| SWARM
@@ -79,7 +85,7 @@ graph LR
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes | | **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes |
| **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes | | **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes |
| **Dev/debug plane** | Full live telemetry for bench development and [HIL](00-glossary.md#infrastructure--delivery) tests | High | **No — physically absent from production builds** | | **Dev/debug plane** | Full live telemetry for bench development and HIL tests | High | **No — physically absent from production builds** |
The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it. The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it.
+15 -15
View File
@@ -1,6 +1,6 @@
# 03 — Data platform # 03 — Data platform
The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every [storage floor](00-glossary.md#t0--hot)** of the system. The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every floor** of the system.
## Datasets ## Datasets
@@ -14,7 +14,7 @@ Three logical datasets, distinct because their consumers and sync policies diffe
## Partitioning layout ## Partitioning layout
Identity dimensions first, calendar time last, everything [Hive-style](00-glossary.md#data--storage) `key=value`: Identity dimensions first, calendar time last, everything Hive-style `key=value`:
``` ```
dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/ dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/
@@ -29,7 +29,7 @@ Design decisions and their reasons:
- **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.) - **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.)
- **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both. - **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both.
- **Flight identifiers.** The canonical flight id is a [**UUIDv7**](00-glossary.md#data--storage) (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest. - **Flight identifiers.** The canonical flight id is a **UUIDv7** (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
- **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time. - **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time.
- **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)). - **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)).
@@ -50,7 +50,7 @@ sequenceDiagram
S->>W: rows (int64 ns, typed) S->>W: rows (int64 ns, typed)
W->>C: flush min_NN.parquet (small, ZSTD-fast) W->>C: flush min_NN.parquet (small, ZSTD-fast)
end end
Note over Z: window closes (hour [sealed](00-glossary.md#data--storage)<br/>or flight ends) Note over Z: window closes (hour sealed<br/>or flight ends)
Z->>C: read all min_*.parquet Z->>C: read all min_*.parquet
Z->>P: write one compact file (ZSTD-high) Z->>P: write one compact file (ZSTD-high)
Z->>C: remove blocks after verify Z->>C: remove blocks after verify
@@ -63,28 +63,28 @@ sequenceDiagram
## Storage floors (tiers) ## Storage floors (tiers)
Same [Parquet](00-glossary.md#data--storage) + [DuckDB](00-glossary.md#data--storage) format on every floor; only volume, retention, and location change. See [glossary](00-glossary.md#t0--hot) for the full T0T4 definitions. Same format on every floor; only volume, retention, and location change:
| Floor | Where | Contents | Retention | | Floor | Where | Contents | Retention |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **[T0 — hot](00-glossary.md#t0--hot)** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes | | **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
| **[T1 — warm](00-glossary.md#t1--warm)** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) | | **T1 — warm** | Drone NVMe (Parquet) | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
| **[T2 — shared](00-glossary.md#t2--shared)** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission | | **T2 — shared** | Peer-derived cache on NVMe (Parquet) | Derived data only (`detections`, `state`) pulled from peers when the link allows | Current mission |
| **[T3 — warehouse](00-glossary.md#t3--warehouse)** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; [source of truth](00-glossary.md#source-of-truth) for replay, analytics, model training | Years | | **T3 — warehouse** | Ground **MinIO** (or equivalent) + Parquet lakehouse | Every flight of every drone; replay, analytics, model training**primary MinIO home** | Years |
| **[T4 — dev](00-glossary.md#t4--dev)** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral | | **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
**MinIO note:** default sketch keeps MinIO on the **ground (T3)**. An on-board MinIO sidecar is a possible exception if the team already standardises on an S3 API in Compose — derived data only, never the 5 Hz pose path. Placement is an open question ([09](09-open-questions.md), [00 — Executive](00-executive.md)).
## Flight offload: a mirror, not a migration ## Flight offload: a mirror, not a migration
Because [T1](00-glossary.md#t1--warm) and [T3](00-glossary.md#t3--warehouse) share the identical layout, [offload](00-glossary.md#offload) after landing is: Because T1 and T3 share the identical layout, offload after landing is:
```bash ```bash
# per drone, at the base station # per drone, at the base station
rsync -a drone-nvme/flights/ warehouse/flights/ # or mc mirror to object store mc mirror drone-nvme/flights/ warehouse/flights/ # or rsync over the wired dock
duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…" duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…"
``` ```
For S3-only backends where rsync is awkward, [rclone](00-glossary.md#rclone) is an optional rsync-class alternative — not the default mesh path ([09 — Open questions](09-open-questions.md) §12).
The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job. The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job.
## Retention and quotas on board ## Retention and quotas on board
@@ -140,6 +140,6 @@ On the ground this powers replay overlays, coverage analysis (did ubiquitous sen
## Schema management ## Schema management
- Every dataset schema is versioned in-repo (planned `schemas/` directory — [roadmap stage 2](08-roadmap.md), [09 — Open questions](09-open-questions.md) §14; SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)). - Every dataset schema is versioned in-repo (`schemas/` SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
- Parquet files carry the schema version in file metadata; readers select the matching deserializer. - Parquet files carry the schema version in file metadata; readers select the matching deserializer.
- Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release. - Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release.
+29 -21
View File
@@ -16,21 +16,27 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex
## Broadcast payload: small on the wire, precise at rest ## Broadcast payload: small on the wire, precise at rest
The `state` broadcast is a fixed compact frame (46 bytes, little-endian): The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
what the prototype uses for bandwidth estimates.
| Field | Type | Notes | | Field | Wire type (PoC) | Notes |
| --- | --- | --- | | --- | --- | --- |
| `magic` | 2 bytes | `b"SH"` | | `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` |
| `version` | uint8 | Protocol version | | `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap |
| `drone_id` | 8 bytes | Zero-padded ASCII fleet id |
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage | | `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision | | `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision |
| `att_roll/pitch/yaw` | int16 | Centi-degrees | | `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model |
| `vel_x/y/z` | int16 | cm/s | | `vel_x/y/z` | 3 × int16 | cm/s |
| `frame_ref` | uint8 | Frame of reference id ([GPS-denied](00-glossary.md#swarm--robotics): local/visual-odometry frames must be explicit) | | `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … | | `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
46 bytes per frame → a 50-drone swarm at 5 Hz is ~11.5 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh. 45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.
An earlier sketch used relative coordinates and a bounding sphere (no attitude).
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
costs almost nothing at this frame size. Relative localization remains a
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison. `detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
@@ -53,39 +59,41 @@ graph LR
SB -->|"store-and-forward relay"| SC SB -->|"store-and-forward relay"| SC
``` ```
### Recommended: [Zenoh](00-glossary.md#swarm--robotics) ### Recommended (proposal): Zenoh
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub. - Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware. - First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
- Tiny footprint, ARM64-native. - Tiny footprint, ARM64-native.
**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.
### Alternatives considered ### Alternatives considered
| Option | Verdict | | Option | Verdict |
| --- | --- | | --- | --- |
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively | | **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh | | **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode | | **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode |
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below | | **MinIO bucket replication** | Keep as a **team option** for bulk derived catch-up if ops already standardise on it; wrong tool for the 5 Hz pose path. Default sketch: MinIO primary on the **ground warehouse**, not on the drone |
## Two sync mechanisms, deliberately separate ## Two sync mechanisms, deliberately separate
1. **Fast path — pub/sub ([Zenoh](00-glossary.md#swarm--robotics)):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache. 1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — [rsync](00-glossary.md#bulk-sync) over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. 2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.
Why SSH-based bulk sync over object-store replication: Why SSH-based bulk sync over object-store replication:
- **Identity is already there.** Every drone holds pre-provisioned [ed25519](00-glossary.md#infrastructure--delivery) keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery. - **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
- **One persistent multiplexed session** ([`ControlMaster`](00-glossary.md#infrastructure--delivery)) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes. - **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing. - **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it. - **Zero extra services** on the flight-critical node by default. **MinIO stays first-class on the ground (T3).** On-board MinIO is an exception only if Compose already depends on an S3 API. In-flight bulk path defaults to SSH/rsync; switch to MinIO replication if that is already how the team moves objects.
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**). Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
Two deliberate non-features of the SSH channel: Two deliberate non-features of the SSH channel:
- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount. - **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH [forced command](00-glossary.md#infrastructure--delivery)** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction. - **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.
## Design decision: SQL-over-SSH as the peer query channel ## Design decision: SQL-over-SSH as the peer query channel
@@ -107,7 +115,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
| Layer | Mechanism | What it stops | | Layer | Mechanism | What it stops |
| --- | --- | --- | | --- | --- | --- |
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement | | 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
| 2. Statement gate | Wrapper accepts a single statement, parses it, rejects anything but `SELECT` (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); parameters bound, not interpolated | SQL-as-a-write-channel, config tampering | | 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering |
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 | | 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store | | 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries | | 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
@@ -116,7 +124,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction. - **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
- **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs. - **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from [T3](00-glossary.md#t3--warehouse) if result sizes outgrow it. - **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it.
What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one. What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.
+4 -4
View File
@@ -23,9 +23,9 @@ graph TB
A["drone A<br/>wg key + certs"] A["drone A<br/>wg key + certs"]
B["drone B<br/>wg key + certs"] B["drone B<br/>wg key + certs"]
C["drone C<br/>wg key + certs"] C["drone C<br/>wg key + certs"]
A <-->|"WireGuard tunnel"| B A <-->|"WireGuard + mTLS"| B
B <-->|"WireGuard tunnel"| C B <-->|"WireGuard + mTLS"| C
A <-->|"WireGuard tunnel"| C A <-->|"WireGuard + mTLS"| C
end end
PROV -->|"per-device identity, peer list"| A PROV -->|"per-device identity, peer list"| A
PROV --> B PROV --> B
@@ -65,7 +65,7 @@ Keep both layers: WireGuard for *who is on the network and whether the wire is r
## Data at rest ## Data at rest
- On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable. - On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable.
- The ground warehouse applies standard at-rest encryption plus role-based access; it is the [source of truth](00-glossary.md#source-of-truth) and the single most valuable asset in the system (every flight ever flown). - The ground warehouse applies standard at-rest encryption plus role-based access; it is the single most valuable asset in the system (every flight ever flown).
## Attack surface, deliberately shortened ## Attack surface, deliberately shortened
+9 -9
View File
@@ -9,15 +9,15 @@ graph LR
subgraph dev [3 — Dev and simulation] subgraph dev [3 — Dev and simulation]
SIM["virtual drone fleet<br/>(simulator, Compose)"] SIM["virtual drone fleet<br/>(simulator, Compose)"]
LAP["engineer laptop<br/>Dev Container"] LAP["engineer laptop<br/>Dev Container"]
SLICE["[T4](00-glossary.md#t4--dev): data slices"] SLICE["T4: data slices"]
end end
subgraph fleet [1 — Fleet, in flight] subgraph fleet [1 — Fleet, in flight]
D1["drone: Compose data plane<br/>[T0](00-glossary.md#t0--hot) hot + [T1](00-glossary.md#t1--warm) NVMe + [T2](00-glossary.md#t2--shared) MinIO"] D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 peer cache"]
D2["drone …"] D2["drone …"]
end end
subgraph ground [2 — Ground, on-prem] subgraph ground [2 — Ground, on-prem]
DOCK["base station docks"] DOCK["base station docks"]
DWH["[T3](00-glossary.md#t3--warehouse) warehouse<br/>object store + Parquet/DuckDB"] DWH["T3 warehouse<br/>MinIO + Parquet/DuckDB"]
K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"] K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
TRAIN["model training (out of scope)<br/>reads T3, ships weights"] TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
end end
@@ -34,7 +34,7 @@ graph LR
## 1 — Fleet infrastructure (on the drones) ## 1 — Fleet infrastructure (on the drones)
- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md). - Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md).
- Storage floors [T0T2](00-glossary.md#t0--hot). Nothing here requires ground contact during a mission. - Storage floors T0T2. Nothing here requires ground contact during a mission.
- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest. - Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.
## 2 — Ground infrastructure (on-prem) ## 2 — Ground infrastructure (on-prem)
@@ -43,7 +43,7 @@ The permanent installation. This **is** allowed to be a cluster — links are wi
| Component | Runs on | Role | | Component | Runs on | Role |
| --- | --- | --- | | --- | --- | --- |
| **Warehouse ([T3](00-glossary.md#t3--warehouse))** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's [source of truth](00-glossary.md#source-of-truth) and long-term memory | | **Warehouse (T3)** | **MinIO** (already in the stack) + Parquet | Every flight of every drone; the system's long-term memory |
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) | | **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see | | **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) | | **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
@@ -60,7 +60,7 @@ The environment engineers live in daily — and deliberately the first thing to
- **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free. - **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free.
- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop. - **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop.
- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story. - **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
- **[T4](00-glossary.md#t4--dev) data slices**: a thin tool pulls partition subsets from [T3](00-glossary.md#t3--warehouse) (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified. - **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets). - The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).
## IaC boundaries ## IaC boundaries
@@ -74,13 +74,13 @@ Infrastructure as code follows the same discipline as the data plane: each tool
| Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. | | Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. |
| On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). | | On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). |
Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, [T1](00-glossary.md#t1--warm)) and **`var/t3/`** (warehouse, [T3](00-glossary.md#t3--warehouse)). Compose, k3d, and DuckDB on the host all read the same Parquet tree. Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, T1) and **`var/t3/`** (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree.
Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle. Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle.
### k3d: the ground segment as an executable miniature ### k3d: the ground segment as an executable miniature
`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the [T1](00-glossary.md#t1--warm)→[T3](00-glossary.md#t3--warehouse) offload CronJob. The sim CronJob uses `cp -ru` plus `mc mirror` as a shortcut; production dock [offload](00-glossary.md#offload) uses rsync or `mc mirror` with checksum audit ([03](03-data-platform.md)). The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API. `ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API.
### Next fidelity step: microVMs ### Next fidelity step: microVMs
@@ -90,7 +90,7 @@ The k3d fleet shares one kernel and one network namespace tree — good enough t
| Operation | What it costs with one layout everywhere | | Operation | What it costs with one layout everywhere |
| --- | --- | | --- | --- |
| Flight offload | `rsync` / `mc mirror` + audit — no transform step ([03](03-data-platform.md)) | | Flight offload | `mc mirror` / `rsync` + audit — no transform step |
| Debugging a field issue | Pull the flight slice, replay locally, same queries | | Debugging a field issue | Pull the flight slice, replay locally, same queries |
| Validating a pipeline change | Run simulator, diff Parquet outputs | | Validating a pipeline change | Run simulator, diff Parquet outputs |
| Training data prep | Read T3 directly, no export pipeline | | Training data prep | Read T3 directly, no export pipeline |
+2 -2
View File
@@ -29,7 +29,7 @@ graph LR
LG["structured logs<br/>= Parquet partitions"] LG["structured logs<br/>= Parquet partitions"]
end end
subgraph ground [Ground k3s] subgraph ground [Ground k3s]
DWH["[T3](00-glossary.md#t3--warehouse) warehouse"] DWH["T3 warehouse"]
PR["Prometheus<br/>(live: docks, CI, sim farm)"] PR["Prometheus<br/>(live: docks, CI, sim farm)"]
LK["Loki<br/>(live logs, dev plane)"] LK["Loki<br/>(live logs, dev plane)"]
GF["Grafana<br/>dashboards + replay"] GF["Grafana<br/>dashboards + replay"]
@@ -44,7 +44,7 @@ graph LR
``` ```
- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane. - **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
- **Flight post-mortems query the [source of truth](00-glossary.md#source-of-truth) warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute. - **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.
## Health questions this design answers cheaply ## Health questions this design answers cheaply
+6 -6
View File
@@ -9,11 +9,11 @@ graph TD
S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"] S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"]
S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"] S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"]
S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"] S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"]
S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"] S4["Stage 4<br/>Serving layer<br/>event hook, query API"]
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"] S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"] S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk replication (rsync over SSH)"] S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk pull"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"] S8["Stage 8<br/>Ground warehouse<br/>offload + audit, MinIO T3, replay tooling"]
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"] S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"] S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"] S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"]
@@ -41,11 +41,11 @@ graph TD
| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — | | **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — |
| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 | | **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 |
| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 | | **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 |
| **4 — Serving layer** | Event hook + query API + on-board MinIO | Event-driven consumption; nothing polls | 3 | | **4 — Serving layer** | Event hook + query API (SQL gate) | Event-driven consumption; nothing polls | 3 |
| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 | | **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 |
| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 | | **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 |
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk replication over the mesh | Staleness budgets met under simulated loss/partitions | 5, 6 | | **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk partition pull | Staleness budgets met under simulated loss/partitions | 5, 6 |
| **8 — Ground warehouse** | Dock offload (mirror + audit), T3 store, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 | | **8 — Ground warehouse** | Dock offload (mirror + audit), **MinIO as T3**, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 | | **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 |
| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 | | **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 |
| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 | | **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 |
+11 -16
View File
@@ -18,7 +18,7 @@ Visual-odometry positions are relative; different drones may hold different loca
By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging. By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging.
*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from the [source of truth](00-glossary.md#source-of-truth) in the warehouse. If a persistent shared structure becomes unavoidable, use a [CRDT](00-glossary.md#infrastructure--delivery) type for it rather than inventing reconciliation. *Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from ground truth in the warehouse. If a persistent shared structure becomes unavoidable, use a CRDT type for it rather than inventing reconciliation.
## 4 — Degradation policy under sustained link loss ## 4 — Degradation policy under sustained link loss
@@ -42,7 +42,7 @@ The degradation ladder ([03](03-data-platform.md)) needs a domain-approved order
The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration. The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration.
*Proposal:* a small [HIL](00-glossary.md#infrastructure--delivery) rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI. *Proposal:* a small HIL rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI.
## 8 — Video retention policy ## 8 — Video retention policy
@@ -64,24 +64,19 @@ Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage).
## 11 — Lake retention and partition pruning ## 11 — Lake retention and partition pruning
The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on [T1](00-glossary.md#t1--warm) disk after offload to [T3](00-glossary.md#t3--warehouse). The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on T1 disk after offload to T3.
*Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job. *Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job.
## 12 — Bulk transfer tool choice ## 12 — MinIO placement (already in the stack)
In-flight peer bulk sync, dock offload, and sim shortcuts have used different wording (rsync, rclone, `mc mirror`, `cp -ru`). Exact how MinIO is used today was not fully specified. It should not be ripped out.
*Proposal:* **rsync over SSH** is the default for peer [bulk sync](00-glossary.md#bulk-sync) and wired dock [offload](00-glossary.md#offload) of the Hive partition tree; **MinIO `mc mirror`** when the far end is the object-store facade; **[rclone](00-glossary.md#rclone)** only when the backend is S3-compatible and rsync is awkward; sim CronJob keeps **`cp -ru` + `mc mirror`** as a documented shortcut. Pick one primary tool per path in provisioning, not all at once. *Proposal (default):*
## 13 — DuckLake vs plain DuckDB + Parquet on T3 - **Ground warehouse (T3):** primary MinIO home — offload target and long-term object API.
- **On the drone:** default **off** — local Parquet + DuckDB cover the hot path; an object store competes with flight-critical CPU/RAM.
- **Exception:** on-board MinIO allowed if Compose already depends on an S3 API — derived datasets only, never 5 Hz pose.
- **Peer bulk catch-up:** prefer sealed-partition pull (SSH/rsync); use MinIO replication only if that is already the ops standard.
The design journey mentions DuckLake; no ADR or implementation commits to it yet. *Settle with the team:* one short diagram of today's MinIO topology (buckets, clients, what syncs). Until then, treat A7 in [01](01-problem-statement.md) as the working assumption. Summary table: [00 — Executive](00-executive.md).
*Proposal:* baseline remains DuckDB `read_parquet` globs over the [T3](00-glossary.md#t3--warehouse) Hive tree with MinIO underneath; evaluate DuckLake only if catalog/versioning pain appears at warehouse scale — decision gets its own ADR before any dependency lands.
## 14 — `schemas/` registry layout
[03 — Data platform](03-data-platform.md) references an in-repo `schemas/` directory that does not exist yet (roadmap stage 2).
*Proposal:* add `schemas/` as part of stage 2 with one SemVer file per dataset version, referenced from the fleet release manifest; until then treat schema versions as manifest-only fields and document the gap here rather than implying the directory already exists.
+6 -6
View File
@@ -9,14 +9,14 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera
| **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) | | **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) |
| **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) | | **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) |
| **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) | | **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) |
| **[Sustainable pulsing](00-glossary.md#swarm--robotics)** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) | | **Sustainable pulsing** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) |
| **[Ubiquitous sensing](00-glossary.md#swarm--robotics)** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) | | **Ubiquitous sensing** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) |
| **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) | | **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) |
| **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) | | **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) |
| **[GPS-denied operation](00-glossary.md#swarm--robotics)** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) | | **GPS-denied operation** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) |
| **[Man-in-the-loop](00-glossary.md#swarm--robotics)** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) | | **Man-in-the-loop** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) |
| **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) | | **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) |
| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the [source of truth](00-glossary.md#source-of-truth) warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) | | **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) |
| **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) | | **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) |
| **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) | | **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) |
@@ -25,7 +25,7 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera
Every one of these principles quietly assumes a working data layer underneath: Every one of these principles quietly assumes a working data layer underneath:
- *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget. - *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget.
- *Ubiquitous sensing* assumes observations are **fused later** — that is the [source of truth](00-glossary.md#source-of-truth) warehouse with aligned timestamps and schemas. - *Ubiquitous sensing* assumes observations are **fused later** — that is a warehouse with aligned timestamps and schemas.
- *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop. - *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop.
- *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data. - *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data.
- *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format. - *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format.
+1 -1
View File
@@ -85,7 +85,7 @@ schemas:
config: config:
compose-bundle: 3.4.0 compose-bundle: 3.4.0
radio-profile: production-2 radio-profile: production-2
peer-registry: fleet-42-r7 # provisioning + revocations, see [05 — Network & security](05-network-security.md) peer-registry: fleet-42-r7 # provisioning + revocations, see 05
``` ```
- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm. - Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
+38 -33
View File
@@ -1,10 +1,12 @@
# 12 — Design journey # 12 — Design journey
How this design came together, told in the order the thinking actually happened. How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report. The formal decisions, with options and It is a walk-through, not a report — and **not a mandate**. The goal is a
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the from-scratch platform sketch that shows how the pieces fit; every concrete
story behind them, and each chapter links straight into the code it produced. choice is an option with trade-offs recorded as
Terms: [Glossary](00-glossary.md). [Architecture Decision Records](adr/README.md). Replace any piece if a better
fit shows up. This is the story behind the decisions, and each chapter links
straight into the code it produced.
--- ---
@@ -47,17 +49,17 @@ conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need. everything locally. Send only what peers truly need.
> **Read more** > **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0T4) > - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
## 4. Three pipeline stages inside each unit ## 4. Three tiers inside each unit
That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0T4: That gave [three layers](03-data-platform.md) on every drone:
1. **Raw capture** (pipeline stage 1), never transformed. 1. **Raw capture**, never transformed.
2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared: 2. **ETL / reduction**, where data is cut down to what has to be shared:
millimeters to centimeters, thinning time series where that is enough, millimetres to centimetres, thinning time series where that is enough,
dropping what nobody downstream reads. dropping what nobody downstream reads.
3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the 3. **A bidirectional interface**: on one side it broadcasts position into the
shared channel, on the other it lets peers pull data. shared channel, on the other it lets peers pull data.
The ETL is deliberately not over-specified. How sensor data is transformed is a The ETL is deliberately not over-specified. How sensor data is transformed is a
@@ -66,20 +68,22 @@ stored and in which structure, not to step into that work.
> **Read the code** > **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight > - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1 > - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
## 5. What actually needs to sync ## 5. What actually needs to sync
The most time-critical item is where each peer is, so every unit has time to The most time-critical item is where each peer is, so every unit has time to
react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and react. The wire frame carries **mission-frame position** (millimetres on the
time) instead of absolute coordinates. Relative is cheaper and enough for wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags —
coordination. To avoid sending orientation, a unit is modelled as a **sphere** 45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the
that bounds its extent. That trades a little compute for much less data, and no fusion pain. An earlier sketch used relative coordinates and a bounding sphere
per-shape encoding. If the units are identical, their 3D model can be provisioned with no orientation; it was dropped. When peers disagree on frames,
ahead instead of transmitted. `frame_ref` makes the mismatch explicit for consumers
([09 — Open questions](09-open-questions.md)).
> **Read the code** > **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only > - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible > - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
## 6. Security as nested layers, not one wall ## 6. Security as nested layers, not one wall
@@ -132,20 +136,21 @@ coming back to foundations DevOps and sysadmins already trust.
## 8. Replication and offload ## 8. Replication and offload
Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock channel. It transfers diffs of the Hive tree efficiently. This matters because a
[offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently. unit can be lost, and if it is, we want its data to already exist elsewhere.
This matters because a unit can be lost, and if it is, we want its data to When peers have a stable link the partition tree is pulled between them, which
already exist elsewhere. When peers have a stable link the partition tree is gives redundancy and a way to cross-check later. On the ground, when the fleet
pulled between them, which gives redundancy and a way to cross-check later. On returns, the same path unifies every unit's data into a
the ground, when the fleet returns, the same path unifies every unit's data into [local warehouse](06-environments.md). The warehouse is read-mostly for
a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)). analytics, so transaction contention is not a concern, which opens a
The warehouse is read-mostly for analytics, so transaction contention is not a DuckDB / DuckLake approach with **MinIO underneath on the ground (T3)** — MinIO
concern — plain DuckDB over Parquet with MinIO underneath is the baseline; is already in the stack, so the warehouse leans on it rather than inventing a
DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md) second object store — the
§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md). [same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a uniform structure is what lets an analyst trust a single picture even when a
unit's data has a gap. unit's data has a gap. On-board MinIO stays an open exception, not the default
([00 — Executive](00-executive.md)).
> **Read the code** > **Read the code**
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob > - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
+10 -5
View File
@@ -32,10 +32,15 @@ wrapper, permissions, and output format a peer drone would use.
## Consequences ## Consequences
- One access path is built, secured, and tested — "what you test is what - One access path is designed for flight and bench — "what you test is what
flies". flies".
- The SQL gate is safety-critical and is covered by unit tests - The statement gate in the PoC (`simulator/explorer/server.py`, covered by
(`simulator/tests/test_sql_gate.py`). `simulator/tests/test_sql_gate.py`) is a **first layer**: keyword allow/deny
- The explorer and the prototype's live mode are *just another read-only over HTTP for the explorer. Production still needs the remaining layers in
consumer* of this same contract ([`../04-swarm-sync.md`](../04-swarm-sync.md)). [04 — Swarm sync](../04-swarm-sync.md) (forced-command key, read-only OS user,
engine hardening, resource caps) and a real SQL parser rather than keywords
alone.
- The explorer and the prototype's live mode are *stand-in consumers* of the
same read-only contract over HTTP today; the proposed flight path is
SQL-over-SSH ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this. - Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
-68
View File
@@ -1,68 +0,0 @@
# Documentation audit — 2026-07-10
Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify T0T4 vocabulary, align rsync/offload wording with code, improve readability.
## Summary
| Metric | Result |
| --- | --- |
| Markdown files checked | 33 |
| Broken relative links (after fixes) | 0 |
| Glossary `###` anchors added | 15 (T0T4, layers, bulk sync, source of truth, …) |
| Open questions added | §12–§14 (tooling, DuckLake, `schemas/`) |
## Findings fixed
### Links
- `infra/terraform/ground/README.md` pointed at non-existent `03-storage-design.md``03-data-platform.md`.
- `11-cicd-delivery.md` bare cross-reference to doc 05 → proper markdown link.
### Factual drift (code vs docs)
| Item | Was | Now (matches `broadcast.py`, `sim.ts`) |
| --- | --- | --- |
| Pose frame | ~40 bytes, uint16 `drone_id` | 46 bytes, 8-byte ASCII `drone_id` |
### Terminology
- Extended [00 — Glossary](../00-glossary.md) with T0T4, architecture layers, pipeline stages, source of truth.
- Disambiguated design journey §4: pipeline stages ≠ storage floors.
- Roadmap Stage 7: rsync bulk path, not “MinIO replication”.
### rsync / rclone / offload
| Path | Canonical tool |
| --- | --- |
| Peer bulk sync | rsync over SSH |
| Dock offload | rsync or `mc mirror` |
| S3 backends (optional) | rclone |
| Sim CronJob | `cp -ru` + `mc mirror` (documented shortcut) |
### Live demos (design journey)
Tracked in [PR #3](https://git.produktor.io/eSlider/swarm-house/pulls/3) — explorer + Grafana links in `12-design-journey.md` §7/§9. Prototype link already on `main` §2.
### README / LICENSE
- Restricted-use notice with link to [LICENSE](../../LICENSE).
- Component READMEs tightened (caveman style).
## Remaining (documented, not bugs)
| Item | Where |
| --- | --- |
| `schemas/` directory absent | OQ §14, roadmap stage 2 |
| DuckLake undecided | OQ §13 |
| rsync bulk sync not in sim code | Design only |
| Glossary table terms link to section headers, not row anchors | Optional polish |
| T4 after “Source of truth” in glossary order | Cosmetic |
| GitOps “source of truth” vs data SSOT | Different meanings in `11-cicd-delivery` |
## Suggested CI checks (for issue #1)
Implemented in [`bin/check_docs.py`](../../bin/check_docs.py) — see [ci-rules.md](ci-rules.md).
## Files touched
`docs/00``12`, `docs/adr/*`, nine READMEs, root `README.md`.
-27
View File
@@ -1,27 +0,0 @@
# Documentation improvement records
Point-in-time audits and follow-up work for links, terminology, readability, and factual alignment with code.
| Report | Date | Scope |
| --- | --- | --- |
| [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0T4; live demo links |
## Collaboration style
Issues, PRs, and reviews default to **caveman-full** ([`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default),
[`AGENTS.md`](../../AGENTS.md#caveman-style-default-for-collaboration)). Gitea templates live under
[`.github/ISSUE_TEMPLATE/`](../../.github/ISSUE_TEMPLATE/) and
[`.github/pull_request_template.md`](../../.github/pull_request_template.md).
## CI automation
Enforced in Gitea Actions (`docs-verify` job). Rules live in [`bin/docs_rules.json`](../../bin/docs_rules.json); see [ci-rules.md](ci-rules.md).
Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1).
## How to add a new report
1. Copy the structure from the latest audit file.
2. Name it `YYYY-MM-DD-short-topic.md`.
3. Add a row to the table above.
4. Cross-link from [09 — Open questions](../09-open-questions.md) if new ambiguities appear.
-58
View File
@@ -1,58 +0,0 @@
# CI documentation rules
Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate).
Implementation: [`bin/check_docs.py`](../../bin/check_docs.py) reads [`bin/docs_rules.json`](../../bin/docs_rules.json).
Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**.
Tracked from [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1) and the [2026-07-10 audit](2026-07-10-audit.md).
## Gates
| Rule | Severity | What it checks |
| --- | --- | --- |
| Relative links | error | Every `[text](path)` and `[text](path#anchor)` to a local `.md` file resolves to an existing file |
| Fragment anchors | error | `#anchor` fragments match a `##` or `###` heading in the target file (GitHub-style slug) |
| Banned targets | error | Known-bad paths (e.g. deleted `03-storage-design.md`) are not linked |
| Glossary fragments | error | All `00-glossary.md#…` links use anchors that exist in the glossary |
| Section vs term links | warning | Links to glossary *section* headers (`#swarm--robotics`, …) are listed; prefer `###` term anchors where they exist |
| Bare cross-refs | error | Prose cross-references to numbered docs without a proper markdown link |
| ADR immutability | error | No modifications to existing `docs/adr/ADR-*.md` (compare `origin/main`); add new ADR to supersede |
Warnings are printed in the job log but do not fail CI. Errors exit non-zero and block merge.
## PR workflow
```mermaid
flowchart LR
PR[Pull request] --> DV[docs-verify]
PR --> V[verify simulator]
V --> S[Trivy scan]
DV --> OK{All required green?}
V --> OK
S --> OK
OK -->|yes| Merge[Mergeable]
```
| Event | `docs-verify` | `verify` + `scan` |
| --- | --- | --- |
| PR → `main` | always | always |
| Push → `main` (code paths) | always | runs |
| Push → `main` (docs/README/AGENTS only) | always | skipped (`paths-ignore`) |
Docs-only changes therefore still get link and glossary validation on `main`; they do not trigger a semantic release.
## Local run
```bash
python bin/check_docs.py
python bin/check_adrs.py --base origin/main
python bin/check_docs.py --warnings-as-errors # treat section-link warnings as failures
```
## Changing rules
1. Edit `bin/docs_rules.json` (paths, banned targets, preferred anchors).
2. Adjust `bin/check_docs.py` only when adding a new rule *type*.
3. Add a row to the table above and mention it in the audit/improvement record if the change is significant.
+18 -9
View File
@@ -1,11 +1,14 @@
# Ansible — host preparation # Ansible — host preparation
Ansible = host prep before workloads run. Terraform ([`../terraform/`](../terraform/)) owns workloads. Boundary: [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries). Ansible owns everything that happens **on a host before workloads run**.
Terraform ([`../terraform/`](../terraform/)) owns the workloads themselves.
The boundary is deliberate and documented in
[06 — Environments](../../docs/06-environments.md#iac-boundaries).
| Playbook | Target | What it does | | Playbook | Target | What it does |
| --- | --- | --- | | --- | --- | --- |
| `drone-provision.yml` | Drone (bench, before mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from fleet manifest | | `drone-provision.yml` | Drone (on the bench, before a mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from the fleet release manifest |
| `sim-cluster.yml` | Local workstation / CI host | k3d cluster miniaturing ground segment; shared data volume; simulator image import | | `sim-cluster.yml` | Local workstation / CI host | k3d cluster that miniatures the ground segment; shared data volume; simulator image import |
## Provision a drone ## Provision a drone
@@ -13,12 +16,16 @@ Ansible = host prep before workloads run. Terraform ([`../terraform/`](../terraf
ansible-playbook -i inventory.example.yml drone-provision.yml ansible-playbook -i inventory.example.yml drone-provision.yml
``` ```
Idempotent. Bench network only. Never provision in flight ([05 — Network & security](../../docs/05-network-security.md)): The playbook is idempotent and runs only over the bench network — drones
are never provisioned in flight (see [05 — Network & security](../../docs/05-network-security.md)):
1. **Identity**: generate drone ed25519 keypair if absent. 1. **Identity**: generate the drone's ed25519 keypair if absent.
2. **Peer trust**: fleet public keys in `authorized_keys`, each pinned to read-only SQL forced command. 2. **Peer trust**: install every fleet member's public key into
3. **WireGuard**: render `wg0.conf` with drone address and peers. `authorized_keys`, each pinned to the read-only SQL forced command —
4. **Data plane**: Compose bundle from fleet manifest, systemd unit enabled. the only thing a peer can execute.
3. **WireGuard**: render `wg0.conf` with the drone's address and peers.
4. **Data plane**: lay down the Compose bundle referenced by the fleet
release manifest and enable it as a systemd unit.
## Create the simulation cluster ## Create the simulation cluster
@@ -27,4 +34,6 @@ ansible-playbook sim-cluster.yml # creates k3d cluster 'swarm-sim'
cd ../terraform/sim-env && terraform init && terraform apply cd ../terraform/sim-env && terraform init && terraform apply
``` ```
Shared host directory mounted as `/data` — drone pods, exporter, explorer see one Parquet lake (same contract as on-board NVMe). The cluster mounts a shared host directory as `/data` on the (single)
node, so drone pods, the exporter, and the explorer see one Parquet lake —
the same contract as the on-board NVMe layout.
+19 -9
View File
@@ -1,16 +1,20 @@
# GitOps — ground segment (Flux) # GitOps — ground segment (Flux)
Flux = long-lived ground config from git. Not on drones. Not fleet manifest. Different lifecycles ([11 — CI/CD](../../docs/11-cicd-delivery.md)). Flux reconciles **long-lived ground configuration** from this repository. It does
not run on drones and does not replace the fleet release manifest — those are
different lifecycles by design ([11 — CI/CD](../../docs/11-cicd-delivery.md)).
## Boundary: Terraform vs Flux vs fleet manifest ## Boundary: Terraform vs Flux vs fleet manifest
| Layer | Tool | Owns | | Layer | Tool | Owns |
| --- | --- | --- | | --- | --- | --- |
| Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts | | Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts — the shape of the simulation |
| Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs | | Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs — things that change without reprovisioning PVCs |
| Drones | **[Fleet release manifest](../../docs/11-cicd-delivery.md)** | Compose digests, model weights, peer registry — dock-only | | Drones | **Fleet release manifest** | Compose bundle digests, model weights, peer registry — atomic, dock-only delivery |
Drones outside GitOps. No reconciler in flight. Dock applies pinned manifest once. No mid-mission drift — no update endpoint in radio profile. Drones are **outside GitOps**: there is no reconciler in flight. A dock applies
the pinned manifest once; mid-mission drift is impossible because the update
endpoint does not exist in the radio profile.
## Layout ## Layout
@@ -22,7 +26,9 @@ infra/gitops/
## Bootstrap on the k3d simulation cluster ## Bootstrap on the k3d simulation cluster
`infra/ansible/sim-cluster.yml` installs Flux and applies CRs. After first `git push`, Flux reconciles `infra/gitops/ground/`. `infra/ansible/sim-cluster.yml` installs Flux controllers and applies the CRs
below. After the first `git push`, Flux polls `origin` and reconciles
`infra/gitops/ground/` into the `ground` namespace.
```bash ```bash
# Manual bootstrap (if you skipped Ansible): # Manual bootstrap (if you skipped Ansible):
@@ -30,13 +36,17 @@ flux install --namespace=flux-system
kubectl apply -k infra/gitops/flux kubectl apply -k infra/gitops/flux
``` ```
Reconcile immediately: To reconcile immediately without waiting for the poll interval:
```bash ```bash
flux reconcile source git swarm-house -n flux-system flux reconcile source git swarm-house -n flux-system
flux reconcile kustomization ground-segment -n flux-system flux reconcile kustomization ground-segment -n flux-system
``` ```
## What Flux manages ## What Flux manages here (example)
[`ground/`](ground/) overlay: GitOps-managed ConfigMap with fleet policy metadata. Production extends to Grafana dashboards, Prometheus rules, offload-schedule ConfigMaps — git-versioned, no `terraform apply` for label tweaks. The [`ground/`](ground/) overlay currently carries a **GitOps-managed ConfigMap**
that tags the warehouse segment with fleet policy metadata. In production this
pattern extends to Grafana dashboard bundles, Prometheus rule files, and
offload-schedule ConfigMaps — all versioned in git, all auditable, none of
them requiring a `terraform apply` to tweak a label.
+1 -12
View File
@@ -1,18 +1,7 @@
# Terraform — ground workloads
Declared workloads on the k3d simulation cluster. See [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries).
| Module | Tier | Purpose |
| --- | --- | --- |
| [`sim-env/`](sim-env/) | [T4](../../docs/00-glossary.md#t4--dev) dev/sim | Virtual fleet + observability on k3d |
| [`ground/`](ground/) | [T3](../../docs/00-glossary.md#t3--warehouse) | Warehouse + [T1→T3](../../docs/03-data-platform.md#storage-floors-tiers) offload |
Two modules, two states. Fleet disposable; warehouse survives `destroy` of sim-env.
## Recovering from partial state ## Recovering from partial state
If resources were created outside Terraform (or state was lost), import them If resources were created outside Terraform (or state was lost), import them
before `terraform apply`. Run from the target module directory (`sim-env/` or `ground/`): before `terraform apply`:
```bash ```bash
# sim-env (namespace: swarm) # sim-env (namespace: swarm)
+21 -9
View File
@@ -1,14 +1,19 @@
# Terraform — ground warehouse ([T3](../../../docs/00-glossary.md#t3--warehouse)) # Terraform — ground warehouse (T3)
Ground warehouse. Receives sealed partitions post-flight ([storage floors](../../../docs/03-data-platform.md#storage-floors-tiers)). Long-lived infra → Terraform. The stationary half of the system: the historical warehouse that receives
sealed partitions after every flight ([03 — Storage tiers](../../../docs/03-storage-design.md)).
It is the longest-lived, most conventional infrastructure in the design —
and therefore the most natural Terraform territory.
Runs in same k3d cluster as [`../sim-env`](../sim-env/) but separate namespace and state: fleet disposable, warehouse not — never share `terraform destroy`. Runs in the same k3d cluster as [`../sim-env`](../sim-env/) but in its own
namespace with its own state: the fleet is disposable, the warehouse is not,
and the two lifecycles must never share a `terraform destroy`.
| Resource | Purpose | | Resource | Purpose |
| --- | --- | | --- | --- |
| Deployment `minio` + PVC + NodePort 30901 | Object-store facade for downstream consumers | | Deployment `minio` + PVC + NodePort 30901 | Object-store facade of the warehouse for downstream consumers (training pipelines, replay) |
| CronJob `offload` | [T1](../../../docs/00-glossary.md#t1--warm) → [T3](../../../docs/00-glossary.md#t3--warehouse) offload: Hive partitions verbatim, then `mc mirror` into MinIO | | CronJob `offload` | Post-flight offload: copies Hive partitions verbatim from the fleet lake (T1) into the warehouse (T3), then mirrors into MinIO |
| Deployment `warehouse-explorer` + NodePort 30089 | Read-only SQL across **all** offloaded flights | | Deployment `warehouse-explorer` + NodePort 30089 | Second explorer instance over the warehouse path — historical read-only SQL across **all** flights |
## Usage ## Usage
@@ -18,9 +23,16 @@ terraform apply
terraform output terraform output
``` ```
After offload runs, warehouse explorer shows same partition tree as live lake, accumulated across flights. [T1](../../../docs/00-glossary.md#t1--warm) layout **is** [T3](../../../docs/00-glossary.md#t3--warehouse) layout. No transform. No schema drift. Same DuckDB queries both ends. After a couple of offload runs, the warehouse explorer shows the same
partition tree as the live lake but accumulated across flights — the T1
layout **is** the T3 layout, which is the whole point: no transform step,
no schema drift, DuckDB queries work identically on both ends.
## Simulation vs production ## Simulation vs production
- Sim CronJob: `cp -ru` + `mc mirror` on a schedule — shortcut for "lands, offloads, prunes". Production: event-driven per dock, rsync or `mc mirror`, checksum audit before prune ([03 — Data platform](../../../docs/03-data-platform.md)). - The CronJob compresses "drone lands, docks, offloads, prunes" into a
- MinIO credentials here are throwaway defaults; production credentials sealed per site, never in state or VCS. periodic rsync-style copy. Production offload is event-driven per docking
and verifies checksums from the flight's partition manifest before pruning
the on-board lake.
- MinIO credentials here are throwaway defaults; production credentials are
sealed per site and never in state or VCS.
+19 -11
View File
@@ -1,14 +1,17 @@
# Terraform — simulation environment ([T4](../../../docs/00-glossary.md#t4--dev)) # Terraform — simulation environment (T4)
Sim fleet + observability on k3d. Cluster from [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml). Executable [T4](../../../docs/00-glossary.md#t4--dev) miniature ([06 — Environments](../../../docs/06-environments.md)). Declares the simulated fleet and its observability stack on the k3d cluster
created by [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml).
This module is the executable miniature of the dev/sim environment from
[06 — Environments](../../../docs/06-environments.md).
| Resource | Purpose | | Resource | Purpose |
| --- | --- | | --- | --- |
| StatefulSet `drone` × `drone_count` | Virtual drones; pod names `DRONE_ID`; shared `/data` lake | | StatefulSet `drone` × `drone_count` | Virtual drones; stable pod names become `DRONE_ID`s; all write Hive-partitioned Parquet into the shared `/data` lake |
| Deployment `exporter` | Prometheus exporter reading lake with DuckDB | | Deployment `exporter` | Prometheus exporter reading the lake with DuckDB |
| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL (prototype live mode) | | Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL console (also feeds the prototype's live mode) |
| Deployment `prometheus` + NodePort 30990 | Scrapes exporter | | Deployment `prometheus` + NodePort 30990 | Scrapes the exporter |
| Deployment `grafana` + NodePort 30300 | Dashboard JSON from ConfigMap | | Deployment `grafana` + NodePort 30300 | Same dashboard JSON as the Compose profile, provisioned from a ConfigMap |
## Usage ## Usage
@@ -19,7 +22,7 @@ terraform apply
terraform output # URLs terraform output # URLs
``` ```
Scale without YAML edits: Scale the fleet without touching YAML:
```bash ```bash
terraform apply -var drone_count=9 -var speedup=4 terraform apply -var drone_count=9 -var speedup=4
@@ -27,6 +30,11 @@ terraform apply -var drone_count=9 -var speedup=4
## Notes ## Notes
- `image_pull_policy = "Never"`image imported by Ansible (`k3d image import`); no registry pull. Air-gap mirror. - `image_pull_policy = "Never"`the simulator image is imported by the
- Pod name → identity (StatefulSet ordinal). `kubectl delete pod` = power cycle: new process, same identity, new flight ID. Ansible playbook (`k3d image import`); the cluster never pulls from a
- Local state only — cluster disposable. registry, mirroring the air-gap doctrine.
- Pods discover their identity from the pod name (StatefulSet ordinal), so
a flight survives `kubectl delete pod` the same way a drone survives a
power cycle: new process, same identity, new flight ID.
- State is local (`terraform.tfstate` in this directory) — the simulation
cluster is disposable; nothing here is shared infrastructure.
+6 -13
View File
@@ -1,8 +1,8 @@
# Swarm visualization prototype # Swarm visualization prototype
2D top-down view of swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol with obstacles, 5 Hz pose broadcasts (blue flashes), opportunistic bulk sync (green links with rate and byte counters). A 2D top-down view of the swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol an area with static and mobile obstacles, exchange 5 Hz pose broadcasts (blue link flashes), and run opportunistic bulk sync of sealed partitions (green links with live rate and cumulative up/down counters).
No backend. Client-side sim. Same behavioral model as Python simulator (Parquet + UDP). No backend — a pure client-side simulation of the same behavioral model the Python simulator implements over real Parquet and UDP.
## Run ## Run
@@ -11,16 +11,9 @@ npm install
npm run dev # http://localhost:5173 npm run dev # http://localhost:5173
``` ```
## Live mode
With k3d fleet + explorer running (`ansible-playbook` + `terraform apply` — see [sim-env README](../infra/terraform/sim-env/README.md)):
- Explorer at `http://localhost:30088`
- Header → **go live** — polls read-only SQL API for real drone positions
## What to look at ## What to look at
- Links form and break by radio range; broadcasts flash active links - **Links appear and disappear** as drones move in and out of radio range; persistent links render semi-transparent, broadcast deliveries flash them briefly.
- Busy links turn green — rate plus `↑` / `↓` totals - **Busy links turn green** and show current rate plus total transferred (`↑` / `↓`) — the bandwidth budget from the sync design made visible.
- Per-drone label: id, Wi-Fi channel, battery - **Per-drone label**: id, current Wi-Fi channel (hopping), battery.
- Controls: drone count, speed, pause - Controls: drone count, simulation speed, pause.
+1 -1
View File
@@ -268,7 +268,7 @@ export default function App(): JSX.Element {
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>} {busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
</Section> </Section>
<Section title="Legend"> <Section title="Legend">
<div style={styles.dim}>blue line pose broadcast (5 Hz, ~46 B)</div> <div style={styles.dim}>blue line pose broadcast (5 Hz, 45 B)</div>
<div style={styles.dim}>green line bulk sync (sealed partitions)</div> <div style={styles.dim}>green line bulk sync (sealed partitions)</div>
<div style={styles.dim}>flash broadcast event delivered</div> <div style={styles.dim}>flash broadcast event delivered</div>
<div style={styles.dim}>red circle transit object crossing the area</div> <div style={styles.dim}>red circle transit object crossing the area</div>
+1 -1
View File
@@ -57,7 +57,7 @@ export function areaWidth(): number {
export const LINK_RANGE = 320; export const LINK_RANGE = 320;
const CRUISE = 28; const CRUISE = 28;
const SEPARATION = 55; const SEPARATION = 55;
const POSE_BYTES = 46; const POSE_BYTES = 45;
const POSE_HZ = 5; const POSE_HZ = 5;
// Steady per-link broadcast throughput (both directions), bytes/s // Steady per-link broadcast throughput (both directions), bytes/s
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ; export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
+15 -14
View File
@@ -1,8 +1,8 @@
# Virtual drone fleet # Virtual drone fleet
Data generator. Same on-board pipeline as [03 — Data platform](../docs/03-data-platform.md): seeded kinematics, sensor streams, detection events, hot `current/` → sealed Parquet, compact UDP state broadcast. A data generator that exercises the **exact on-board pipeline** described in [03 — Data platform](../docs/03-data-platform.md): seeded kinematics along a patrol route, sensor streams at realistic rates, detection events, the hot `current/` → sealed Parquet write path, and the compact UDP state broadcast between drones.
One process = one drone. Scale swarm with Compose flag. One process = one drone. Scaling the swarm is a Compose flag.
## Run a swarm ## Run a swarm
@@ -14,7 +14,7 @@ FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --build --scale drone=5
DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10 DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10
``` ```
Output lands in [`../var/t1/`](../var/t1/) ([T1](../docs/00-glossary.md#t1--warm) live lake — same path k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed: Output lands in [`../var/t1/`](../var/t1/) (the shared T1 lake — same path the k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed:
``` ```
var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet
@@ -22,7 +22,7 @@ var/t1/dataset=detections/flight=…/drone=…/…
var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts
``` ```
Historical flights after offload: [`../var/t3/`](../var/t3/) ([T3](../docs/00-glossary.md#t3--warehouse) warehouse). Historical flights after offload live under [`../var/t3/`](../var/t3/) (T3 warehouse).
## Run a single drone without Docker ## Run a single drone without Docker
@@ -52,32 +52,33 @@ print(con.sql("""
EOF EOF
``` ```
## Monitoring ## Monitoring: Grafana over the generated data
`monitoring` profile: DuckDB exporter → Prometheus → Grafana ([07 — Observability](../docs/07-observability.md)): The `monitoring` profile spins up a small metrics chain — a DuckDB-based exporter that scans the generated Parquet every few seconds, Prometheus, and a pre-provisioned Grafana dashboard:
```bash ```bash
docker compose --profile monitoring up -d docker compose --profile monitoring up -d # exporter + prometheus + grafana
# generate some flights in parallel or beforehand:
FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5 FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5
``` ```
- Grafana: `http://localhost:3000` **Swarm Fleet — generated data** - Grafana: `http://localhost:3000` (anonymous admin — demo only) → dashboard **Swarm Fleet — generated data**
- Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics` - Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics`
Fleet stats derived from Parquet via SQL — no on-drone agent. Panels: telemetry rows by drone/sensor, detections by class, pose frames sent/received, battery per drone, RSSI and estimated distance per link, Parquet bytes/files on disk. The exporter is deliberately a demonstration of the observability doctrine from [07 — Observability](../docs/07-observability.md): fleet statistics are *derived from the data platform itself* — no agent on the drone, just SQL over the same Parquet everyone else reads.
## Data-plane explorer ## Data-plane explorer
`monitoring` profile also starts explorer at `http://localhost:8088` (k3d: `:30088`): The `monitoring` profile also starts a **data-plane explorer** at `http://localhost:8088` — a single-page web view over the lake itself, complementing Grafana (which shows aggregates, not the structure):
- Partition tree live: `dataset → flight → drone → sensor → hour → file` - **Partition tree**, live: `dataset → flight → drone → sensor → year/…/hour → file`, with file counts and bytes rolled up at every level. You can watch `current/` files appear and get sealed while a swarm is flying.
- Read-only SQL console — same [statement gate](../docs/04-swarm-sync.md) as peer query channel: `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only - **Read-only SQL console** with the datasets pre-registered as views (`telemetry`, `detections`, `state`) and one-click sample queries (fleet overview, battery timeline, detections by class, peer link quality, schema).
Point: bench exploration uses same read-only SQL contract as flight peers. The console enforces the **same statement gate as the peer query channel** from [04 — Swarm sync](../docs/04-swarm-sync.md): a single statement, `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only, write/config/extension keywords rejected. That is the point: exploring the data plane on the bench uses the same read-only SQL contract a peer drone uses in flight — the explorer is the "no debug API" principle made visible.
## Reproducibility ## Reproducibility
Deterministic per `(SEED, DRONE_ID)`. Bug report = seed + config. Every run is deterministic per `(SEED, DRONE_ID)`: same route jitter, same sensor noise, same detection sequence. A bug report is a seed and a config, not a description.
## Knobs ## Knobs
+1 -1
View File
@@ -9,7 +9,7 @@ from virtual_drone.flight import Pose
def test_frame_size_matches_wire_spec() -> None: def test_frame_size_matches_wire_spec() -> None:
# Documented as 46 B in docs/04; struct packs to FRAME.size (45 B on this layout). # Keep docs/04, the journey, and prototype/src/sim.ts POSE_BYTES in lockstep.
assert FRAME.size == 45 assert FRAME.size == 45
-70
View File
@@ -1,70 +0,0 @@
"""Tests for bin/check_adrs.py — ADR immutability gate."""
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
import pytest
SCRIPTS = Path(__file__).resolve().parents[2] / "bin"
sys.path.insert(0, str(SCRIPTS))
from check_adrs import find_modified_adrs, run_check # noqa: E402
def _git(cwd: Path, *args: str) -> None:
subprocess.check_call(["git", *args], cwd=cwd, stdout=subprocess.DEVNULL)
def test_no_modified_adrs_on_current_branch() -> None:
assert run_check(base="origin/main") == 0
def test_detects_adr_edit_in_sandbox(tmp_path: Path) -> None:
_git(tmp_path, "init")
_git(tmp_path, "config", "user.email", "ci@test")
_git(tmp_path, "config", "user.name", "ci")
adr = tmp_path / "docs/adr"
adr.mkdir(parents=True)
(adr / "ADR-0001-test.md").write_text("v1\n")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "add adr")
_git(tmp_path, "branch", "-M", "main")
(adr / "ADR-0001-test.md").write_text("v2\n")
_git(tmp_path, "checkout", "-b", "feature")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "edit adr")
monkeypatch = pytest.MonkeyPatch()
monkeypatch.setattr("check_adrs.ROOT", tmp_path)
try:
assert find_modified_adrs("main") == ["docs/adr/ADR-0001-test.md"]
assert run_check(base="main") == 1
finally:
monkeypatch.undo()
def test_new_adr_allowed(tmp_path: Path) -> None:
_git(tmp_path, "init")
_git(tmp_path, "config", "user.email", "ci@test")
_git(tmp_path, "config", "user.name", "ci")
adr = tmp_path / "docs/adr"
adr.mkdir(parents=True)
(adr / "ADR-0001-test.md").write_text("v1\n")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "add adr")
_git(tmp_path, "branch", "-M", "main")
(adr / "ADR-0002-new.md").write_text("new\n")
_git(tmp_path, "checkout", "-b", "feature")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "add adr 2")
monkeypatch = pytest.MonkeyPatch()
monkeypatch.setattr("check_adrs.ROOT", tmp_path)
try:
assert find_modified_adrs("main") == []
assert run_check(base="main") == 0
finally:
monkeypatch.undo()
-77
View File
@@ -1,77 +0,0 @@
"""Tests for bin/check_docs.py — documentation CI rules."""
from __future__ import annotations
import json
import sys
from pathlib import Path
import pytest
SCRIPTS = Path(__file__).resolve().parents[2] / "bin"
sys.path.insert(0, str(SCRIPTS))
from check_docs import ( # noqa: E402
collect_anchors,
run_checks,
slug_heading,
split_link_target,
)
ROOT = Path(__file__).resolve().parents[2]
GLOSSARY = ROOT / "docs/00-glossary.md"
def test_slug_heading_matches_glossary_anchors() -> None:
assert slug_heading("T0 — hot") == "t0--hot"
assert slug_heading("T3 — warehouse") == "t3--warehouse"
assert slug_heading("Source of truth") == "source-of-truth"
assert slug_heading("Data & storage") == "data--storage"
def test_glossary_term_anchors_exist() -> None:
anchors = collect_anchors(GLOSSARY)
for term in (
"t0--hot",
"t1--warm",
"t2--shared",
"t3--warehouse",
"t4--dev",
"source-of-truth",
"bulk-sync",
"offload",
):
assert term in anchors, f"missing glossary anchor #{term}"
def test_split_link_target() -> None:
assert split_link_target("foo.md#bar") == ("foo.md", "bar")
assert split_link_target("foo.md") == ("foo.md", None)
def test_repo_docs_pass() -> None:
assert run_checks() == 0
def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "00-glossary.md").write_text("# Glossary\n\n## Data & storage\n")
(docs / "bad.md").write_text("See [missing](missing.md).\n")
rules = tmp_path / "bin"
rules.mkdir()
(rules / "docs_rules.json").write_text(
json.dumps(
{
"markdown_paths": ["docs/**/*.md"],
"glossary": {"file": "docs/00-glossary.md"},
"banned_link_targets": [],
"glossary_section_anchors": [],
"preferred_term_anchors": [],
"readability": {"forbid_bare_see_refs": False},
}
)
)
monkeypatch.setattr("check_docs.ROOT", tmp_path)
monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.json")
assert run_checks() == 1
+4 -3
View File
@@ -1,12 +1,13 @@
"""State broadcast over UDP: the compact pose frame from the sync design. """State broadcast over UDP: the compact pose frame from the sync design.
Frame layout (little-endian, 46 bytes): Frame layout (little-endian, 45 bytes) — source of truth for docs and the
prototype bandwidth estimate:
magic 2s b"SH" magic 2s b"SH"
version B version B
drone_id 8s zero-padded ascii drone_id 8s zero-padded ascii (PoC; production may switch to uint16 registry)
ts_ns q epoch nanoseconds ts_ns q epoch nanoseconds
pos_mm 3i position, millimeters (quantized on the wire only) pos_mm 3i position in the mission frame, millimeters (wire quantization only)
att_cdeg 3h roll/pitch/yaw, centi-degrees att_cdeg 3h roll/pitch/yaw, centi-degrees
vel_cms 3h velocity, cm/s vel_cms 3h velocity, cm/s
frame_ref B frame_ref B