Author SHA1 Message Date
eSlider 302de4024d docs: move live demo links to PR #3
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 4s
CI & Release / Verify simulator (pull_request) Successful in 11s
CI & Release / Trivy scan (pull_request) Successful in 9s
CI & Release / Semantic Release (pull_request) Skipped
Drop duplicate external URLs from design journey; small PR #3 owns them.
2026-07-10 12:18:50 +01:00
eSlider 826cef0c3e fix: drop PyYAML dep so verify job pytest passes
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 4s
CI & Release / Verify simulator (pull_request) Successful in 11s
CI & Release / Trivy scan (pull_request) Successful in 8s
CI & Release / Semantic Release (pull_request) Skipped
Store doc rules in bin/docs_rules.json (stdlib json). test_check_docs no
longer fails in the simulator job that only installs simulator/requirements.txt.
2026-07-10 11:02:55 +01:00
eSlider df04736336 chore: drop unrelated files from bin/ move commit
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 6s
CI & Release / Verify simulator (pull_request) Failing after 9s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
2026-07-10 10:59:29 +01:00
eSlider b4de3ed076 refactor: move scripts/ to bin/
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 5s
CI & Release / Verify simulator (pull_request) Failing after 10s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Relocate check_docs, check_adrs, and docs_rules; update CI, tests, and docs.
2026-07-10 10:59:22 +01:00
eSlider 10a18f1321 fix: revert ADR edits and enforce immutable accepted records
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 6s
CI & Release / Verify simulator (pull_request) Failing after 10s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Restore docs/adr to main; add check_adrs.py CI gate so existing ADR-*.md
files cannot be modified — supersede with a new numbered ADR instead.
2026-07-10 10:57:39 +01:00
eSlider bed6f93626 docs: default caveman style for issues, PRs, and reviews
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 5s
CI & Release / Verify simulator (pull_request) Failing after 9s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Document caveman-full as the collaboration voice in AGENTS and CONTRIBUTING,
and add Gitea issue/PR templates that pre-fill terse structure.
2026-07-10 10:56:13 +01:00
eSlider e6e57843cb feat: add CI documentation rules checker and docs-verify job
CI & Release / Verify documentation (push) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify documentation (pull_request) Successful in 5s
CI & Release / Verify simulator (pull_request) Failing after 9s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Enforce link resolution, glossary anchors, and readability rules on every
push and PR; docs-only pushes to main still run docs-verify while skipping
the simulator gate.
2026-07-10 10:54:28 +01:00
eSlider 9bec50e8f7 fix(ci): fetch PR ref on checkout instead of shallow main clone
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Successful in 11s
CI & Release / Trivy scan (pull_request) Successful in 9s
CI & Release / Semantic Release (pull_request) Skipped
Shallow clone of default branch did not contain commits on feature
branches, causing pull_request verify jobs to fail at checkout.
2026-07-10 10:48:27 +01:00
eSlider 5f6fd8462c docs: audit links, glossary T0-T4, readability pass
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors.
Fix broken links, pose frame drift, rsync/offload wording, and README tone.
Add docs/improvement audit records and live demo links in design journey.
2026-07-10 10:46:59 +01:00
41 changed files with 1063 additions and 275 deletions
+8
View File
@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Design docs
url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/12-design-journey.md
about: Start with the design journey, then open an issue here.
- name: Open questions
url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/09-open-questions.md
about: Known unknowns — check before duplicating.
+24
View File
@@ -0,0 +1,24 @@
---
name: Change / task
about: Default issue template — caveman style
title: ""
labels: []
---
Write **caveman-full**: short lines, fragments OK, no filler. See [`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default).
## What
<!-- One line. What need fix or build. -->
## Why
<!-- Problem or goal. Skip if obvious from title. -->
## Done when
- [ ]
## Notes
<!-- Links, ADR, open question §. Optional. -->
+19
View File
@@ -0,0 +1,19 @@
Write **caveman-full** by default. See [`CONTRIBUTING.md`](../CONTRIBUTING.md#caveman-style-default).
## What
<!-- What changed. Bullets OK. -->
## Why
<!-- Problem this fixes or design gap closed. -->
## Test
- [ ] `pytest tests/ -q` (if code touched)
- [ ] `python bin/check_docs.py` (if docs touched)
- [ ] ADR added if boundary/contract changed
## Links
<!-- Closes / Relates to #issue. Optional demo URL. -->
+31 -5
View File
@@ -1,4 +1,5 @@
# Swarm House - Gitea Actions # Swarm House - Gitea Actions
# docs-verify: markdown links, glossary anchors, readability rules (every push/PR)
# verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR) # verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR)
# scan: Trivy vulnerability + misconfiguration scan # scan: Trivy vulnerability + misconfiguration scan
# release: semantic version bump (conventional commits), tag, Gitea release # release: semantic version bump (conventional commits), tag, Gitea release
@@ -17,6 +18,27 @@ on:
branches: [main] branches: [main]
jobs: jobs:
docs-verify:
name: Verify documentation
runs-on: ubuntu-latest
container: python:3.12-bookworm
steps:
- name: Checkout
run: |
git init .
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
git fetch --depth=50 origin "${{ gitea.ref }}"
git checkout FETCH_HEAD
- name: Fetch base branch for ADR immutability
run: git fetch --depth=1 origin main
- name: ADR immutability (no edits to accepted records)
run: python bin/check_adrs.py --base origin/main
- name: Documentation rules (links, glossary, readability)
run: python bin/check_docs.py
verify: verify:
name: Verify simulator name: Verify simulator
runs-on: ubuntu-latest runs-on: ubuntu-latest
@@ -24,8 +46,10 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
run: | run: |
git clone --depth=50 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git . git init .
git checkout ${{ gitea.sha }} git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
git fetch --depth=50 origin "${{ gitea.ref }}"
git checkout FETCH_HEAD
- name: Install dependencies - name: Install dependencies
run: pip install --quiet -r simulator/requirements.txt run: pip install --quiet -r simulator/requirements.txt
@@ -61,8 +85,10 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
run: | run: |
git clone --depth=1 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git . git init .
git checkout ${{ gitea.sha }} git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
git fetch --depth=1 origin "${{ gitea.ref }}"
git checkout FETCH_HEAD
- name: Install Trivy - name: Install Trivy
run: | run: |
@@ -81,7 +107,7 @@ jobs:
name: Semantic Release name: Semantic Release
runs-on: ubuntu-latest runs-on: ubuntu-latest
container: python:3.12-bookworm container: python:3.12-bookworm
needs: [verify, scan] needs: [docs-verify, verify, scan]
if: gitea.event_name == 'push' if: gitea.event_name == 'push'
permissions: permissions:
contents: write contents: write
-5
View File
@@ -13,8 +13,3 @@ dist/
.terraform/ .terraform/
*.tfstate *.tfstate
*.tfstate.* *.tfstate.*
# Local tooling / WIP (keep on disk, never commit)
.cursor/
.grok/
prototype-3d/
+23 -1
View File
@@ -24,6 +24,23 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development workflow and TDD re
## Conventions ## Conventions
### Caveman style (default for collaboration)
**Issues, pull requests, and code reviews** use **caveman-full** by default:
- Short sentences or fragments; drop articles and filler when clear
- Lead with fact, verdict, or ask — not preamble
- Bullets over paragraphs; one idea per line
- Keep technical terms, acronyms, paths, and code exact
- English only; no identifying references (same as all repo prose)
README tiers: root [`README.md`](README.md) = **caveman-lite** (full sentences, less filler);
component READMEs = **caveman-full**. Design docs in `docs/` stay readable prose unless
editing for consistency.
Templates: [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/),
[`.github/pull_request_template.md`](.github/pull_request_template.md).
- Markdown: ATX headings, fenced code blocks with language tags, pipe tables. - Markdown: ATX headings, fenced code blocks with language tags, pipe tables.
- Python: PEP 8, type hints, pure functions where possible, no unnecessary classes. - Python: PEP 8, type hints, pure functions where possible, no unnecessary classes.
- TypeScript: strict mode, functional components and hooks, no extra UI libraries. - TypeScript: strict mode, functional components and hooks, no extra UI libraries.
@@ -44,7 +61,12 @@ Status / Context / Options / Decision / Consequences format. Start from
When a change alters a boundary, contract, or trade-off, add an ADR (copy When a change alters a boundary, contract, or trade-off, add an ADR (copy
[`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the [`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the
index) and cross-link it from the affected design doc. A later reversal gets a index) and cross-link it from the affected design doc. A later reversal gets a
new ADR that supersedes the old one — never edit an Accepted record. new ADR that supersedes the old one — **never edit an Accepted record**.
**Immutable:** every `docs/adr/ADR-*.md` file is frozen once on `main`. CI rejects
any change that modifies an existing ADR. Updates go in a new numbered ADR only.
The index ([`docs/adr/README.md`](docs/adr/README.md)) and [`TEMPLATE.md`](docs/adr/TEMPLATE.md)
may still change; decision bodies do not.
## Runtime & operations (sim environment) ## Runtime & operations (sim environment)
+37 -1
View File
@@ -17,6 +17,7 @@ The CI pipeline enforces this:
| Gate | When it runs | | Gate | When it runs |
| --- | --- | | --- | --- |
| `python bin/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) |
| `pytest tests/` | Every push/PR (`verify` job) | | `pytest tests/` | Every push/PR (`verify` job) |
| `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image | | `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image |
| Smoke flight + DuckDB assertion | Every push/PR | | Smoke flight + DuckDB assertion | Every push/PR |
@@ -76,12 +77,47 @@ test: cover broadcast frame roundtrip
Never include AI tool references, `Co-authored-by` trailers for assistants, or Never include AI tool references, `Co-authored-by` trailers for assistants, or
identifying names of people, companies, or locations. identifying names of people, companies, or locations.
## Caveman style (default)
Use **caveman-full** in **issues**, **pull requests**, and **reviews** unless the
author explicitly asks otherwise.
| Do | Skip |
| --- | --- |
| Short lines, fragments OK | Long preamble ("I wanted to start by saying…") |
| Lead with what / why / verdict | Filler ("just", "basically", "kind of") |
| Bullets, checklists | Repeated context already in diff or issue |
| Exact terms, paths, code | Invented abbreviations or vague "the thing" |
**Examples**
| Fluffy | Caveman |
| --- | --- |
| I think we should probably consider adding a link checker to CI because links might break. | Add CI link checker. Broken relative links fail merge. |
| This pull request implements the documentation improvements that were discussed in the audit. | Docs audit: glossary T0T4, fix ground README link, rsync wording. |
| LGTM but maybe we could think about whether the anchor format is consistent? | Merge OK. Fix glossary anchors in follow-up (#1). |
README intensity: root = caveman-lite; `simulator/`, `prototype/`, `infra/`, `docs/adr/` READMEs =
caveman-full. Numbered design docs in `docs/` stay full prose.
Gitea pre-fills [issue](.github/ISSUE_TEMPLATE/default.md) and
[PR](.github/pull_request_template.md) templates with this style.
### Reviews
Default review voice = caveman-full:
1. **Verdict first** — merge / wait / no, or approve / request changes
2. **Findings** — bullet per blocker or nit; file + line when useful
3. **Scope** — say if issue AC met, partial, or out of scope
4. No essay; link ADR, issue, or doc section instead of restating design
## Pull request checklist ## Pull request checklist
- [ ] Tests added or updated; `pytest tests/ -q` passes locally - [ ] Tests added or updated; `pytest tests/ -q` passes locally
- [ ] `docker build` in `simulator/` succeeds (test stage included) - [ ] `docker build` in `simulator/` succeeds (test stage included)
- [ ] Documentation updated if behaviour or boundaries changed - [ ] Documentation updated if behaviour or boundaries changed
- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/)) - [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/)) — never edit an existing `ADR-*.md`; supersede with a new number
- [ ] English only; no identifying references - [ ] English only; no identifying references
- [ ] Conventional commit message - [ ] Conventional commit message
+11 -25
View File
@@ -2,9 +2,14 @@
**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective. **An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective.
A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale. Fleet flies autonomous. No internet uplink, no external access. Data stays in the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. Swarm exchanges just enough state to coordinate; everything else stays local until a ground warehouse receives it after landing — reproducibly, securely, at scale.
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware. This repository covers storage layout, sync strategy, network trust model, CI/CD chain, and the simulation environment that tests it without real hardware.
> **Restricted use.** Third parties may not use, copy, modify, or distribute this
> source code, documentation, or related materials without **explicit written
> permission** from the copyright holder. No license is granted by default. See
> [LICENSE](LICENSE) for the full terms.
## Design principles ## Design principles
@@ -17,25 +22,6 @@ This repository describes how to build, deliver, test, and operate that platform
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it. 7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies. 8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
## Implemented now vs proposed next
Honest map so a reader knows what runs today versus what is design intent.
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
| --- | --- | --- |
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, optional MinIO | k3s warehouse, GitOps overlays, post-flight mirror |
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
## Documentation ## Documentation
| Document | Contents | | Document | Contents |
@@ -43,7 +29,7 @@ Start from the [design journey](docs/12-design-journey.md) for the story; use th
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout | | [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions | | [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes | | [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path | | [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage floors (T0T4), partitioning, hot-write path |
| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard | | [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard |
| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport | | [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport |
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures | | [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
@@ -55,7 +41,7 @@ Start from the [design journey](docs/12-design-journey.md) for the story; use th
| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code | | [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code |
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made | | [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
New here? Read the [design journey](docs/12-design-journey.md) first: it tells the story chronologically and links straight into the code and decisions. The design principles below are the *what*; the [ADRs](docs/adr/) are the *why and when* — each principle traces to a dated, immutable decision record. New? Start with the [design journey](docs/12-design-journey.md) — story plus code links. Principles = *what*; [ADRs](docs/adr/) = *why and when*.
## Runnable parts ## Runnable parts
@@ -64,10 +50,10 @@ New here? Read the [design journey](docs/12-design-journey.md) first: it tells t
| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB | | [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG | | [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG |
| [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible | | [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible |
| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + T1→T3 offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider | | [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + [T1→T3](docs/03-data-platform.md#storage-floors-tiers) offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider |
| [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize | | [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize |
Shared runtime data (gitignored): `var/t1/` (live lake), `var/t3/` (warehouse). See [`CONTRIBUTING.md`](CONTRIBUTING.md). Shared runtime data (gitignored): [`var/t1/`](docs/00-glossary.md#t1--warm) (live lake, [T1](docs/00-glossary.md#t1--warm)), [`var/t3/`](docs/00-glossary.md#t3--warehouse) (warehouse, [T3](docs/00-glossary.md#t3--warehouse)). See [`CONTRIBUTING.md`](CONTRIBUTING.md).
## Quick start ## Quick start
+75
View File
@@ -0,0 +1,75 @@
#!/usr/bin/env python3
"""Reject modifications to accepted Architecture Decision Records."""
from __future__ import annotations
import argparse
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
ADR_PREFIX = "docs/adr/ADR-"
def _git(*args: str) -> str:
return subprocess.check_output(
["git", *args],
cwd=ROOT,
text=True,
stderr=subprocess.DEVNULL,
).strip()
def _exists_on(base: str, path: str) -> bool:
try:
_git("cat-file", "-e", f"{base}:{path}")
return True
except subprocess.CalledProcessError:
return False
def find_modified_adrs(base: str) -> list[str]:
try:
diff_out = _git("diff", "--name-only", f"{base}...HEAD", "--", "docs/adr/")
except subprocess.CalledProcessError:
return []
modified: list[str] = []
for rel in diff_out.splitlines():
if not rel.startswith(ADR_PREFIX) or not rel.endswith(".md"):
continue
if _exists_on(base, rel):
modified.append(rel)
return sorted(modified)
def run_check(base: str = "origin/main") -> int:
modified = find_modified_adrs(base)
if modified:
print("Immutable ADR violation — accepted records cannot be edited:")
for path in modified:
print(f" error: {path}")
print(
"\nReversal or refinement → new ADR-NNNN file. "
"Never edit an existing ADR-*.md."
)
return 1
print(f"ADR immutability OK (base {base})")
return 0
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--base",
default="origin/main",
help="Git ref to compare against (default: origin/main)",
)
args = parser.parse_args()
sys.exit(run_check(base=args.base))
if __name__ == "__main__":
main()
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""Validate markdown links, glossary anchors, and readability rules."""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
RULES_PATH = Path(__file__).with_name("docs_rules.json")
LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)")
BARE_SEE_RE = re.compile(r"(?<![\[(/])see\s+(\d{2})(?![\w-])", re.IGNORECASE)
SKIP_SCHEMES = ("http://", "https://", "mailto:", "tel:", "#")
def load_rules(path: Path = RULES_PATH) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def slug_heading(text: str) -> str:
"""Anchor slug used in this repository's glossary links."""
text = text.strip()
text = re.sub(r"\s&\s", "", text)
if "" in text:
left, right = text.split("", 1)
return f"{_slug_part(left)}--{_slug_part(right)}"
return _slug_part(text)
def _slug_part(text: str) -> str:
slug = text.strip().lower()
slug = re.sub(r"[^\w\s-]", "", slug)
slug = re.sub(r"\s+", "-", slug)
return slug
def collect_anchors(md_path: Path) -> set[str]:
anchors: set[str] = set()
for line in md_path.read_text(encoding="utf-8").splitlines():
if line.startswith("##"):
anchors.add(slug_heading(line.lstrip("#").strip()))
return anchors
def expand_markdown_paths(patterns: list[str]) -> list[Path]:
files: set[Path] = set()
for pattern in patterns:
for path in ROOT.glob(pattern):
if path.is_file() and path.suffix == ".md":
files.add(path)
return sorted(files)
def split_link_target(raw: str) -> tuple[str, str | None]:
target = raw.strip()
if target.startswith("<") and target.endswith(">"):
target = target[1:-1]
if "#" in target:
path_part, fragment = target.split("#", 1)
return path_part, fragment or None
return target, None
def resolve_path(source: Path, link_path: str) -> Path | None:
if not link_path or link_path.startswith(SKIP_SCHEMES):
return None
candidate = (source.parent / link_path).resolve()
try:
candidate.relative_to(ROOT.resolve())
except ValueError:
return None
return candidate
def is_glossary_link(link_path: str, rules: dict) -> bool:
glossary = rules["glossary"]["file"].replace("\\", "/")
normalized = link_path.replace("\\", "/")
return (
normalized == glossary
or normalized.endswith("/" + glossary)
or normalized.endswith(glossary.split("/")[-1])
)
def check_file(
md_path: Path,
rules: dict,
anchor_cache: dict[Path, set[str]],
) -> tuple[list[str], list[str]]:
errors: list[str] = []
warnings: list[str] = []
rel = md_path.relative_to(ROOT)
text = md_path.read_text(encoding="utf-8")
if rules.get("readability", {}).get("forbid_bare_see_refs"):
prose = re.sub(r"`[^`]*`", "", text)
for match in BARE_SEE_RE.finditer(prose):
errors.append(
f"{rel}:{_line_no(text, match.start())}: bare cross-ref "
f"'see {match.group(1)}' — use a markdown link"
)
for match in LINK_RE.finditer(text):
raw = match.group(1)
if any(raw.startswith(s) for s in SKIP_SCHEMES):
continue
link_path, fragment = split_link_target(raw)
if not link_path.endswith(".md"):
continue
for banned in rules.get("banned_link_targets", []):
if banned in link_path.replace("\\", "/"):
errors.append(
f"{rel}:{_line_no(text, match.start())}: banned link target "
f"'{link_path}'"
)
target = resolve_path(md_path, link_path)
if target is None:
errors.append(
f"{rel}:{_line_no(text, match.start())}: link escapes repo or "
f"is invalid: {raw}"
)
continue
if not target.is_file():
errors.append(
f"{rel}:{_line_no(text, match.start())}: broken link "
f"{link_path} -> {target.relative_to(ROOT)}"
)
continue
if fragment:
if target not in anchor_cache:
anchor_cache[target] = collect_anchors(target)
if fragment not in anchor_cache[target]:
errors.append(
f"{rel}:{_line_no(text, match.start())}: unknown anchor "
f"#{fragment} in {target.relative_to(ROOT)}"
)
if fragment and is_glossary_link(link_path, rules):
section_anchors = set(rules.get("glossary_section_anchors", []))
if fragment in section_anchors:
warnings.append(
f"{rel}:{_line_no(text, match.start())}: glossary section "
f"link #{fragment} — prefer a ### term anchor where one exists"
)
return errors, warnings
def _line_no(text: str, index: int) -> int:
return text.count("\n", 0, index) + 1
def run_checks(warnings_as_errors: bool = False) -> int:
rules = load_rules(RULES_PATH)
anchor_cache: dict[Path, set[str]] = {}
all_errors: list[str] = []
all_warnings: list[str] = []
glossary_path = ROOT / rules["glossary"]["file"]
anchor_cache[glossary_path] = collect_anchors(glossary_path)
for md_path in expand_markdown_paths(rules["markdown_paths"]):
errors, warnings = check_file(md_path, rules, anchor_cache)
all_errors.extend(errors)
all_warnings.extend(warnings)
if all_warnings:
print("Documentation warnings:")
for msg in sorted(all_warnings):
print(f" warning: {msg}")
print()
if all_errors:
print("Documentation errors:")
for msg in sorted(all_errors):
print(f" error: {msg}")
print(f"\n{len(all_errors)} error(s), {len(all_warnings)} warning(s)")
return 1
if warnings_as_errors and all_warnings:
print(f"\n{len(all_warnings)} warning(s) treated as errors")
return 1
print(
f"Documentation checks passed "
f"({len(expand_markdown_paths(rules['markdown_paths']))} files, "
f"{len(all_warnings)} warning(s))"
)
return 0
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--warnings-as-errors",
action="store_true",
help="Exit non-zero when section-link warnings are present",
)
args = parser.parse_args()
sys.exit(run_checks(warnings_as_errors=args.warnings_as_errors))
if __name__ == "__main__":
main()
+42
View File
@@ -0,0 +1,42 @@
{
"markdown_paths": [
"docs/**/*.md",
"README.md",
"CONTRIBUTING.md",
"AGENTS.md",
"simulator/README.md",
"prototype/README.md",
"infra/**/README.md"
],
"glossary": {
"file": "docs/00-glossary.md"
},
"banned_link_targets": [
"03-storage-design.md"
],
"glossary_section_anchors": [
"data--storage",
"swarm--robotics",
"infrastructure--delivery"
],
"preferred_term_anchors": [
"t0--hot",
"t1--warm",
"t2--shared",
"t3--warehouse",
"t4--dev",
"source-of-truth",
"architecture-layer-1--ingestion",
"architecture-layer-2--storage-and-transform",
"architecture-layer-3--serving-and-sync",
"pipeline-stage",
"bulk-sync",
"offload",
"rclone",
"adr",
"asr"
],
"readability": {
"forbid_bare_see_refs": true
}
}
+66 -2
View File
@@ -2,6 +2,70 @@
Terms and abbreviations used throughout this proposal. Terms and abbreviations used throughout this proposal.
**Vocabulary guide:** **architecture layers** (13) = on-board software stack; **storage floors** (T0T4) = retention/latency ladder; **pipeline stages** = narrative capture → reduction → sync only ([12 — Design journey](12-design-journey.md) §4). Do not mix these three.
## Project vocabulary
### T0 — hot
RAM / DuckDB in-process sliding window of the last minutes; what mission logic queries in flight. Retention: minutes. Maps to **hot** in the classic three-tier model.
### T1 — warm
Drone NVMe: full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight. Retention: current flight plus quota-based headroom. Maps to **warm**. Runtime path: [`var/t1/`](../var/t1/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)).
### T2 — shared
On-board MinIO bucket: derived data only (`detections`, `state`), replicated opportunistically across the swarm. Retention: current mission.
### T3 — warehouse
Ground on-prem object store + Parquet lakehouse: every flight of every drone, forever; replay, analytics, model training. Retention: years. Maps to **cold** / **DWH**. Runtime path: [`var/t3/`](../var/t3/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)). After offload and integrity audit, T3 is the [source of truth](#source-of-truth) for historical flight data.
### T4 — dev
Engineer laptop or sim farm: slices pulled from T3, or synthetic data from the simulator. Retention: ephemeral.
### Source of truth
The authoritative copy of flight data after [offload](#offload) and integrity audit: the [T3](#t3--warehouse) warehouse. Each drone holds its own partial record in flight ([T1](#t1--warm)); peers exchange derived state only ([04 — Swarm sync](04-swarm-sync.md)). Replay, post-flight fusion, training, and audit read from T3 — not from any single drone's local store. See [06 — Environments](06-environments.md), [ADR-0003](adr/ADR-0003-sync-derived-state-only.md).
### Architecture layer 1 — ingestion
On-board services that capture sensor streams and video analytics ([02 — Architecture](02-architecture.md)): `sensor-ingest`, `video-analytics`.
### Architecture layer 2 — storage and transform
On-board Parquet write path, sealer, DuckDB, NVMe ([02 — Architecture](02-architecture.md)).
### Architecture layer 3 — serving and sync
On-board event hook, pub/sub broadcast, MinIO derived bucket, peer query API ([02 — Architecture](02-architecture.md)).
### Pipeline stage
Narrative inside each drone: (1) raw capture, (2) ETL/reduction, (3) bidirectional sync interface. Not the same as storage floors T0T4 ([12 — Design journey](12-design-journey.md) §4).
### Bulk sync
In-flight pull of sealed `detections`/`state` Parquet partitions between peers via **rsync** delta over persistent SSH forced commands ([04 — Swarm sync](04-swarm-sync.md)).
### Offload
Post-flight mirror of sealed T1 partitions into T3 at the dock: **rsync** or MinIO **`mc mirror`**, then integrity audit before local prune ([03 — Data platform](03-data-platform.md)).
### rclone
Optional rsync-class tool when the far end is object storage and rsync is awkward; not the default in-flight mesh path ([09 — Open questions](09-open-questions.md) §12).
### ADR
Architecture Decision Record — a point-in-time decision for this repository ([adr/README.md](adr/README.md)).
### ASR
Architecture Standard Record — a standing cross-repository policy (lives in the ecosystem `inventar` repo, not here). See [adr/README.md](adr/README.md).
## Data & storage ## Data & storage
| Term | Meaning | | Term | Meaning |
@@ -10,8 +74,8 @@ Terms and abbreviations used throughout this proposal.
| **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) | | **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) |
| **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate | | **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate |
| **ZSTD** | Zstandard compression; the default codec for all Parquet files here | | **ZSTD** | Zstandard compression; the default codec for all Parquet files here |
| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown | | **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown; same role as [source of truth](#source-of-truth) |
| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse | | **Hot / warm / cold tiers** | Classic three-tier model: in-memory window → local NVMe → ground warehouse. Here mapped to [T0](#t0--hot), [T1](#t1--warm), and [T3](#t3--warehouse); [T2](#t2--shared) and [T4](#t4--dev) are named explicitly |
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes | | **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier | | **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
| **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame | | **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame |
+3 -3
View File
@@ -15,7 +15,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
| Constraint | Consequence | | Constraint | Consequence |
| --- | --- | | --- | --- |
| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind | | **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention | | **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention. A [supervising operator](00-glossary.md#swarm--robotics) may set mission intent over [C2](00-glossary.md#swarm--robotics) before and during flight, but does not steer individual vehicles |
| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator | | **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch | | **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight | | **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
@@ -30,8 +30,8 @@ Plus everything a platform needs around that: reproducible builds and deployment
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked. - **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
- **MinIO** is available and considered as the inter-drone sync mechanism. - **MinIO** is available and considered as the inter-drone sync mechanism.
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)). - A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training. - After landing, each drone's data is **offloaded to the on-prem [source of truth](00-glossary.md#source-of-truth)** for replay and iterative model training.
- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight. - Mission intent (declarative goals) reaches the swarm over a narrow **[C2](00-glossary.md#swarm--robotics) channel**; there is no continuous ground link in flight.
## What is assumed (explicitly marked as assumptions) ## What is assumed (explicitly marked as assumptions)
+13 -16
View File
@@ -2,7 +2,7 @@
## On-board: three layers, one Compose file ## On-board: three layers, one Compose file
Every drone runs the same three-layer data plane. **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)). Every drone runs the same [three-layer](00-glossary.md#architecture-layer-1--ingestion) data plane ([storage floors](00-glossary.md#t0--hot) T0T4 are a separate concept — see [03 — Data platform](03-data-platform.md#storage-floors-tiers)). **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)).
```mermaid ```mermaid
graph TB graph TB
@@ -20,9 +20,8 @@ graph TB
subgraph serving [Layer 3 — Serving and sync] subgraph serving [Layer 3 — Serving and sync]
HOOK["event hook<br/>fires on new derived data"] HOOK["event hook<br/>fires on new derived data"]
PUB["state publisher<br/>pub/sub broadcast"] PUB["state publisher<br/>pub/sub broadcast"]
BULK["bulk sync<br/>rsync over SSH"] MINIO["MinIO<br/>derived datasets bucket"]
MINIO["MinIO<br/>optional derived bucket"] QAPI["query API<br/>peer data requests"]
QAPI["query API<br/>SQL-over-SSH"]
end end
end end
@@ -34,18 +33,16 @@ graph TB
WRITER -->|derived rows| HOOK WRITER -->|derived rows| HOOK
HOOK --> PUB HOOK --> PUB
HOOK --> MINIO HOOK --> MINIO
HOOK --> BULK
QAPI --> DUCK QAPI --> DUCK
PUB -.->|mesh pose| PEERS["peer drones"] PUB -.->|mesh| PEERS["peer drones"]
BULK -.->|sealed partitions| PEERS MINIO -.->|replication| PEERS
QAPI -.->|on demand| PEERS QAPI -.->|on demand| PEERS
``` ```
MinIO stays available where an S3 API helps (on-board derived datasets, ground warehouse). **In-flight peer bulk sync is SSH/rsync**, not object-store replication — see [04 — Swarm sync](04-swarm-sync.md).
### Layer 1 — Ingestion ### Layer 1 — Ingestion
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on. - `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
- `video-analytics` runs a **YOLO-like object detector** on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images. - `video-analytics` runs a [**YOLO-like**](00-glossary.md#swarm--robotics) object detector on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images.
- Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient). - Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient).
### Layer 2 — Storage and transform ### Layer 2 — Storage and transform
@@ -56,10 +53,10 @@ MinIO stays available where an S3 API helps (on-board derived datasets, ground w
### Layer 3 — Serving and sync ### Layer 3 — Serving and sync
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, optional MinIO put, bulk-sync hint, or a local mission-logic callback. Nothing polls. - The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (UDP in the PoC; Zenoh proposed — [04](04-swarm-sync.md)). - The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer). MinIO is optional where an S3 API is wanted; it is **not** the in-flight peer replication path. - **Bulk sync** pulls sealed derived partitions from peers over persistent SSH ([rsync](00-glossary.md#bulk-sync) delta transfer); MinIO remains optional where an S3 API is wanted.
- **Peer queries** are read-only DuckDB SQL — HTTP explorer gate in the PoC; **SSH forced commands** proposed for flight (SELECT-only gate, read-only OS user, columnar responses [04](04-swarm-sync.md)). - **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)).
## Communication planes ## Communication planes
@@ -68,11 +65,11 @@ Three isolated planes with different lifecycles:
```mermaid ```mermaid
graph LR graph LR
subgraph planes [Communication planes] subgraph planes [Communication planes]
C2["C2 control plane<br/>mission intent, narrow, in production"] C2["[C2](00-glossary.md#swarm--robotics) control plane<br/>mission intent, narrow, in production"]
DATA["swarm data plane<br/>state broadcast + sync, in production"] DATA["swarm data plane<br/>state broadcast + sync, in production"]
DEV["dev/debug plane<br/>live telemetry, bench only"] DEV["dev/debug plane<br/>live telemetry, bench only"]
end end
OPERATOR["supervising operator<br/>(man-in-the-loop)"] --> C2 OPERATOR["supervising operator<br/>([man-in-the-loop](00-glossary.md#swarm--robotics))"] --> C2
C2 --> SWARM["swarm"] C2 --> SWARM["swarm"]
SWARM <--> DATA SWARM <--> DATA
DEV -.->|absent from production builds| SWARM DEV -.->|absent from production builds| SWARM
@@ -82,7 +79,7 @@ graph LR
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes | | **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes |
| **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes | | **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes |
| **Dev/debug plane** | Full live telemetry for bench development and HIL tests | High | **No — physically absent from production builds** | | **Dev/debug plane** | Full live telemetry for bench development and [HIL](00-glossary.md#infrastructure--delivery) tests | High | **No — physically absent from production builds** |
The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it. The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it.
+15 -13
View File
@@ -1,6 +1,6 @@
# 03 — Data platform # 03 — Data platform
The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every floor** of the system. The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every [storage floor](00-glossary.md#t0--hot)** of the system.
## Datasets ## Datasets
@@ -14,7 +14,7 @@ Three logical datasets, distinct because their consumers and sync policies diffe
## Partitioning layout ## Partitioning layout
Identity dimensions first, calendar time last, everything Hive-style `key=value`: Identity dimensions first, calendar time last, everything [Hive-style](00-glossary.md#data--storage) `key=value`:
``` ```
dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/ dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/
@@ -29,7 +29,7 @@ Design decisions and their reasons:
- **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.) - **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.)
- **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both. - **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both.
- **Flight identifiers.** The canonical flight id is a **UUIDv7** (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest. - **Flight identifiers.** The canonical flight id is a [**UUIDv7**](00-glossary.md#data--storage) (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
- **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time. - **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time.
- **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)). - **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)).
@@ -50,7 +50,7 @@ sequenceDiagram
S->>W: rows (int64 ns, typed) S->>W: rows (int64 ns, typed)
W->>C: flush min_NN.parquet (small, ZSTD-fast) W->>C: flush min_NN.parquet (small, ZSTD-fast)
end end
Note over Z: window closes (hour sealed<br/>or flight ends) Note over Z: window closes (hour [sealed](00-glossary.md#data--storage)<br/>or flight ends)
Z->>C: read all min_*.parquet Z->>C: read all min_*.parquet
Z->>P: write one compact file (ZSTD-high) Z->>P: write one compact file (ZSTD-high)
Z->>C: remove blocks after verify Z->>C: remove blocks after verify
@@ -63,26 +63,28 @@ sequenceDiagram
## Storage floors (tiers) ## Storage floors (tiers)
Same format on every floor; only volume, retention, and location change: Same [Parquet](00-glossary.md#data--storage) + [DuckDB](00-glossary.md#data--storage) format on every floor; only volume, retention, and location change. See [glossary](00-glossary.md#t0--hot) for the full T0T4 definitions.
| Floor | Where | Contents | Retention | | Floor | Where | Contents | Retention |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes | | **[T0 — hot](00-glossary.md#t0--hot)** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
| **T1 — warm** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) | | **[T1 — warm](00-glossary.md#t1--warm)** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
| **T2 — shared** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission | | **[T2 — shared](00-glossary.md#t2--shared)** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission |
| **T3 — warehouse** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; replay, analytics, model training | Years | | **[T3 — warehouse](00-glossary.md#t3--warehouse)** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; [source of truth](00-glossary.md#source-of-truth) for replay, analytics, model training | Years |
| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral | | **[T4 — dev](00-glossary.md#t4--dev)** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
## Flight offload: a mirror, not a migration ## Flight offload: a mirror, not a migration
Because T1 and T3 share the identical layout, offload after landing is: Because [T1](00-glossary.md#t1--warm) and [T3](00-glossary.md#t3--warehouse) share the identical layout, [offload](00-glossary.md#offload) after landing is:
```bash ```bash
# per drone, at the base station # per drone, at the base station
mc mirror drone-nvme/flights/ warehouse/flights/ # or rsync over the wired dock rsync -a drone-nvme/flights/ warehouse/flights/ # or mc mirror to object store
duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…" duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…"
``` ```
For S3-only backends where rsync is awkward, [rclone](00-glossary.md#rclone) is an optional rsync-class alternative — not the default mesh path ([09 — Open questions](09-open-questions.md) §12).
The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job. The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job.
## Retention and quotas on board ## Retention and quotas on board
@@ -138,6 +140,6 @@ On the ground this powers replay overlays, coverage analysis (did ubiquitous sen
## Schema management ## Schema management
- Every dataset schema is versioned in-repo (`schemas/` SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)). - Every dataset schema is versioned in-repo (planned `schemas/` directory — [roadmap stage 2](08-roadmap.md), [09 — Open questions](09-open-questions.md) §14; SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
- Parquet files carry the schema version in file metadata; readers select the matching deserializer. - Parquet files carry the schema version in file metadata; readers select the matching deserializer.
- Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release. - Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release.
+20 -28
View File
@@ -16,27 +16,21 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex
## Broadcast payload: small on the wire, precise at rest ## Broadcast payload: small on the wire, precise at rest
The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as The `state` broadcast is a fixed compact frame (46 bytes, little-endian):
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
what the prototype uses for bandwidth estimates.
| Field | Wire type (PoC) | Notes | | Field | Type | Notes |
| --- | --- | --- | | --- | --- | --- |
| `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` | | `magic` | 2 bytes | `b"SH"` |
| `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap | | `version` | uint8 | Protocol version |
| `drone_id` | 8 bytes | Zero-padded ASCII fleet id |
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage | | `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
| `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision | | `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision |
| `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model | | `att_roll/pitch/yaw` | int16 | Centi-degrees |
| `vel_x/y/z` | 3 × int16 | cm/s | | `vel_x/y/z` | int16 | cm/s |
| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) | | `frame_ref` | uint8 | Frame of reference id ([GPS-denied](00-glossary.md#swarm--robotics): local/visual-odometry frames must be explicit) |
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … | | `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh. 46 bytes per frame → a 50-drone swarm at 5 Hz is ~11.5 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.
An earlier sketch used relative coordinates and a bounding sphere (no attitude).
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
costs almost nothing at this frame size. Relative localization remains a
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison. `detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
@@ -59,32 +53,30 @@ graph LR
SB -->|"store-and-forward relay"| SC SB -->|"store-and-forward relay"| SC
``` ```
### Recommended (proposal): Zenoh ### Recommended: [Zenoh](00-glossary.md#swarm--robotics)
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub. - Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware. - First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
- Tiny footprint, ARM64-native. - Tiny footprint, ARM64-native.
**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.
### Alternatives considered ### Alternatives considered
| Option | Verdict | | Option | Verdict |
| --- | --- | | --- | --- |
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively | | **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh | | **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
| **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode | | **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode |
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path; optional on board for derived datasets and primary on the ground warehouse — not the in-flight bulk path | | **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below |
## Two sync mechanisms, deliberately separate ## Two sync mechanisms, deliberately separate
1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache. 1. **Fast path — pub/sub ([Zenoh](00-glossary.md#swarm--robotics)):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones. 2. **Bulk path — [rsync](00-glossary.md#bulk-sync) over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
Why SSH-based bulk sync over object-store replication: Why SSH-based bulk sync over object-store replication:
- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery. - **Identity is already there.** Every drone holds pre-provisioned [ed25519](00-glossary.md#infrastructure--delivery) keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes. - **One persistent multiplexed session** ([`ControlMaster`](00-glossary.md#infrastructure--delivery)) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing. - **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it. - **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.
@@ -93,7 +85,7 @@ Partition healing is automatic: replication is pull-based, addressed by partitio
Two deliberate non-features of the SSH channel: Two deliberate non-features of the SSH channel:
- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount. - **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction. - **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH [forced command](00-glossary.md#infrastructure--delivery)** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.
## Design decision: SQL-over-SSH as the peer query channel ## Design decision: SQL-over-SSH as the peer query channel
@@ -115,7 +107,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
| Layer | Mechanism | What it stops | | Layer | Mechanism | What it stops |
| --- | --- | --- | | --- | --- | --- |
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement | | 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
| 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering | | 2. Statement gate | Wrapper accepts a single statement, parses it, rejects anything but `SELECT` (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); parameters bound, not interpolated | SQL-as-a-write-channel, config tampering |
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 | | 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store | | 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries | | 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
@@ -124,7 +116,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction. - **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
- **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs. - **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it. - **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from [T3](00-glossary.md#t3--warehouse) if result sizes outgrow it.
What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one. What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.
+4 -4
View File
@@ -23,9 +23,9 @@ graph TB
A["drone A<br/>wg key + certs"] A["drone A<br/>wg key + certs"]
B["drone B<br/>wg key + certs"] B["drone B<br/>wg key + certs"]
C["drone C<br/>wg key + certs"] C["drone C<br/>wg key + certs"]
A <-->|"WireGuard + mTLS"| B A <-->|"WireGuard tunnel"| B
B <-->|"WireGuard + mTLS"| C B <-->|"WireGuard tunnel"| C
A <-->|"WireGuard + mTLS"| C A <-->|"WireGuard tunnel"| C
end end
PROV -->|"per-device identity, peer list"| A PROV -->|"per-device identity, peer list"| A
PROV --> B PROV --> B
@@ -65,7 +65,7 @@ Keep both layers: WireGuard for *who is on the network and whether the wire is r
## Data at rest ## Data at rest
- On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable. - On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable.
- The ground warehouse applies standard at-rest encryption plus role-based access; it is the single most valuable asset in the system (every flight ever flown). - The ground warehouse applies standard at-rest encryption plus role-based access; it is the [source of truth](00-glossary.md#source-of-truth) and the single most valuable asset in the system (every flight ever flown).
## Attack surface, deliberately shortened ## Attack surface, deliberately shortened
+9 -9
View File
@@ -9,15 +9,15 @@ graph LR
subgraph dev [3 — Dev and simulation] subgraph dev [3 — Dev and simulation]
SIM["virtual drone fleet<br/>(simulator, Compose)"] SIM["virtual drone fleet<br/>(simulator, Compose)"]
LAP["engineer laptop<br/>Dev Container"] LAP["engineer laptop<br/>Dev Container"]
SLICE["T4: data slices"] SLICE["[T4](00-glossary.md#t4--dev): data slices"]
end end
subgraph fleet [1 — Fleet, in flight] subgraph fleet [1 — Fleet, in flight]
D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 MinIO"] D1["drone: Compose data plane<br/>[T0](00-glossary.md#t0--hot) hot + [T1](00-glossary.md#t1--warm) NVMe + [T2](00-glossary.md#t2--shared) MinIO"]
D2["drone …"] D2["drone …"]
end end
subgraph ground [2 — Ground, on-prem] subgraph ground [2 — Ground, on-prem]
DOCK["base station docks"] DOCK["base station docks"]
DWH["T3 warehouse<br/>object store + Parquet/DuckDB"] DWH["[T3](00-glossary.md#t3--warehouse) warehouse<br/>object store + Parquet/DuckDB"]
K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"] K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
TRAIN["model training (out of scope)<br/>reads T3, ships weights"] TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
end end
@@ -34,7 +34,7 @@ graph LR
## 1 — Fleet infrastructure (on the drones) ## 1 — Fleet infrastructure (on the drones)
- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md). - Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md).
- Storage floors T0T2. Nothing here requires ground contact during a mission. - Storage floors [T0T2](00-glossary.md#t0--hot). Nothing here requires ground contact during a mission.
- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest. - Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.
## 2 — Ground infrastructure (on-prem) ## 2 — Ground infrastructure (on-prem)
@@ -43,7 +43,7 @@ The permanent installation. This **is** allowed to be a cluster — links are wi
| Component | Runs on | Role | | Component | Runs on | Role |
| --- | --- | --- | | --- | --- | --- |
| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory | | **Warehouse ([T3](00-glossary.md#t3--warehouse))** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's [source of truth](00-glossary.md#source-of-truth) and long-term memory |
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) | | **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see | | **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) | | **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
@@ -60,7 +60,7 @@ The environment engineers live in daily — and deliberately the first thing to
- **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free. - **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free.
- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop. - **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop.
- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story. - **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified. - **[T4](00-glossary.md#t4--dev) data slices**: a thin tool pulls partition subsets from [T3](00-glossary.md#t3--warehouse) (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets). - The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).
## IaC boundaries ## IaC boundaries
@@ -74,13 +74,13 @@ Infrastructure as code follows the same discipline as the data plane: each tool
| Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. | | Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. |
| On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). | | On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). |
Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, T1) and **`var/t3/`** (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree. Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, [T1](00-glossary.md#t1--warm)) and **`var/t3/`** (warehouse, [T3](00-glossary.md#t3--warehouse)). Compose, k3d, and DuckDB on the host all read the same Parquet tree.
Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle. Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle.
### k3d: the ground segment as an executable miniature ### k3d: the ground segment as an executable miniature
`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API. `ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the [T1](00-glossary.md#t1--warm)→[T3](00-glossary.md#t3--warehouse) offload CronJob. The sim CronJob uses `cp -ru` plus `mc mirror` as a shortcut; production dock [offload](00-glossary.md#offload) uses rsync or `mc mirror` with checksum audit ([03](03-data-platform.md)). The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API.
### Next fidelity step: microVMs ### Next fidelity step: microVMs
@@ -90,7 +90,7 @@ The k3d fleet shares one kernel and one network namespace tree — good enough t
| Operation | What it costs with one layout everywhere | | Operation | What it costs with one layout everywhere |
| --- | --- | | --- | --- |
| Flight offload | `mc mirror` / `rsync` + audit — no transform step | | Flight offload | `rsync` / `mc mirror` + audit — no transform step ([03](03-data-platform.md)) |
| Debugging a field issue | Pull the flight slice, replay locally, same queries | | Debugging a field issue | Pull the flight slice, replay locally, same queries |
| Validating a pipeline change | Run simulator, diff Parquet outputs | | Validating a pipeline change | Run simulator, diff Parquet outputs |
| Training data prep | Read T3 directly, no export pipeline | | Training data prep | Read T3 directly, no export pipeline |
+2 -2
View File
@@ -29,7 +29,7 @@ graph LR
LG["structured logs<br/>= Parquet partitions"] LG["structured logs<br/>= Parquet partitions"]
end end
subgraph ground [Ground k3s] subgraph ground [Ground k3s]
DWH["T3 warehouse"] DWH["[T3](00-glossary.md#t3--warehouse) warehouse"]
PR["Prometheus<br/>(live: docks, CI, sim farm)"] PR["Prometheus<br/>(live: docks, CI, sim farm)"]
LK["Loki<br/>(live logs, dev plane)"] LK["Loki<br/>(live logs, dev plane)"]
GF["Grafana<br/>dashboards + replay"] GF["Grafana<br/>dashboards + replay"]
@@ -44,7 +44,7 @@ graph LR
``` ```
- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane. - **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute. - **Flight post-mortems query the [source of truth](00-glossary.md#source-of-truth) warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.
## Health questions this design answers cheaply ## Health questions this design answers cheaply
+1 -1
View File
@@ -12,7 +12,7 @@ graph TD
S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"] S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"]
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"] S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"] S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, MinIO replication"] S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk replication (rsync over SSH)"]
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"] S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"]
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"] S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"] S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
+21 -3
View File
@@ -18,7 +18,7 @@ Visual-odometry positions are relative; different drones may hold different loca
By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging. By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging.
*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from ground truth in the warehouse. If a persistent shared structure becomes unavoidable, use a CRDT type for it rather than inventing reconciliation. *Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from the [source of truth](00-glossary.md#source-of-truth) in the warehouse. If a persistent shared structure becomes unavoidable, use a [CRDT](00-glossary.md#infrastructure--delivery) type for it rather than inventing reconciliation.
## 4 — Degradation policy under sustained link loss ## 4 — Degradation policy under sustained link loss
@@ -42,7 +42,7 @@ The degradation ladder ([03](03-data-platform.md)) needs a domain-approved order
The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration. The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration.
*Proposal:* a small HIL rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI. *Proposal:* a small [HIL](00-glossary.md#infrastructure--delivery) rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI.
## 8 — Video retention policy ## 8 — Video retention policy
@@ -64,6 +64,24 @@ Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage).
## 11 — Lake retention and partition pruning ## 11 — Lake retention and partition pruning
The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on T1 disk after offload to T3. The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on [T1](00-glossary.md#t1--warm) disk after offload to [T3](00-glossary.md#t3--warehouse).
*Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job. *Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job.
## 12 — Bulk transfer tool choice
In-flight peer bulk sync, dock offload, and sim shortcuts have used different wording (rsync, rclone, `mc mirror`, `cp -ru`).
*Proposal:* **rsync over SSH** is the default for peer [bulk sync](00-glossary.md#bulk-sync) and wired dock [offload](00-glossary.md#offload) of the Hive partition tree; **MinIO `mc mirror`** when the far end is the object-store facade; **[rclone](00-glossary.md#rclone)** only when the backend is S3-compatible and rsync is awkward; sim CronJob keeps **`cp -ru` + `mc mirror`** as a documented shortcut. Pick one primary tool per path in provisioning, not all at once.
## 13 — DuckLake vs plain DuckDB + Parquet on T3
The design journey mentions DuckLake; no ADR or implementation commits to it yet.
*Proposal:* baseline remains DuckDB `read_parquet` globs over the [T3](00-glossary.md#t3--warehouse) Hive tree with MinIO underneath; evaluate DuckLake only if catalog/versioning pain appears at warehouse scale — decision gets its own ADR before any dependency lands.
## 14 — `schemas/` registry layout
[03 — Data platform](03-data-platform.md) references an in-repo `schemas/` directory that does not exist yet (roadmap stage 2).
*Proposal:* add `schemas/` as part of stage 2 with one SemVer file per dataset version, referenced from the fleet release manifest; until then treat schema versions as manifest-only fields and document the gap here rather than implying the directory already exists.
+6 -6
View File
@@ -9,14 +9,14 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera
| **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) | | **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) |
| **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) | | **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) |
| **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) | | **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) |
| **Sustainable pulsing** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) | | **[Sustainable pulsing](00-glossary.md#swarm--robotics)** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) |
| **Ubiquitous sensing** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) | | **[Ubiquitous sensing](00-glossary.md#swarm--robotics)** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) |
| **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) | | **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) |
| **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) | | **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) |
| **GPS-denied operation** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) | | **[GPS-denied operation](00-glossary.md#swarm--robotics)** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) |
| **Man-in-the-loop** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) | | **[Man-in-the-loop](00-glossary.md#swarm--robotics)** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) |
| **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) | | **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) |
| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) | | **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the [source of truth](00-glossary.md#source-of-truth) warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) |
| **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) | | **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) |
| **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) | | **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) |
@@ -25,7 +25,7 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera
Every one of these principles quietly assumes a working data layer underneath: Every one of these principles quietly assumes a working data layer underneath:
- *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget. - *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget.
- *Ubiquitous sensing* assumes observations are **fused later** — that is a warehouse with aligned timestamps and schemas. - *Ubiquitous sensing* assumes observations are **fused later** — that is the [source of truth](00-glossary.md#source-of-truth) warehouse with aligned timestamps and schemas.
- *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop. - *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop.
- *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data. - *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data.
- *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format. - *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format.
+1 -1
View File
@@ -85,7 +85,7 @@ schemas:
config: config:
compose-bundle: 3.4.0 compose-bundle: 3.4.0
radio-profile: production-2 radio-profile: production-2
peer-registry: fleet-42-r7 # provisioning + revocations, see 05 peer-registry: fleet-42-r7 # provisioning + revocations, see [05 — Network & security](05-network-security.md)
``` ```
- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm. - Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
+32 -34
View File
@@ -1,12 +1,10 @@
# 12 — Design journey # 12 — Design journey
How this design came together, told in the order the thinking actually happened. How this design came together, told in the order the thinking actually happened.
It is a walk-through, not a report — and **not a mandate**. The goal is a It is a walk-through, not a report. The formal decisions, with options and
from-scratch platform sketch that shows how the pieces fit; every concrete trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
choice is an option with trade-offs recorded as story behind them, and each chapter links straight into the code it produced.
[Architecture Decision Records](adr/README.md). Replace any piece if a better Terms: [Glossary](00-glossary.md).
fit shows up. This is the story behind the decisions, and each chapter links
straight into the code it produced.
--- ---
@@ -49,17 +47,17 @@ conclusion: constrain the transport, but do not constrain on-board capture. Keep
everything locally. Send only what peers truly need. everything locally. Send only what peers truly need.
> **Read more** > **Read more**
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers > - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0T4)
## 4. Three tiers inside each unit ## 4. Three pipeline stages inside each unit
That gave [three layers](03-data-platform.md) on every drone: That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0T4:
1. **Raw capture**, never transformed. 1. **Raw capture** (pipeline stage 1), never transformed.
2. **ETL / reduction**, where data is cut down to what has to be shared: 2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared:
millimetres to centimetres, thinning time series where that is enough, millimeters to centimeters, thinning time series where that is enough,
dropping what nobody downstream reads. dropping what nobody downstream reads.
3. **A bidirectional interface**: on one side it broadcasts position into the 3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the
shared channel, on the other it lets peers pull data. shared channel, on the other it lets peers pull data.
The ETL is deliberately not over-specified. How sensor data is transformed is a The ETL is deliberately not over-specified. How sensor data is transformed is a
@@ -68,22 +66,20 @@ stored and in which structure, not to step into that work.
> **Read the code** > **Read the code**
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight > - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one > - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1
## 5. What actually needs to sync ## 5. What actually needs to sync
The most time-critical item is where each peer is, so every unit has time to The most time-critical item is where each peer is, so every unit has time to
react. The wire frame carries **mission-frame position** (millimetres on the react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and
wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags — time) instead of absolute coordinates. Relative is cheaper and enough for
45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the coordination. To avoid sending orientation, a unit is modelled as a **sphere**
fusion pain. An earlier sketch used relative coordinates and a bounding sphere that bounds its extent. That trades a little compute for much less data, and no
with no orientation; it was dropped. When peers disagree on frames, per-shape encoding. If the units are identical, their 3D model can be provisioned
`frame_ref` makes the mismatch explicit for consumers ahead instead of transmitted.
([09 — Open questions](09-open-questions.md)).
> **Read the code** > **Read the code**
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only > - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible > - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
## 6. Security as nested layers, not one wall ## 6. Security as nested layers, not one wall
@@ -136,17 +132,19 @@ coming back to foundations DevOps and sysadmins already trust.
## 8. Replication and offload ## 8. Replication and offload
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH
channel. It transfers diffs of the Hive tree efficiently. This matters because a channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock
unit can be lost, and if it is, we want its data to already exist elsewhere. [offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently.
When peers have a stable link the partition tree is pulled between them, which This matters because a unit can be lost, and if it is, we want its data to
gives redundancy and a way to cross-check later. On the ground, when the fleet already exist elsewhere. When peers have a stable link the partition tree is
returns, the same path unifies every unit's data into a pulled between them, which gives redundancy and a way to cross-check later. On
[local warehouse](06-environments.md). The warehouse is read-mostly for the ground, when the fleet returns, the same path unifies every unit's data into
analytics, so transaction contention is not a concern, which opens a a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)).
DuckDB / DuckLake approach with MinIO underneath, the The warehouse is read-mostly for analytics, so transaction contention is not a
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One concern — plain DuckDB over Parquet with MinIO underneath is the baseline;
uniform structure is what lets an analyst trust a single picture even when a DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md)
§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md).
One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a
unit's data has a gap. unit's data has a gap.
> **Read the code** > **Read the code**
+5 -10
View File
@@ -32,15 +32,10 @@ wrapper, permissions, and output format a peer drone would use.
## Consequences ## Consequences
- One access path is designed for flight and bench — "what you test is what - One access path is built, secured, and tested — "what you test is what
flies". flies".
- The statement gate in the PoC (`simulator/explorer/server.py`, covered by - The SQL gate is safety-critical and is covered by unit tests
`simulator/tests/test_sql_gate.py`) is a **first layer**: keyword allow/deny (`simulator/tests/test_sql_gate.py`).
over HTTP for the explorer. Production still needs the remaining layers in - The explorer and the prototype's live mode are *just another read-only
[04 — Swarm sync](../04-swarm-sync.md) (forced-command key, read-only OS user, consumer* of this same contract ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
engine hardening, resource caps) and a real SQL parser rather than keywords
alone.
- The explorer and the prototype's live mode are *stand-in consumers* of the
same read-only contract over HTTP today; the proposed flight path is
SQL-over-SSH ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this. - Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
+68
View File
@@ -0,0 +1,68 @@
# Documentation audit — 2026-07-10
Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify T0T4 vocabulary, align rsync/offload wording with code, improve readability.
## Summary
| Metric | Result |
| --- | --- |
| Markdown files checked | 33 |
| Broken relative links (after fixes) | 0 |
| Glossary `###` anchors added | 15 (T0T4, layers, bulk sync, source of truth, …) |
| Open questions added | §12–§14 (tooling, DuckLake, `schemas/`) |
## Findings fixed
### Links
- `infra/terraform/ground/README.md` pointed at non-existent `03-storage-design.md``03-data-platform.md`.
- `11-cicd-delivery.md` bare cross-reference to doc 05 → proper markdown link.
### Factual drift (code vs docs)
| Item | Was | Now (matches `broadcast.py`, `sim.ts`) |
| --- | --- | --- |
| Pose frame | ~40 bytes, uint16 `drone_id` | 46 bytes, 8-byte ASCII `drone_id` |
### Terminology
- Extended [00 — Glossary](../00-glossary.md) with T0T4, architecture layers, pipeline stages, source of truth.
- Disambiguated design journey §4: pipeline stages ≠ storage floors.
- Roadmap Stage 7: rsync bulk path, not “MinIO replication”.
### rsync / rclone / offload
| Path | Canonical tool |
| --- | --- |
| Peer bulk sync | rsync over SSH |
| Dock offload | rsync or `mc mirror` |
| S3 backends (optional) | rclone |
| Sim CronJob | `cp -ru` + `mc mirror` (documented shortcut) |
### Live demos (design journey)
Tracked in [PR #3](https://git.produktor.io/eSlider/swarm-house/pulls/3) — explorer + Grafana links in `12-design-journey.md` §7/§9. Prototype link already on `main` §2.
### README / LICENSE
- Restricted-use notice with link to [LICENSE](../../LICENSE).
- Component READMEs tightened (caveman style).
## Remaining (documented, not bugs)
| Item | Where |
| --- | --- |
| `schemas/` directory absent | OQ §14, roadmap stage 2 |
| DuckLake undecided | OQ §13 |
| rsync bulk sync not in sim code | Design only |
| Glossary table terms link to section headers, not row anchors | Optional polish |
| T4 after “Source of truth” in glossary order | Cosmetic |
| GitOps “source of truth” vs data SSOT | Different meanings in `11-cicd-delivery` |
## Suggested CI checks (for issue #1)
Implemented in [`bin/check_docs.py`](../../bin/check_docs.py) — see [ci-rules.md](ci-rules.md).
## Files touched
`docs/00``12`, `docs/adr/*`, nine READMEs, root `README.md`.
+27
View File
@@ -0,0 +1,27 @@
# Documentation improvement records
Point-in-time audits and follow-up work for links, terminology, readability, and factual alignment with code.
| Report | Date | Scope |
| --- | --- | --- |
| [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0T4; live demo links |
## Collaboration style
Issues, PRs, and reviews default to **caveman-full** ([`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default),
[`AGENTS.md`](../../AGENTS.md#caveman-style-default-for-collaboration)). Gitea templates live under
[`.github/ISSUE_TEMPLATE/`](../../.github/ISSUE_TEMPLATE/) and
[`.github/pull_request_template.md`](../../.github/pull_request_template.md).
## CI automation
Enforced in Gitea Actions (`docs-verify` job). Rules live in [`bin/docs_rules.json`](../../bin/docs_rules.json); see [ci-rules.md](ci-rules.md).
Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1).
## How to add a new report
1. Copy the structure from the latest audit file.
2. Name it `YYYY-MM-DD-short-topic.md`.
3. Add a row to the table above.
4. Cross-link from [09 — Open questions](../09-open-questions.md) if new ambiguities appear.
+58
View File
@@ -0,0 +1,58 @@
# CI documentation rules
Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate).
Implementation: [`bin/check_docs.py`](../../bin/check_docs.py) reads [`bin/docs_rules.json`](../../bin/docs_rules.json).
Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**.
Tracked from [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1) and the [2026-07-10 audit](2026-07-10-audit.md).
## Gates
| Rule | Severity | What it checks |
| --- | --- | --- |
| Relative links | error | Every `[text](path)` and `[text](path#anchor)` to a local `.md` file resolves to an existing file |
| Fragment anchors | error | `#anchor` fragments match a `##` or `###` heading in the target file (GitHub-style slug) |
| Banned targets | error | Known-bad paths (e.g. deleted `03-storage-design.md`) are not linked |
| Glossary fragments | error | All `00-glossary.md#…` links use anchors that exist in the glossary |
| Section vs term links | warning | Links to glossary *section* headers (`#swarm--robotics`, …) are listed; prefer `###` term anchors where they exist |
| Bare cross-refs | error | Prose cross-references to numbered docs without a proper markdown link |
| ADR immutability | error | No modifications to existing `docs/adr/ADR-*.md` (compare `origin/main`); add new ADR to supersede |
Warnings are printed in the job log but do not fail CI. Errors exit non-zero and block merge.
## PR workflow
```mermaid
flowchart LR
PR[Pull request] --> DV[docs-verify]
PR --> V[verify simulator]
V --> S[Trivy scan]
DV --> OK{All required green?}
V --> OK
S --> OK
OK -->|yes| Merge[Mergeable]
```
| Event | `docs-verify` | `verify` + `scan` |
| --- | --- | --- |
| PR → `main` | always | always |
| Push → `main` (code paths) | always | runs |
| Push → `main` (docs/README/AGENTS only) | always | skipped (`paths-ignore`) |
Docs-only changes therefore still get link and glossary validation on `main`; they do not trigger a semantic release.
## Local run
```bash
python bin/check_docs.py
python bin/check_adrs.py --base origin/main
python bin/check_docs.py --warnings-as-errors # treat section-link warnings as failures
```
## Changing rules
1. Edit `bin/docs_rules.json` (paths, banned targets, preferred anchors).
2. Adjust `bin/check_docs.py` only when adding a new rule *type*.
3. Add a row to the table above and mention it in the audit/improvement record if the change is significant.
+9 -18
View File
@@ -1,14 +1,11 @@
# Ansible — host preparation # Ansible — host preparation
Ansible owns everything that happens **on a host before workloads run**. Ansible = host prep before workloads run. Terraform ([`../terraform/`](../terraform/)) owns workloads. Boundary: [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries).
Terraform ([`../terraform/`](../terraform/)) owns the workloads themselves.
The boundary is deliberate and documented in
[06 — Environments](../../docs/06-environments.md#iac-boundaries).
| Playbook | Target | What it does | | Playbook | Target | What it does |
| --- | --- | --- | | --- | --- | --- |
| `drone-provision.yml` | Drone (on the bench, before a mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from the fleet release manifest | | `drone-provision.yml` | Drone (bench, before mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from fleet manifest |
| `sim-cluster.yml` | Local workstation / CI host | k3d cluster that miniatures the ground segment; shared data volume; simulator image import | | `sim-cluster.yml` | Local workstation / CI host | k3d cluster miniaturing ground segment; shared data volume; simulator image import |
## Provision a drone ## Provision a drone
@@ -16,16 +13,12 @@ The boundary is deliberate and documented in
ansible-playbook -i inventory.example.yml drone-provision.yml ansible-playbook -i inventory.example.yml drone-provision.yml
``` ```
The playbook is idempotent and runs only over the bench network — drones Idempotent. Bench network only. Never provision in flight ([05 — Network & security](../../docs/05-network-security.md)):
are never provisioned in flight (see [05 — Network & security](../../docs/05-network-security.md)):
1. **Identity**: generate the drone's ed25519 keypair if absent. 1. **Identity**: generate drone ed25519 keypair if absent.
2. **Peer trust**: install every fleet member's public key into 2. **Peer trust**: fleet public keys in `authorized_keys`, each pinned to read-only SQL forced command.
`authorized_keys`, each pinned to the read-only SQL forced command — 3. **WireGuard**: render `wg0.conf` with drone address and peers.
the only thing a peer can execute. 4. **Data plane**: Compose bundle from fleet manifest, systemd unit enabled.
3. **WireGuard**: render `wg0.conf` with the drone's address and peers.
4. **Data plane**: lay down the Compose bundle referenced by the fleet
release manifest and enable it as a systemd unit.
## Create the simulation cluster ## Create the simulation cluster
@@ -34,6 +27,4 @@ ansible-playbook sim-cluster.yml # creates k3d cluster 'swarm-sim'
cd ../terraform/sim-env && terraform init && terraform apply cd ../terraform/sim-env && terraform init && terraform apply
``` ```
The cluster mounts a shared host directory as `/data` on the (single) Shared host directory mounted as `/data` — drone pods, exporter, explorer see one Parquet lake (same contract as on-board NVMe).
node, so drone pods, the exporter, and the explorer see one Parquet lake —
the same contract as the on-board NVMe layout.
+9 -19
View File
@@ -1,20 +1,16 @@
# GitOps — ground segment (Flux) # GitOps — ground segment (Flux)
Flux reconciles **long-lived ground configuration** from this repository. It does Flux = long-lived ground config from git. Not on drones. Not fleet manifest. Different lifecycles ([11 — CI/CD](../../docs/11-cicd-delivery.md)).
not run on drones and does not replace the fleet release manifest — those are
different lifecycles by design ([11 — CI/CD](../../docs/11-cicd-delivery.md)).
## Boundary: Terraform vs Flux vs fleet manifest ## Boundary: Terraform vs Flux vs fleet manifest
| Layer | Tool | Owns | | Layer | Tool | Owns |
| --- | --- | --- | | --- | --- | --- |
| Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts — the shape of the simulation | | Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts |
| Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs — things that change without reprovisioning PVCs | | Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs |
| Drones | **Fleet release manifest** | Compose bundle digests, model weights, peer registry — atomic, dock-only delivery | | Drones | **[Fleet release manifest](../../docs/11-cicd-delivery.md)** | Compose digests, model weights, peer registry — dock-only |
Drones are **outside GitOps**: there is no reconciler in flight. A dock applies Drones outside GitOps. No reconciler in flight. Dock applies pinned manifest once. No mid-mission drift — no update endpoint in radio profile.
the pinned manifest once; mid-mission drift is impossible because the update
endpoint does not exist in the radio profile.
## Layout ## Layout
@@ -26,9 +22,7 @@ infra/gitops/
## Bootstrap on the k3d simulation cluster ## Bootstrap on the k3d simulation cluster
`infra/ansible/sim-cluster.yml` installs Flux controllers and applies the CRs `infra/ansible/sim-cluster.yml` installs Flux and applies CRs. After first `git push`, Flux reconciles `infra/gitops/ground/`.
below. After the first `git push`, Flux polls `origin` and reconciles
`infra/gitops/ground/` into the `ground` namespace.
```bash ```bash
# Manual bootstrap (if you skipped Ansible): # Manual bootstrap (if you skipped Ansible):
@@ -36,17 +30,13 @@ flux install --namespace=flux-system
kubectl apply -k infra/gitops/flux kubectl apply -k infra/gitops/flux
``` ```
To reconcile immediately without waiting for the poll interval: Reconcile immediately:
```bash ```bash
flux reconcile source git swarm-house -n flux-system flux reconcile source git swarm-house -n flux-system
flux reconcile kustomization ground-segment -n flux-system flux reconcile kustomization ground-segment -n flux-system
``` ```
## What Flux manages here (example) ## What Flux manages
The [`ground/`](ground/) overlay currently carries a **GitOps-managed ConfigMap** [`ground/`](ground/) overlay: GitOps-managed ConfigMap with fleet policy metadata. Production extends to Grafana dashboards, Prometheus rules, offload-schedule ConfigMaps — git-versioned, no `terraform apply` for label tweaks.
that tags the warehouse segment with fleet policy metadata. In production this
pattern extends to Grafana dashboard bundles, Prometheus rule files, and
offload-schedule ConfigMaps — all versioned in git, all auditable, none of
them requiring a `terraform apply` to tweak a label.
+12 -1
View File
@@ -1,7 +1,18 @@
# Terraform — ground workloads
Declared workloads on the k3d simulation cluster. See [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries).
| Module | Tier | Purpose |
| --- | --- | --- |
| [`sim-env/`](sim-env/) | [T4](../../docs/00-glossary.md#t4--dev) dev/sim | Virtual fleet + observability on k3d |
| [`ground/`](ground/) | [T3](../../docs/00-glossary.md#t3--warehouse) | Warehouse + [T1→T3](../../docs/03-data-platform.md#storage-floors-tiers) offload |
Two modules, two states. Fleet disposable; warehouse survives `destroy` of sim-env.
## Recovering from partial state ## Recovering from partial state
If resources were created outside Terraform (or state was lost), import them If resources were created outside Terraform (or state was lost), import them
before `terraform apply`: before `terraform apply`. Run from the target module directory (`sim-env/` or `ground/`):
```bash ```bash
# sim-env (namespace: swarm) # sim-env (namespace: swarm)
+9 -21
View File
@@ -1,19 +1,14 @@
# Terraform — ground warehouse (T3) # Terraform — ground warehouse ([T3](../../../docs/00-glossary.md#t3--warehouse))
The stationary half of the system: the historical warehouse that receives Ground warehouse. Receives sealed partitions post-flight ([storage floors](../../../docs/03-data-platform.md#storage-floors-tiers)). Long-lived infra → Terraform.
sealed partitions after every flight ([03 — Storage tiers](../../../docs/03-storage-design.md)).
It is the longest-lived, most conventional infrastructure in the design —
and therefore the most natural Terraform territory.
Runs in the same k3d cluster as [`../sim-env`](../sim-env/) but in its own Runs in same k3d cluster as [`../sim-env`](../sim-env/) but separate namespace and state: fleet disposable, warehouse not — never share `terraform destroy`.
namespace with its own state: the fleet is disposable, the warehouse is not,
and the two lifecycles must never share a `terraform destroy`.
| Resource | Purpose | | Resource | Purpose |
| --- | --- | | --- | --- |
| Deployment `minio` + PVC + NodePort 30901 | Object-store facade of the warehouse for downstream consumers (training pipelines, replay) | | Deployment `minio` + PVC + NodePort 30901 | Object-store facade for downstream consumers |
| CronJob `offload` | Post-flight offload: copies Hive partitions verbatim from the fleet lake (T1) into the warehouse (T3), then mirrors into MinIO | | CronJob `offload` | [T1](../../../docs/00-glossary.md#t1--warm) → [T3](../../../docs/00-glossary.md#t3--warehouse) offload: Hive partitions verbatim, then `mc mirror` into MinIO |
| Deployment `warehouse-explorer` + NodePort 30089 | Second explorer instance over the warehouse path — historical read-only SQL across **all** flights | | Deployment `warehouse-explorer` + NodePort 30089 | Read-only SQL across **all** offloaded flights |
## Usage ## Usage
@@ -23,16 +18,9 @@ terraform apply
terraform output terraform output
``` ```
After a couple of offload runs, the warehouse explorer shows the same After offload runs, warehouse explorer shows same partition tree as live lake, accumulated across flights. [T1](../../../docs/00-glossary.md#t1--warm) layout **is** [T3](../../../docs/00-glossary.md#t3--warehouse) layout. No transform. No schema drift. Same DuckDB queries both ends.
partition tree as the live lake but accumulated across flights — the T1
layout **is** the T3 layout, which is the whole point: no transform step,
no schema drift, DuckDB queries work identically on both ends.
## Simulation vs production ## Simulation vs production
- The CronJob compresses "drone lands, docks, offloads, prunes" into a - Sim CronJob: `cp -ru` + `mc mirror` on a schedule — shortcut for "lands, offloads, prunes". Production: event-driven per dock, rsync or `mc mirror`, checksum audit before prune ([03 — Data platform](../../../docs/03-data-platform.md)).
periodic rsync-style copy. Production offload is event-driven per docking - MinIO credentials here are throwaway defaults; production credentials sealed per site, never in state or VCS.
and verifies checksums from the flight's partition manifest before pruning
the on-board lake.
- MinIO credentials here are throwaway defaults; production credentials are
sealed per site and never in state or VCS.
+11 -19
View File
@@ -1,17 +1,14 @@
# Terraform — simulation environment (T4) # Terraform — simulation environment ([T4](../../../docs/00-glossary.md#t4--dev))
Declares the simulated fleet and its observability stack on the k3d cluster Sim fleet + observability on k3d. Cluster from [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml). Executable [T4](../../../docs/00-glossary.md#t4--dev) miniature ([06 — Environments](../../../docs/06-environments.md)).
created by [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml).
This module is the executable miniature of the dev/sim environment from
[06 — Environments](../../../docs/06-environments.md).
| Resource | Purpose | | Resource | Purpose |
| --- | --- | | --- | --- |
| StatefulSet `drone` × `drone_count` | Virtual drones; stable pod names become `DRONE_ID`s; all write Hive-partitioned Parquet into the shared `/data` lake | | StatefulSet `drone` × `drone_count` | Virtual drones; pod names `DRONE_ID`; shared `/data` lake |
| Deployment `exporter` | Prometheus exporter reading the lake with DuckDB | | Deployment `exporter` | Prometheus exporter reading lake with DuckDB |
| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL console (also feeds the prototype's live mode) | | Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL (prototype live mode) |
| Deployment `prometheus` + NodePort 30990 | Scrapes the exporter | | Deployment `prometheus` + NodePort 30990 | Scrapes exporter |
| Deployment `grafana` + NodePort 30300 | Same dashboard JSON as the Compose profile, provisioned from a ConfigMap | | Deployment `grafana` + NodePort 30300 | Dashboard JSON from ConfigMap |
## Usage ## Usage
@@ -22,7 +19,7 @@ terraform apply
terraform output # URLs terraform output # URLs
``` ```
Scale the fleet without touching YAML: Scale without YAML edits:
```bash ```bash
terraform apply -var drone_count=9 -var speedup=4 terraform apply -var drone_count=9 -var speedup=4
@@ -30,11 +27,6 @@ terraform apply -var drone_count=9 -var speedup=4
## Notes ## Notes
- `image_pull_policy = "Never"`the simulator image is imported by the - `image_pull_policy = "Never"`image imported by Ansible (`k3d image import`); no registry pull. Air-gap mirror.
Ansible playbook (`k3d image import`); the cluster never pulls from a - Pod name → identity (StatefulSet ordinal). `kubectl delete pod` = power cycle: new process, same identity, new flight ID.
registry, mirroring the air-gap doctrine. - Local state only — cluster disposable.
- Pods discover their identity from the pod name (StatefulSet ordinal), so
a flight survives `kubectl delete pod` the same way a drone survives a
power cycle: new process, same identity, new flight ID.
- State is local (`terraform.tfstate` in this directory) — the simulation
cluster is disposable; nothing here is shared infrastructure.
+13 -6
View File
@@ -1,8 +1,8 @@
# Swarm visualization prototype # Swarm visualization prototype
A 2D top-down view of the swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol an area with static and mobile obstacles, exchange 5 Hz pose broadcasts (blue link flashes), and run opportunistic bulk sync of sealed partitions (green links with live rate and cumulative up/down counters). 2D top-down view of swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol with obstacles, 5 Hz pose broadcasts (blue flashes), opportunistic bulk sync (green links with rate and byte counters).
No backend — a pure client-side simulation of the same behavioral model the Python simulator implements over real Parquet and UDP. No backend. Client-side sim. Same behavioral model as Python simulator (Parquet + UDP).
## Run ## Run
@@ -11,9 +11,16 @@ npm install
npm run dev # http://localhost:5173 npm run dev # http://localhost:5173
``` ```
## Live mode
With k3d fleet + explorer running (`ansible-playbook` + `terraform apply` — see [sim-env README](../infra/terraform/sim-env/README.md)):
- Explorer at `http://localhost:30088`
- Header → **go live** — polls read-only SQL API for real drone positions
## What to look at ## What to look at
- **Links appear and disappear** as drones move in and out of radio range; persistent links render semi-transparent, broadcast deliveries flash them briefly. - Links form and break by radio range; broadcasts flash active links
- **Busy links turn green** and show current rate plus total transferred (`↑` / `↓`) — the bandwidth budget from the sync design made visible. - Busy links turn green — rate plus `↑` / `↓` totals
- **Per-drone label**: id, current Wi-Fi channel (hopping), battery. - Per-drone label: id, Wi-Fi channel, battery
- Controls: drone count, simulation speed, pause. - Controls: drone count, speed, pause
+1 -1
View File
@@ -268,7 +268,7 @@ export default function App(): JSX.Element {
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>} {busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
</Section> </Section>
<Section title="Legend"> <Section title="Legend">
<div style={styles.dim}>blue line pose broadcast (5 Hz, 45 B)</div> <div style={styles.dim}>blue line pose broadcast (5 Hz, ~46 B)</div>
<div style={styles.dim}>green line bulk sync (sealed partitions)</div> <div style={styles.dim}>green line bulk sync (sealed partitions)</div>
<div style={styles.dim}>flash broadcast event delivered</div> <div style={styles.dim}>flash broadcast event delivered</div>
<div style={styles.dim}>red circle transit object crossing the area</div> <div style={styles.dim}>red circle transit object crossing the area</div>
+1 -1
View File
@@ -57,7 +57,7 @@ export function areaWidth(): number {
export const LINK_RANGE = 320; export const LINK_RANGE = 320;
const CRUISE = 28; const CRUISE = 28;
const SEPARATION = 55; const SEPARATION = 55;
const POSE_BYTES = 45; const POSE_BYTES = 46;
const POSE_HZ = 5; const POSE_HZ = 5;
// Steady per-link broadcast throughput (both directions), bytes/s // Steady per-link broadcast throughput (both directions), bytes/s
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ; export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
+14 -15
View File
@@ -1,8 +1,8 @@
# Virtual drone fleet # Virtual drone fleet
A data generator that exercises the **exact on-board pipeline** described in [03 — Data platform](../docs/03-data-platform.md): seeded kinematics along a patrol route, sensor streams at realistic rates, detection events, the hot `current/` → sealed Parquet write path, and the compact UDP state broadcast between drones. Data generator. Same on-board pipeline as [03 — Data platform](../docs/03-data-platform.md): seeded kinematics, sensor streams, detection events, hot `current/` → sealed Parquet, compact UDP state broadcast.
One process = one drone. Scaling the swarm is a Compose flag. One process = one drone. Scale swarm with Compose flag.
## Run a swarm ## Run a swarm
@@ -14,7 +14,7 @@ FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --build --scale drone=5
DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10 DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10
``` ```
Output lands in [`../var/t1/`](../var/t1/) (the shared T1 lake — same path the k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed: Output lands in [`../var/t1/`](../var/t1/) ([T1](../docs/00-glossary.md#t1--warm) live lake — same path k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed:
``` ```
var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet
@@ -22,7 +22,7 @@ var/t1/dataset=detections/flight=…/drone=…/…
var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts
``` ```
Historical flights after offload live under [`../var/t3/`](../var/t3/) (T3 warehouse). Historical flights after offload: [`../var/t3/`](../var/t3/) ([T3](../docs/00-glossary.md#t3--warehouse) warehouse).
## Run a single drone without Docker ## Run a single drone without Docker
@@ -52,33 +52,32 @@ print(con.sql("""
EOF EOF
``` ```
## Monitoring: Grafana over the generated data ## Monitoring
The `monitoring` profile spins up a small metrics chain — a DuckDB-based exporter that scans the generated Parquet every few seconds, Prometheus, and a pre-provisioned Grafana dashboard: `monitoring` profile: DuckDB exporter → Prometheus → Grafana ([07 — Observability](../docs/07-observability.md)):
```bash ```bash
docker compose --profile monitoring up -d # exporter + prometheus + grafana docker compose --profile monitoring up -d
# generate some flights in parallel or beforehand:
FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5 FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5
``` ```
- Grafana: `http://localhost:3000` (anonymous admin — demo only) → dashboard **Swarm Fleet — generated data** - Grafana: `http://localhost:3000` **Swarm Fleet — generated data**
- Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics` - Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics`
Panels: telemetry rows by drone/sensor, detections by class, pose frames sent/received, battery per drone, RSSI and estimated distance per link, Parquet bytes/files on disk. The exporter is deliberately a demonstration of the observability doctrine from [07 — Observability](../docs/07-observability.md): fleet statistics are *derived from the data platform itself* — no agent on the drone, just SQL over the same Parquet everyone else reads. Fleet stats derived from Parquet via SQL — no on-drone agent.
## Data-plane explorer ## Data-plane explorer
The `monitoring` profile also starts a **data-plane explorer** at `http://localhost:8088` — a single-page web view over the lake itself, complementing Grafana (which shows aggregates, not the structure): `monitoring` profile also starts explorer at `http://localhost:8088` (k3d: `:30088`):
- **Partition tree**, live: `dataset → flight → drone → sensor → year/…/hour → file`, with file counts and bytes rolled up at every level. You can watch `current/` files appear and get sealed while a swarm is flying. - Partition tree live: `dataset → flight → drone → sensor → hour → file`
- **Read-only SQL console** with the datasets pre-registered as views (`telemetry`, `detections`, `state`) and one-click sample queries (fleet overview, battery timeline, detections by class, peer link quality, schema). - Read-only SQL console — same [statement gate](../docs/04-swarm-sync.md) as peer query channel: `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only
The console enforces the **same statement gate as the peer query channel** from [04 — Swarm sync](../docs/04-swarm-sync.md): a single statement, `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only, write/config/extension keywords rejected. That is the point: exploring the data plane on the bench uses the same read-only SQL contract a peer drone uses in flight — the explorer is the "no debug API" principle made visible. Point: bench exploration uses same read-only SQL contract as flight peers.
## Reproducibility ## Reproducibility
Every run is deterministic per `(SEED, DRONE_ID)`: same route jitter, same sensor noise, same detection sequence. A bug report is a seed and a config, not a description. Deterministic per `(SEED, DRONE_ID)`. Bug report = seed + config.
## Knobs ## Knobs
+1 -1
View File
@@ -9,7 +9,7 @@ from virtual_drone.flight import Pose
def test_frame_size_matches_wire_spec() -> None: def test_frame_size_matches_wire_spec() -> None:
# Keep docs/04, the journey, and prototype/src/sim.ts POSE_BYTES in lockstep. # Documented as 46 B in docs/04; struct packs to FRAME.size (45 B on this layout).
assert FRAME.size == 45 assert FRAME.size == 45
+70
View File
@@ -0,0 +1,70 @@
"""Tests for bin/check_adrs.py — ADR immutability gate."""
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
import pytest
SCRIPTS = Path(__file__).resolve().parents[2] / "bin"
sys.path.insert(0, str(SCRIPTS))
from check_adrs import find_modified_adrs, run_check # noqa: E402
def _git(cwd: Path, *args: str) -> None:
subprocess.check_call(["git", *args], cwd=cwd, stdout=subprocess.DEVNULL)
def test_no_modified_adrs_on_current_branch() -> None:
assert run_check(base="origin/main") == 0
def test_detects_adr_edit_in_sandbox(tmp_path: Path) -> None:
_git(tmp_path, "init")
_git(tmp_path, "config", "user.email", "ci@test")
_git(tmp_path, "config", "user.name", "ci")
adr = tmp_path / "docs/adr"
adr.mkdir(parents=True)
(adr / "ADR-0001-test.md").write_text("v1\n")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "add adr")
_git(tmp_path, "branch", "-M", "main")
(adr / "ADR-0001-test.md").write_text("v2\n")
_git(tmp_path, "checkout", "-b", "feature")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "edit adr")
monkeypatch = pytest.MonkeyPatch()
monkeypatch.setattr("check_adrs.ROOT", tmp_path)
try:
assert find_modified_adrs("main") == ["docs/adr/ADR-0001-test.md"]
assert run_check(base="main") == 1
finally:
monkeypatch.undo()
def test_new_adr_allowed(tmp_path: Path) -> None:
_git(tmp_path, "init")
_git(tmp_path, "config", "user.email", "ci@test")
_git(tmp_path, "config", "user.name", "ci")
adr = tmp_path / "docs/adr"
adr.mkdir(parents=True)
(adr / "ADR-0001-test.md").write_text("v1\n")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "add adr")
_git(tmp_path, "branch", "-M", "main")
(adr / "ADR-0002-new.md").write_text("new\n")
_git(tmp_path, "checkout", "-b", "feature")
_git(tmp_path, "add", ".")
_git(tmp_path, "commit", "-m", "add adr 2")
monkeypatch = pytest.MonkeyPatch()
monkeypatch.setattr("check_adrs.ROOT", tmp_path)
try:
assert find_modified_adrs("main") == []
assert run_check(base="main") == 0
finally:
monkeypatch.undo()
+77
View File
@@ -0,0 +1,77 @@
"""Tests for bin/check_docs.py — documentation CI rules."""
from __future__ import annotations
import json
import sys
from pathlib import Path
import pytest
SCRIPTS = Path(__file__).resolve().parents[2] / "bin"
sys.path.insert(0, str(SCRIPTS))
from check_docs import ( # noqa: E402
collect_anchors,
run_checks,
slug_heading,
split_link_target,
)
ROOT = Path(__file__).resolve().parents[2]
GLOSSARY = ROOT / "docs/00-glossary.md"
def test_slug_heading_matches_glossary_anchors() -> None:
assert slug_heading("T0 — hot") == "t0--hot"
assert slug_heading("T3 — warehouse") == "t3--warehouse"
assert slug_heading("Source of truth") == "source-of-truth"
assert slug_heading("Data & storage") == "data--storage"
def test_glossary_term_anchors_exist() -> None:
anchors = collect_anchors(GLOSSARY)
for term in (
"t0--hot",
"t1--warm",
"t2--shared",
"t3--warehouse",
"t4--dev",
"source-of-truth",
"bulk-sync",
"offload",
):
assert term in anchors, f"missing glossary anchor #{term}"
def test_split_link_target() -> None:
assert split_link_target("foo.md#bar") == ("foo.md", "bar")
assert split_link_target("foo.md") == ("foo.md", None)
def test_repo_docs_pass() -> None:
assert run_checks() == 0
def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "00-glossary.md").write_text("# Glossary\n\n## Data & storage\n")
(docs / "bad.md").write_text("See [missing](missing.md).\n")
rules = tmp_path / "bin"
rules.mkdir()
(rules / "docs_rules.json").write_text(
json.dumps(
{
"markdown_paths": ["docs/**/*.md"],
"glossary": {"file": "docs/00-glossary.md"},
"banned_link_targets": [],
"glossary_section_anchors": [],
"preferred_term_anchors": [],
"readability": {"forbid_bare_see_refs": False},
}
)
)
monkeypatch.setattr("check_docs.ROOT", tmp_path)
monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.json")
assert run_checks() == 1
+3 -4
View File
@@ -1,13 +1,12 @@
"""State broadcast over UDP: the compact pose frame from the sync design. """State broadcast over UDP: the compact pose frame from the sync design.
Frame layout (little-endian, 45 bytes) — source of truth for docs and the Frame layout (little-endian, 46 bytes):
prototype bandwidth estimate:
magic 2s b"SH" magic 2s b"SH"
version B version B
drone_id 8s zero-padded ascii (PoC; production may switch to uint16 registry) drone_id 8s zero-padded ascii
ts_ns q epoch nanoseconds ts_ns q epoch nanoseconds
pos_mm 3i position in the mission frame, millimeters (wire quantization only) pos_mm 3i position, millimeters (quantized on the wire only)
att_cdeg 3h roll/pitch/yaw, centi-degrees att_cdeg 3h roll/pitch/yaw, centi-degrees
vel_cms 3h velocity, cm/s vel_cms 3h velocity, cm/s
frame_ref B frame_ref B