Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
87551fc1c1 | ||
|
|
2003ff69fb | ||
|
|
2a6b50929c |
@@ -1,8 +0,0 @@
|
|||||||
blank_issues_enabled: false
|
|
||||||
contact_links:
|
|
||||||
- name: Design docs
|
|
||||||
url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/12-design-journey.md
|
|
||||||
about: Start with the design journey, then open an issue here.
|
|
||||||
- name: Open questions
|
|
||||||
url: https://git.produktor.io/eSlider/swarm-house/src/branch/main/docs/09-open-questions.md
|
|
||||||
about: Known unknowns — check before duplicating.
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
---
|
|
||||||
name: Change / task
|
|
||||||
about: Default issue template — caveman style
|
|
||||||
title: ""
|
|
||||||
labels: []
|
|
||||||
---
|
|
||||||
|
|
||||||
Write **caveman-full**: short lines, fragments OK, no filler. See [`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default).
|
|
||||||
|
|
||||||
## What
|
|
||||||
|
|
||||||
<!-- One line. What need fix or build. -->
|
|
||||||
|
|
||||||
## Why
|
|
||||||
|
|
||||||
<!-- Problem or goal. Skip if obvious from title. -->
|
|
||||||
|
|
||||||
## Done when
|
|
||||||
|
|
||||||
- [ ]
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
<!-- Links, ADR, open question §. Optional. -->
|
|
||||||
@@ -1,19 +0,0 @@
|
|||||||
Write **caveman-full** by default. See [`CONTRIBUTING.md`](../CONTRIBUTING.md#caveman-style-default).
|
|
||||||
|
|
||||||
## What
|
|
||||||
|
|
||||||
<!-- What changed. Bullets OK. -->
|
|
||||||
|
|
||||||
## Why
|
|
||||||
|
|
||||||
<!-- Problem this fixes or design gap closed. -->
|
|
||||||
|
|
||||||
## Test
|
|
||||||
|
|
||||||
- [ ] `pytest tests/ -q` (if code touched)
|
|
||||||
- [ ] `python bin/check_docs.py` (if docs touched)
|
|
||||||
- [ ] ADR added if boundary/contract changed
|
|
||||||
|
|
||||||
## Links
|
|
||||||
|
|
||||||
<!-- Closes / Relates to #issue. Optional demo URL. -->
|
|
||||||
@@ -1,5 +1,4 @@
|
|||||||
# Swarm House - Gitea Actions
|
# Swarm House - Gitea Actions
|
||||||
# docs-verify: markdown links, glossary anchors, readability rules (every push/PR)
|
|
||||||
# verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR)
|
# verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR)
|
||||||
# scan: Trivy vulnerability + misconfiguration scan
|
# scan: Trivy vulnerability + misconfiguration scan
|
||||||
# release: semantic version bump (conventional commits), tag, Gitea release
|
# release: semantic version bump (conventional commits), tag, Gitea release
|
||||||
@@ -18,27 +17,6 @@ on:
|
|||||||
branches: [main]
|
branches: [main]
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
docs-verify:
|
|
||||||
name: Verify documentation
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
container: python:3.12-bookworm
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
run: |
|
|
||||||
git init .
|
|
||||||
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
|
|
||||||
git fetch --depth=50 origin "${{ gitea.ref }}"
|
|
||||||
git checkout FETCH_HEAD
|
|
||||||
|
|
||||||
- name: Fetch base branch for ADR immutability
|
|
||||||
run: git fetch --depth=1 origin main
|
|
||||||
|
|
||||||
- name: ADR immutability (no edits to accepted records)
|
|
||||||
run: python bin/check_adrs.py --base origin/main
|
|
||||||
|
|
||||||
- name: Documentation rules (links, glossary, readability)
|
|
||||||
run: python bin/check_docs.py
|
|
||||||
|
|
||||||
verify:
|
verify:
|
||||||
name: Verify simulator
|
name: Verify simulator
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
@@ -46,10 +24,8 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
run: |
|
run: |
|
||||||
git init .
|
git clone --depth=50 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git .
|
||||||
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
|
git checkout ${{ gitea.sha }}
|
||||||
git fetch --depth=50 origin "${{ gitea.ref }}"
|
|
||||||
git checkout FETCH_HEAD
|
|
||||||
|
|
||||||
- name: Install dependencies
|
- name: Install dependencies
|
||||||
run: pip install --quiet -r simulator/requirements.txt
|
run: pip install --quiet -r simulator/requirements.txt
|
||||||
@@ -85,10 +61,8 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
run: |
|
run: |
|
||||||
git init .
|
git clone --depth=1 https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git .
|
||||||
git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git
|
git checkout ${{ gitea.sha }}
|
||||||
git fetch --depth=1 origin "${{ gitea.ref }}"
|
|
||||||
git checkout FETCH_HEAD
|
|
||||||
|
|
||||||
- name: Install Trivy
|
- name: Install Trivy
|
||||||
run: |
|
run: |
|
||||||
@@ -107,7 +81,7 @@ jobs:
|
|||||||
name: Semantic Release
|
name: Semantic Release
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
container: python:3.12-bookworm
|
container: python:3.12-bookworm
|
||||||
needs: [docs-verify, verify, scan]
|
needs: [verify, scan]
|
||||||
if: gitea.event_name == 'push'
|
if: gitea.event_name == 'push'
|
||||||
permissions:
|
permissions:
|
||||||
contents: write
|
contents: write
|
||||||
|
|||||||
@@ -13,3 +13,8 @@ dist/
|
|||||||
.terraform/
|
.terraform/
|
||||||
*.tfstate
|
*.tfstate
|
||||||
*.tfstate.*
|
*.tfstate.*
|
||||||
|
|
||||||
|
# Local tooling / WIP (keep on disk, never commit)
|
||||||
|
.cursor/
|
||||||
|
.grok/
|
||||||
|
prototype-3d/
|
||||||
|
|||||||
@@ -24,23 +24,6 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development workflow and TDD re
|
|||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
### Caveman style (default for collaboration)
|
|
||||||
|
|
||||||
**Issues, pull requests, and code reviews** use **caveman-full** by default:
|
|
||||||
|
|
||||||
- Short sentences or fragments; drop articles and filler when clear
|
|
||||||
- Lead with fact, verdict, or ask — not preamble
|
|
||||||
- Bullets over paragraphs; one idea per line
|
|
||||||
- Keep technical terms, acronyms, paths, and code exact
|
|
||||||
- English only; no identifying references (same as all repo prose)
|
|
||||||
|
|
||||||
README tiers: root [`README.md`](README.md) = **caveman-lite** (full sentences, less filler);
|
|
||||||
component READMEs = **caveman-full**. Design docs in `docs/` stay readable prose unless
|
|
||||||
editing for consistency.
|
|
||||||
|
|
||||||
Templates: [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/),
|
|
||||||
[`.github/pull_request_template.md`](.github/pull_request_template.md).
|
|
||||||
|
|
||||||
- Markdown: ATX headings, fenced code blocks with language tags, pipe tables.
|
- Markdown: ATX headings, fenced code blocks with language tags, pipe tables.
|
||||||
- Python: PEP 8, type hints, pure functions where possible, no unnecessary classes.
|
- Python: PEP 8, type hints, pure functions where possible, no unnecessary classes.
|
||||||
- TypeScript: strict mode, functional components and hooks, no extra UI libraries.
|
- TypeScript: strict mode, functional components and hooks, no extra UI libraries.
|
||||||
@@ -61,12 +44,7 @@ Status / Context / Options / Decision / Consequences format. Start from
|
|||||||
When a change alters a boundary, contract, or trade-off, add an ADR (copy
|
When a change alters a boundary, contract, or trade-off, add an ADR (copy
|
||||||
[`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the
|
[`docs/adr/TEMPLATE.md`](docs/adr/TEMPLATE.md), take the next number, update the
|
||||||
index) and cross-link it from the affected design doc. A later reversal gets a
|
index) and cross-link it from the affected design doc. A later reversal gets a
|
||||||
new ADR that supersedes the old one — **never edit an Accepted record**.
|
new ADR that supersedes the old one — never edit an Accepted record.
|
||||||
|
|
||||||
**Immutable:** every `docs/adr/ADR-*.md` file is frozen once on `main`. CI rejects
|
|
||||||
any change that modifies an existing ADR. Updates go in a new numbered ADR only.
|
|
||||||
The index ([`docs/adr/README.md`](docs/adr/README.md)) and [`TEMPLATE.md`](docs/adr/TEMPLATE.md)
|
|
||||||
may still change; decision bodies do not.
|
|
||||||
|
|
||||||
## Runtime & operations (sim environment)
|
## Runtime & operations (sim environment)
|
||||||
|
|
||||||
|
|||||||
+1
-37
@@ -17,7 +17,6 @@ The CI pipeline enforces this:
|
|||||||
|
|
||||||
| Gate | When it runs |
|
| Gate | When it runs |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `python bin/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) |
|
|
||||||
| `pytest tests/` | Every push/PR (`verify` job) |
|
| `pytest tests/` | Every push/PR (`verify` job) |
|
||||||
| `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image |
|
| `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image |
|
||||||
| Smoke flight + DuckDB assertion | Every push/PR |
|
| Smoke flight + DuckDB assertion | Every push/PR |
|
||||||
@@ -77,47 +76,12 @@ test: cover broadcast frame roundtrip
|
|||||||
Never include AI tool references, `Co-authored-by` trailers for assistants, or
|
Never include AI tool references, `Co-authored-by` trailers for assistants, or
|
||||||
identifying names of people, companies, or locations.
|
identifying names of people, companies, or locations.
|
||||||
|
|
||||||
## Caveman style (default)
|
|
||||||
|
|
||||||
Use **caveman-full** in **issues**, **pull requests**, and **reviews** unless the
|
|
||||||
author explicitly asks otherwise.
|
|
||||||
|
|
||||||
| Do | Skip |
|
|
||||||
| --- | --- |
|
|
||||||
| Short lines, fragments OK | Long preamble ("I wanted to start by saying…") |
|
|
||||||
| Lead with what / why / verdict | Filler ("just", "basically", "kind of") |
|
|
||||||
| Bullets, checklists | Repeated context already in diff or issue |
|
|
||||||
| Exact terms, paths, code | Invented abbreviations or vague "the thing" |
|
|
||||||
|
|
||||||
**Examples**
|
|
||||||
|
|
||||||
| Fluffy | Caveman |
|
|
||||||
| --- | --- |
|
|
||||||
| I think we should probably consider adding a link checker to CI because links might break. | Add CI link checker. Broken relative links fail merge. |
|
|
||||||
| This pull request implements the documentation improvements that were discussed in the audit. | Docs audit: glossary T0–T4, fix ground README link, rsync wording. |
|
|
||||||
| LGTM but maybe we could think about whether the anchor format is consistent? | Merge OK. Fix glossary anchors in follow-up (#1). |
|
|
||||||
|
|
||||||
README intensity: root = caveman-lite; `simulator/`, `prototype/`, `infra/`, `docs/adr/` READMEs =
|
|
||||||
caveman-full. Numbered design docs in `docs/` stay full prose.
|
|
||||||
|
|
||||||
Gitea pre-fills [issue](.github/ISSUE_TEMPLATE/default.md) and
|
|
||||||
[PR](.github/pull_request_template.md) templates with this style.
|
|
||||||
|
|
||||||
### Reviews
|
|
||||||
|
|
||||||
Default review voice = caveman-full:
|
|
||||||
|
|
||||||
1. **Verdict first** — merge / wait / no, or approve / request changes
|
|
||||||
2. **Findings** — bullet per blocker or nit; file + line when useful
|
|
||||||
3. **Scope** — say if issue AC met, partial, or out of scope
|
|
||||||
4. No essay; link ADR, issue, or doc section instead of restating design
|
|
||||||
|
|
||||||
## Pull request checklist
|
## Pull request checklist
|
||||||
|
|
||||||
- [ ] Tests added or updated; `pytest tests/ -q` passes locally
|
- [ ] Tests added or updated; `pytest tests/ -q` passes locally
|
||||||
- [ ] `docker build` in `simulator/` succeeds (test stage included)
|
- [ ] `docker build` in `simulator/` succeeds (test stage included)
|
||||||
- [ ] Documentation updated if behaviour or boundaries changed
|
- [ ] Documentation updated if behaviour or boundaries changed
|
||||||
- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/)) — never edit an existing `ADR-*.md`; supersede with a new number
|
- [ ] **ADR added** if the change alters an architectural boundary, contract, or trade-off (see [`docs/adr/`](docs/adr/))
|
||||||
- [ ] English only; no identifying references
|
- [ ] English only; no identifying references
|
||||||
- [ ] Conventional commit message
|
- [ ] Conventional commit message
|
||||||
|
|
||||||
|
|||||||
@@ -2,14 +2,11 @@
|
|||||||
|
|
||||||
**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective.
|
**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective.
|
||||||
|
|
||||||
Fleet flies autonomous. No internet uplink, no external access. Data stays in the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. Swarm exchanges just enough state to coordinate; everything else stays local until a ground warehouse receives it after landing — reproducibly, securely, at scale.
|
A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale.
|
||||||
|
|
||||||
This repository covers storage layout, sync strategy, network trust model, CI/CD chain, and the simulation environment that tests it without real hardware.
|
This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware.
|
||||||
|
|
||||||
> **Restricted use.** Third parties may not use, copy, modify, or distribute this
|
**New here?** Start with the one-pager: [00 — Executive overview](docs/00-executive.md) (includes how **MinIO** fits). Then the [design journey](docs/12-design-journey.md) if you want the narrative.
|
||||||
> source code, documentation, or related materials without **explicit written
|
|
||||||
> permission** from the copyright holder. No license is granted by default. See
|
|
||||||
> [LICENSE](LICENSE) for the full terms.
|
|
||||||
|
|
||||||
## Design principles
|
## Design principles
|
||||||
|
|
||||||
@@ -22,26 +19,46 @@ This repository covers storage layout, sync strategy, network trust model, CI/CD
|
|||||||
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
|
7. **SQL is the contract.** Peer data access is read-only DuckDB SQL over SSH forced commands — the query language already lives on both ends, so no service, port, or protocol is invented for it.
|
||||||
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
|
8. **No debug API.** The bench and the flight use the same channel: an engineer debugging on the ground runs the identical query through the identical wrapper, permissions, and output format a peer drone would use. What you test is what flies.
|
||||||
|
|
||||||
|
None of the concrete tool picks above are mandates. This repo is a **from-scratch platform sketch**: enough structure to hire and build against, with every decision recorded so the team can replace a piece when a better fit appears.
|
||||||
|
|
||||||
|
## Implemented now vs proposed next
|
||||||
|
|
||||||
|
Honest map so a reader knows what runs today versus what is design intent.
|
||||||
|
|
||||||
|
| Area | Implemented in this repo (runnable PoC) | Proposed for a production air-gapped fleet |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| On-board layout | Hive-partitioned Parquet writer, seal step, DuckDB views | Same contract; Compose services under systemd |
|
||||||
|
| Pose path | Fixed **45-byte** UDP frame + 2D bandwidth visualisation | Zenoh pub/sub (UDP kept as degraded minimal profile) |
|
||||||
|
| Peer query | Read-only SQL **gate** over HTTP explorer (keyword allow-list) | Same gate idea via **SSH forced command** + OS/engine hardening |
|
||||||
|
| Bulk sync | Visualised opportunistic transfer volume | rsync/rclone over persistent SSH between peers |
|
||||||
|
| Mesh trust | Ansible templates for WireGuard + ed25519 forced commands | Provisioned per-device keys; nothing joins at runtime |
|
||||||
|
| Ground segment | k3d/Terraform sim: lake, Grafana, offload CronJob, **MinIO as T3** | k3s warehouse with MinIO, GitOps overlays, post-flight mirror |
|
||||||
|
| CI / delivery | GitHub Actions: pytest, smoke flight, Trivy, semver release + fleet manifest artifact | Self-hosted GitLab + registry inside the air gap (same stages) |
|
||||||
|
| Docs | Problem, architecture, ADRs, design journey, open questions | Living ADRs owned by the team |
|
||||||
|
|
||||||
|
Start from the [design journey](docs/12-design-journey.md) for the story; use the table above when reviewing scope.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
| Document | Contents |
|
| Document | Contents |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
|
| [00 — Executive](docs/00-executive.md) | **Start here** — one-page goal, sync rule, MinIO stance |
|
||||||
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
|
| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout |
|
||||||
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
|
| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions |
|
||||||
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
|
| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes |
|
||||||
| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage floors (T0–T4), partitioning, hot-write path |
|
| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path |
|
||||||
| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard |
|
| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard |
|
||||||
| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport |
|
| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport |
|
||||||
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
|
| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures |
|
||||||
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
|
| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry |
|
||||||
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
|
| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex |
|
||||||
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers |
|
| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers (incl. MinIO placement) |
|
||||||
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
|
| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on |
|
||||||
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
|
| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases |
|
||||||
| [12 — Design journey](docs/12-design-journey.md) | **Start here** — a narrative walk-through of how the design came together, linking into the code |
|
| [12 — Design journey](docs/12-design-journey.md) | Narrative walk-through linking into the code |
|
||||||
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
|
| [ADRs](docs/adr/) | Architecture Decision Records — the decisions behind the above, in the order they were made |
|
||||||
|
|
||||||
New? Start with the [design journey](docs/12-design-journey.md) — story plus code links. Principles = *what*; [ADRs](docs/adr/) = *why and when*.
|
New here? Read the [executive overview](docs/00-executive.md) first, then the [design journey](docs/12-design-journey.md) if you want the story. The design principles above are the *what*; the [ADRs](docs/adr/) are the *why and when*.
|
||||||
|
|
||||||
## Runnable parts
|
## Runnable parts
|
||||||
|
|
||||||
@@ -50,10 +67,10 @@ New? Start with the [design journey](docs/12-design-journey.md) — story plus c
|
|||||||
| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
|
| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB |
|
||||||
| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG |
|
| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics; **live mode** renders real positions from the k3d fleet | React, TypeScript, Vite, SVG |
|
||||||
| [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible |
|
| [`infra/ansible/`](infra/ansible/) | Host preparation: drone bench provisioning (keys, forced commands, WireGuard, Compose bundle) and local k3d simulation cluster | Ansible |
|
||||||
| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + [T1→T3](docs/03-data-platform.md#storage-floors-tiers) offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider |
|
| [`infra/terraform/`](infra/terraform/) | Declared workloads on the sim cluster: virtual fleet + observability ([`sim-env`](infra/terraform/sim-env/)), warehouse + T1→T3 offload ([`ground`](infra/terraform/ground/)) | Terraform, kubernetes provider |
|
||||||
| [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize |
|
| [`infra/gitops/`](infra/gitops/) | Flux overlays for the ground segment — policy labels, dashboard bundles; drones stay on the fleet manifest | Flux, Kustomize |
|
||||||
|
|
||||||
Shared runtime data (gitignored): [`var/t1/`](docs/00-glossary.md#t1--warm) (live lake, [T1](docs/00-glossary.md#t1--warm)), [`var/t3/`](docs/00-glossary.md#t3--warehouse) (warehouse, [T3](docs/00-glossary.md#t3--warehouse)). See [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
Shared runtime data (gitignored): `var/t1/` (live lake), `var/t3/` (warehouse). See [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
|
|||||||
@@ -1,75 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Reject modifications to accepted Architecture Decision Records."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
ADR_PREFIX = "docs/adr/ADR-"
|
|
||||||
|
|
||||||
|
|
||||||
def _git(*args: str) -> str:
|
|
||||||
return subprocess.check_output(
|
|
||||||
["git", *args],
|
|
||||||
cwd=ROOT,
|
|
||||||
text=True,
|
|
||||||
stderr=subprocess.DEVNULL,
|
|
||||||
).strip()
|
|
||||||
|
|
||||||
|
|
||||||
def _exists_on(base: str, path: str) -> bool:
|
|
||||||
try:
|
|
||||||
_git("cat-file", "-e", f"{base}:{path}")
|
|
||||||
return True
|
|
||||||
except subprocess.CalledProcessError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def find_modified_adrs(base: str) -> list[str]:
|
|
||||||
try:
|
|
||||||
diff_out = _git("diff", "--name-only", f"{base}...HEAD", "--", "docs/adr/")
|
|
||||||
except subprocess.CalledProcessError:
|
|
||||||
return []
|
|
||||||
|
|
||||||
modified: list[str] = []
|
|
||||||
for rel in diff_out.splitlines():
|
|
||||||
if not rel.startswith(ADR_PREFIX) or not rel.endswith(".md"):
|
|
||||||
continue
|
|
||||||
if _exists_on(base, rel):
|
|
||||||
modified.append(rel)
|
|
||||||
return sorted(modified)
|
|
||||||
|
|
||||||
|
|
||||||
def run_check(base: str = "origin/main") -> int:
|
|
||||||
modified = find_modified_adrs(base)
|
|
||||||
if modified:
|
|
||||||
print("Immutable ADR violation — accepted records cannot be edited:")
|
|
||||||
for path in modified:
|
|
||||||
print(f" error: {path}")
|
|
||||||
print(
|
|
||||||
"\nReversal or refinement → new ADR-NNNN file. "
|
|
||||||
"Never edit an existing ADR-*.md."
|
|
||||||
)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
print(f"ADR immutability OK (base {base})")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument(
|
|
||||||
"--base",
|
|
||||||
default="origin/main",
|
|
||||||
help="Git ref to compare against (default: origin/main)",
|
|
||||||
)
|
|
||||||
args = parser.parse_args()
|
|
||||||
sys.exit(run_check(base=args.base))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,212 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Validate markdown links, glossary anchors, and readability rules."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
RULES_PATH = Path(__file__).with_name("docs_rules.json")
|
|
||||||
|
|
||||||
LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)")
|
|
||||||
BARE_SEE_RE = re.compile(r"(?<![\[(/])see\s+(\d{2})(?![\w-])", re.IGNORECASE)
|
|
||||||
SKIP_SCHEMES = ("http://", "https://", "mailto:", "tel:", "#")
|
|
||||||
|
|
||||||
|
|
||||||
def load_rules(path: Path = RULES_PATH) -> dict:
|
|
||||||
return json.loads(path.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def slug_heading(text: str) -> str:
|
|
||||||
"""Anchor slug used in this repository's glossary links."""
|
|
||||||
text = text.strip()
|
|
||||||
text = re.sub(r"\s&\s", " — ", text)
|
|
||||||
if " — " in text:
|
|
||||||
left, right = text.split(" — ", 1)
|
|
||||||
return f"{_slug_part(left)}--{_slug_part(right)}"
|
|
||||||
return _slug_part(text)
|
|
||||||
|
|
||||||
|
|
||||||
def _slug_part(text: str) -> str:
|
|
||||||
slug = text.strip().lower()
|
|
||||||
slug = re.sub(r"[^\w\s-]", "", slug)
|
|
||||||
slug = re.sub(r"\s+", "-", slug)
|
|
||||||
return slug
|
|
||||||
|
|
||||||
|
|
||||||
def collect_anchors(md_path: Path) -> set[str]:
|
|
||||||
anchors: set[str] = set()
|
|
||||||
for line in md_path.read_text(encoding="utf-8").splitlines():
|
|
||||||
if line.startswith("##"):
|
|
||||||
anchors.add(slug_heading(line.lstrip("#").strip()))
|
|
||||||
return anchors
|
|
||||||
|
|
||||||
|
|
||||||
def expand_markdown_paths(patterns: list[str]) -> list[Path]:
|
|
||||||
files: set[Path] = set()
|
|
||||||
for pattern in patterns:
|
|
||||||
for path in ROOT.glob(pattern):
|
|
||||||
if path.is_file() and path.suffix == ".md":
|
|
||||||
files.add(path)
|
|
||||||
return sorted(files)
|
|
||||||
|
|
||||||
|
|
||||||
def split_link_target(raw: str) -> tuple[str, str | None]:
|
|
||||||
target = raw.strip()
|
|
||||||
if target.startswith("<") and target.endswith(">"):
|
|
||||||
target = target[1:-1]
|
|
||||||
if "#" in target:
|
|
||||||
path_part, fragment = target.split("#", 1)
|
|
||||||
return path_part, fragment or None
|
|
||||||
return target, None
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_path(source: Path, link_path: str) -> Path | None:
|
|
||||||
if not link_path or link_path.startswith(SKIP_SCHEMES):
|
|
||||||
return None
|
|
||||||
candidate = (source.parent / link_path).resolve()
|
|
||||||
try:
|
|
||||||
candidate.relative_to(ROOT.resolve())
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
return candidate
|
|
||||||
|
|
||||||
|
|
||||||
def is_glossary_link(link_path: str, rules: dict) -> bool:
|
|
||||||
glossary = rules["glossary"]["file"].replace("\\", "/")
|
|
||||||
normalized = link_path.replace("\\", "/")
|
|
||||||
return (
|
|
||||||
normalized == glossary
|
|
||||||
or normalized.endswith("/" + glossary)
|
|
||||||
or normalized.endswith(glossary.split("/")[-1])
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def check_file(
|
|
||||||
md_path: Path,
|
|
||||||
rules: dict,
|
|
||||||
anchor_cache: dict[Path, set[str]],
|
|
||||||
) -> tuple[list[str], list[str]]:
|
|
||||||
errors: list[str] = []
|
|
||||||
warnings: list[str] = []
|
|
||||||
rel = md_path.relative_to(ROOT)
|
|
||||||
text = md_path.read_text(encoding="utf-8")
|
|
||||||
|
|
||||||
if rules.get("readability", {}).get("forbid_bare_see_refs"):
|
|
||||||
prose = re.sub(r"`[^`]*`", "", text)
|
|
||||||
for match in BARE_SEE_RE.finditer(prose):
|
|
||||||
errors.append(
|
|
||||||
f"{rel}:{_line_no(text, match.start())}: bare cross-ref "
|
|
||||||
f"'see {match.group(1)}' — use a markdown link"
|
|
||||||
)
|
|
||||||
|
|
||||||
for match in LINK_RE.finditer(text):
|
|
||||||
raw = match.group(1)
|
|
||||||
if any(raw.startswith(s) for s in SKIP_SCHEMES):
|
|
||||||
continue
|
|
||||||
|
|
||||||
link_path, fragment = split_link_target(raw)
|
|
||||||
if not link_path.endswith(".md"):
|
|
||||||
continue
|
|
||||||
|
|
||||||
for banned in rules.get("banned_link_targets", []):
|
|
||||||
if banned in link_path.replace("\\", "/"):
|
|
||||||
errors.append(
|
|
||||||
f"{rel}:{_line_no(text, match.start())}: banned link target "
|
|
||||||
f"'{link_path}'"
|
|
||||||
)
|
|
||||||
|
|
||||||
target = resolve_path(md_path, link_path)
|
|
||||||
if target is None:
|
|
||||||
errors.append(
|
|
||||||
f"{rel}:{_line_no(text, match.start())}: link escapes repo or "
|
|
||||||
f"is invalid: {raw}"
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
if not target.is_file():
|
|
||||||
errors.append(
|
|
||||||
f"{rel}:{_line_no(text, match.start())}: broken link "
|
|
||||||
f"{link_path} -> {target.relative_to(ROOT)}"
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
if fragment:
|
|
||||||
if target not in anchor_cache:
|
|
||||||
anchor_cache[target] = collect_anchors(target)
|
|
||||||
if fragment not in anchor_cache[target]:
|
|
||||||
errors.append(
|
|
||||||
f"{rel}:{_line_no(text, match.start())}: unknown anchor "
|
|
||||||
f"#{fragment} in {target.relative_to(ROOT)}"
|
|
||||||
)
|
|
||||||
|
|
||||||
if fragment and is_glossary_link(link_path, rules):
|
|
||||||
section_anchors = set(rules.get("glossary_section_anchors", []))
|
|
||||||
if fragment in section_anchors:
|
|
||||||
warnings.append(
|
|
||||||
f"{rel}:{_line_no(text, match.start())}: glossary section "
|
|
||||||
f"link #{fragment} — prefer a ### term anchor where one exists"
|
|
||||||
)
|
|
||||||
|
|
||||||
return errors, warnings
|
|
||||||
|
|
||||||
|
|
||||||
def _line_no(text: str, index: int) -> int:
|
|
||||||
return text.count("\n", 0, index) + 1
|
|
||||||
|
|
||||||
|
|
||||||
def run_checks(warnings_as_errors: bool = False) -> int:
|
|
||||||
rules = load_rules(RULES_PATH)
|
|
||||||
anchor_cache: dict[Path, set[str]] = {}
|
|
||||||
all_errors: list[str] = []
|
|
||||||
all_warnings: list[str] = []
|
|
||||||
|
|
||||||
glossary_path = ROOT / rules["glossary"]["file"]
|
|
||||||
anchor_cache[glossary_path] = collect_anchors(glossary_path)
|
|
||||||
|
|
||||||
for md_path in expand_markdown_paths(rules["markdown_paths"]):
|
|
||||||
errors, warnings = check_file(md_path, rules, anchor_cache)
|
|
||||||
all_errors.extend(errors)
|
|
||||||
all_warnings.extend(warnings)
|
|
||||||
|
|
||||||
if all_warnings:
|
|
||||||
print("Documentation warnings:")
|
|
||||||
for msg in sorted(all_warnings):
|
|
||||||
print(f" warning: {msg}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
if all_errors:
|
|
||||||
print("Documentation errors:")
|
|
||||||
for msg in sorted(all_errors):
|
|
||||||
print(f" error: {msg}")
|
|
||||||
print(f"\n{len(all_errors)} error(s), {len(all_warnings)} warning(s)")
|
|
||||||
return 1
|
|
||||||
|
|
||||||
if warnings_as_errors and all_warnings:
|
|
||||||
print(f"\n{len(all_warnings)} warning(s) treated as errors")
|
|
||||||
return 1
|
|
||||||
|
|
||||||
print(
|
|
||||||
f"Documentation checks passed "
|
|
||||||
f"({len(expand_markdown_paths(rules['markdown_paths']))} files, "
|
|
||||||
f"{len(all_warnings)} warning(s))"
|
|
||||||
)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument(
|
|
||||||
"--warnings-as-errors",
|
|
||||||
action="store_true",
|
|
||||||
help="Exit non-zero when section-link warnings are present",
|
|
||||||
)
|
|
||||||
args = parser.parse_args()
|
|
||||||
sys.exit(run_checks(warnings_as_errors=args.warnings_as_errors))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
{
|
|
||||||
"markdown_paths": [
|
|
||||||
"docs/**/*.md",
|
|
||||||
"README.md",
|
|
||||||
"CONTRIBUTING.md",
|
|
||||||
"AGENTS.md",
|
|
||||||
"simulator/README.md",
|
|
||||||
"prototype/README.md",
|
|
||||||
"infra/**/README.md"
|
|
||||||
],
|
|
||||||
"glossary": {
|
|
||||||
"file": "docs/00-glossary.md"
|
|
||||||
},
|
|
||||||
"banned_link_targets": [
|
|
||||||
"03-storage-design.md"
|
|
||||||
],
|
|
||||||
"glossary_section_anchors": [
|
|
||||||
"data--storage",
|
|
||||||
"swarm--robotics",
|
|
||||||
"infrastructure--delivery"
|
|
||||||
],
|
|
||||||
"preferred_term_anchors": [
|
|
||||||
"t0--hot",
|
|
||||||
"t1--warm",
|
|
||||||
"t2--shared",
|
|
||||||
"t3--warehouse",
|
|
||||||
"t4--dev",
|
|
||||||
"source-of-truth",
|
|
||||||
"architecture-layer-1--ingestion",
|
|
||||||
"architecture-layer-2--storage-and-transform",
|
|
||||||
"architecture-layer-3--serving-and-sync",
|
|
||||||
"pipeline-stage",
|
|
||||||
"bulk-sync",
|
|
||||||
"offload",
|
|
||||||
"rclone",
|
|
||||||
"adr",
|
|
||||||
"asr"
|
|
||||||
],
|
|
||||||
"readability": {
|
|
||||||
"forbid_bare_see_refs": true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# 00 — Executive overview
|
||||||
|
|
||||||
|
A one-page map of the proposal. Details and trade-offs live in the linked docs.
|
||||||
|
Nothing here is a mandate: it is a from-scratch platform sketch the team can
|
||||||
|
reshape once real constraints are on the table.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Build an **on-prem, air-gapped data platform** for an autonomous drone swarm:
|
||||||
|
accumulate sensor and detection data on each unit, share only what peers need in
|
||||||
|
flight, and offload complete flights to a ground warehouse for replay and
|
||||||
|
training.
|
||||||
|
|
||||||
|
## Constraints (given)
|
||||||
|
|
||||||
|
- No internet / no external access in flight; data stays inside the system.
|
||||||
|
- Each drone is autonomous; mesh links are intermittent — **no swarm-wide orchestrator**.
|
||||||
|
- On-board today: **Docker Compose**, **DuckDB**, YOLO-like video analytics; **Parquet** under evaluation.
|
||||||
|
- **MinIO is already in the stack** — exact placement not fully known yet (see below).
|
||||||
|
- Ground side can run lightweight Kubernetes; this proposal keeps k3s **on the ground only**.
|
||||||
|
|
||||||
|
## Data contract (proposed)
|
||||||
|
|
||||||
|
| What | Where | Crosses the air? |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Raw telemetry | Local NVMe (Parquet / Hive layout) | **No** — offloads after landing |
|
||||||
|
| Pose / detections (derived) | Local store + peer pub/sub | **Yes** — small, budgeted |
|
||||||
|
| Full flight archive | Ground warehouse (Parquet + object store) | After landing only |
|
||||||
|
|
||||||
|
One layout on every floor so offload is a **mirror**, not a migration
|
||||||
|
([03 — Data platform](03-data-platform.md)).
|
||||||
|
|
||||||
|
## Sync rule
|
||||||
|
|
||||||
|
1. **Fast path:** compact pose frames (~45 B @ 5 Hz) + event-shaped detections.
|
||||||
|
2. **Bulk path (peers):** sealed derived partitions when the link allows
|
||||||
|
(rsync/SSH proposed; alternatives open).
|
||||||
|
3. **Never** push raw high-rate telemetry peer-to-peer in flight.
|
||||||
|
|
||||||
|
## MinIO (first-class, placement TBD)
|
||||||
|
|
||||||
|
MinIO is treated as **already chosen infrastructure**, not something to rip out.
|
||||||
|
|
||||||
|
| Role | Stance in this sketch |
|
||||||
|
| --- | --- |
|
||||||
|
| **Ground warehouse (T3)** | **Primary home** — S3 API under the Parquet lake / offload target |
|
||||||
|
| **On-board (drone)** | **Default: no** — competes with flight-critical CPU/RAM; local Parquet + DuckDB is enough for the hot path |
|
||||||
|
| **On-board exception** | Not excluded if a team already relies on an S3 API in Compose; then derived-only, never the 5 Hz pose path |
|
||||||
|
| **5 Hz pose / collision state** | **Not MinIO** — pub/sub (UDP in the PoC; Zenoh proposed) |
|
||||||
|
| **Inter-drone history catch-up** | Prefer pull of sealed partitions (SSH/rsync); MinIO replication only if ops already standardised on it |
|
||||||
|
|
||||||
|
Open item for the team: document how MinIO is used today, then lock T2/T3 roles
|
||||||
|
([09 — Open questions](09-open-questions.md)).
|
||||||
|
|
||||||
|
## What to read next
|
||||||
|
|
||||||
|
1. [12 — Design journey](12-design-journey.md) — narrative + deep links into code
|
||||||
|
2. [Implemented vs proposed](../README.md#implemented-now-vs-proposed-next) — PoC vs production intent
|
||||||
|
3. [09 — Open questions](09-open-questions.md) — including MinIO placement
|
||||||
|
|
||||||
|
Live visual PoC: [swarm.produktor.io](https://swarm.produktor.io/) (same access as this repository).
|
||||||
+3
-67
@@ -2,70 +2,6 @@
|
|||||||
|
|
||||||
Terms and abbreviations used throughout this proposal.
|
Terms and abbreviations used throughout this proposal.
|
||||||
|
|
||||||
**Vocabulary guide:** **architecture layers** (1–3) = on-board software stack; **storage floors** (T0–T4) = retention/latency ladder; **pipeline stages** = narrative capture → reduction → sync only ([12 — Design journey](12-design-journey.md) §4). Do not mix these three.
|
|
||||||
|
|
||||||
## Project vocabulary
|
|
||||||
|
|
||||||
### T0 — hot
|
|
||||||
|
|
||||||
RAM / DuckDB in-process sliding window of the last minutes; what mission logic queries in flight. Retention: minutes. Maps to **hot** in the classic three-tier model.
|
|
||||||
|
|
||||||
### T1 — warm
|
|
||||||
|
|
||||||
Drone NVMe: full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight. Retention: current flight plus quota-based headroom. Maps to **warm**. Runtime path: [`var/t1/`](../var/t1/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)).
|
|
||||||
|
|
||||||
### T2 — shared
|
|
||||||
|
|
||||||
On-board MinIO bucket: derived data only (`detections`, `state`), replicated opportunistically across the swarm. Retention: current mission.
|
|
||||||
|
|
||||||
### T3 — warehouse
|
|
||||||
|
|
||||||
Ground on-prem object store + Parquet lakehouse: every flight of every drone, forever; replay, analytics, model training. Retention: years. Maps to **cold** / **DWH**. Runtime path: [`var/t3/`](../var/t3/) ([ADR-0008](adr/ADR-0008-runtime-data-paths.md)). After offload and integrity audit, T3 is the [source of truth](#source-of-truth) for historical flight data.
|
|
||||||
|
|
||||||
### T4 — dev
|
|
||||||
|
|
||||||
Engineer laptop or sim farm: slices pulled from T3, or synthetic data from the simulator. Retention: ephemeral.
|
|
||||||
|
|
||||||
### Source of truth
|
|
||||||
|
|
||||||
The authoritative copy of flight data after [offload](#offload) and integrity audit: the [T3](#t3--warehouse) warehouse. Each drone holds its own partial record in flight ([T1](#t1--warm)); peers exchange derived state only ([04 — Swarm sync](04-swarm-sync.md)). Replay, post-flight fusion, training, and audit read from T3 — not from any single drone's local store. See [06 — Environments](06-environments.md), [ADR-0003](adr/ADR-0003-sync-derived-state-only.md).
|
|
||||||
|
|
||||||
### Architecture layer 1 — ingestion
|
|
||||||
|
|
||||||
On-board services that capture sensor streams and video analytics ([02 — Architecture](02-architecture.md)): `sensor-ingest`, `video-analytics`.
|
|
||||||
|
|
||||||
### Architecture layer 2 — storage and transform
|
|
||||||
|
|
||||||
On-board Parquet write path, sealer, DuckDB, NVMe ([02 — Architecture](02-architecture.md)).
|
|
||||||
|
|
||||||
### Architecture layer 3 — serving and sync
|
|
||||||
|
|
||||||
On-board event hook, pub/sub broadcast, MinIO derived bucket, peer query API ([02 — Architecture](02-architecture.md)).
|
|
||||||
|
|
||||||
### Pipeline stage
|
|
||||||
|
|
||||||
Narrative inside each drone: (1) raw capture, (2) ETL/reduction, (3) bidirectional sync interface. Not the same as storage floors T0–T4 ([12 — Design journey](12-design-journey.md) §4).
|
|
||||||
|
|
||||||
### Bulk sync
|
|
||||||
|
|
||||||
In-flight pull of sealed `detections`/`state` Parquet partitions between peers via **rsync** delta over persistent SSH forced commands ([04 — Swarm sync](04-swarm-sync.md)).
|
|
||||||
|
|
||||||
### Offload
|
|
||||||
|
|
||||||
Post-flight mirror of sealed T1 partitions into T3 at the dock: **rsync** or MinIO **`mc mirror`**, then integrity audit before local prune ([03 — Data platform](03-data-platform.md)).
|
|
||||||
|
|
||||||
### rclone
|
|
||||||
|
|
||||||
Optional rsync-class tool when the far end is object storage and rsync is awkward; not the default in-flight mesh path ([09 — Open questions](09-open-questions.md) §12).
|
|
||||||
|
|
||||||
### ADR
|
|
||||||
|
|
||||||
Architecture Decision Record — a point-in-time decision for this repository ([adr/README.md](adr/README.md)).
|
|
||||||
|
|
||||||
### ASR
|
|
||||||
|
|
||||||
Architecture Standard Record — a standing cross-repository policy (lives in the ecosystem `inventar` repo, not here). See [adr/README.md](adr/README.md).
|
|
||||||
|
|
||||||
## Data & storage
|
## Data & storage
|
||||||
|
|
||||||
| Term | Meaning |
|
| Term | Meaning |
|
||||||
@@ -74,8 +10,8 @@ Architecture Standard Record — a standing cross-repository policy (lives in th
|
|||||||
| **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) |
|
| **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) |
|
||||||
| **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate |
|
| **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate |
|
||||||
| **ZSTD** | Zstandard compression; the default codec for all Parquet files here |
|
| **ZSTD** | Zstandard compression; the default codec for all Parquet files here |
|
||||||
| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown; same role as [source of truth](#source-of-truth) |
|
| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown |
|
||||||
| **Hot / warm / cold tiers** | Classic three-tier model: in-memory window → local NVMe → ground warehouse. Here mapped to [T0](#t0--hot), [T1](#t1--warm), and [T3](#t3--warehouse); [T2](#t2--shared) and [T4](#t4--dev) are named explicitly |
|
| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse |
|
||||||
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
|
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
|
||||||
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
|
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
|
||||||
| **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame |
|
| **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame |
|
||||||
@@ -119,7 +55,7 @@ Architecture Standard Record — a standing cross-repository policy (lives in th
|
|||||||
| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone |
|
| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone |
|
||||||
| **k3s** | Lightweight Kubernetes distribution; used on the ground only |
|
| **k3s** | Lightweight Kubernetes distribution; used on the ground only |
|
||||||
| **Dev Container** | Reproducible containerized development environment definition |
|
| **Dev Container** | Reproducible containerized development environment definition |
|
||||||
| **MinIO** | S3-compatible object store; runs on-board for derived data and on the ground as the warehouse backend |
|
| **MinIO** | S3-compatible object store **already in the stack**; default home is the **ground warehouse (T3)**. On-board use is an exception, not the baseline |
|
||||||
| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it |
|
| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it |
|
||||||
| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight |
|
| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight |
|
||||||
| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version |
|
| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version |
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
|
|||||||
| Constraint | Consequence |
|
| Constraint | Consequence |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
|
| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind |
|
||||||
| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention. A [supervising operator](00-glossary.md#swarm--robotics) may set mission intent over [C2](00-glossary.md#swarm--robotics) before and during flight, but does not steer individual vehicles |
|
| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention |
|
||||||
| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
|
| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator |
|
||||||
| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
|
| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch |
|
||||||
| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
|
| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight |
|
||||||
@@ -28,10 +28,10 @@ Plus everything a platform needs around that: reproducible builds and deployment
|
|||||||
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
|
- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary).
|
||||||
- **DuckDB** is already in use for local data handling.
|
- **DuckDB** is already in use for local data handling.
|
||||||
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
|
- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked.
|
||||||
- **MinIO** is available and considered as the inter-drone sync mechanism.
|
- **MinIO** is already in the stack. Exact placement (ground vs on-board vs sync path) was not fully specified in the brief; this proposal treats MinIO as **first-class ground-warehouse infrastructure** by default and keeps on-board MinIO as an open option — see [00 — Executive](00-executive.md) and [09 — Open questions](09-open-questions.md).
|
||||||
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
|
- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)).
|
||||||
- After landing, each drone's data is **offloaded to the on-prem [source of truth](00-glossary.md#source-of-truth)** for replay and iterative model training.
|
- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training.
|
||||||
- Mission intent (declarative goals) reaches the swarm over a narrow **[C2](00-glossary.md#swarm--robotics) channel**; there is no continuous ground link in flight.
|
- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight.
|
||||||
|
|
||||||
## What is assumed (explicitly marked as assumptions)
|
## What is assumed (explicitly marked as assumptions)
|
||||||
|
|
||||||
@@ -43,6 +43,7 @@ Plus everything a platform needs around that: reproducible builds and deployment
|
|||||||
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
|
| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions |
|
||||||
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
|
| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns |
|
||||||
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
|
| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only |
|
||||||
|
| A7 | **MinIO's primary home is the ground warehouse (T3), not the drone** | On-board object store competes with flight-critical CPU/RAM; local Parquet + DuckDB already cover the hot path. On-board MinIO is not excluded if ops already depend on an S3 API in Compose |
|
||||||
|
|
||||||
## Out of scope
|
## Out of scope
|
||||||
|
|
||||||
|
|||||||
+20
-14
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
## On-board: three layers, one Compose file
|
## On-board: three layers, one Compose file
|
||||||
|
|
||||||
Every drone runs the same [three-layer](00-glossary.md#architecture-layer-1--ingestion) data plane ([storage floors](00-glossary.md#t0--hot) T0–T4 are a separate concept — see [03 — Data platform](03-data-platform.md#storage-floors-tiers)). **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)).
|
Every drone runs the same three-layer data plane. **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)).
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph TB
|
graph TB
|
||||||
@@ -20,11 +20,13 @@ graph TB
|
|||||||
subgraph serving [Layer 3 — Serving and sync]
|
subgraph serving [Layer 3 — Serving and sync]
|
||||||
HOOK["event hook<br/>fires on new derived data"]
|
HOOK["event hook<br/>fires on new derived data"]
|
||||||
PUB["state publisher<br/>pub/sub broadcast"]
|
PUB["state publisher<br/>pub/sub broadcast"]
|
||||||
MINIO["MinIO<br/>derived datasets bucket"]
|
BULK["bulk sync<br/>rsync over SSH"]
|
||||||
QAPI["query API<br/>peer data requests"]
|
QAPI["query API<br/>SQL-over-SSH"]
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
MINIO[("MinIO — ground warehouse T3<br/>already in the stack")]
|
||||||
|
|
||||||
SENSORS --> WRITER
|
SENSORS --> WRITER
|
||||||
VIDEO -->|detections| WRITER
|
VIDEO -->|detections| WRITER
|
||||||
WRITER --> NVME
|
WRITER --> NVME
|
||||||
@@ -32,17 +34,20 @@ graph TB
|
|||||||
DUCK --> NVME
|
DUCK --> NVME
|
||||||
WRITER -->|derived rows| HOOK
|
WRITER -->|derived rows| HOOK
|
||||||
HOOK --> PUB
|
HOOK --> PUB
|
||||||
HOOK --> MINIO
|
HOOK --> BULK
|
||||||
QAPI --> DUCK
|
QAPI --> DUCK
|
||||||
PUB -.->|mesh| PEERS["peer drones"]
|
PUB -.->|mesh pose| PEERS["peer drones"]
|
||||||
MINIO -.->|replication| PEERS
|
BULK -.->|sealed partitions| PEERS
|
||||||
QAPI -.->|on demand| PEERS
|
QAPI -.->|on demand| PEERS
|
||||||
|
NVME -->|post-flight offload| MINIO
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**MinIO** is already in the stack and is the **default ground warehouse (T3)** behind offload — keep it, do not rip it out. Default sketch: **not on the drone** (CPU/RAM budget); on-board MinIO remains an allowed exception if Compose already depends on an S3 API. **In-flight peer bulk sync is SSH/rsync** (or team-standard MinIO replication if that is already how ops works) — never the 5 Hz pose path. See [00 — Executive](00-executive.md) and [04 — Swarm sync](04-swarm-sync.md).
|
||||||
|
|
||||||
### Layer 1 — Ingestion
|
### Layer 1 — Ingestion
|
||||||
|
|
||||||
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
|
- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on.
|
||||||
- `video-analytics` runs a [**YOLO-like**](00-glossary.md#swarm--robotics) object detector on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images.
|
- `video-analytics` runs a **YOLO-like object detector** on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images.
|
||||||
- Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient).
|
- Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient).
|
||||||
|
|
||||||
### Layer 2 — Storage and transform
|
### Layer 2 — Storage and transform
|
||||||
@@ -53,10 +58,11 @@ graph TB
|
|||||||
|
|
||||||
### Layer 3 — Serving and sync
|
### Layer 3 — Serving and sync
|
||||||
|
|
||||||
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
|
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, bulk-sync hint, or a local mission-logic callback. Nothing polls.
|
||||||
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
|
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (UDP in the PoC; Zenoh proposed — [04](04-swarm-sync.md)).
|
||||||
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH ([rsync](00-glossary.md#bulk-sync) delta transfer); MinIO remains optional where an S3 API is wanted.
|
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer). If the team already operates MinIO replication for that job, reuse it — do not invent a second path. Pose traffic never goes through MinIO.
|
||||||
- **Peer queries** are read-only DuckDB SQL over SSH forced commands — SELECT-only gate, read-only OS user, columnar responses (the design decision and its reasoning are in [04](04-swarm-sync.md)).
|
- **Peer queries** are read-only DuckDB SQL — HTTP explorer gate in the PoC; **SSH forced commands** proposed for flight (SELECT-only gate, read-only OS user, columnar responses — [04](04-swarm-sync.md)).
|
||||||
|
- **MinIO** sits on the **ground** as T3 by default ([00 — Executive](00-executive.md)); on-board MinIO is an explicit exception, not the baseline.
|
||||||
|
|
||||||
## Communication planes
|
## Communication planes
|
||||||
|
|
||||||
@@ -65,11 +71,11 @@ Three isolated planes with different lifecycles:
|
|||||||
```mermaid
|
```mermaid
|
||||||
graph LR
|
graph LR
|
||||||
subgraph planes [Communication planes]
|
subgraph planes [Communication planes]
|
||||||
C2["[C2](00-glossary.md#swarm--robotics) control plane<br/>mission intent, narrow, in production"]
|
C2["C2 control plane<br/>mission intent, narrow, in production"]
|
||||||
DATA["swarm data plane<br/>state broadcast + sync, in production"]
|
DATA["swarm data plane<br/>state broadcast + sync, in production"]
|
||||||
DEV["dev/debug plane<br/>live telemetry, bench only"]
|
DEV["dev/debug plane<br/>live telemetry, bench only"]
|
||||||
end
|
end
|
||||||
OPERATOR["supervising operator<br/>([man-in-the-loop](00-glossary.md#swarm--robotics))"] --> C2
|
OPERATOR["supervising operator<br/>(man-in-the-loop)"] --> C2
|
||||||
C2 --> SWARM["swarm"]
|
C2 --> SWARM["swarm"]
|
||||||
SWARM <--> DATA
|
SWARM <--> DATA
|
||||||
DEV -.->|absent from production builds| SWARM
|
DEV -.->|absent from production builds| SWARM
|
||||||
@@ -79,7 +85,7 @@ graph LR
|
|||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes |
|
| **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes |
|
||||||
| **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes |
|
| **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes |
|
||||||
| **Dev/debug plane** | Full live telemetry for bench development and [HIL](00-glossary.md#infrastructure--delivery) tests | High | **No — physically absent from production builds** |
|
| **Dev/debug plane** | Full live telemetry for bench development and HIL tests | High | **No — physically absent from production builds** |
|
||||||
|
|
||||||
The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it.
|
The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it.
|
||||||
|
|
||||||
|
|||||||
+15
-15
@@ -1,6 +1,6 @@
|
|||||||
# 03 — Data platform
|
# 03 — Data platform
|
||||||
|
|
||||||
The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every [storage floor](00-glossary.md#t0--hot)** of the system.
|
The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every floor** of the system.
|
||||||
|
|
||||||
## Datasets
|
## Datasets
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@ Three logical datasets, distinct because their consumers and sync policies diffe
|
|||||||
|
|
||||||
## Partitioning layout
|
## Partitioning layout
|
||||||
|
|
||||||
Identity dimensions first, calendar time last, everything [Hive-style](00-glossary.md#data--storage) `key=value`:
|
Identity dimensions first, calendar time last, everything Hive-style `key=value`:
|
||||||
|
|
||||||
```
|
```
|
||||||
dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/
|
dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/
|
||||||
@@ -29,7 +29,7 @@ Design decisions and their reasons:
|
|||||||
|
|
||||||
- **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.)
|
- **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.)
|
||||||
- **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both.
|
- **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both.
|
||||||
- **Flight identifiers.** The canonical flight id is a [**UUIDv7**](00-glossary.md#data--storage) (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
|
- **Flight identifiers.** The canonical flight id is a **UUIDv7** (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
|
||||||
- **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time.
|
- **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time.
|
||||||
- **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)).
|
- **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)).
|
||||||
|
|
||||||
@@ -50,7 +50,7 @@ sequenceDiagram
|
|||||||
S->>W: rows (int64 ns, typed)
|
S->>W: rows (int64 ns, typed)
|
||||||
W->>C: flush min_NN.parquet (small, ZSTD-fast)
|
W->>C: flush min_NN.parquet (small, ZSTD-fast)
|
||||||
end
|
end
|
||||||
Note over Z: window closes (hour [sealed](00-glossary.md#data--storage)<br/>or flight ends)
|
Note over Z: window closes (hour sealed<br/>or flight ends)
|
||||||
Z->>C: read all min_*.parquet
|
Z->>C: read all min_*.parquet
|
||||||
Z->>P: write one compact file (ZSTD-high)
|
Z->>P: write one compact file (ZSTD-high)
|
||||||
Z->>C: remove blocks after verify
|
Z->>C: remove blocks after verify
|
||||||
@@ -63,28 +63,28 @@ sequenceDiagram
|
|||||||
|
|
||||||
## Storage floors (tiers)
|
## Storage floors (tiers)
|
||||||
|
|
||||||
Same [Parquet](00-glossary.md#data--storage) + [DuckDB](00-glossary.md#data--storage) format on every floor; only volume, retention, and location change. See [glossary](00-glossary.md#t0--hot) for the full T0–T4 definitions.
|
Same format on every floor; only volume, retention, and location change:
|
||||||
|
|
||||||
| Floor | Where | Contents | Retention |
|
| Floor | Where | Contents | Retention |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **[T0 — hot](00-glossary.md#t0--hot)** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
|
| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
|
||||||
| **[T1 — warm](00-glossary.md#t1--warm)** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
|
| **T1 — warm** | Drone NVMe (Parquet) | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
|
||||||
| **[T2 — shared](00-glossary.md#t2--shared)** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission |
|
| **T2 — shared** | Peer-derived cache on NVMe (Parquet) | Derived data only (`detections`, `state`) pulled from peers when the link allows | Current mission |
|
||||||
| **[T3 — warehouse](00-glossary.md#t3--warehouse)** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; [source of truth](00-glossary.md#source-of-truth) for replay, analytics, model training | Years |
|
| **T3 — warehouse** | Ground **MinIO** (or equivalent) + Parquet lakehouse | Every flight of every drone; replay, analytics, model training — **primary MinIO home** | Years |
|
||||||
| **[T4 — dev](00-glossary.md#t4--dev)** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
|
| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
|
||||||
|
|
||||||
|
**MinIO note:** default sketch keeps MinIO on the **ground (T3)**. An on-board MinIO sidecar is a possible exception if the team already standardises on an S3 API in Compose — derived data only, never the 5 Hz pose path. Placement is an open question ([09](09-open-questions.md), [00 — Executive](00-executive.md)).
|
||||||
|
|
||||||
## Flight offload: a mirror, not a migration
|
## Flight offload: a mirror, not a migration
|
||||||
|
|
||||||
Because [T1](00-glossary.md#t1--warm) and [T3](00-glossary.md#t3--warehouse) share the identical layout, [offload](00-glossary.md#offload) after landing is:
|
Because T1 and T3 share the identical layout, offload after landing is:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# per drone, at the base station
|
# per drone, at the base station
|
||||||
rsync -a drone-nvme/flights/ warehouse/flights/ # or mc mirror to object store
|
mc mirror drone-nvme/flights/ warehouse/flights/ # or rsync over the wired dock
|
||||||
duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…"
|
duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…"
|
||||||
```
|
```
|
||||||
|
|
||||||
For S3-only backends where rsync is awkward, [rclone](00-glossary.md#rclone) is an optional rsync-class alternative — not the default mesh path ([09 — Open questions](09-open-questions.md) §12).
|
|
||||||
|
|
||||||
The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job.
|
The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job.
|
||||||
|
|
||||||
## Retention and quotas on board
|
## Retention and quotas on board
|
||||||
@@ -140,6 +140,6 @@ On the ground this powers replay overlays, coverage analysis (did ubiquitous sen
|
|||||||
|
|
||||||
## Schema management
|
## Schema management
|
||||||
|
|
||||||
- Every dataset schema is versioned in-repo (planned `schemas/` directory — [roadmap stage 2](08-roadmap.md), [09 — Open questions](09-open-questions.md) §14; SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
|
- Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
|
||||||
- Parquet files carry the schema version in file metadata; readers select the matching deserializer.
|
- Parquet files carry the schema version in file metadata; readers select the matching deserializer.
|
||||||
- Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release.
|
- Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release.
|
||||||
|
|||||||
+29
-21
@@ -16,21 +16,27 @@ Bandwidth is the scarcest resource in the system. Every message class gets an ex
|
|||||||
|
|
||||||
## Broadcast payload: small on the wire, precise at rest
|
## Broadcast payload: small on the wire, precise at rest
|
||||||
|
|
||||||
The `state` broadcast is a fixed compact frame (46 bytes, little-endian):
|
The `state` broadcast is a fixed compact frame. The runnable PoC encodes it as
|
||||||
|
**45 bytes** little-endian (`simulator/virtual_drone/broadcast.py`); that size is
|
||||||
|
what the prototype uses for bandwidth estimates.
|
||||||
|
|
||||||
| Field | Type | Notes |
|
| Field | Wire type (PoC) | Notes |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `magic` | 2 bytes | `b"SH"` |
|
| `magic` + `version` | 2s + uint8 | `b"SH"`, version `1` |
|
||||||
| `version` | uint8 | Protocol version |
|
| `drone_id` | 8s ascii | Zero-padded; a fleet `uint16` registry id is a natural production swap |
|
||||||
| `drone_id` | 8 bytes | Zero-padded ASCII fleet id |
|
|
||||||
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
|
| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage |
|
||||||
| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision |
|
| `pos_x/y/z` | 3 × int32 | **Millimeters** in the **mission frame** — quantized on the wire only; storage keeps full float precision |
|
||||||
| `att_roll/pitch/yaw` | int16 | Centi-degrees |
|
| `att_roll/pitch/yaw` | 3 × int16 | Centi-degrees — attitude stays on the wire so peers need no local shape model |
|
||||||
| `vel_x/y/z` | int16 | cm/s |
|
| `vel_x/y/z` | 3 × int16 | cm/s |
|
||||||
| `frame_ref` | uint8 | Frame of reference id ([GPS-denied](00-glossary.md#swarm--robotics): local/visual-odometry frames must be explicit) |
|
| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) |
|
||||||
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
|
| `flags` | uint8 | Battery-low, returning, degraded-sensors, … |
|
||||||
|
|
||||||
46 bytes per frame → a 50-drone swarm at 5 Hz is ~11.5 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh.
|
45 bytes × 5 Hz × 50 drones ≈ 11 KB/s of pose traffic before transport overhead — still trivial on a congested mesh.
|
||||||
|
|
||||||
|
An earlier sketch used relative coordinates and a bounding sphere (no attitude).
|
||||||
|
It was dropped: mission-frame pose + attitude is simpler to fuse post-flight and
|
||||||
|
costs almost nothing at this frame size. Relative localization remains a
|
||||||
|
consumer concern when `frame_ref` differs across peers ([09](09-open-questions.md)).
|
||||||
|
|
||||||
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
|
`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison.
|
||||||
|
|
||||||
@@ -53,39 +59,41 @@ graph LR
|
|||||||
SB -->|"store-and-forward relay"| SC
|
SB -->|"store-and-forward relay"| SC
|
||||||
```
|
```
|
||||||
|
|
||||||
### Recommended: [Zenoh](00-glossary.md#swarm--robotics)
|
### Recommended (proposal): Zenoh
|
||||||
|
|
||||||
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
|
- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub.
|
||||||
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
|
- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware.
|
||||||
- Tiny footprint, ARM64-native.
|
- Tiny footprint, ARM64-native.
|
||||||
|
|
||||||
|
**PoC today:** the simulator and the 2D prototype exercise the **raw UDP** pose path only — the minimal degraded profile below. Zenoh is the proposed production pub/sub, not yet wired into the runnable stack.
|
||||||
|
|
||||||
### Alternatives considered
|
### Alternatives considered
|
||||||
|
|
||||||
| Option | Verdict |
|
| Option | Verdict |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
|
| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively |
|
||||||
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
|
| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh |
|
||||||
| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode |
|
| **Raw UDP multicast** | **Implemented in the PoC** for pose broadcast (fixed frame, no discovery); no query layer, no reliability — also the documented degraded mode |
|
||||||
| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below |
|
| **MinIO bucket replication** | Keep as a **team option** for bulk derived catch-up if ops already standardise on it; wrong tool for the 5 Hz pose path. Default sketch: MinIO primary on the **ground warehouse**, not on the drone |
|
||||||
|
|
||||||
## Two sync mechanisms, deliberately separate
|
## Two sync mechanisms, deliberately separate
|
||||||
|
|
||||||
1. **Fast path — pub/sub ([Zenoh](00-glossary.md#swarm--robotics)):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
|
1. **Fast path — pub/sub (Zenoh proposed; UDP in the PoC):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
|
||||||
2. **Bulk path — [rsync](00-glossary.md#bulk-sync) over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
|
2. **Bulk path — rsync over persistent SSH (proposed):** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. The PoC visualises bulk volume; it does not yet run real rsync between virtual drones.
|
||||||
|
|
||||||
Why SSH-based bulk sync over object-store replication:
|
Why SSH-based bulk sync over object-store replication:
|
||||||
|
|
||||||
- **Identity is already there.** Every drone holds pre-provisioned [ed25519](00-glossary.md#infrastructure--delivery) keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
|
- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
|
||||||
- **One persistent multiplexed session** ([`ControlMaster`](00-glossary.md#infrastructure--delivery)) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
|
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
|
||||||
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
|
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
|
||||||
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.
|
- **Zero extra services** on the flight-critical node by default. **MinIO stays first-class on the ground (T3).** On-board MinIO is an exception only if Compose already depends on an S3 API. In-flight bulk path defaults to SSH/rsync; switch to MinIO replication if that is already how the team moves objects.
|
||||||
|
|
||||||
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
|
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
|
||||||
|
|
||||||
Two deliberate non-features of the SSH channel:
|
Two deliberate non-features of the SSH channel:
|
||||||
|
|
||||||
- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
|
- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
|
||||||
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH [forced command](00-glossary.md#infrastructure--delivery)** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.
|
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.
|
||||||
|
|
||||||
## Design decision: SQL-over-SSH as the peer query channel
|
## Design decision: SQL-over-SSH as the peer query channel
|
||||||
|
|
||||||
@@ -107,7 +115,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
|
|||||||
| Layer | Mechanism | What it stops |
|
| Layer | Mechanism | What it stops |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
|
| 1. Key = operation | Forced command: the query key can only invoke the query wrapper, nothing else | Arbitrary exec, lateral movement |
|
||||||
| 2. Statement gate | Wrapper accepts a single statement, parses it, rejects anything but `SELECT` (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); parameters bound, not interpolated | SQL-as-a-write-channel, config tampering |
|
| 2. Statement gate | Wrapper accepts a single statement, rejects anything but `SELECT`/`WITH`/… (no `COPY`, `ATTACH`, `INSTALL`, `SET`, multi-statements); production should prefer a real parser + bound parameters, not keywords alone | SQL-as-a-write-channel, config tampering |
|
||||||
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
|
| 3. OS permissions | Wrapper runs as a dedicated user with **read-only filesystem access** to the data root and write access to nothing | Any write that slips past layer 2 |
|
||||||
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
|
| 4. Engine hardening | `:memory:` database, external access disabled except the data-root glob, extension loading off | Reaching outside the store |
|
||||||
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
|
| 5. Resource caps | Timeout, memory cap, niced CPU (flight software always wins), response size budget | Denial of service via expensive queries |
|
||||||
@@ -116,7 +124,7 @@ SQL access must not become a write channel. A single "read-only connection" flag
|
|||||||
|
|
||||||
- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
|
- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. Schema version is negotiated from the fleet release manifest — peers on one release speak one schema by construction.
|
||||||
- **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
|
- **Read-only SELECT over the local Parquet store**, streamed back as Parquet/Arrow batches over the multiplexed SSH session. Never JSON blobs.
|
||||||
- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from [T3](00-glossary.md#t3--warehouse) if result sizes outgrow it.
|
- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if result sizes outgrow it.
|
||||||
|
|
||||||
What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.
|
What this buys operationally: the dev/bench experience and the flight protocol are the same thing. An engineer debugging on the bench runs the identical query a peer drone would run — same wrapper, same permissions, same output format. There is no "debug API" that behaves differently from the real one.
|
||||||
|
|
||||||
|
|||||||
@@ -23,9 +23,9 @@ graph TB
|
|||||||
A["drone A<br/>wg key + certs"]
|
A["drone A<br/>wg key + certs"]
|
||||||
B["drone B<br/>wg key + certs"]
|
B["drone B<br/>wg key + certs"]
|
||||||
C["drone C<br/>wg key + certs"]
|
C["drone C<br/>wg key + certs"]
|
||||||
A <-->|"WireGuard tunnel"| B
|
A <-->|"WireGuard + mTLS"| B
|
||||||
B <-->|"WireGuard tunnel"| C
|
B <-->|"WireGuard + mTLS"| C
|
||||||
A <-->|"WireGuard tunnel"| C
|
A <-->|"WireGuard + mTLS"| C
|
||||||
end
|
end
|
||||||
PROV -->|"per-device identity, peer list"| A
|
PROV -->|"per-device identity, peer list"| A
|
||||||
PROV --> B
|
PROV --> B
|
||||||
@@ -65,7 +65,7 @@ Keep both layers: WireGuard for *who is on the network and whether the wire is r
|
|||||||
## Data at rest
|
## Data at rest
|
||||||
|
|
||||||
- On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable.
|
- On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable.
|
||||||
- The ground warehouse applies standard at-rest encryption plus role-based access; it is the [source of truth](00-glossary.md#source-of-truth) and the single most valuable asset in the system (every flight ever flown).
|
- The ground warehouse applies standard at-rest encryption plus role-based access; it is the single most valuable asset in the system (every flight ever flown).
|
||||||
|
|
||||||
## Attack surface, deliberately shortened
|
## Attack surface, deliberately shortened
|
||||||
|
|
||||||
|
|||||||
@@ -9,15 +9,15 @@ graph LR
|
|||||||
subgraph dev [3 — Dev and simulation]
|
subgraph dev [3 — Dev and simulation]
|
||||||
SIM["virtual drone fleet<br/>(simulator, Compose)"]
|
SIM["virtual drone fleet<br/>(simulator, Compose)"]
|
||||||
LAP["engineer laptop<br/>Dev Container"]
|
LAP["engineer laptop<br/>Dev Container"]
|
||||||
SLICE["[T4](00-glossary.md#t4--dev): data slices"]
|
SLICE["T4: data slices"]
|
||||||
end
|
end
|
||||||
subgraph fleet [1 — Fleet, in flight]
|
subgraph fleet [1 — Fleet, in flight]
|
||||||
D1["drone: Compose data plane<br/>[T0](00-glossary.md#t0--hot) hot + [T1](00-glossary.md#t1--warm) NVMe + [T2](00-glossary.md#t2--shared) MinIO"]
|
D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 peer cache"]
|
||||||
D2["drone …"]
|
D2["drone …"]
|
||||||
end
|
end
|
||||||
subgraph ground [2 — Ground, on-prem]
|
subgraph ground [2 — Ground, on-prem]
|
||||||
DOCK["base station docks"]
|
DOCK["base station docks"]
|
||||||
DWH["[T3](00-glossary.md#t3--warehouse) warehouse<br/>object store + Parquet/DuckDB"]
|
DWH["T3 warehouse<br/>MinIO + Parquet/DuckDB"]
|
||||||
K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
|
K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
|
||||||
TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
|
TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
|
||||||
end
|
end
|
||||||
@@ -34,7 +34,7 @@ graph LR
|
|||||||
## 1 — Fleet infrastructure (on the drones)
|
## 1 — Fleet infrastructure (on the drones)
|
||||||
|
|
||||||
- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md).
|
- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md).
|
||||||
- Storage floors [T0–T2](00-glossary.md#t0--hot). Nothing here requires ground contact during a mission.
|
- Storage floors T0–T2. Nothing here requires ground contact during a mission.
|
||||||
- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.
|
- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.
|
||||||
|
|
||||||
## 2 — Ground infrastructure (on-prem)
|
## 2 — Ground infrastructure (on-prem)
|
||||||
@@ -43,7 +43,7 @@ The permanent installation. This **is** allowed to be a cluster — links are wi
|
|||||||
|
|
||||||
| Component | Runs on | Role |
|
| Component | Runs on | Role |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| **Warehouse ([T3](00-glossary.md#t3--warehouse))** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's [source of truth](00-glossary.md#source-of-truth) and long-term memory |
|
| **Warehouse (T3)** | **MinIO** (already in the stack) + Parquet | Every flight of every drone; the system's long-term memory |
|
||||||
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
|
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
|
||||||
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
|
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
|
||||||
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
|
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
|
||||||
@@ -60,7 +60,7 @@ The environment engineers live in daily — and deliberately the first thing to
|
|||||||
- **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free.
|
- **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free.
|
||||||
- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop.
|
- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop.
|
||||||
- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
|
- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
|
||||||
- **[T4](00-glossary.md#t4--dev) data slices**: a thin tool pulls partition subsets from [T3](00-glossary.md#t3--warehouse) (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
|
- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
|
||||||
- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).
|
- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).
|
||||||
|
|
||||||
## IaC boundaries
|
## IaC boundaries
|
||||||
@@ -74,13 +74,13 @@ Infrastructure as code follows the same discipline as the data plane: each tool
|
|||||||
| Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. |
|
| Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. |
|
||||||
| On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). |
|
| On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). |
|
||||||
|
|
||||||
Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, [T1](00-glossary.md#t1--warm)) and **`var/t3/`** (warehouse, [T3](00-glossary.md#t3--warehouse)). Compose, k3d, and DuckDB on the host all read the same Parquet tree.
|
Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, T1) and **`var/t3/`** (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree.
|
||||||
|
|
||||||
Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle.
|
Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle.
|
||||||
|
|
||||||
### k3d: the ground segment as an executable miniature
|
### k3d: the ground segment as an executable miniature
|
||||||
|
|
||||||
`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the [T1](00-glossary.md#t1--warm)→[T3](00-glossary.md#t3--warehouse) offload CronJob. The sim CronJob uses `cp -ru` plus `mc mirror` as a shortcut; production dock [offload](00-glossary.md#offload) uses rsync or `mc mirror` with checksum audit ([03](03-data-platform.md)). The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API.
|
`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API.
|
||||||
|
|
||||||
### Next fidelity step: microVMs
|
### Next fidelity step: microVMs
|
||||||
|
|
||||||
@@ -90,7 +90,7 @@ The k3d fleet shares one kernel and one network namespace tree — good enough t
|
|||||||
|
|
||||||
| Operation | What it costs with one layout everywhere |
|
| Operation | What it costs with one layout everywhere |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Flight offload | `rsync` / `mc mirror` + audit — no transform step ([03](03-data-platform.md)) |
|
| Flight offload | `mc mirror` / `rsync` + audit — no transform step |
|
||||||
| Debugging a field issue | Pull the flight slice, replay locally, same queries |
|
| Debugging a field issue | Pull the flight slice, replay locally, same queries |
|
||||||
| Validating a pipeline change | Run simulator, diff Parquet outputs |
|
| Validating a pipeline change | Run simulator, diff Parquet outputs |
|
||||||
| Training data prep | Read T3 directly, no export pipeline |
|
| Training data prep | Read T3 directly, no export pipeline |
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ graph LR
|
|||||||
LG["structured logs<br/>= Parquet partitions"]
|
LG["structured logs<br/>= Parquet partitions"]
|
||||||
end
|
end
|
||||||
subgraph ground [Ground k3s]
|
subgraph ground [Ground k3s]
|
||||||
DWH["[T3](00-glossary.md#t3--warehouse) warehouse"]
|
DWH["T3 warehouse"]
|
||||||
PR["Prometheus<br/>(live: docks, CI, sim farm)"]
|
PR["Prometheus<br/>(live: docks, CI, sim farm)"]
|
||||||
LK["Loki<br/>(live logs, dev plane)"]
|
LK["Loki<br/>(live logs, dev plane)"]
|
||||||
GF["Grafana<br/>dashboards + replay"]
|
GF["Grafana<br/>dashboards + replay"]
|
||||||
@@ -44,7 +44,7 @@ graph LR
|
|||||||
```
|
```
|
||||||
|
|
||||||
- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
|
- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
|
||||||
- **Flight post-mortems query the [source of truth](00-glossary.md#source-of-truth) warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.
|
- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.
|
||||||
|
|
||||||
## Health questions this design answers cheaply
|
## Health questions this design answers cheaply
|
||||||
|
|
||||||
|
|||||||
+6
-6
@@ -9,11 +9,11 @@ graph TD
|
|||||||
S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"]
|
S1["Stage 1<br/>Dev environment + CI skeleton<br/>Dev Containers, GitLab templates, multi-arch builds"]
|
||||||
S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"]
|
S2["Stage 2<br/>Registry and artifacts<br/>images, model weights, schemas, cleanup policies"]
|
||||||
S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"]
|
S3["Stage 3<br/>Single-drone data pipeline<br/>ingest → current/ → sealed Parquet → DuckDB"]
|
||||||
S4["Stage 4<br/>Serving layer<br/>event hook, query API, local MinIO"]
|
S4["Stage 4<br/>Serving layer<br/>event hook, query API"]
|
||||||
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
|
S5["Stage 5<br/>Virtual swarm simulation<br/>N drones in Compose, seeded scenarios, CI regression"]
|
||||||
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
|
S6["Stage 6<br/>Mesh network and security<br/>WireGuard overlay, PKI provisioning, mTLS"]
|
||||||
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk replication (rsync over SSH)"]
|
S7["Stage 7<br/>Swarm sync<br/>pose broadcast, detections pub/sub, bulk pull"]
|
||||||
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, T3 store, replay tooling"]
|
S8["Stage 8<br/>Ground warehouse<br/>offload + audit, MinIO T3, replay tooling"]
|
||||||
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
|
S9["Stage 9<br/>Observability<br/>platform metrics as sensor stream, ground dashboards"]
|
||||||
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
|
S10["Stage 10<br/>Fleet releases<br/>manifest, mirror, dock delivery, atomic rollback"]
|
||||||
S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"]
|
S11["Stage 11<br/>Query standard<br/>GraphQL contract, schema registry, multi-team surface"]
|
||||||
@@ -41,11 +41,11 @@ graph TD
|
|||||||
| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — |
|
| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — |
|
||||||
| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 |
|
| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 |
|
||||||
| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 |
|
| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 |
|
||||||
| **4 — Serving layer** | Event hook + query API + on-board MinIO | Event-driven consumption; nothing polls | 3 |
|
| **4 — Serving layer** | Event hook + query API (SQL gate) | Event-driven consumption; nothing polls | 3 |
|
||||||
| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 |
|
| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 |
|
||||||
| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 |
|
| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 |
|
||||||
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk replication over the mesh | Staleness budgets met under simulated loss/partitions | 5, 6 |
|
| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk partition pull | Staleness budgets met under simulated loss/partitions | 5, 6 |
|
||||||
| **8 — Ground warehouse** | Dock offload (mirror + audit), T3 store, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
|
| **8 — Ground warehouse** | Dock offload (mirror + audit), **MinIO as T3**, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 |
|
||||||
| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 |
|
| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 |
|
||||||
| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 |
|
| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 |
|
||||||
| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 |
|
| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 |
|
||||||
|
|||||||
+11
-16
@@ -18,7 +18,7 @@ Visual-odometry positions are relative; different drones may hold different loca
|
|||||||
|
|
||||||
By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging.
|
By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging.
|
||||||
|
|
||||||
*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from the [source of truth](00-glossary.md#source-of-truth) in the warehouse. If a persistent shared structure becomes unavoidable, use a [CRDT](00-glossary.md#infrastructure--delivery) type for it rather than inventing reconciliation.
|
*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from ground truth in the warehouse. If a persistent shared structure becomes unavoidable, use a CRDT type for it rather than inventing reconciliation.
|
||||||
|
|
||||||
## 4 — Degradation policy under sustained link loss
|
## 4 — Degradation policy under sustained link loss
|
||||||
|
|
||||||
@@ -42,7 +42,7 @@ The degradation ladder ([03](03-data-platform.md)) needs a domain-approved order
|
|||||||
|
|
||||||
The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration.
|
The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration.
|
||||||
|
|
||||||
*Proposal:* a small [HIL](00-glossary.md#infrastructure--delivery) rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI.
|
*Proposal:* a small HIL rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI.
|
||||||
|
|
||||||
## 8 — Video retention policy
|
## 8 — Video retention policy
|
||||||
|
|
||||||
@@ -64,24 +64,19 @@ Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage).
|
|||||||
|
|
||||||
## 11 — Lake retention and partition pruning
|
## 11 — Lake retention and partition pruning
|
||||||
|
|
||||||
The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on [T1](00-glossary.md#t1--warm) disk after offload to [T3](00-glossary.md#t3--warehouse).
|
The observability path now scans only the most recent flights ([ADR-0009](adr/ADR-0009-bounded-lake-scans.md)), but old `flight=` partitions still accumulate on T1 disk after offload to T3.
|
||||||
|
|
||||||
*Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job.
|
*Proposal:* a scheduled prune of T1 partitions whose flights are confirmed present in the T3 warehouse (offload as the retention gate), configurable by age and free-space watermark. On the drone the same policy is bounded by the NVMe quota ladder ([03](03-data-platform.md)); in the sim environment it is a ground CronJob alongside the offload job.
|
||||||
|
|
||||||
## 12 — Bulk transfer tool choice
|
## 12 — MinIO placement (already in the stack)
|
||||||
|
|
||||||
In-flight peer bulk sync, dock offload, and sim shortcuts have used different wording (rsync, rclone, `mc mirror`, `cp -ru`).
|
Exact how MinIO is used today was not fully specified. It should not be ripped out.
|
||||||
|
|
||||||
*Proposal:* **rsync over SSH** is the default for peer [bulk sync](00-glossary.md#bulk-sync) and wired dock [offload](00-glossary.md#offload) of the Hive partition tree; **MinIO `mc mirror`** when the far end is the object-store facade; **[rclone](00-glossary.md#rclone)** only when the backend is S3-compatible and rsync is awkward; sim CronJob keeps **`cp -ru` + `mc mirror`** as a documented shortcut. Pick one primary tool per path in provisioning, not all at once.
|
*Proposal (default):*
|
||||||
|
|
||||||
## 13 — DuckLake vs plain DuckDB + Parquet on T3
|
- **Ground warehouse (T3):** primary MinIO home — offload target and long-term object API.
|
||||||
|
- **On the drone:** default **off** — local Parquet + DuckDB cover the hot path; an object store competes with flight-critical CPU/RAM.
|
||||||
|
- **Exception:** on-board MinIO allowed if Compose already depends on an S3 API — derived datasets only, never 5 Hz pose.
|
||||||
|
- **Peer bulk catch-up:** prefer sealed-partition pull (SSH/rsync); use MinIO replication only if that is already the ops standard.
|
||||||
|
|
||||||
The design journey mentions DuckLake; no ADR or implementation commits to it yet.
|
*Settle with the team:* one short diagram of today's MinIO topology (buckets, clients, what syncs). Until then, treat A7 in [01](01-problem-statement.md) as the working assumption. Summary table: [00 — Executive](00-executive.md).
|
||||||
|
|
||||||
*Proposal:* baseline remains DuckDB `read_parquet` globs over the [T3](00-glossary.md#t3--warehouse) Hive tree with MinIO underneath; evaluate DuckLake only if catalog/versioning pain appears at warehouse scale — decision gets its own ADR before any dependency lands.
|
|
||||||
|
|
||||||
## 14 — `schemas/` registry layout
|
|
||||||
|
|
||||||
[03 — Data platform](03-data-platform.md) references an in-repo `schemas/` directory that does not exist yet (roadmap stage 2).
|
|
||||||
|
|
||||||
*Proposal:* add `schemas/` as part of stage 2 with one SemVer file per dataset version, referenced from the fleet release manifest; until then treat schema versions as manifest-only fields and document the gap here rather than implying the directory already exists.
|
|
||||||
|
|||||||
@@ -9,14 +9,14 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera
|
|||||||
| **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) |
|
| **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) |
|
||||||
| **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) |
|
| **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) |
|
||||||
| **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) |
|
| **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) |
|
||||||
| **[Sustainable pulsing](00-glossary.md#swarm--robotics)** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) |
|
| **Sustainable pulsing** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) |
|
||||||
| **[Ubiquitous sensing](00-glossary.md#swarm--robotics)** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) |
|
| **Ubiquitous sensing** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) |
|
||||||
| **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) |
|
| **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) |
|
||||||
| **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) |
|
| **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) |
|
||||||
| **[GPS-denied operation](00-glossary.md#swarm--robotics)** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) |
|
| **GPS-denied operation** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) |
|
||||||
| **[Man-in-the-loop](00-glossary.md#swarm--robotics)** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) |
|
| **Man-in-the-loop** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) |
|
||||||
| **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) |
|
| **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) |
|
||||||
| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the [source of truth](00-glossary.md#source-of-truth) warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) |
|
| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) |
|
||||||
| **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) |
|
| **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) |
|
||||||
| **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) |
|
| **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) |
|
||||||
|
|
||||||
@@ -25,7 +25,7 @@ Swarm autonomy has moved past biomimicry-driven research toward practical, opera
|
|||||||
Every one of these principles quietly assumes a working data layer underneath:
|
Every one of these principles quietly assumes a working data layer underneath:
|
||||||
|
|
||||||
- *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget.
|
- *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget.
|
||||||
- *Ubiquitous sensing* assumes observations are **fused later** — that is the [source of truth](00-glossary.md#source-of-truth) warehouse with aligned timestamps and schemas.
|
- *Ubiquitous sensing* assumes observations are **fused later** — that is a warehouse with aligned timestamps and schemas.
|
||||||
- *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop.
|
- *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop.
|
||||||
- *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data.
|
- *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data.
|
||||||
- *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format.
|
- *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format.
|
||||||
|
|||||||
@@ -85,7 +85,7 @@ schemas:
|
|||||||
config:
|
config:
|
||||||
compose-bundle: 3.4.0
|
compose-bundle: 3.4.0
|
||||||
radio-profile: production-2
|
radio-profile: production-2
|
||||||
peer-registry: fleet-42-r7 # provisioning + revocations, see [05 — Network & security](05-network-security.md)
|
peer-registry: fleet-42-r7 # provisioning + revocations, see 05
|
||||||
```
|
```
|
||||||
|
|
||||||
- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
|
- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm.
|
||||||
|
|||||||
+38
-33
@@ -1,10 +1,12 @@
|
|||||||
# 12 — Design journey
|
# 12 — Design journey
|
||||||
|
|
||||||
How this design came together, told in the order the thinking actually happened.
|
How this design came together, told in the order the thinking actually happened.
|
||||||
It is a walk-through, not a report. The formal decisions, with options and
|
It is a walk-through, not a report — and **not a mandate**. The goal is a
|
||||||
trade-offs, live as [Architecture Decision Records](adr/README.md). This is the
|
from-scratch platform sketch that shows how the pieces fit; every concrete
|
||||||
story behind them, and each chapter links straight into the code it produced.
|
choice is an option with trade-offs recorded as
|
||||||
Terms: [Glossary](00-glossary.md).
|
[Architecture Decision Records](adr/README.md). Replace any piece if a better
|
||||||
|
fit shows up. This is the story behind the decisions, and each chapter links
|
||||||
|
straight into the code it produced.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -47,17 +49,17 @@ conclusion: constrain the transport, but do not constrain on-board capture. Keep
|
|||||||
everything locally. Send only what peers truly need.
|
everything locally. Send only what peers truly need.
|
||||||
|
|
||||||
> **Read more**
|
> **Read more**
|
||||||
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the [storage floors](00-glossary.md#t0--hot) (T0–T4)
|
> - [01 — Problem statement](01-problem-statement.md) and [03 — Data platform](03-data-platform.md) — the constraints and the storage tiers
|
||||||
|
|
||||||
## 4. Three pipeline stages inside each unit
|
## 4. Three tiers inside each unit
|
||||||
|
|
||||||
That gave [three architecture layers](02-architecture.md) on every drone — not the same as [storage floors](00-glossary.md#t0--hot) T0–T4:
|
That gave [three layers](03-data-platform.md) on every drone:
|
||||||
|
|
||||||
1. **Raw capture** (pipeline stage 1), never transformed.
|
1. **Raw capture**, never transformed.
|
||||||
2. **ETL / reduction** (pipeline stage 2), where data is cut down to what has to be shared:
|
2. **ETL / reduction**, where data is cut down to what has to be shared:
|
||||||
millimeters to centimeters, thinning time series where that is enough,
|
millimetres to centimetres, thinning time series where that is enough,
|
||||||
dropping what nobody downstream reads.
|
dropping what nobody downstream reads.
|
||||||
3. **A bidirectional interface** (pipeline stage 3): on one side it broadcasts position into the
|
3. **A bidirectional interface**: on one side it broadcasts position into the
|
||||||
shared channel, on the other it lets peers pull data.
|
shared channel, on the other it lets peers pull data.
|
||||||
|
|
||||||
The ETL is deliberately not over-specified. How sensor data is transformed is a
|
The ETL is deliberately not over-specified. How sensor data is transformed is a
|
||||||
@@ -66,20 +68,22 @@ stored and in which structure, not to step into that work.
|
|||||||
|
|
||||||
> **Read the code**
|
> **Read the code**
|
||||||
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
|
> - [`simulator/virtual_drone/writer.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/writer.py#L23-L80) — the Hive-partition layout and the seal step that compacts a flight
|
||||||
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds pipeline stage 1
|
> - [`simulator/virtual_drone/sensors.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/sensors.py) — the raw capture that feeds tier one
|
||||||
|
|
||||||
## 5. What actually needs to sync
|
## 5. What actually needs to sync
|
||||||
|
|
||||||
The most time-critical item is where each peer is, so every unit has time to
|
The most time-critical item is where each peer is, so every unit has time to
|
||||||
react. The choice was to broadcast [relative pose](04-swarm-sync.md) (x, y, z and
|
react. The wire frame carries **mission-frame position** (millimetres on the
|
||||||
time) instead of absolute coordinates. Relative is cheaper and enough for
|
wire, full float at rest), **attitude**, **velocity**, a `frame_ref`, and flags —
|
||||||
coordination. To avoid sending orientation, a unit is modelled as a **sphere**
|
45 bytes at 5 Hz. That is cheap enough that shrinking further is not worth the
|
||||||
that bounds its extent. That trades a little compute for much less data, and no
|
fusion pain. An earlier sketch used relative coordinates and a bounding sphere
|
||||||
per-shape encoding. If the units are identical, their 3D model can be provisioned
|
with no orientation; it was dropped. When peers disagree on frames,
|
||||||
ahead instead of transmitted.
|
`frame_ref` makes the mismatch explicit for consumers
|
||||||
|
([09 — Open questions](09-open-questions.md)).
|
||||||
|
|
||||||
> **Read the code**
|
> **Read the code**
|
||||||
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L40) — the 46-byte pose frame, position quantized on the wire only
|
> - [`simulator/virtual_drone/broadcast.py`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/simulator/virtual_drone/broadcast.py#L1-L45) — the 45-byte pose frame, position quantized on the wire only
|
||||||
|
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L60-L63) — `POSE_BYTES = 45` drives the volume estimate
|
||||||
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
|
> - [`prototype/src/sim.ts`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/prototype/src/sim.ts#L260-L277) — pose broadcasts at 5 Hz and opportunistic bulk sync, made visible
|
||||||
|
|
||||||
## 6. Security as nested layers, not one wall
|
## 6. Security as nested layers, not one wall
|
||||||
@@ -132,20 +136,21 @@ coming back to foundations DevOps and sysadmins already trust.
|
|||||||
|
|
||||||
## 8. Replication and offload
|
## 8. Replication and offload
|
||||||
|
|
||||||
Moving accumulated data uses [rsync](00-glossary.md#bulk-sync) over the same SSH
|
Moving accumulated data uses rsync-class tooling (rclone) over the same SSH
|
||||||
channel for in-flight peer [bulk sync](00-glossary.md#bulk-sync) and dock
|
channel. It transfers diffs of the Hive tree efficiently. This matters because a
|
||||||
[offload](00-glossary.md#offload). It transfers diffs of the Hive tree efficiently.
|
unit can be lost, and if it is, we want its data to already exist elsewhere.
|
||||||
This matters because a unit can be lost, and if it is, we want its data to
|
When peers have a stable link the partition tree is pulled between them, which
|
||||||
already exist elsewhere. When peers have a stable link the partition tree is
|
gives redundancy and a way to cross-check later. On the ground, when the fleet
|
||||||
pulled between them, which gives redundancy and a way to cross-check later. On
|
returns, the same path unifies every unit's data into a
|
||||||
the ground, when the fleet returns, the same path unifies every unit's data into
|
[local warehouse](06-environments.md). The warehouse is read-mostly for
|
||||||
a [local warehouse](06-environments.md) ([T3](00-glossary.md#t3--warehouse)).
|
analytics, so transaction contention is not a concern, which opens a
|
||||||
The warehouse is read-mostly for analytics, so transaction contention is not a
|
DuckDB / DuckLake approach with **MinIO underneath on the ground (T3)** — MinIO
|
||||||
concern — plain DuckDB over Parquet with MinIO underneath is the baseline;
|
is already in the stack, so the warehouse leans on it rather than inventing a
|
||||||
DuckLake remains under evaluation ([09 — Open questions](09-open-questions.md)
|
second object store — the
|
||||||
§13). The [same storage model on both ends](adr/ADR-0002-one-storage-format.md).
|
[same storage model on both ends](adr/ADR-0002-one-storage-format.md). One
|
||||||
One uniform structure is what lets an analyst trust a [single picture](00-glossary.md#source-of-truth) even when a
|
uniform structure is what lets an analyst trust a single picture even when a
|
||||||
unit's data has a gap.
|
unit's data has a gap. On-board MinIO stays an open exception, not the default
|
||||||
|
([00 — Executive](00-executive.md)).
|
||||||
|
|
||||||
> **Read the code**
|
> **Read the code**
|
||||||
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
|
> - [`infra/terraform/ground/main.tf`](https://git.produktor.io/eSlider/swarm-house/src/branch/main/infra/terraform/ground/main.tf#L124-L160) — the post-flight T1 to T3 offload CronJob
|
||||||
|
|||||||
@@ -32,10 +32,15 @@ wrapper, permissions, and output format a peer drone would use.
|
|||||||
|
|
||||||
## Consequences
|
## Consequences
|
||||||
|
|
||||||
- One access path is built, secured, and tested — "what you test is what
|
- One access path is designed for flight and bench — "what you test is what
|
||||||
flies".
|
flies".
|
||||||
- The SQL gate is safety-critical and is covered by unit tests
|
- The statement gate in the PoC (`simulator/explorer/server.py`, covered by
|
||||||
(`simulator/tests/test_sql_gate.py`).
|
`simulator/tests/test_sql_gate.py`) is a **first layer**: keyword allow/deny
|
||||||
- The explorer and the prototype's live mode are *just another read-only
|
over HTTP for the explorer. Production still needs the remaining layers in
|
||||||
consumer* of this same contract ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
|
[04 — Swarm sync](../04-swarm-sync.md) (forced-command key, read-only OS user,
|
||||||
|
engine hardening, resource caps) and a real SQL parser rather than keywords
|
||||||
|
alone.
|
||||||
|
- The explorer and the prototype's live mode are *stand-in consumers* of the
|
||||||
|
same read-only contract over HTTP today; the proposed flight path is
|
||||||
|
SQL-over-SSH ([`../04-swarm-sync.md`](../04-swarm-sync.md)).
|
||||||
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
|
- Design principles 7 and 8 in [`../../README.md`](../../README.md) restate this.
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
# Documentation audit — 2026-07-10
|
|
||||||
|
|
||||||
Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify T0–T4 vocabulary, align rsync/offload wording with code, improve readability.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
| Metric | Result |
|
|
||||||
| --- | --- |
|
|
||||||
| Markdown files checked | 33 |
|
|
||||||
| Broken relative links (after fixes) | 0 |
|
|
||||||
| Glossary `###` anchors added | 15 (T0–T4, layers, bulk sync, source of truth, …) |
|
|
||||||
| Open questions added | §12–§14 (tooling, DuckLake, `schemas/`) |
|
|
||||||
|
|
||||||
## Findings fixed
|
|
||||||
|
|
||||||
### Links
|
|
||||||
|
|
||||||
- `infra/terraform/ground/README.md` pointed at non-existent `03-storage-design.md` → `03-data-platform.md`.
|
|
||||||
- `11-cicd-delivery.md` bare cross-reference to doc 05 → proper markdown link.
|
|
||||||
|
|
||||||
### Factual drift (code vs docs)
|
|
||||||
|
|
||||||
| Item | Was | Now (matches `broadcast.py`, `sim.ts`) |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Pose frame | ~40 bytes, uint16 `drone_id` | 46 bytes, 8-byte ASCII `drone_id` |
|
|
||||||
|
|
||||||
### Terminology
|
|
||||||
|
|
||||||
- Extended [00 — Glossary](../00-glossary.md) with T0–T4, architecture layers, pipeline stages, source of truth.
|
|
||||||
- Disambiguated design journey §4: pipeline stages ≠ storage floors.
|
|
||||||
- Roadmap Stage 7: rsync bulk path, not “MinIO replication”.
|
|
||||||
|
|
||||||
### rsync / rclone / offload
|
|
||||||
|
|
||||||
| Path | Canonical tool |
|
|
||||||
| --- | --- |
|
|
||||||
| Peer bulk sync | rsync over SSH |
|
|
||||||
| Dock offload | rsync or `mc mirror` |
|
|
||||||
| S3 backends (optional) | rclone |
|
|
||||||
| Sim CronJob | `cp -ru` + `mc mirror` (documented shortcut) |
|
|
||||||
|
|
||||||
### Live demos (design journey)
|
|
||||||
|
|
||||||
Tracked in [PR #3](https://git.produktor.io/eSlider/swarm-house/pulls/3) — explorer + Grafana links in `12-design-journey.md` §7/§9. Prototype link already on `main` §2.
|
|
||||||
|
|
||||||
### README / LICENSE
|
|
||||||
|
|
||||||
- Restricted-use notice with link to [LICENSE](../../LICENSE).
|
|
||||||
- Component READMEs tightened (caveman style).
|
|
||||||
|
|
||||||
## Remaining (documented, not bugs)
|
|
||||||
|
|
||||||
| Item | Where |
|
|
||||||
| --- | --- |
|
|
||||||
| `schemas/` directory absent | OQ §14, roadmap stage 2 |
|
|
||||||
| DuckLake undecided | OQ §13 |
|
|
||||||
| rsync bulk sync not in sim code | Design only |
|
|
||||||
| Glossary table terms link to section headers, not row anchors | Optional polish |
|
|
||||||
| T4 after “Source of truth” in glossary order | Cosmetic |
|
|
||||||
| GitOps “source of truth” vs data SSOT | Different meanings in `11-cicd-delivery` |
|
|
||||||
|
|
||||||
## Suggested CI checks (for issue #1)
|
|
||||||
|
|
||||||
Implemented in [`bin/check_docs.py`](../../bin/check_docs.py) — see [ci-rules.md](ci-rules.md).
|
|
||||||
|
|
||||||
## Files touched
|
|
||||||
|
|
||||||
`docs/00`–`12`, `docs/adr/*`, nine READMEs, root `README.md`.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Documentation improvement records
|
|
||||||
|
|
||||||
Point-in-time audits and follow-up work for links, terminology, readability, and factual alignment with code.
|
|
||||||
|
|
||||||
| Report | Date | Scope |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0–T4; live demo links |
|
|
||||||
|
|
||||||
## Collaboration style
|
|
||||||
|
|
||||||
Issues, PRs, and reviews default to **caveman-full** ([`CONTRIBUTING.md`](../../CONTRIBUTING.md#caveman-style-default),
|
|
||||||
[`AGENTS.md`](../../AGENTS.md#caveman-style-default-for-collaboration)). Gitea templates live under
|
|
||||||
[`.github/ISSUE_TEMPLATE/`](../../.github/ISSUE_TEMPLATE/) and
|
|
||||||
[`.github/pull_request_template.md`](../../.github/pull_request_template.md).
|
|
||||||
|
|
||||||
## CI automation
|
|
||||||
|
|
||||||
Enforced in Gitea Actions (`docs-verify` job). Rules live in [`bin/docs_rules.json`](../../bin/docs_rules.json); see [ci-rules.md](ci-rules.md).
|
|
||||||
|
|
||||||
Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1).
|
|
||||||
|
|
||||||
## How to add a new report
|
|
||||||
|
|
||||||
1. Copy the structure from the latest audit file.
|
|
||||||
2. Name it `YYYY-MM-DD-short-topic.md`.
|
|
||||||
3. Add a row to the table above.
|
|
||||||
4. Cross-link from [09 — Open questions](../09-open-questions.md) if new ambiguities appear.
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
# CI documentation rules
|
|
||||||
|
|
||||||
Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate).
|
|
||||||
|
|
||||||
Implementation: [`bin/check_docs.py`](../../bin/check_docs.py) reads [`bin/docs_rules.json`](../../bin/docs_rules.json).
|
|
||||||
|
|
||||||
Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**.
|
|
||||||
|
|
||||||
Tracked from [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1) and the [2026-07-10 audit](2026-07-10-audit.md).
|
|
||||||
|
|
||||||
## Gates
|
|
||||||
|
|
||||||
| Rule | Severity | What it checks |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Relative links | error | Every `[text](path)` and `[text](path#anchor)` to a local `.md` file resolves to an existing file |
|
|
||||||
| Fragment anchors | error | `#anchor` fragments match a `##` or `###` heading in the target file (GitHub-style slug) |
|
|
||||||
| Banned targets | error | Known-bad paths (e.g. deleted `03-storage-design.md`) are not linked |
|
|
||||||
| Glossary fragments | error | All `00-glossary.md#…` links use anchors that exist in the glossary |
|
|
||||||
| Section vs term links | warning | Links to glossary *section* headers (`#swarm--robotics`, …) are listed; prefer `###` term anchors where they exist |
|
|
||||||
| Bare cross-refs | error | Prose cross-references to numbered docs without a proper markdown link |
|
|
||||||
| ADR immutability | error | No modifications to existing `docs/adr/ADR-*.md` (compare `origin/main`); add new ADR to supersede |
|
|
||||||
|
|
||||||
Warnings are printed in the job log but do not fail CI. Errors exit non-zero and block merge.
|
|
||||||
|
|
||||||
## PR workflow
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart LR
|
|
||||||
PR[Pull request] --> DV[docs-verify]
|
|
||||||
PR --> V[verify simulator]
|
|
||||||
V --> S[Trivy scan]
|
|
||||||
DV --> OK{All required green?}
|
|
||||||
V --> OK
|
|
||||||
S --> OK
|
|
||||||
OK -->|yes| Merge[Mergeable]
|
|
||||||
```
|
|
||||||
|
|
||||||
| Event | `docs-verify` | `verify` + `scan` |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| PR → `main` | always | always |
|
|
||||||
| Push → `main` (code paths) | always | runs |
|
|
||||||
| Push → `main` (docs/README/AGENTS only) | always | skipped (`paths-ignore`) |
|
|
||||||
|
|
||||||
Docs-only changes therefore still get link and glossary validation on `main`; they do not trigger a semantic release.
|
|
||||||
|
|
||||||
## Local run
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python bin/check_docs.py
|
|
||||||
python bin/check_adrs.py --base origin/main
|
|
||||||
python bin/check_docs.py --warnings-as-errors # treat section-link warnings as failures
|
|
||||||
```
|
|
||||||
|
|
||||||
## Changing rules
|
|
||||||
|
|
||||||
1. Edit `bin/docs_rules.json` (paths, banned targets, preferred anchors).
|
|
||||||
2. Adjust `bin/check_docs.py` only when adding a new rule *type*.
|
|
||||||
3. Add a row to the table above and mention it in the audit/improvement record if the change is significant.
|
|
||||||
+18
-9
@@ -1,11 +1,14 @@
|
|||||||
# Ansible — host preparation
|
# Ansible — host preparation
|
||||||
|
|
||||||
Ansible = host prep before workloads run. Terraform ([`../terraform/`](../terraform/)) owns workloads. Boundary: [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries).
|
Ansible owns everything that happens **on a host before workloads run**.
|
||||||
|
Terraform ([`../terraform/`](../terraform/)) owns the workloads themselves.
|
||||||
|
The boundary is deliberate and documented in
|
||||||
|
[06 — Environments](../../docs/06-environments.md#iac-boundaries).
|
||||||
|
|
||||||
| Playbook | Target | What it does |
|
| Playbook | Target | What it does |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `drone-provision.yml` | Drone (bench, before mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from fleet manifest |
|
| `drone-provision.yml` | Drone (on the bench, before a mission) | Identity keys, peer trust with forced commands, WireGuard, Compose bundle from the fleet release manifest |
|
||||||
| `sim-cluster.yml` | Local workstation / CI host | k3d cluster miniaturing ground segment; shared data volume; simulator image import |
|
| `sim-cluster.yml` | Local workstation / CI host | k3d cluster that miniatures the ground segment; shared data volume; simulator image import |
|
||||||
|
|
||||||
## Provision a drone
|
## Provision a drone
|
||||||
|
|
||||||
@@ -13,12 +16,16 @@ Ansible = host prep before workloads run. Terraform ([`../terraform/`](../terraf
|
|||||||
ansible-playbook -i inventory.example.yml drone-provision.yml
|
ansible-playbook -i inventory.example.yml drone-provision.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
Idempotent. Bench network only. Never provision in flight ([05 — Network & security](../../docs/05-network-security.md)):
|
The playbook is idempotent and runs only over the bench network — drones
|
||||||
|
are never provisioned in flight (see [05 — Network & security](../../docs/05-network-security.md)):
|
||||||
|
|
||||||
1. **Identity**: generate drone ed25519 keypair if absent.
|
1. **Identity**: generate the drone's ed25519 keypair if absent.
|
||||||
2. **Peer trust**: fleet public keys in `authorized_keys`, each pinned to read-only SQL forced command.
|
2. **Peer trust**: install every fleet member's public key into
|
||||||
3. **WireGuard**: render `wg0.conf` with drone address and peers.
|
`authorized_keys`, each pinned to the read-only SQL forced command —
|
||||||
4. **Data plane**: Compose bundle from fleet manifest, systemd unit enabled.
|
the only thing a peer can execute.
|
||||||
|
3. **WireGuard**: render `wg0.conf` with the drone's address and peers.
|
||||||
|
4. **Data plane**: lay down the Compose bundle referenced by the fleet
|
||||||
|
release manifest and enable it as a systemd unit.
|
||||||
|
|
||||||
## Create the simulation cluster
|
## Create the simulation cluster
|
||||||
|
|
||||||
@@ -27,4 +34,6 @@ ansible-playbook sim-cluster.yml # creates k3d cluster 'swarm-sim'
|
|||||||
cd ../terraform/sim-env && terraform init && terraform apply
|
cd ../terraform/sim-env && terraform init && terraform apply
|
||||||
```
|
```
|
||||||
|
|
||||||
Shared host directory mounted as `/data` — drone pods, exporter, explorer see one Parquet lake (same contract as on-board NVMe).
|
The cluster mounts a shared host directory as `/data` on the (single)
|
||||||
|
node, so drone pods, the exporter, and the explorer see one Parquet lake —
|
||||||
|
the same contract as the on-board NVMe layout.
|
||||||
|
|||||||
+19
-9
@@ -1,16 +1,20 @@
|
|||||||
# GitOps — ground segment (Flux)
|
# GitOps — ground segment (Flux)
|
||||||
|
|
||||||
Flux = long-lived ground config from git. Not on drones. Not fleet manifest. Different lifecycles ([11 — CI/CD](../../docs/11-cicd-delivery.md)).
|
Flux reconciles **long-lived ground configuration** from this repository. It does
|
||||||
|
not run on drones and does not replace the fleet release manifest — those are
|
||||||
|
different lifecycles by design ([11 — CI/CD](../../docs/11-cicd-delivery.md)).
|
||||||
|
|
||||||
## Boundary: Terraform vs Flux vs fleet manifest
|
## Boundary: Terraform vs Flux vs fleet manifest
|
||||||
|
|
||||||
| Layer | Tool | Owns |
|
| Layer | Tool | Owns |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts |
|
| Cluster + first boot | **Terraform** ([`../terraform/`](../terraform/)) | Namespaces, PVCs, Deployments, CronJobs, NodePorts — the shape of the simulation |
|
||||||
| Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs |
|
| Ongoing ground config | **Flux** (this directory) | Policy labels, dashboard bundles, offload knobs — things that change without reprovisioning PVCs |
|
||||||
| Drones | **[Fleet release manifest](../../docs/11-cicd-delivery.md)** | Compose digests, model weights, peer registry — dock-only |
|
| Drones | **Fleet release manifest** | Compose bundle digests, model weights, peer registry — atomic, dock-only delivery |
|
||||||
|
|
||||||
Drones outside GitOps. No reconciler in flight. Dock applies pinned manifest once. No mid-mission drift — no update endpoint in radio profile.
|
Drones are **outside GitOps**: there is no reconciler in flight. A dock applies
|
||||||
|
the pinned manifest once; mid-mission drift is impossible because the update
|
||||||
|
endpoint does not exist in the radio profile.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
@@ -22,7 +26,9 @@ infra/gitops/
|
|||||||
|
|
||||||
## Bootstrap on the k3d simulation cluster
|
## Bootstrap on the k3d simulation cluster
|
||||||
|
|
||||||
`infra/ansible/sim-cluster.yml` installs Flux and applies CRs. After first `git push`, Flux reconciles `infra/gitops/ground/`.
|
`infra/ansible/sim-cluster.yml` installs Flux controllers and applies the CRs
|
||||||
|
below. After the first `git push`, Flux polls `origin` and reconciles
|
||||||
|
`infra/gitops/ground/` into the `ground` namespace.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Manual bootstrap (if you skipped Ansible):
|
# Manual bootstrap (if you skipped Ansible):
|
||||||
@@ -30,13 +36,17 @@ flux install --namespace=flux-system
|
|||||||
kubectl apply -k infra/gitops/flux
|
kubectl apply -k infra/gitops/flux
|
||||||
```
|
```
|
||||||
|
|
||||||
Reconcile immediately:
|
To reconcile immediately without waiting for the poll interval:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
flux reconcile source git swarm-house -n flux-system
|
flux reconcile source git swarm-house -n flux-system
|
||||||
flux reconcile kustomization ground-segment -n flux-system
|
flux reconcile kustomization ground-segment -n flux-system
|
||||||
```
|
```
|
||||||
|
|
||||||
## What Flux manages
|
## What Flux manages here (example)
|
||||||
|
|
||||||
[`ground/`](ground/) overlay: GitOps-managed ConfigMap with fleet policy metadata. Production extends to Grafana dashboards, Prometheus rules, offload-schedule ConfigMaps — git-versioned, no `terraform apply` for label tweaks.
|
The [`ground/`](ground/) overlay currently carries a **GitOps-managed ConfigMap**
|
||||||
|
that tags the warehouse segment with fleet policy metadata. In production this
|
||||||
|
pattern extends to Grafana dashboard bundles, Prometheus rule files, and
|
||||||
|
offload-schedule ConfigMaps — all versioned in git, all auditable, none of
|
||||||
|
them requiring a `terraform apply` to tweak a label.
|
||||||
|
|||||||
@@ -1,18 +1,7 @@
|
|||||||
# Terraform — ground workloads
|
|
||||||
|
|
||||||
Declared workloads on the k3d simulation cluster. See [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries).
|
|
||||||
|
|
||||||
| Module | Tier | Purpose |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| [`sim-env/`](sim-env/) | [T4](../../docs/00-glossary.md#t4--dev) dev/sim | Virtual fleet + observability on k3d |
|
|
||||||
| [`ground/`](ground/) | [T3](../../docs/00-glossary.md#t3--warehouse) | Warehouse + [T1→T3](../../docs/03-data-platform.md#storage-floors-tiers) offload |
|
|
||||||
|
|
||||||
Two modules, two states. Fleet disposable; warehouse survives `destroy` of sim-env.
|
|
||||||
|
|
||||||
## Recovering from partial state
|
## Recovering from partial state
|
||||||
|
|
||||||
If resources were created outside Terraform (or state was lost), import them
|
If resources were created outside Terraform (or state was lost), import them
|
||||||
before `terraform apply`. Run from the target module directory (`sim-env/` or `ground/`):
|
before `terraform apply`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# sim-env (namespace: swarm)
|
# sim-env (namespace: swarm)
|
||||||
|
|||||||
@@ -1,14 +1,19 @@
|
|||||||
# Terraform — ground warehouse ([T3](../../../docs/00-glossary.md#t3--warehouse))
|
# Terraform — ground warehouse (T3)
|
||||||
|
|
||||||
Ground warehouse. Receives sealed partitions post-flight ([storage floors](../../../docs/03-data-platform.md#storage-floors-tiers)). Long-lived infra → Terraform.
|
The stationary half of the system: the historical warehouse that receives
|
||||||
|
sealed partitions after every flight ([03 — Storage tiers](../../../docs/03-storage-design.md)).
|
||||||
|
It is the longest-lived, most conventional infrastructure in the design —
|
||||||
|
and therefore the most natural Terraform territory.
|
||||||
|
|
||||||
Runs in same k3d cluster as [`../sim-env`](../sim-env/) but separate namespace and state: fleet disposable, warehouse not — never share `terraform destroy`.
|
Runs in the same k3d cluster as [`../sim-env`](../sim-env/) but in its own
|
||||||
|
namespace with its own state: the fleet is disposable, the warehouse is not,
|
||||||
|
and the two lifecycles must never share a `terraform destroy`.
|
||||||
|
|
||||||
| Resource | Purpose |
|
| Resource | Purpose |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Deployment `minio` + PVC + NodePort 30901 | Object-store facade for downstream consumers |
|
| Deployment `minio` + PVC + NodePort 30901 | Object-store facade of the warehouse for downstream consumers (training pipelines, replay) |
|
||||||
| CronJob `offload` | [T1](../../../docs/00-glossary.md#t1--warm) → [T3](../../../docs/00-glossary.md#t3--warehouse) offload: Hive partitions verbatim, then `mc mirror` into MinIO |
|
| CronJob `offload` | Post-flight offload: copies Hive partitions verbatim from the fleet lake (T1) into the warehouse (T3), then mirrors into MinIO |
|
||||||
| Deployment `warehouse-explorer` + NodePort 30089 | Read-only SQL across **all** offloaded flights |
|
| Deployment `warehouse-explorer` + NodePort 30089 | Second explorer instance over the warehouse path — historical read-only SQL across **all** flights |
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
@@ -18,9 +23,16 @@ terraform apply
|
|||||||
terraform output
|
terraform output
|
||||||
```
|
```
|
||||||
|
|
||||||
After offload runs, warehouse explorer shows same partition tree as live lake, accumulated across flights. [T1](../../../docs/00-glossary.md#t1--warm) layout **is** [T3](../../../docs/00-glossary.md#t3--warehouse) layout. No transform. No schema drift. Same DuckDB queries both ends.
|
After a couple of offload runs, the warehouse explorer shows the same
|
||||||
|
partition tree as the live lake but accumulated across flights — the T1
|
||||||
|
layout **is** the T3 layout, which is the whole point: no transform step,
|
||||||
|
no schema drift, DuckDB queries work identically on both ends.
|
||||||
|
|
||||||
## Simulation vs production
|
## Simulation vs production
|
||||||
|
|
||||||
- Sim CronJob: `cp -ru` + `mc mirror` on a schedule — shortcut for "lands, offloads, prunes". Production: event-driven per dock, rsync or `mc mirror`, checksum audit before prune ([03 — Data platform](../../../docs/03-data-platform.md)).
|
- The CronJob compresses "drone lands, docks, offloads, prunes" into a
|
||||||
- MinIO credentials here are throwaway defaults; production credentials sealed per site, never in state or VCS.
|
periodic rsync-style copy. Production offload is event-driven per docking
|
||||||
|
and verifies checksums from the flight's partition manifest before pruning
|
||||||
|
the on-board lake.
|
||||||
|
- MinIO credentials here are throwaway defaults; production credentials are
|
||||||
|
sealed per site and never in state or VCS.
|
||||||
|
|||||||
@@ -1,14 +1,17 @@
|
|||||||
# Terraform — simulation environment ([T4](../../../docs/00-glossary.md#t4--dev))
|
# Terraform — simulation environment (T4)
|
||||||
|
|
||||||
Sim fleet + observability on k3d. Cluster from [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml). Executable [T4](../../../docs/00-glossary.md#t4--dev) miniature ([06 — Environments](../../../docs/06-environments.md)).
|
Declares the simulated fleet and its observability stack on the k3d cluster
|
||||||
|
created by [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml).
|
||||||
|
This module is the executable miniature of the dev/sim environment from
|
||||||
|
[06 — Environments](../../../docs/06-environments.md).
|
||||||
|
|
||||||
| Resource | Purpose |
|
| Resource | Purpose |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| StatefulSet `drone` × `drone_count` | Virtual drones; pod names → `DRONE_ID`; shared `/data` lake |
|
| StatefulSet `drone` × `drone_count` | Virtual drones; stable pod names become `DRONE_ID`s; all write Hive-partitioned Parquet into the shared `/data` lake |
|
||||||
| Deployment `exporter` | Prometheus exporter reading lake with DuckDB |
|
| Deployment `exporter` | Prometheus exporter reading the lake with DuckDB |
|
||||||
| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL (prototype live mode) |
|
| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL console (also feeds the prototype's live mode) |
|
||||||
| Deployment `prometheus` + NodePort 30990 | Scrapes exporter |
|
| Deployment `prometheus` + NodePort 30990 | Scrapes the exporter |
|
||||||
| Deployment `grafana` + NodePort 30300 | Dashboard JSON from ConfigMap |
|
| Deployment `grafana` + NodePort 30300 | Same dashboard JSON as the Compose profile, provisioned from a ConfigMap |
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
@@ -19,7 +22,7 @@ terraform apply
|
|||||||
terraform output # URLs
|
terraform output # URLs
|
||||||
```
|
```
|
||||||
|
|
||||||
Scale without YAML edits:
|
Scale the fleet without touching YAML:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
terraform apply -var drone_count=9 -var speedup=4
|
terraform apply -var drone_count=9 -var speedup=4
|
||||||
@@ -27,6 +30,11 @@ terraform apply -var drone_count=9 -var speedup=4
|
|||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- `image_pull_policy = "Never"` — image imported by Ansible (`k3d image import`); no registry pull. Air-gap mirror.
|
- `image_pull_policy = "Never"` — the simulator image is imported by the
|
||||||
- Pod name → identity (StatefulSet ordinal). `kubectl delete pod` = power cycle: new process, same identity, new flight ID.
|
Ansible playbook (`k3d image import`); the cluster never pulls from a
|
||||||
- Local state only — cluster disposable.
|
registry, mirroring the air-gap doctrine.
|
||||||
|
- Pods discover their identity from the pod name (StatefulSet ordinal), so
|
||||||
|
a flight survives `kubectl delete pod` the same way a drone survives a
|
||||||
|
power cycle: new process, same identity, new flight ID.
|
||||||
|
- State is local (`terraform.tfstate` in this directory) — the simulation
|
||||||
|
cluster is disposable; nothing here is shared infrastructure.
|
||||||
|
|||||||
+6
-13
@@ -1,8 +1,8 @@
|
|||||||
# Swarm visualization prototype
|
# Swarm visualization prototype
|
||||||
|
|
||||||
2D top-down view of swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol with obstacles, 5 Hz pose broadcasts (blue flashes), opportunistic bulk sync (green links with rate and byte counters).
|
A 2D top-down view of the swarm data plane from [04 — Swarm sync](../docs/04-swarm-sync.md): drones patrol an area with static and mobile obstacles, exchange 5 Hz pose broadcasts (blue link flashes), and run opportunistic bulk sync of sealed partitions (green links with live rate and cumulative up/down counters).
|
||||||
|
|
||||||
No backend. Client-side sim. Same behavioral model as Python simulator (Parquet + UDP).
|
No backend — a pure client-side simulation of the same behavioral model the Python simulator implements over real Parquet and UDP.
|
||||||
|
|
||||||
## Run
|
## Run
|
||||||
|
|
||||||
@@ -11,16 +11,9 @@ npm install
|
|||||||
npm run dev # http://localhost:5173
|
npm run dev # http://localhost:5173
|
||||||
```
|
```
|
||||||
|
|
||||||
## Live mode
|
|
||||||
|
|
||||||
With k3d fleet + explorer running (`ansible-playbook` + `terraform apply` — see [sim-env README](../infra/terraform/sim-env/README.md)):
|
|
||||||
|
|
||||||
- Explorer at `http://localhost:30088`
|
|
||||||
- Header → **go live** — polls read-only SQL API for real drone positions
|
|
||||||
|
|
||||||
## What to look at
|
## What to look at
|
||||||
|
|
||||||
- Links form and break by radio range; broadcasts flash active links
|
- **Links appear and disappear** as drones move in and out of radio range; persistent links render semi-transparent, broadcast deliveries flash them briefly.
|
||||||
- Busy links turn green — rate plus `↑` / `↓` totals
|
- **Busy links turn green** and show current rate plus total transferred (`↑` / `↓`) — the bandwidth budget from the sync design made visible.
|
||||||
- Per-drone label: id, Wi-Fi channel, battery
|
- **Per-drone label**: id, current Wi-Fi channel (hopping), battery.
|
||||||
- Controls: drone count, speed, pause
|
- Controls: drone count, simulation speed, pause.
|
||||||
|
|||||||
@@ -268,7 +268,7 @@ export default function App(): JSX.Element {
|
|||||||
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
|
{busiest.length === 0 && <div style={styles.dim}>no links in range</div>}
|
||||||
</Section>
|
</Section>
|
||||||
<Section title="Legend">
|
<Section title="Legend">
|
||||||
<div style={styles.dim}>blue line — pose broadcast (5 Hz, ~46 B)</div>
|
<div style={styles.dim}>blue line — pose broadcast (5 Hz, 45 B)</div>
|
||||||
<div style={styles.dim}>green line — bulk sync (sealed partitions)</div>
|
<div style={styles.dim}>green line — bulk sync (sealed partitions)</div>
|
||||||
<div style={styles.dim}>flash — broadcast event delivered</div>
|
<div style={styles.dim}>flash — broadcast event delivered</div>
|
||||||
<div style={styles.dim}>red circle — transit object crossing the area</div>
|
<div style={styles.dim}>red circle — transit object crossing the area</div>
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ export function areaWidth(): number {
|
|||||||
export const LINK_RANGE = 320;
|
export const LINK_RANGE = 320;
|
||||||
const CRUISE = 28;
|
const CRUISE = 28;
|
||||||
const SEPARATION = 55;
|
const SEPARATION = 55;
|
||||||
const POSE_BYTES = 46;
|
const POSE_BYTES = 45;
|
||||||
const POSE_HZ = 5;
|
const POSE_HZ = 5;
|
||||||
// Steady per-link broadcast throughput (both directions), bytes/s
|
// Steady per-link broadcast throughput (both directions), bytes/s
|
||||||
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
|
export const POSE_RATE_BYTES = POSE_BYTES * POSE_HZ;
|
||||||
|
|||||||
+15
-14
@@ -1,8 +1,8 @@
|
|||||||
# Virtual drone fleet
|
# Virtual drone fleet
|
||||||
|
|
||||||
Data generator. Same on-board pipeline as [03 — Data platform](../docs/03-data-platform.md): seeded kinematics, sensor streams, detection events, hot `current/` → sealed Parquet, compact UDP state broadcast.
|
A data generator that exercises the **exact on-board pipeline** described in [03 — Data platform](../docs/03-data-platform.md): seeded kinematics along a patrol route, sensor streams at realistic rates, detection events, the hot `current/` → sealed Parquet write path, and the compact UDP state broadcast between drones.
|
||||||
|
|
||||||
One process = one drone. Scale swarm with Compose flag.
|
One process = one drone. Scaling the swarm is a Compose flag.
|
||||||
|
|
||||||
## Run a swarm
|
## Run a swarm
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@ FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --build --scale drone=5
|
|||||||
DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10
|
DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10
|
||||||
```
|
```
|
||||||
|
|
||||||
Output lands in [`../var/t1/`](../var/t1/) ([T1](../docs/00-glossary.md#t1--warm) live lake — same path k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed:
|
Output lands in [`../var/t1/`](../var/t1/) (the shared T1 lake — same path the k3d fleet and warehouse offload use). Override with `SWARM_T1_DIR` if needed:
|
||||||
|
|
||||||
```
|
```
|
||||||
var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet
|
var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet
|
||||||
@@ -22,7 +22,7 @@ var/t1/dataset=detections/flight=…/drone=…/…
|
|||||||
var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts
|
var/t1/dataset=state/flight=…/drone=…/… ← sent + received broadcasts
|
||||||
```
|
```
|
||||||
|
|
||||||
Historical flights after offload: [`../var/t3/`](../var/t3/) ([T3](../docs/00-glossary.md#t3--warehouse) warehouse).
|
Historical flights after offload live under [`../var/t3/`](../var/t3/) (T3 warehouse).
|
||||||
|
|
||||||
## Run a single drone without Docker
|
## Run a single drone without Docker
|
||||||
|
|
||||||
@@ -52,32 +52,33 @@ print(con.sql("""
|
|||||||
EOF
|
EOF
|
||||||
```
|
```
|
||||||
|
|
||||||
## Monitoring
|
## Monitoring: Grafana over the generated data
|
||||||
|
|
||||||
`monitoring` profile: DuckDB exporter → Prometheus → Grafana ([07 — Observability](../docs/07-observability.md)):
|
The `monitoring` profile spins up a small metrics chain — a DuckDB-based exporter that scans the generated Parquet every few seconds, Prometheus, and a pre-provisioned Grafana dashboard:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose --profile monitoring up -d
|
docker compose --profile monitoring up -d # exporter + prometheus + grafana
|
||||||
|
# generate some flights in parallel or beforehand:
|
||||||
FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5
|
FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5
|
||||||
```
|
```
|
||||||
|
|
||||||
- Grafana: `http://localhost:3000` → **Swarm Fleet — generated data**
|
- Grafana: `http://localhost:3000` (anonymous admin — demo only) → dashboard **Swarm Fleet — generated data**
|
||||||
- Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics`
|
- Prometheus: `http://localhost:9090` · exporter: `http://localhost:9105/metrics`
|
||||||
|
|
||||||
Fleet stats derived from Parquet via SQL — no on-drone agent.
|
Panels: telemetry rows by drone/sensor, detections by class, pose frames sent/received, battery per drone, RSSI and estimated distance per link, Parquet bytes/files on disk. The exporter is deliberately a demonstration of the observability doctrine from [07 — Observability](../docs/07-observability.md): fleet statistics are *derived from the data platform itself* — no agent on the drone, just SQL over the same Parquet everyone else reads.
|
||||||
|
|
||||||
## Data-plane explorer
|
## Data-plane explorer
|
||||||
|
|
||||||
`monitoring` profile also starts explorer at `http://localhost:8088` (k3d: `:30088`):
|
The `monitoring` profile also starts a **data-plane explorer** at `http://localhost:8088` — a single-page web view over the lake itself, complementing Grafana (which shows aggregates, not the structure):
|
||||||
|
|
||||||
- Partition tree live: `dataset → flight → drone → sensor → hour → file`
|
- **Partition tree**, live: `dataset → flight → drone → sensor → year/…/hour → file`, with file counts and bytes rolled up at every level. You can watch `current/` files appear and get sealed while a swarm is flying.
|
||||||
- Read-only SQL console — same [statement gate](../docs/04-swarm-sync.md) as peer query channel: `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only
|
- **Read-only SQL console** with the datasets pre-registered as views (`telemetry`, `detections`, `state`) and one-click sample queries (fleet overview, battery timeline, detections by class, peer link quality, schema).
|
||||||
|
|
||||||
Point: bench exploration uses same read-only SQL contract as flight peers.
|
The console enforces the **same statement gate as the peer query channel** from [04 — Swarm sync](../docs/04-swarm-sync.md): a single statement, `SELECT`/`WITH`/`DESCRIBE`/`SUMMARIZE`/`SHOW` only, write/config/extension keywords rejected. That is the point: exploring the data plane on the bench uses the same read-only SQL contract a peer drone uses in flight — the explorer is the "no debug API" principle made visible.
|
||||||
|
|
||||||
## Reproducibility
|
## Reproducibility
|
||||||
|
|
||||||
Deterministic per `(SEED, DRONE_ID)`. Bug report = seed + config.
|
Every run is deterministic per `(SEED, DRONE_ID)`: same route jitter, same sensor noise, same detection sequence. A bug report is a seed and a config, not a description.
|
||||||
|
|
||||||
## Knobs
|
## Knobs
|
||||||
|
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ from virtual_drone.flight import Pose
|
|||||||
|
|
||||||
|
|
||||||
def test_frame_size_matches_wire_spec() -> None:
|
def test_frame_size_matches_wire_spec() -> None:
|
||||||
# Documented as 46 B in docs/04; struct packs to FRAME.size (45 B on this layout).
|
# Keep docs/04, the journey, and prototype/src/sim.ts POSE_BYTES in lockstep.
|
||||||
assert FRAME.size == 45
|
assert FRAME.size == 45
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,70 +0,0 @@
|
|||||||
"""Tests for bin/check_adrs.py — ADR immutability gate."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
SCRIPTS = Path(__file__).resolve().parents[2] / "bin"
|
|
||||||
sys.path.insert(0, str(SCRIPTS))
|
|
||||||
|
|
||||||
from check_adrs import find_modified_adrs, run_check # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def _git(cwd: Path, *args: str) -> None:
|
|
||||||
subprocess.check_call(["git", *args], cwd=cwd, stdout=subprocess.DEVNULL)
|
|
||||||
|
|
||||||
|
|
||||||
def test_no_modified_adrs_on_current_branch() -> None:
|
|
||||||
assert run_check(base="origin/main") == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_detects_adr_edit_in_sandbox(tmp_path: Path) -> None:
|
|
||||||
_git(tmp_path, "init")
|
|
||||||
_git(tmp_path, "config", "user.email", "ci@test")
|
|
||||||
_git(tmp_path, "config", "user.name", "ci")
|
|
||||||
adr = tmp_path / "docs/adr"
|
|
||||||
adr.mkdir(parents=True)
|
|
||||||
(adr / "ADR-0001-test.md").write_text("v1\n")
|
|
||||||
_git(tmp_path, "add", ".")
|
|
||||||
_git(tmp_path, "commit", "-m", "add adr")
|
|
||||||
_git(tmp_path, "branch", "-M", "main")
|
|
||||||
(adr / "ADR-0001-test.md").write_text("v2\n")
|
|
||||||
_git(tmp_path, "checkout", "-b", "feature")
|
|
||||||
_git(tmp_path, "add", ".")
|
|
||||||
_git(tmp_path, "commit", "-m", "edit adr")
|
|
||||||
|
|
||||||
monkeypatch = pytest.MonkeyPatch()
|
|
||||||
monkeypatch.setattr("check_adrs.ROOT", tmp_path)
|
|
||||||
try:
|
|
||||||
assert find_modified_adrs("main") == ["docs/adr/ADR-0001-test.md"]
|
|
||||||
assert run_check(base="main") == 1
|
|
||||||
finally:
|
|
||||||
monkeypatch.undo()
|
|
||||||
|
|
||||||
|
|
||||||
def test_new_adr_allowed(tmp_path: Path) -> None:
|
|
||||||
_git(tmp_path, "init")
|
|
||||||
_git(tmp_path, "config", "user.email", "ci@test")
|
|
||||||
_git(tmp_path, "config", "user.name", "ci")
|
|
||||||
adr = tmp_path / "docs/adr"
|
|
||||||
adr.mkdir(parents=True)
|
|
||||||
(adr / "ADR-0001-test.md").write_text("v1\n")
|
|
||||||
_git(tmp_path, "add", ".")
|
|
||||||
_git(tmp_path, "commit", "-m", "add adr")
|
|
||||||
_git(tmp_path, "branch", "-M", "main")
|
|
||||||
(adr / "ADR-0002-new.md").write_text("new\n")
|
|
||||||
_git(tmp_path, "checkout", "-b", "feature")
|
|
||||||
_git(tmp_path, "add", ".")
|
|
||||||
_git(tmp_path, "commit", "-m", "add adr 2")
|
|
||||||
|
|
||||||
monkeypatch = pytest.MonkeyPatch()
|
|
||||||
monkeypatch.setattr("check_adrs.ROOT", tmp_path)
|
|
||||||
try:
|
|
||||||
assert find_modified_adrs("main") == []
|
|
||||||
assert run_check(base="main") == 0
|
|
||||||
finally:
|
|
||||||
monkeypatch.undo()
|
|
||||||
@@ -1,77 +0,0 @@
|
|||||||
"""Tests for bin/check_docs.py — documentation CI rules."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
SCRIPTS = Path(__file__).resolve().parents[2] / "bin"
|
|
||||||
sys.path.insert(0, str(SCRIPTS))
|
|
||||||
|
|
||||||
from check_docs import ( # noqa: E402
|
|
||||||
collect_anchors,
|
|
||||||
run_checks,
|
|
||||||
slug_heading,
|
|
||||||
split_link_target,
|
|
||||||
)
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[2]
|
|
||||||
GLOSSARY = ROOT / "docs/00-glossary.md"
|
|
||||||
|
|
||||||
|
|
||||||
def test_slug_heading_matches_glossary_anchors() -> None:
|
|
||||||
assert slug_heading("T0 — hot") == "t0--hot"
|
|
||||||
assert slug_heading("T3 — warehouse") == "t3--warehouse"
|
|
||||||
assert slug_heading("Source of truth") == "source-of-truth"
|
|
||||||
assert slug_heading("Data & storage") == "data--storage"
|
|
||||||
|
|
||||||
|
|
||||||
def test_glossary_term_anchors_exist() -> None:
|
|
||||||
anchors = collect_anchors(GLOSSARY)
|
|
||||||
for term in (
|
|
||||||
"t0--hot",
|
|
||||||
"t1--warm",
|
|
||||||
"t2--shared",
|
|
||||||
"t3--warehouse",
|
|
||||||
"t4--dev",
|
|
||||||
"source-of-truth",
|
|
||||||
"bulk-sync",
|
|
||||||
"offload",
|
|
||||||
):
|
|
||||||
assert term in anchors, f"missing glossary anchor #{term}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_split_link_target() -> None:
|
|
||||||
assert split_link_target("foo.md#bar") == ("foo.md", "bar")
|
|
||||||
assert split_link_target("foo.md") == ("foo.md", None)
|
|
||||||
|
|
||||||
|
|
||||||
def test_repo_docs_pass() -> None:
|
|
||||||
assert run_checks() == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
||||||
docs = tmp_path / "docs"
|
|
||||||
docs.mkdir()
|
|
||||||
(docs / "00-glossary.md").write_text("# Glossary\n\n## Data & storage\n")
|
|
||||||
(docs / "bad.md").write_text("See [missing](missing.md).\n")
|
|
||||||
rules = tmp_path / "bin"
|
|
||||||
rules.mkdir()
|
|
||||||
(rules / "docs_rules.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"markdown_paths": ["docs/**/*.md"],
|
|
||||||
"glossary": {"file": "docs/00-glossary.md"},
|
|
||||||
"banned_link_targets": [],
|
|
||||||
"glossary_section_anchors": [],
|
|
||||||
"preferred_term_anchors": [],
|
|
||||||
"readability": {"forbid_bare_see_refs": False},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
)
|
|
||||||
monkeypatch.setattr("check_docs.ROOT", tmp_path)
|
|
||||||
monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.json")
|
|
||||||
assert run_checks() == 1
|
|
||||||
@@ -1,12 +1,13 @@
|
|||||||
"""State broadcast over UDP: the compact pose frame from the sync design.
|
"""State broadcast over UDP: the compact pose frame from the sync design.
|
||||||
|
|
||||||
Frame layout (little-endian, 46 bytes):
|
Frame layout (little-endian, 45 bytes) — source of truth for docs and the
|
||||||
|
prototype bandwidth estimate:
|
||||||
|
|
||||||
magic 2s b"SH"
|
magic 2s b"SH"
|
||||||
version B
|
version B
|
||||||
drone_id 8s zero-padded ascii
|
drone_id 8s zero-padded ascii (PoC; production may switch to uint16 registry)
|
||||||
ts_ns q epoch nanoseconds
|
ts_ns q epoch nanoseconds
|
||||||
pos_mm 3i position, millimeters (quantized on the wire only)
|
pos_mm 3i position in the mission frame, millimeters (wire quantization only)
|
||||||
att_cdeg 3h roll/pitch/yaw, centi-degrees
|
att_cdeg 3h roll/pitch/yaw, centi-degrees
|
||||||
vel_cms 3h velocity, cm/s
|
vel_cms 3h velocity, cm/s
|
||||||
frame_ref B
|
frame_ref B
|
||||||
|
|||||||
Reference in New Issue
Block a user