commit 57e48e0b55c75fcfb0c38e6af2ea6f8667282553 Author: Andriy Oblivantsev Date: Wed Jul 8 13:03:23 2026 +0100 docs: on-prem data platform proposal for autonomous drone swarms Problem statement, on-board architecture, Parquet/DuckDB storage design, swarm sync strategy, zero-trust networking, environments, observability, CI/CD delivery with fleet release manifests, roadmap and open questions. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..28d98fe --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +data/ +__pycache__/ +*.pyc +.venv/ +node_modules/ +dist/ +.env diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..70647c1 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,26 @@ +# AGENTS.md + +Rules for anyone (human or tooling) contributing to this repository. + +## Hard rules + +1. **English only.** All documentation, code, comments, commit messages, and file names are written in English. +2. **No identifying references.** Never mention names of people, companies, organizations, or locations anywhere in this repository — not in docs, code, comments, or git history. This is a generic, self-contained design proposal for an on-prem data platform for autonomous drone swarms. +3. **Clean git history.** Commit messages must never reference AI assistants, code generation tools, or editors. No `Co-authored-by` trailers pointing to tools, no "Generated with ..." lines. Conventional commit format (`feat:`, `fix:`, `docs:`, `refactor:`). +4. **Push target.** The only remote is `origin` → `git.produktor.io`. Never mirror this repository anywhere else. + +## Project layout + +``` +docs/ Design documentation (Markdown + Mermaid) +simulator/ Virtual drone fleet — Python data generator (Parquet/DuckDB pipeline) +prototype/ 2D swarm visualization — React + TypeScript (Vite) +``` + +## Conventions + +- Markdown: ATX headings, fenced code blocks with language tags, pipe tables. +- Python: PEP 8, type hints, pure functions where possible, no unnecessary classes. +- TypeScript: strict mode, functional components and hooks, no extra UI libraries. +- YAML: 2-space indent. +- Diagrams: Mermaid inside Markdown. diff --git a/README.md b/README.md new file mode 100644 index 0000000..4a2c797 --- /dev/null +++ b/README.md @@ -0,0 +1,50 @@ +# Swarm House + +**An on-prem data platform for autonomous drone swarms** — a design proposal from a DevOps perspective. + +A fleet of drones flies fully autonomously: no internet uplink, no external access, all data stays inside the swarm. Each drone collects high-rate sensor telemetry and runs on-board video object detection. The swarm must exchange just enough state to coordinate flight, store everything else locally, and hand its data over to a ground warehouse after landing — reproducibly, securely, and at scale. + +This repository describes how to build, deliver, test, and operate that platform: the storage layout, the sync strategy, the network trust model, the CI/CD chain, and the simulation environment that makes it all testable without touching real hardware. + +## Design principles + +1. **Every drone is autonomous.** No cluster orchestrator spans the swarm; intermittent mesh connectivity makes that an anti-pattern. Coordination happens through data exchange, not through a control plane. +2. **One storage format on every floor.** Parquet + DuckDB with an identical Hive-partitioned layout on the drone, in the ground warehouse, and in the dev environment. Flight offload is a plain mirror operation. +3. **Sync results, not raw data.** Only derived state (position, attitude, detections) crosses the air. Raw telemetry stays on local NVMe until the drone lands. +4. **Event-driven, not polled.** New derived data triggers downstream actions through hooks; nothing burns cycles asking "anything new yet?". +5. **Zero trust, provisioned ahead.** Keys are issued per device before deployment; nothing joins the swarm at runtime. Dev channels are physically absent from production builds. +6. **The whole fleet has one version.** A release manifest pins every image digest, model weight, schema, and config. Rollback is atomic. + +## Documentation + +| Document | Contents | +| --- | --- | +| [00 — Glossary](docs/00-glossary.md) | Terms and abbreviations used throughout | +| [01 — Problem statement](docs/01-problem-statement.md) | Goal, constraints, knowns vs assumptions | +| [02 — Architecture](docs/02-architecture.md) | On-board layers, service composition, communication planes | +| [03 — Data platform](docs/03-data-platform.md) | Parquet/DuckDB storage tiers, partitioning, hot-write path | +| [04 — Swarm sync](docs/04-swarm-sync.md) | What syncs, what does not, pub/sub transport, query standard | +| [05 — Network & security](docs/05-network-security.md) | Zero-trust mesh, key provisioning, encrypted transport | +| [06 — Environments](docs/06-environments.md) | Drone / ground warehouse / dev-simulation infrastructures | +| [07 — Observability](docs/07-observability.md) | Logs, service metrics, hardware telemetry | +| [08 — Roadmap](docs/08-roadmap.md) | Dependency graph, simple to complex | +| [09 — Open questions](docs/09-open-questions.md) | Known unknowns and proposed answers | +| [10 — Domain context](docs/10-domain-context.md) | Swarm autonomy principles this design builds on | +| [11 — CI/CD & delivery](docs/11-cicd-delivery.md) | Pipelines, registry, dev containers, fleet releases | + +## Runnable parts + +| Component | Purpose | Stack | +| --- | --- | --- | +| [`simulator/`](simulator/) | Virtual drone fleet: generates realistic telemetry through the same Parquet/DuckDB pipeline a real drone would use; N drones via Docker Compose | Python, pyarrow, DuckDB | +| [`prototype/`](prototype/) | 2D top-down swarm visualization: drones, obstacles, inter-drone links with live transfer metrics | React, TypeScript, Vite, SVG | + +## Quick start + +```bash +# Spin up a virtual swarm of 10 drones +cd simulator && DRONE_COUNT=10 docker compose up + +# Run the visualization +cd prototype && npm install && npm run dev +``` diff --git a/docs/00-glossary.md b/docs/00-glossary.md new file mode 100644 index 0000000..88c73f4 --- /dev/null +++ b/docs/00-glossary.md @@ -0,0 +1,63 @@ +# 00 — Glossary + +Terms and abbreviations used throughout this proposal. + +## Data & storage + +| Term | Meaning | +| --- | --- | +| **Parquet** | Columnar file format; compresses time series extremely well (ZSTD + delta encoding), supports predicate pushdown | +| **Hive partitioning** | Directory layout `key=value/…` understood by most engines; enables partition pruning (only relevant directories are read) | +| **DuckDB** | In-process analytical SQL engine; queries Parquet globs directly, no server to operate | +| **ZSTD** | Zstandard compression; the default codec for all Parquet files here | +| **DWH** | Data warehouse — here: the on-prem ground store holding every flight ever flown | +| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse | +| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes | +| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier | +| **Schema registry** | Versioned catalog of dataset schemas so producers and consumers can evolve independently | +| **ETL** | Extract, transform, load — the pipelines that move and reshape data between tiers | + +## Swarm & robotics + +| Term | Meaning | +| --- | --- | +| **UAV / UGV / USV / UUV** | Unmanned aerial / ground / surface / underwater vehicle | +| **Swarm** | Group of autonomous vehicles coordinating without a central controller | +| **Sustainable pulsing** | Swarm ability to repeatedly engage a target area from multiple directions and self-reorganize as conditions evolve | +| **Ubiquitous sensing** | The swarm acting as one distributed sensor, fusing local observations into shared situational awareness ("top sight") | +| **Man-in-the-loop** | A human supervises intent and boundaries but does not steer individual vehicles | +| **GPS-denied** | Operating without satellite positioning; position comes from visual odometry and relative localization | +| **Visual odometry** | Estimating a vehicle's motion from its own camera stream | +| **ROS 2** | Robot Operating System 2 — the de-facto middleware for robotics software; nodes exchange typed messages over topics | +| **DDS** | Data Distribution Service — the default ROS 2 transport middleware | +| **Zenoh** | Pub/sub + query protocol designed for constrained, dynamic networks; official ROS 2 RMW implementation exists | +| **YOLO-like detector** | Family of single-pass object-detection models; the on-board video service runs one of them, weights are swappable | +| **IMU** | Inertial measurement unit — accelerometer + gyroscope (+ magnetometer) | +| **LiDAR** | Laser-based distance sensing | +| **RSSI** | Received signal strength indicator; Wi-Fi signal strength, usable as a coarse relative-positioning signal | +| **C2** | Command and control — the narrow channel that delivers mission intent to the swarm | + +## Infrastructure & delivery + +| Term | Meaning | +| --- | --- | +| **Air-gapped** | No route to the internet, physically or logically | +| **On-prem** | Runs on owned hardware; no public cloud involvement | +| **Zero trust** | No implicit trust from network position; every peer authenticates with pre-provisioned credentials | +| **mTLS** | Mutual TLS — both sides of a connection present certificates | +| **PKI** | Public key infrastructure — the certificate authority hierarchy behind mTLS | +| **WireGuard** | Lightweight encrypted tunnel protocol; basis for the swarm mesh overlay | +| **Ad-hoc mesh** | Wireless network formed directly between peers, no fixed infrastructure | +| **Docker Compose** | Declarative multi-container runtime; the only orchestrator on board a drone | +| **k3s** | Lightweight Kubernetes distribution; used on the ground only | +| **Dev Container** | Reproducible containerized development environment definition | +| **MinIO** | S3-compatible object store; runs on-board for derived data and on the ground as the warehouse backend | +| **Registry mirror** | Local copy of a container registry inside the air gap; drones pull images from it | +| **OTA** | Over-the-air update — delivered before a mission while docked, never mid-flight | +| **Fleet release manifest** | A versioned lockfile pinning every artifact (image digests, model weights, schemas, configs) that defines one fleet version | +| **SemVer** | Semantic versioning (`MAJOR.MINOR.PATCH`) applied to every artifact and to the fleet manifest itself | +| **Trivy** | Open-source scanner for container images and dependencies | +| **SonarQube** | Static code-quality and security analysis platform | +| **nvidia-container-toolkit** | Runtime hook exposing GPUs to containers; needed for on-board inference | +| **HIL** | Hardware-in-the-loop testing — real flight hardware wired into a simulated environment | +| **CRDT** | Conflict-free replicated data type; merges concurrent updates without coordination | diff --git a/docs/01-problem-statement.md b/docs/01-problem-statement.md new file mode 100644 index 0000000..4f3e809 --- /dev/null +++ b/docs/01-problem-statement.md @@ -0,0 +1,59 @@ +# 01 — Problem statement + +## Goal + +Design the **data and delivery platform** for a fully autonomous drone swarm: + +1. **Accumulate** — ingest high-rate sensor and detection data locally on every drone, in flight, in real time. +2. **Transform & store** — convert raw streams into a compressed, queryable local store that survives the whole mission on on-board NVMe. +3. **Serve & sync** — expose derived results (position, attitude, detections) to the rest of the swarm with minimal bandwidth, and offload complete flight data to a ground warehouse after landing. + +Plus everything a platform needs around that: reproducible builds and deployments, a simulation environment, observability, and a zero-trust network — all fully on-prem and air-gapped. + +## Constraints + +| Constraint | Consequence | +| --- | --- | +| **No internet, no external access, ever** | All infrastructure self-hosted; updates through an in-air-gap registry mirror; no cloud services of any kind | +| **Autonomous operation** | No human in the control loop during flight; the platform must not require operator intervention | +| **Intermittent mesh connectivity** | No component may assume a stable link between any two drones; no swarm-wide orchestrator | +| **Mixed hardware** | ARM64 and AMD64 Linux targets; some nodes carry GPUs for inference; all images built multi-arch | +| **Bounded on-board resources** | Multi-TB NVMe but finite; high-rate ingest must be compressed and retention-managed in flight | +| **Data never leaves the system** | Raw data stays on the drone until physical offload at the base; only derived state crosses the air, encrypted | +| **Small on-board footprint** | The data plane must not compete with flight-critical software for CPU/RAM | + +## What is known (given) + +- Drones run **Linux** on ARM64 (and AMD64 variants); on-board compute is substantial (NVMe, many sensors, GPU-class accelerators for vision). +- The current on-board software is **Docker Compose** with two services: **sensor ingestion** and **video-stream object detection** (a YOLO-like model; weights and approaches vary). +- **DuckDB** is already in use for local data handling. +- **Parquet** adoption is at the evaluation stage — the storage layout in this proposal is the core of what is being asked. +- **MinIO** is available and considered as the inter-drone sync mechanism. +- A **lightweight Kubernetes** exists in the ecosystem; this proposal scopes it to ground infrastructure only (see [02 — Architecture](02-architecture.md)). +- After landing, each drone's data is **offloaded to an on-prem warehouse** for replay and iterative model training. +- Mission intent (declarative goals) reaches the swarm over a narrow **C2 channel**; there is no continuous ground link in flight. + +## What is assumed (explicitly marked as assumptions) + +| # | Assumption | Basis | +| --- | --- | --- | +| A1 | The shared state each drone must publish is small: 3D position + attitude + timestamp + drone id, plus detection events | Coordination and collision avoidance need exactly this; anything more wastes air time | +| A2 | Raw sensor data is never needed by peer drones in flight | Offload happens at base; peers act on derived state only | +| A3 | Sensors emit at rates from single Hz (barometer) to hundreds of Hz (IMU); video detections are event-shaped | Typical for this vehicle class; drives nanosecond timestamps and partition sizing | +| A4 | Missions are bounded (battery), so a "flight" is the natural unit of data lifecycle | Return-to-base on low energy implies discrete flight sessions | +| A5 | Mission logic consumes the data platform as a service and is out of scope here | Flight control, planning, and model training are separate concerns | +| A6 | A development-only telemetry channel exists on the bench and is absent from production builds | Standard practice; production radio profile carries C2 + swarm data plane only | + +## Out of scope + +- Flight-control algorithms, trajectory planning, collision-avoidance logic — **consumers** of this platform. +- Model training pipelines — they read from the ground warehouse; the warehouse contract is in scope, training itself is not. +- Radio hardware selection — the design assumes an IP-capable ad-hoc wireless link and treats bandwidth as scarce. + +## Success criteria + +1. A single drone can ingest, compress, and locally query its full sensor load for an entire flight without operator action. +2. Any drone can obtain any peer's latest derived state within a bounded delay while links are up, and reconcile automatically after partitions heal. +3. Flight offload to the ground warehouse is a plain mirror operation with an integrity audit — no bespoke migration step. +4. The whole fleet can be built, scanned, versioned, and rolled back as one release artifact. +5. An engineer can spin up a virtual swarm on a laptop and run the identical data path end to end. diff --git a/docs/02-architecture.md b/docs/02-architecture.md new file mode 100644 index 0000000..ce75c63 --- /dev/null +++ b/docs/02-architecture.md @@ -0,0 +1,88 @@ +# 02 — Architecture + +## On-board: three layers, one Compose file + +Every drone runs the same three-layer data plane. **Docker Compose under systemd is the only on-board orchestrator.** Cluster orchestrators (Kubernetes, Swarm mode) assume a stable control plane and continuous connectivity — in an ad-hoc mesh with intermittent links that is an anti-pattern. Each drone is fully autonomous; coordination happens through data exchange, not through a shared control plane. Lightweight Kubernetes (k3s) is used on the ground only (warehouse, CI runners, simulation farm — see [06 — Environments](06-environments.md)). + +```mermaid +graph TB + subgraph drone [Single drone — Docker Compose under systemd] + subgraph ingestion [Layer 1 — Ingestion] + SENSORS["sensor-ingest
ROS 2 topics / drivers → stream"] + VIDEO["video-analytics
YOLO-like detector, GPU"] + end + subgraph storage [Layer 2 — Storage and transform] + WRITER["parquet-writer
stream → current/ blocks"] + SEALER["sealer
compact closed windows"] + DUCK["DuckDB
in-process SQL over Parquet"] + NVME[("NVMe
Hive-partitioned Parquet")] + end + subgraph serving [Layer 3 — Serving and sync] + HOOK["event hook
fires on new derived data"] + PUB["state publisher
pub/sub broadcast"] + MINIO["MinIO
derived datasets bucket"] + QAPI["query API
peer data requests"] + end + end + + SENSORS --> WRITER + VIDEO -->|detections| WRITER + WRITER --> NVME + SEALER --> NVME + DUCK --> NVME + WRITER -->|derived rows| HOOK + HOOK --> PUB + HOOK --> MINIO + QAPI --> DUCK + PUB -.->|mesh| PEERS["peer drones"] + MINIO -.->|replication| PEERS + QAPI -.->|on demand| PEERS +``` + +### Layer 1 — Ingestion + +- `sensor-ingest` subscribes to sensor sources (ROS 2 topics where available, raw drivers otherwise) and normalizes them into typed streams: IMU, barometer, temperature, LiDAR, RSSI, power, and so on. +- `video-analytics` runs a **YOLO-like object detector** on the camera stream. The model is deliberately treated as a swappable, versioned artifact (weights differ per mission and improve over time — see [11 — CI/CD & delivery](11-cicd-delivery.md)). GPU access via nvidia-container-toolkit; ARM64 builds use the vendor's L4T-class base images. +- Both services emit rows, not files. Timestamps are `int64` epoch **nanoseconds** end to end (native for ROS 2; IMU-class rates make milliseconds insufficient). + +### Layer 2 — Storage and transform + +- `parquet-writer` appends incoming rows into small per-minute Parquet blocks under the partition's `current/` directory (details in [03 — Data platform](03-data-platform.md)). +- `sealer` compacts each closed window into one ZSTD-compressed file and enforces retention quotas on the NVMe. +- DuckDB runs **in-process** inside whichever service needs SQL — there is no database server to babysit. + +### Layer 3 — Serving and sync + +- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls. +- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)). +- **MinIO** holds derived datasets only, replicated opportunistically between drones. +- The **query API** answers ad-hoc peer requests against local DuckDB (the query standard is also in [04](04-swarm-sync.md)). + +## Communication planes + +Three isolated planes with different lifecycles: + +```mermaid +graph LR + subgraph planes [Communication planes] + C2["C2 control plane
mission intent, narrow, in production"] + DATA["swarm data plane
state broadcast + sync, in production"] + DEV["dev/debug plane
live telemetry, bench only"] + end + OPERATOR["supervising operator
(man-in-the-loop)"] --> C2 + C2 --> SWARM["swarm"] + SWARM <--> DATA + DEV -.->|absent from production builds| SWARM +``` + +| Plane | Purpose | Bandwidth | In production | +| --- | --- | --- | --- | +| **C2 control plane** | Deliver declarative mission goals and boundaries; receive high-level status. Human supervises intent, not actions | Very low | Yes | +| **Swarm data plane** | State broadcast, derived-data replication, peer queries | The scarce resource; budget per message class | Yes | +| **Dev/debug plane** | Full live telemetry for bench development and HIL tests | High | **No — physically absent from production builds** | + +The dev plane is not "disabled by config": production images and radio profiles simply do not contain it. That removes an entire attack surface instead of guarding it. + +## Out of scope (consumed as services) + +Flight control, trajectory planning, mission logic, and model training sit **on top of** this platform: they subscribe to the event hook, query DuckDB, and read the warehouse. Their internals do not affect the platform design beyond the data contracts defined here. diff --git a/docs/03-data-platform.md b/docs/03-data-platform.md new file mode 100644 index 0000000..8ed3399 --- /dev/null +++ b/docs/03-data-platform.md @@ -0,0 +1,116 @@ +# 03 — Data platform + +The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every floor** of the system. + +## Datasets + +Three logical datasets, distinct because their consumers and sync policies differ: + +| Dataset | Contents | Rate | Leaves the drone in flight? | +| --- | --- | --- | --- | +| `telemetry` | Raw sensor rows: IMU, barometer, temperature, LiDAR, RSSI, power, magnetometer, … | Up to hundreds of Hz per sensor | **Never** | +| `detections` | Object-detection events from the video service: class, confidence, bounding box, ego-position at detection time | Event-shaped, bursty | Yes — derived, shareable | +| `state` | The drone's own pose stream: position, attitude, velocity, timestamp; plus the log of every broadcast sent/received | 1–10 Hz effective | Yes — this *is* the shared state | + +## Partitioning layout + +Identity dimensions first, calendar time last, everything Hive-style `key=value`: + +``` +dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/ + current/min_00.parquet ← hot write path (open window) + current/min_01.parquet + data.parquet ← sealed once the hour closes +dataset=detections/flight=…/drone=…/year=…/month=…/day=…/hour=…/ +dataset=state/flight=…/drone=…/year=…/month=…/day=…/hour=…/ +``` + +Design decisions and their reasons: + +- **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.) +- **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both. +- **Flight identifiers.** The canonical flight id is a **UUIDv7** (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest. +- **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time. +- **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)). + +## Hot write path: `current/` → sealed + +Streaming many small rows straight into big Parquet files is impossible (Parquet is immutable); buffering a whole hour in RAM is unacceptable (crash = lost data). The proven pattern: + +```mermaid +sequenceDiagram + participant S as sensor-ingest + participant W as parquet-writer + participant C as current/ (open window) + participant Z as sealer + participant P as data.parquet (sealed) + participant D as DuckDB + + loop every minute block + S->>W: rows (int64 ns, typed) + W->>C: flush min_NN.parquet (small, ZSTD-fast) + end + Note over Z: window closes (hour sealed
or flight ends) + Z->>C: read all min_*.parquet + Z->>P: write one compact file (ZSTD-high) + Z->>C: remove blocks after verify + D->>P: read_parquet('…/**/*.parquet') — glob covers both states +``` + +- Crash tolerance is one minute-block, tunable per sensor rate. +- Readers never care about the sealing state: a single `read_parquet` glob matches both `current/*.parquet` and `data.parquet`. +- The sealer runs at low priority — flight-critical software always wins the CPU. + +## Storage floors (tiers) + +Same format on every floor; only volume, retention, and location change: + +| Floor | Where | Contents | Retention | +| --- | --- | --- | --- | +| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes | +| **T1 — warm** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) | +| **T2 — shared** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission | +| **T3 — warehouse** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; replay, analytics, model training | Years | +| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral | + +## Flight offload: a mirror, not a migration + +Because T1 and T3 share the identical layout, offload after landing is: + +```bash +# per drone, at the base station +mc mirror drone-nvme/flights/ warehouse/flights/ # or rsync over the wired dock +duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…" +``` + +The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job. + +## Retention and quotas on board + +- Per-dataset NVMe quotas enforced by the sealer (oldest sealed partitions of *previous* flights evicted first; the current flight is never evicted). +- If ingest pressure exceeds the write budget, degradation is explicit and ordered: reduce optional high-rate sensors' sampling first, never drop `state` or `detections`. + +## Query examples (DuckDB) + +```sql +-- All detections of one flight, fused across the fleet (runs on T3) +SELECT drone, ts_ns, class, confidence, pos_x, pos_y, pos_z +FROM read_parquet('warehouse/dataset=detections/flight=20260708T1100Z-a3f2/**/*.parquet') +ORDER BY ts_ns; + +-- IMU of one drone for one minute (runs identically on T1 or T3) +SELECT * +FROM read_parquet('…/dataset=telemetry/flight=…/drone=dr-017/sensor=imu/**/*.parquet') +WHERE ts_ns BETWEEN 1783941000000000000 AND 1783941060000000000; + +-- Downsample for a dashboard: 1 Hz means over a 400 Hz stream +SELECT ts_ns // 1000000000 AS second, avg(accel_x), avg(accel_y), avg(accel_z) +FROM read_parquet('…/sensor=imu/**/*.parquet') +GROUP BY second ORDER BY second; +``` + +## Schema management + +- Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)). +- Parquet files carry the schema version in file metadata; readers select the matching deserializer. +- Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release. diff --git a/docs/04-swarm-sync.md b/docs/04-swarm-sync.md new file mode 100644 index 0000000..469c32a --- /dev/null +++ b/docs/04-swarm-sync.md @@ -0,0 +1,106 @@ +# 04 — Swarm sync + +What crosses the air, what never does, and how it moves. + +## The golden rule + +> **Sync results, not raw data — and only when they are actually needed.** + +| Data | Synced in flight? | Why | +| --- | --- | --- | +| `state` (pose) | **Yes — broadcast** | Every drone needs every peer's position to coordinate and avoid collisions | +| `detections` | **Yes — published, pulled on demand** | Peers re-task around detections; but not every peer needs every frame of context | +| `telemetry` (raw) | **No** | Nobody in the air needs a peer's raw IMU; it offloads at the base ([03](03-data-platform.md)) | + +Bandwidth is the scarcest resource in the system. Every message class gets an explicit budget; anything unbudgeted stays local. + +## Broadcast payload: small on the wire, precise at rest + +The `state` broadcast is a fixed compact frame: + +| Field | Type | Notes | +| --- | --- | --- | +| `drone_id` | uint16 | Fleet-scoped registry | +| `ts_ns` | int64 | Epoch nanoseconds, same clock domain as storage | +| `pos_x/y/z` | int32 | **Millimeters** in the mission frame — quantized only here, storage keeps full float precision | +| `att_roll/pitch/yaw` | int16 | Centi-degrees | +| `vel_x/y/z` | int16 | cm/s | +| `frame_ref` | uint8 | Frame of reference id (GPS-denied: local/visual-odometry frames must be explicit) | +| `flags` | uint8 | Battery-low, returning, degraded-sensors, … | + +~40 bytes per frame → a 50-drone swarm at 5 Hz is ~10 KB/s of pose traffic before transport overhead. Trivial even on a congested mesh. + +`detections` events are slightly larger (class, confidence, bounding volume, ego-pose) but event-shaped and rare by comparison. + +## Transport: event-driven pub/sub over an ad-hoc mesh + +Requirements: peer discovery without infrastructure, pub/sub with late-joiner catch-up, graceful behavior under partitions, tiny footprint, ARM64 support. + +```mermaid +graph LR + subgraph droneA [Drone A] + HA["event hook"] --> PA["publisher"] + end + subgraph droneB [Drone B] + SB["subscriber"] --> LB["local state cache + DuckDB"] + end + subgraph droneC [Drone C — out of direct range] + SC["subscriber"] + end + PA -->|"pose @5Hz, detections on event"| SB + SB -->|"store-and-forward relay"| SC +``` + +### Recommended: Zenoh + +- Designed exactly for constrained, dynamic networks: built-in peer discovery, brokerless peer-to-peer mode, store-and-forward, and a query layer on top of pub/sub. +- First-class robotics citizenship: an official ROS 2 RMW implementation exists, so the ingestion side and the sync side can share one middleware. +- Tiny footprint, ARM64-native. + +### Alternatives considered + +| Option | Verdict | +| --- | --- | +| **DDS multicast** (ROS 2 default) | Works, battle-tested; but discovery storms and tuning pain on lossy wireless meshes are well documented. Keep as fallback since ROS 2 speaks it natively | +| **MQTT** | Needs a broker — a per-drone broker bridge is possible but adds moving parts for no gain over Zenoh | +| **Raw UDP multicast** | Perfect as a last-resort minimal profile for the pose broadcast alone (fixed frame, no discovery); no query layer, no reliability — documented as the degraded mode | +| **MinIO bucket replication** | Wrong tool for the 5 Hz pose path, right tool for bulk derived datasets — see below | + +## Two sync mechanisms, deliberately separate + +1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache. +2. **Bulk path — MinIO replication:** sealed `detections`/`state` Parquet partitions replicate opportunistically between on-board MinIO instances when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. + +Partition healing is automatic: replication is pull-based, versioned by partition path, and idempotent (same layout = same keys; last-writer-wins is safe because each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**). + +## Peer query standard + +For everything that is not broadcast, drones (and mission services) query each other explicitly. The contract: + +- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. +- **Recommended interface: GraphQL** over the mesh transport — one flexible, introspectable contract for "give me detections of class X since T from your flight" without inventing endpoints per use case. It also becomes the natural standard for the future multi-team integration surface. +- **Alternative: Arrow Flight** — far more efficient for bulk columnar transfer, worth adopting on the ground (T3 warehouse serving) where result sets are large; overkill as the in-flight peer protocol. +- Responses stream Parquet/Arrow batches, never JSON blobs, so the receiving side lands data straight into its own store. + +## Event-driven end to end + +No component polls. The chain from sensor to swarm reaction: + +```mermaid +sequenceDiagram + participant V as video-analytics + participant W as parquet-writer + participant H as event hook + participant Z as pub/sub mesh + participant P as peer drone + participant M as peer mission logic + + V->>W: detection row + W->>H: derived-data event + H->>Z: publish detection + Z->>P: deliver (direct or relayed) + P->>M: local hook fires + M->>M: re-task decision (out of scope) +``` + +The same hook mechanism that feeds the mesh also feeds local mission logic — one eventing model on board and across the swarm. diff --git a/docs/05-network-security.md b/docs/05-network-security.md new file mode 100644 index 0000000..f8be61d --- /dev/null +++ b/docs/05-network-security.md @@ -0,0 +1,57 @@ +# 05 — Network & security + +Zero trust, fully air-gapped, everything provisioned before wheels-up. + +## Trust model + +- **Nothing joins the swarm at runtime.** Every device receives its identity (key pair + certificate) during ground provisioning, signed by the fleet's offline CA. There is no trust-on-first-use, no runtime enrollment endpoint, no exception path. +- **Network position grants nothing.** Being inside the mesh does not authorize anything; every connection authenticates mutually. +- **Two layers of cryptography, by design:** + 1. **WireGuard mesh overlay** — every drone holds the public keys of every fleet member (distributed at provisioning). All swarm traffic runs inside these tunnels; a device without a provisioned key cannot even complete a handshake. + 2. **mTLS between services** — inside the overlay, service-to-service calls (query API, MinIO replication) present per-service certificates from the fleet PKI. Compromising the network layer alone still yields nothing readable. + +```mermaid +graph TB + subgraph ground [Ground provisioning, before deployment] + CA["offline fleet CA"] + PROV["provisioning station"] + CA -->|"sign device + service certs"| PROV + end + subgraph mesh [In-flight ad-hoc mesh] + A["drone A
wg key + certs"] + B["drone B
wg key + certs"] + C["drone C
wg key + certs"] + A <-->|"WireGuard + mTLS"| B + B <-->|"WireGuard + mTLS"| C + A <-->|"WireGuard + mTLS"| C + end + PROV -->|"per-device identity, peer list"| A + PROV --> B + PROV --> C +``` + +## Radio reality + +- Transport is assumed to be ad-hoc Wi-Fi class (2.4/5 GHz), with channel hopping expected. The platform treats the link as **lossy, variable, and adversarial** — all sync mechanisms already tolerate partitions ([04](04-swarm-sync.md)). +- **RSSI is data.** Per-peer signal strength is captured into `telemetry` like any other sensor — it feeds relative-positioning estimates and post-flight link-quality analysis, and it is free. +- Electromagnetic environment sensing (whatever the radio can observe) is likewise recorded — cheap in flight, valuable in replay. + +## Key lifecycle + +| Phase | Action | +| --- | --- | +| Fleet build | Offline CA generates device identities; keys injected at provisioning, never transmitted over any network | +| Mission prep | Peer lists and certificates refreshed as part of the fleet release bundle ([11](11-cicd-delivery.md)) | +| Rotation | Standard cadence between missions; emergency rotation = new fleet release | +| Loss of a drone | Its keys are revoked in the next release; mesh peers drop the revoked identity at the next mission load. Data-at-rest exposure is bounded by NVMe encryption (below) | + +## Data at rest + +- On-board NVMe is encrypted (LUKS) with keys held in the device's secure element / TPM where the hardware provides one; the disk alone is unreadable. +- The ground warehouse applies standard at-rest encryption plus role-based access; it is the single most valuable asset in the system (every flight ever flown). + +## Attack surface, deliberately shortened + +- The dev/debug plane **does not exist** in production builds ([02](02-architecture.md)) — not firewalled, absent. +- No inbound listeners except the authenticated mesh services; no management SSH in production radio profiles (bench access is wired, at the dock). +- Images are scanned (Trivy) and signed in CI; drones verify signatures against the pinned digests in the fleet release manifest before running anything ([11](11-cicd-delivery.md)). diff --git a/docs/06-environments.md b/docs/06-environments.md new file mode 100644 index 0000000..eda45b6 --- /dev/null +++ b/docs/06-environments.md @@ -0,0 +1,71 @@ +# 06 — Environments + +Three infrastructures, one data contract. The layout, schemas, and pipelines are identical everywhere; only scale and lifetime differ. + +```mermaid +graph LR + subgraph dev [3 — Dev and simulation] + SIM["virtual drone fleet
(simulator, Compose)"] + LAP["engineer laptop
Dev Container"] + SLICE["T4: data slices"] + end + subgraph fleet [1 — Fleet, in flight] + D1["drone: Compose data plane
T0 hot + T1 NVMe + T2 MinIO"] + D2["drone …"] + end + subgraph ground [2 — Ground, on-prem] + DOCK["base station docks"] + DWH["T3 warehouse
object store + Parquet/DuckDB"] + K3S["k3s: CI runners, registry mirror,
sim farm, dashboards"] + TRAIN["model training (out of scope)
reads T3, ships weights"] + end + + D1 <-.->|mesh| D2 + D1 -->|"offload: mirror + audit"| DOCK + DOCK --> DWH + DWH -->|"slices for development"| SLICE + SIM -->|"same pipeline, synthetic flights"| SLICE + DWH --> TRAIN + TRAIN -->|"versioned weights"| K3S +``` + +## 1 — Fleet infrastructure (on the drones) + +- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md). +- Storage floors T0–T2. Nothing here requires ground contact during a mission. +- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest. + +## 2 — Ground infrastructure (on-prem) + +The permanent installation. This **is** allowed to be a cluster — links are wired and stable here, so k3s earns its keep: + +| Component | Runs on | Role | +| --- | --- | --- | +| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory | +| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) | +| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see | +| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) | +| **Simulation farm** | k3s | Scaled-up virtual swarm runs for regression and pre-mission validation | +| **Dashboards** | k3s | Observability stack ([07](07-observability.md)) + flight replay tooling | + +Replay is the warehouse's flagship feature: because every broadcast, detection, and re-tasking trigger is preserved with nanosecond timestamps in one layout, any mission can be reconstructed step by step — for engineering analysis, for audit/traceability, and as training input. + +## 3 — Dev & simulation infrastructure + +The environment engineers live in daily — and deliberately the first thing to build (see [08 — Roadmap](08-roadmap.md)), because everything else is validated inside it. + +- **Dev Containers** give every engineer the same toolchain in minutes; no snowflake laptops. +- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop. +- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story. +- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified. +- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets). + +## Why identical layouts matter (the payoff) + +| Operation | What it costs with one layout everywhere | +| --- | --- | +| Flight offload | `mc mirror` / `rsync` + audit — no transform step | +| Debugging a field issue | Pull the flight slice, replay locally, same queries | +| Validating a pipeline change | Run simulator, diff Parquet outputs | +| Training data prep | Read T3 directly, no export pipeline | +| Onboarding an engineer | One Dev Container, one `docker compose up` | diff --git a/docs/07-observability.md b/docs/07-observability.md new file mode 100644 index 0000000..c149fb0 --- /dev/null +++ b/docs/07-observability.md @@ -0,0 +1,56 @@ +# 07 — Observability + +The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline. + +## Two kinds of signals, one storage + +| Signal | Examples | Where it goes | +| --- | --- | --- | +| **Hardware / environment telemetry** | Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment | The regular `telemetry` dataset — it is sensor data ([03](03-data-platform.md)) | +| **Platform operational metrics** | Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames | `telemetry` dataset, `sensor=platform` — same pipeline, zero extra moving parts on board | + +The deliberate trick: **on board there is no separate metrics stack.** A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window. + +## Logs + +- Services log structured JSON to journald (Compose default); a lightweight shipper appends them into `dataset=logs/flight=…/drone=…/service=…` Parquet partitions on the same write path. +- Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles. + +## On the ground: the standard stack + +Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs: + +```mermaid +graph LR + subgraph flight [In flight] + PM["platform metrics
= sensor stream"] + LG["structured logs
= Parquet partitions"] + end + subgraph ground [Ground k3s] + DWH["T3 warehouse"] + PR["Prometheus
(live: docks, CI, sim farm)"] + LK["Loki
(live logs, dev plane)"] + GF["Grafana
dashboards + replay"] + DK["DuckDB dashboards
flight post-mortems"] + end + PM --> DWH + LG --> DWH + DWH --> DK + PR --> GF + LK --> GF + DK --> GF +``` + +- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane. +- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute. + +## Health questions this design answers cheaply + +| Question | Source | +| --- | --- | +| Did any drone approach its NVMe quota last mission? | `sensor=platform` quota gauges, one SQL query | +| How stale was peer state for drone N during the partition at minute 14? | Broadcast log in `dataset=state` (sent + received both recorded) | +| Which link degraded first, and was it distance or interference? | RSSI telemetry joined with pose distance | +| Is the new writer version flushing slower on ARM64? | CI simulation runs emit the same metrics; diff across fleet releases | + +Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem. diff --git a/docs/08-roadmap.md b/docs/08-roadmap.md new file mode 100644 index 0000000..0a8420b --- /dev/null +++ b/docs/08-roadmap.md @@ -0,0 +1,59 @@ +# 08 — Roadmap + +From simple to complex, ordered by dependency — each stage is independently useful and validates the next. The guiding principle: **build the environment that makes everything else testable first.** + +## Dependency graph + +```mermaid +graph TD + S1["Stage 1
Dev environment + CI skeleton
Dev Containers, GitLab templates, multi-arch builds"] + S2["Stage 2
Registry and artifacts
images, model weights, schemas, cleanup policies"] + S3["Stage 3
Single-drone data pipeline
ingest → current/ → sealed Parquet → DuckDB"] + S4["Stage 4
Serving layer
event hook, query API, local MinIO"] + S5["Stage 5
Virtual swarm simulation
N drones in Compose, seeded scenarios, CI regression"] + S6["Stage 6
Mesh network and security
WireGuard overlay, PKI provisioning, mTLS"] + S7["Stage 7
Swarm sync
pose broadcast, detections pub/sub, MinIO replication"] + S8["Stage 8
Ground warehouse
offload + audit, T3 store, replay tooling"] + S9["Stage 9
Observability
platform metrics as sensor stream, ground dashboards"] + S10["Stage 10
Fleet releases
manifest, mirror, dock delivery, atomic rollback"] + S11["Stage 11
Query standard
GraphQL contract, schema registry, multi-team surface"] + + S1 --> S2 + S1 --> S3 + S2 --> S5 + S3 --> S4 + S4 --> S5 + S5 --> S7 + S6 --> S7 + S3 --> S8 + S7 --> S9 + S8 --> S9 + S2 --> S10 + S5 --> S10 + S8 --> S11 + S7 --> S11 +``` + +## Stage detail + +| Stage | Deliverable | Proves | Depends on | +| --- | --- | --- | --- | +| **1 — Dev environment + CI skeleton** | Dev Containers, GitLab shared templates, multi-arch runner fleet | An engineer builds and tests on day one; ARM64/AMD64 both green | — | +| **2 — Registry & artifacts** | GitLab registry live, model weights + schemas as versioned packages | One artifact store, scanning wired in (Trivy, SonarQube) | 1 | +| **3 — Single-drone data pipeline** | `sensor-ingest` → writer → `current/` → sealed → DuckDB queries on one node | The storage core: rates sustained, compression measured, quotas enforced | 1 | +| **4 — Serving layer** | Event hook + query API + on-board MinIO | Event-driven consumption; nothing polls | 3 | +| **5 — Virtual swarm** | Simulator, N drones via Compose, seeded scenarios, CI regression gate | Everything after this ships with a test bench | 2, 4 | +| **6 — Mesh & security** | WireGuard overlay, offline CA, provisioning flow, mTLS | Zero-trust fabric exists before any real sync traffic | 1 | +| **7 — Swarm sync** | Pose broadcast + detections pub/sub + bulk replication over the mesh | Staleness budgets met under simulated loss/partitions | 5, 6 | +| **8 — Ground warehouse** | Dock offload (mirror + audit), T3 store, replay queries | A full flight round-trips: fly (simulated) → offload → replay | 3 | +| **9 — Observability** | Platform metrics as sensor stream; ground Prometheus/Grafana/Loki; post-mortem dashboards | Every health question from [07](07-observability.md) answerable | 7, 8 | +| **10 — Fleet releases** | Release manifest, registry mirror, dock delivery, atomic rollback drill | The whole fleet moves as one version, rollback rehearsed | 2, 5 | +| **11 — Query standard** | GraphQL contract over versioned schemas; documentation for other teams | The multi-team integration surface | 7, 8 | + +## Sequencing rationale + +- **Stages 1–2 before everything:** unglamorous, but every later stage is only as fast as its build/test loop. +- **Stage 3 is the heart** — and it needs no network, no security, no swarm: a single machine and real (or simulated) sensor rates. Risk is retired early where iteration is cheapest. +- **Stage 5 before sync (7):** building a distributed protocol without a reproducible multi-node bench means debugging it in the field. The virtual swarm makes stage 7 a CI problem instead. +- **Stage 6 in parallel:** the trust fabric has no dependency on the data path and can mature alongside it. +- **Stages 8–11 stack on a proven core** — by then, every layer under them is regression-tested. diff --git a/docs/09-open-questions.md b/docs/09-open-questions.md new file mode 100644 index 0000000..f97e20b --- /dev/null +++ b/docs/09-open-questions.md @@ -0,0 +1,63 @@ +# 09 — Open questions + +Known unknowns, each with a proposed default so the project never blocks on an answer. These are the questions to settle with the domain teams early. + +## 1 — Time synchronization across the swarm + +**The single most critical open item.** Merging state and detections from multiple drones is only meaningful if their clocks agree. GPS time may be unavailable (GPS-denied operation is a design case). + +*Proposal:* PTP over the mesh with one elected (or designated) grandmaster per mission, disciplined by GPS when available; record per-drone clock-offset estimates into `telemetry` so post-flight fusion can correct residual skew. Budget: peer clocks within ≤1 ms keeps 5 Hz pose fusion honest; IMU-grade fusion needs tighter, but that happens on-board within one clock domain. + +## 2 — Frame of reference in GPS-denied flight + +Visual-odometry positions are relative; different drones may hold different local frames. + +*Proposal:* the `frame_ref` field in the state schema is mandatory from day one; frame alignment (co-registration) is a mission-logic concern, but the platform must carry the metadata — retrofitting it is painful. + +## 3 — Conflict resolution after partitions + +By construction there are no write conflicts in bulk sync (each drone writes only its own `drone=` subtree). But *derived swarm-level* products (shared maps, fused detections) may need merging. + +*Proposal:* keep swarm-level fusion out of the storage layer entirely: every drone stores its own observations; fusion is computed, not stored, in flight — and recomputed from ground truth in the warehouse. If a persistent shared structure becomes unavoidable, use a CRDT type for it rather than inventing reconciliation. + +## 4 — Degradation policy under sustained link loss + +What does a drone do when it has seen no peer state for N seconds? + +*Proposal:* the platform's job is to expose staleness honestly (per-peer last-seen in the state cache) and keep store-and-forward running. The behavioral response (loiter, climb, return) is mission logic — but the staleness contract must be a documented, tested number per message class. + +## 5 — Data disposal on loss of a drone + +A drone that does not return carries a full flight on NVMe. + +*Proposal:* LUKS at rest ([05](05-network-security.md)) bounds exposure; key revocation removes the lost unit from the mesh. Open question for the domain owner: is remote crypto-erase over C2 required, and what evidence standard applies to disposal? + +## 6 — Sensor priority order under resource pressure + +The degradation ladder ([03](03-data-platform.md)) needs a domain-approved ordering: which high-rate sensors are sacrificed first when NVMe or CPU budgets are hit? + +*Proposal:* default order — reduce optional high-rate environmental sensors first; never degrade `state`, `detections`, or IMU. To be reviewed per mission profile. + +## 7 — Hardware-in-the-loop coverage + +The virtual swarm covers logic and protocol; it does not cover radio behavior, GPU thermals, or NVMe write endurance under vibration. + +*Proposal:* a small HIL rig at the bench — real compute module + radio, simulated sensor feeds — as stage 5.5 of the roadmap, exercised by the same seeded scenarios as CI. + +## 8 — Video retention policy + +Detections are events, but how much raw video context should be kept? Full stream on NVMe is possible but dominates the storage budget; keeping only detection-triggered clips aligns with event-driven sensing principles and privacy-proportionality, but reduces training data. + +*Proposal:* configurable per mission: detection-triggered clips with pre/post-roll as the default profile; full-stream retention as an explicit opt-in for data-collection flights. + +## 9 — Fleet identity and registry governance + +Who owns the device registry (drone ids, key issuance, revocation) organizationally, and what is the audit process for provisioning? + +*Proposal:* treat the peer registry as a versioned artifact in the release manifest ([11](11-cicd-delivery.md)) so every change is reviewed, signed, and rollback-able like code. + +## 10 — Nano-platform profile + +Sub-250 g class units cannot run the full stack (no GPU, minimal CPU/storage). + +*Proposal:* define a minimal profile early — state publisher + UDP-multicast pose broadcast, no local Parquet store, no video — so the swarm protocol never assumes full-stack peers. diff --git a/docs/10-domain-context.md b/docs/10-domain-context.md new file mode 100644 index 0000000..476cb3d --- /dev/null +++ b/docs/10-domain-context.md @@ -0,0 +1,33 @@ +# 10 — Domain context + +Swarm autonomy has moved past biomimicry-driven research toward practical, operational systems. This proposal builds on a set of publicly discussed principles from that field; each maps directly onto a platform component. + +## Principles and where this design answers them + +| Principle | What it means | Where it lands in this design | +| --- | --- | --- | +| **Decentralization** | No single controller; every unit perceives locally and decides autonomously; no single point of failure | No swarm-wide orchestrator; each drone runs its own full data plane ([02](02-architecture.md)) | +| **Local interactions** | Behavior emerges from neighbor-to-neighbor exchange, not top-down commands | State broadcast between peers; event hooks trigger local reactions ([04](04-swarm-sync.md)) | +| **Self-organization** | The group coordinates without pre-planned orchestration and survives the loss of members | Opportunistic replication; store-and-forward through intermediate peers; automatic reconciliation after partitions ([04](04-swarm-sync.md)) | +| **Sustainable pulsing** | The swarm repeatedly engages a target area from multiple directions and re-forms as conditions change | Requires every unit to know peer state with bounded staleness — exactly the state-sync contract ([04](04-swarm-sync.md)) | +| **Ubiquitous sensing** | The swarm acts as one distributed sensor, fusing observations into shared "top sight" | Detections are first-class derived data: locally stored, selectively shared, fully preserved for post-flight fusion ([03](03-data-platform.md)) | +| **Edge intelligence** | Detection and classification happen on the unit; a local event can autonomously cue nearby units without central validation | On-board YOLO-like inference + event hook that publishes detections to the mesh immediately ([02](02-architecture.md)) | +| **Mesh networking** | Units relay data for each other; loss of any single link degrades nothing | Encrypted ad-hoc mesh, no infrastructure dependency ([05](05-network-security.md)) | +| **GPS-denied operation** | Position comes from visual odometry and relative localization, not satellites | The state schema carries a frame-of-reference field and covariance, not just raw coordinates ([03](03-data-platform.md), [09](09-open-questions.md)) | +| **Man-in-the-loop** | A supervisor defines intent and boundaries; the system distributes tasks itself | Narrow C2 plane for declarative goals; no per-vehicle steering ([02](02-architecture.md)) | +| **Adaptive re-tasking** | A detected event re-prioritizes nearby units without manual replanning | Event-driven data plane: detections propagate as events, mission logic subscribes ([02](02-architecture.md)) | +| **Ethical autonomy / traceability** | Autonomous actions must be explainable: event-driven sensing over indiscriminate collection, decisions logged and auditable | Every broadcast, sync, and re-tasking trigger is recorded in the flight data; the warehouse preserves the full decision trail for replay ([03](03-data-platform.md), [06](06-environments.md)) | +| **Adaptive learning** | Experience gathered by one unit improves the whole team over iterations | Complete flights land in the ground warehouse; training reads from it and ships improved model weights through the fleet release cycle ([06](06-environments.md), [11](11-cicd-delivery.md)) | +| **Multi-domain scaling** | The same swarm concepts apply to aerial, ground, surface, and underwater units — including sub-250 g platforms | The data plane is a set of small independent services; the minimal profile (writer + publisher, no GPU stack) fits constrained units ([02](02-architecture.md)) | + +## Why this matters for a data platform + +Every one of these principles quietly assumes a working data layer underneath: + +- *Pulsing* and *re-tasking* assume each unit **knows peer state** — that is a sync latency and staleness budget. +- *Ubiquitous sensing* assumes observations are **fused later** — that is a warehouse with aligned timestamps and schemas. +- *Edge intelligence* assumes detection events **reach neighbors fast** — that is an event-driven publish path, not a polling loop. +- *Traceability* assumes actions are **replayable** — that is complete, immutable, partitioned flight data. +- *Learning* assumes flights are **comparable across the fleet and across time** — that is schema versioning and a single storage format. + +The swarm behaviors are the visible product; the platform in this repository is the substrate they stand on. diff --git a/docs/11-cicd-delivery.md b/docs/11-cicd-delivery.md new file mode 100644 index 0000000..51de789 --- /dev/null +++ b/docs/11-cicd-delivery.md @@ -0,0 +1,104 @@ +# 11 — CI/CD & delivery + +How software, models, and configuration reach the fleet — reproducibly, scanned, versioned, and atomically rollback-able. Everything lives inside the air gap. + +## Source and pipelines: GitLab + +GitLab (self-managed, on-prem) is the backbone: repositories, CI/CD, and the container registry in one system. + +- **Shared pipeline templates** — one template library (`ci-templates` repo) defines the standard stages; service repos include and parameterize them instead of copy-pasting YAML. +- **Multi-arch by default** — every image builds for `linux/amd64` and `linux/arm64` via buildx on dedicated runners; GPU-inference images additionally build against the vendor's L4T-class base for the ARM targets. +- **Runner fleet on ground k3s** — build runners (amd64 + arm64), a GPU runner for inference smoke tests, and simulation runners that execute virtual-swarm regression scenarios ([06](06-environments.md)). + +```mermaid +graph LR + subgraph gitlab [GitLab, on-prem] + SRC["service repos
+ ci-templates"] + CI["CI pipelines
build · test · scan"] + REG["GitLab Container Registry
images + generic packages"] + end + subgraph quality [Quality gates] + SQ["SonarQube
code quality"] + TR["Trivy
image + dependency scan"] + SIMT["virtual swarm
regression scenarios"] + end + subgraph delivery [Delivery] + MAN["fleet release manifest
semver, digests pinned"] + MIR["registry mirror
base station"] + DOCK["dock: verify + apply"] + end + SRC --> CI + CI --> SQ + CI --> TR + CI --> SIMT + CI --> REG + REG --> MAN + MAN --> MIR + MIR --> DOCK +``` + +## Artifacts: GitLab Registry as the single store + +One registry for everything, next to the pipelines that produce it: + +| Artifact | Stored as | +| --- | --- | +| Service images (multi-arch) | Container registry, immutable tags + digests | +| **Detection model weights** (YOLO-like family — architectures and weights vary per mission and improve over iterations) | Generic package registry, semver-versioned, checksummed | +| Dataset schemas | Generic packages, semver ([03](03-data-platform.md)) | +| Compose bundles, radio profiles, provisioning configs | Generic packages, semver | + +Registry hygiene is part of the design: cleanup policies per repository, immutable release tags, access split between CI (write) and mirrors (read). + +> Consolidating on the GitLab registry removes a separate artifact platform from the stack — one fewer system to run inside the air gap, one auth domain, artifacts adjacent to the pipelines that build them. Scanning moves to Trivy (below). + +## Quality gates + +| Gate | Tool | Blocks merge when | +| --- | --- | --- | +| Code quality, coverage, static analysis | **SonarQube** | Quality gate red | +| Image and dependency vulnerabilities | **Trivy** (in CI + scheduled re-scan of released images) | Critical findings without an accepted waiver | +| Behavioral regression | **Virtual swarm scenarios** — canonical seeds replayed, Parquet outputs asserted (row counts, coverage, staleness budgets) | Any budget exceeded | + +Scheduled Trivy re-scans matter in an air gap: a released image that was clean in March may carry a known CVE by June; the scan flags it for the next fleet release even though the image never changed. + +## The fleet release manifest + +The central delivery idea: **the fleet has exactly one version.** + +```yaml +# fleet-release: 3.4.1 +schema_version: 1 +images: + sensor-ingest: registry.internal/fleet/sensor-ingest@sha256:9f2c… # 2.1.0 + video-analytics: registry.internal/fleet/video-analytics@sha256:5e11… # 3.0.2 + parquet-writer: registry.internal/fleet/parquet-writer@sha256:aa04… # 1.8.0 + state-publisher: registry.internal/fleet/state-publisher@sha256:c7d9… # 1.4.3 +models: + detector: { package: detector-weights, version: 5.2.0, sha256: "e3b0…" } +schemas: + telemetry: 2.1.0 + detections: 1.3.0 + state: 1.2.0 +config: + compose-bundle: 3.4.0 + radio-profile: production-2 + peer-registry: fleet-42-r7 # provisioning + revocations, see 05 +``` + +- Everything is **semver-versioned individually**, and the manifest itself carries the fleet version — a lockfile for the whole swarm. +- Images are referenced **by digest**; docks verify signatures and checksums before applying. +- **Rollback is atomic:** re-apply the previous manifest. No per-service drift, ever — a drone either runs release 3.4.1 in full or 3.4.0 in full. +- The manifest is what the simulation farm certifies: regression scenarios run against the exact manifest that will ship. + +## Delivery to the drones + +1. CI publishes a release manifest; the base-station **registry mirror** pulls all referenced artifacts inside the air gap. +2. Docked drones fetch the manifest, verify digests/signatures ([05](05-network-security.md)), stage the new Compose bundle, and switch on the next boot cycle. +3. **Never mid-flight.** Updates are a dock-only operation by construction — the update endpoint does not exist in the flight radio profile. + +## Developer experience + +- **Dev Containers** define the full toolchain (Python data tooling, DuckDB, compose, linters) — identical on any machine, onboarding in minutes. +- **Linux dev VMs** are provisioned from the same configuration standard (cloud-init + the provisioning role), so "my VM" and "the CI runner" cannot diverge. +- The virtual swarm ([`simulator/`](../simulator/)) is the daily inner loop: change the writer, `docker compose up`, query the output Parquet, done.