docs: SSH-based bulk sync and peer queries, spatial extension

Replace object-store replication with rsync over persistent multiplexed
SSH (ed25519, pinned known_hosts, forced commands as the only remote
API); demote GraphQL to the ground integration standard. Add DuckDB
spatial extension for separation audits, perimeter predicates and
trajectory analysis in the local planar frame.
This commit is contained in:
2026-07-08 13:21:47 +01:00
parent c0b2ee806b
commit df78a7c5db
5 changed files with 51 additions and 8 deletions
+4
View File
@@ -14,6 +14,7 @@ Terms and abbreviations used throughout this proposal.
| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse | | **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse |
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes | | **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier | | **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
| **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame |
| **Schema registry** | Versioned catalog of dataset schemas so producers and consumers can evolve independently | | **Schema registry** | Versioned catalog of dataset schemas so producers and consumers can evolve independently |
| **ETL** | Extract, transform, load — the pipelines that move and reshape data between tiers | | **ETL** | Extract, transform, load — the pipelines that move and reshape data between tiers |
@@ -45,6 +46,9 @@ Terms and abbreviations used throughout this proposal.
| **On-prem** | Runs on owned hardware; no public cloud involvement | | **On-prem** | Runs on owned hardware; no public cloud involvement |
| **Zero trust** | No implicit trust from network position; every peer authenticates with pre-provisioned credentials | | **Zero trust** | No implicit trust from network position; every peer authenticates with pre-provisioned credentials |
| **mTLS** | Mutual TLS — both sides of a connection present certificates | | **mTLS** | Mutual TLS — both sides of a connection present certificates |
| **ed25519** | Modern elliptic-curve signature scheme; the SSH key type provisioned per device and per operation |
| **Forced command** | `command="…"` restriction in `authorized_keys`: the key can execute exactly one predefined operation — the key *is* the API |
| **ControlMaster** | SSH connection multiplexing: one persistent authenticated session carries many transfers/commands |
| **PKI** | Public key infrastructure — the certificate authority hierarchy behind mTLS | | **PKI** | Public key infrastructure — the certificate authority hierarchy behind mTLS |
| **WireGuard** | Lightweight encrypted tunnel protocol; basis for the swarm mesh overlay | | **WireGuard** | Lightweight encrypted tunnel protocol; basis for the swarm mesh overlay |
| **Ad-hoc mesh** | Wireless network formed directly between peers, no fixed infrastructure | | **Ad-hoc mesh** | Wireless network formed directly between peers, no fixed infrastructure |
+2 -2
View File
@@ -55,8 +55,8 @@ graph TB
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls. - The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)). - The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
- **MinIO** holds derived datasets only, replicated opportunistically between drones. - **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer); MinIO remains optional where an S3 API is wanted.
- The **query API** answers ad-hoc peer requests against local DuckDB (the query standard is also in [04](04-swarm-sync.md)). - **Peer queries** run against local DuckDB via SSH forced commands — allow-listed query templates, columnar responses (the standard is in [04](04-swarm-sync.md)).
## Communication planes ## Communication planes
+27
View File
@@ -109,6 +109,33 @@ FROM read_parquet('…/sensor=imu/**/*.parquet')
GROUP BY second ORDER BY second; GROUP BY second ORDER BY second;
``` ```
## Spatial queries: DuckDB spatial extension
Positions and detections are inherently spatial, and the [DuckDB spatial extension](https://github.com/duckdb/duckdb-spatial) makes them queryable as geometry without leaving the engine (`INSTALL spatial; LOAD spatial;`). Its current lack of spherical-coordinate support is irrelevant here: the swarm operates in a **local planar mission frame in meters**, which is exactly the geometry model the extension handles best.
```sql
LOAD spatial;
-- Minimum separation between two drones over a flight (replay safety audit)
SELECT min(ST_Distance(ST_Point(a.pos_x, a.pos_y), ST_Point(b.pos_x, b.pos_y))) AS min_sep_m
FROM read_parquet('…/dataset=state/flight=…/drone=dr-01/**/*.parquet') a
JOIN read_parquet('…/dataset=state/flight=…/drone=dr-02/**/*.parquet') b
ON a.ts_ns // 200000000 = b.ts_ns // 200000000; -- align to 200 ms buckets
-- Detections inside the mission perimeter polygon
SELECT cls, count(*)
FROM read_parquet('…/dataset=detections/flight=…/**/*.parquet')
WHERE ST_Within(ST_Point(obj_x, obj_y), ST_GeomFromText('POLYGON((…))'))
GROUP BY cls;
-- Trajectory as a geometry (export for replay overlays)
SELECT drone, ST_MakeLine(list(ST_Point(pos_x, pos_y) ORDER BY ts_ns)) AS path
FROM read_parquet('…/dataset=state/flight=…/**/*.parquet')
GROUP BY drone;
```
On the ground this powers replay overlays, coverage analysis (did ubiquitous sensing actually cover the area?), and separation audits; on board it is available for cheap geo-predicates (e.g. "detections within R meters of me") with no extra service.
## Schema management ## Schema management
- Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)). - Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
+17 -5
View File
@@ -69,18 +69,30 @@ graph LR
## Two sync mechanisms, deliberately separate ## Two sync mechanisms, deliberately separate
1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache. 1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
2. **Bulk path — MinIO replication:** sealed `detections`/`state` Parquet partitions replicate opportunistically between on-board MinIO instances when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events. 2. **Bulk path — rsync over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
Partition healing is automatic: replication is pull-based, versioned by partition path, and idempotent (same layout = same keys; last-writer-wins is safe because each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**). Why SSH-based bulk sync over object-store replication:
- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
Two deliberate non-features of the SSH channel:
- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.
## Peer query standard ## Peer query standard
For everything that is not broadcast, drones (and mission services) query each other explicitly. The contract: For everything that is not broadcast, drones (and mission services) query each other explicitly. The contract:
- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`. - **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`.
- **Recommended interface: GraphQL** over the mesh transport — one flexible, introspectable contract for "give me detections of class X since T from your flight" without inventing endpoints per use case. It also becomes the natural standard for the future multi-team integration surface. - **In flight — query over forced command:** a peer query is a DuckDB statement (from an allow-listed template set) executed via its dedicated SSH forced command, streaming the result back as Parquet/Arrow over the same multiplexed session. No server process, no new port, no new protocol — and the receiving side lands data straight into its own store.
- **Alternative: Arrow Flight** — far more efficient for bulk columnar transfer, worth adopting on the ground (T3 warehouse serving) where result sets are large; overkill as the in-flight peer protocol. - **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if GraphQL result sizes become the bottleneck.
- Responses stream Parquet/Arrow batches, never JSON blobs, so the receiving side lands data straight into its own store. - Responses are always columnar batches (Parquet/Arrow), never JSON blobs.
## Event-driven end to end ## Event-driven end to end
+1 -1
View File
@@ -8,7 +8,7 @@ Zero trust, fully air-gapped, everything provisioned before wheels-up.
- **Network position grants nothing.** Being inside the mesh does not authorize anything; every connection authenticates mutually. - **Network position grants nothing.** Being inside the mesh does not authorize anything; every connection authenticates mutually.
- **Two layers of cryptography, by design:** - **Two layers of cryptography, by design:**
1. **WireGuard mesh overlay** — every drone holds the public keys of every fleet member (distributed at provisioning). All swarm traffic runs inside these tunnels; a device without a provisioned key cannot even complete a handshake. 1. **WireGuard mesh overlay** — every drone holds the public keys of every fleet member (distributed at provisioning). All swarm traffic runs inside these tunnels; a device without a provisioned key cannot even complete a handshake.
2. **mTLS between services** — inside the overlay, service-to-service calls (query API, MinIO replication) present per-service certificates from the fleet PKI. Compromising the network layer alone still yields nothing readable. 2. **Authenticated channels inside the overlay** — the bulk-sync and peer-query channel is SSH with **ed25519** keys and pinned `known_hosts` (no first-use prompts, no runtime key acceptance); every permitted operation is a separate key pair bound to a **forced command**, so a stolen key authorizes exactly one narrow action. Services that speak TLS (MinIO, ground APIs) present per-service certificates from the fleet PKI (mTLS). Compromising the network layer alone still yields nothing readable — and compromising one credential yields one operation, not a shell.
```mermaid ```mermaid
graph TB graph TB