docs: SSH-based bulk sync and peer queries, spatial extension
Replace object-store replication with rsync over persistent multiplexed SSH (ed25519, pinned known_hosts, forced commands as the only remote API); demote GraphQL to the ground integration standard. Add DuckDB spatial extension for separation audits, perimeter predicates and trajectory analysis in the local planar frame.
This commit is contained in:
@@ -14,6 +14,7 @@ Terms and abbreviations used throughout this proposal.
|
||||
| **Hot / warm / cold tiers** | Storage floors ordered by access latency and retention: in-memory window → local NVMe → ground warehouse |
|
||||
| **Sealing** | Compacting many small streamed Parquet files into one final file per partition once a time window closes |
|
||||
| **UUIDv7** | Time-ordered UUID; sortable by creation time, used as the canonical flight identifier |
|
||||
| **DuckDB spatial** | Extension adding a GEOMETRY type and `ST_*` functions; used for separation audits, perimeter predicates, and trajectory analysis in the local planar frame |
|
||||
| **Schema registry** | Versioned catalog of dataset schemas so producers and consumers can evolve independently |
|
||||
| **ETL** | Extract, transform, load — the pipelines that move and reshape data between tiers |
|
||||
|
||||
@@ -45,6 +46,9 @@ Terms and abbreviations used throughout this proposal.
|
||||
| **On-prem** | Runs on owned hardware; no public cloud involvement |
|
||||
| **Zero trust** | No implicit trust from network position; every peer authenticates with pre-provisioned credentials |
|
||||
| **mTLS** | Mutual TLS — both sides of a connection present certificates |
|
||||
| **ed25519** | Modern elliptic-curve signature scheme; the SSH key type provisioned per device and per operation |
|
||||
| **Forced command** | `command="…"` restriction in `authorized_keys`: the key can execute exactly one predefined operation — the key *is* the API |
|
||||
| **ControlMaster** | SSH connection multiplexing: one persistent authenticated session carries many transfers/commands |
|
||||
| **PKI** | Public key infrastructure — the certificate authority hierarchy behind mTLS |
|
||||
| **WireGuard** | Lightweight encrypted tunnel protocol; basis for the swarm mesh overlay |
|
||||
| **Ad-hoc mesh** | Wireless network formed directly between peers, no fixed infrastructure |
|
||||
|
||||
@@ -55,8 +55,8 @@ graph TB
|
||||
|
||||
- The **event hook** is the on-board "lambda": when the writer lands new *derived* rows (state, detections), it triggers registered actions — broadcast, MinIO upload, or a local mission-logic callback. Nothing polls.
|
||||
- The **state publisher** broadcasts compact position/attitude/detection payloads over the mesh pub/sub (transport analysis in [04 — Swarm sync](04-swarm-sync.md)).
|
||||
- **MinIO** holds derived datasets only, replicated opportunistically between drones.
|
||||
- The **query API** answers ad-hoc peer requests against local DuckDB (the query standard is also in [04](04-swarm-sync.md)).
|
||||
- **Bulk sync** pulls sealed derived partitions from peers over persistent SSH (rsync delta transfer); MinIO remains optional where an S3 API is wanted.
|
||||
- **Peer queries** run against local DuckDB via SSH forced commands — allow-listed query templates, columnar responses (the standard is in [04](04-swarm-sync.md)).
|
||||
|
||||
## Communication planes
|
||||
|
||||
|
||||
@@ -109,6 +109,33 @@ FROM read_parquet('…/sensor=imu/**/*.parquet')
|
||||
GROUP BY second ORDER BY second;
|
||||
```
|
||||
|
||||
## Spatial queries: DuckDB spatial extension
|
||||
|
||||
Positions and detections are inherently spatial, and the [DuckDB spatial extension](https://github.com/duckdb/duckdb-spatial) makes them queryable as geometry without leaving the engine (`INSTALL spatial; LOAD spatial;`). Its current lack of spherical-coordinate support is irrelevant here: the swarm operates in a **local planar mission frame in meters**, which is exactly the geometry model the extension handles best.
|
||||
|
||||
```sql
|
||||
LOAD spatial;
|
||||
|
||||
-- Minimum separation between two drones over a flight (replay safety audit)
|
||||
SELECT min(ST_Distance(ST_Point(a.pos_x, a.pos_y), ST_Point(b.pos_x, b.pos_y))) AS min_sep_m
|
||||
FROM read_parquet('…/dataset=state/flight=…/drone=dr-01/**/*.parquet') a
|
||||
JOIN read_parquet('…/dataset=state/flight=…/drone=dr-02/**/*.parquet') b
|
||||
ON a.ts_ns // 200000000 = b.ts_ns // 200000000; -- align to 200 ms buckets
|
||||
|
||||
-- Detections inside the mission perimeter polygon
|
||||
SELECT cls, count(*)
|
||||
FROM read_parquet('…/dataset=detections/flight=…/**/*.parquet')
|
||||
WHERE ST_Within(ST_Point(obj_x, obj_y), ST_GeomFromText('POLYGON((…))'))
|
||||
GROUP BY cls;
|
||||
|
||||
-- Trajectory as a geometry (export for replay overlays)
|
||||
SELECT drone, ST_MakeLine(list(ST_Point(pos_x, pos_y) ORDER BY ts_ns)) AS path
|
||||
FROM read_parquet('…/dataset=state/flight=…/**/*.parquet')
|
||||
GROUP BY drone;
|
||||
```
|
||||
|
||||
On the ground this powers replay overlays, coverage analysis (did ubiquitous sensing actually cover the area?), and separation audits; on board it is available for cheap geo-predicates (e.g. "detections within R meters of me") with no extra service.
|
||||
|
||||
## Schema management
|
||||
|
||||
- Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
|
||||
|
||||
+17
-5
@@ -69,18 +69,30 @@ graph LR
|
||||
## Two sync mechanisms, deliberately separate
|
||||
|
||||
1. **Fast path — pub/sub (Zenoh):** pose frames and detection events. Fire-and-forget with bounded staleness; consumers keep a peer-state cache.
|
||||
2. **Bulk path — MinIO replication:** sealed `detections`/`state` Parquet partitions replicate opportunistically between on-board MinIO instances when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
|
||||
2. **Bulk path — rsync over persistent SSH:** sealed `detections`/`state` Parquet partitions are pulled opportunistically between drones when links allow. This is how a drone that was out of range catches up on mission history without anyone re-sending events.
|
||||
|
||||
Partition healing is automatic: replication is pull-based, versioned by partition path, and idempotent (same layout = same keys; last-writer-wins is safe because each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
|
||||
Why SSH-based bulk sync over object-store replication:
|
||||
|
||||
- **Identity is already there.** Every drone holds pre-provisioned ed25519 keys and a fixed `known_hosts`/`authorized_keys` set from ground provisioning ([05](05-network-security.md)) — the trust model needs no new machinery.
|
||||
- **One persistent multiplexed session** (`ControlMaster`) per peer costs almost nothing at idle and survives as a single TCP stream; every transfer rides it without new handshakes.
|
||||
- **rsync delta transfer is resumable** across link drops — exactly the failure mode of an ad-hoc mesh — and the shared partition layout makes it trivially incremental: same paths, same files, pull only what is missing.
|
||||
- **Zero extra services** on the flight-critical node. MinIO remains available where an S3 API is genuinely wanted (ground warehouse, and optionally on board), but the in-flight bulk path does not depend on it.
|
||||
|
||||
Partition healing is automatic: replication is pull-based, addressed by partition path, and idempotent (each drone only ever writes its own `drone=` subtree — **no write conflicts by construction**).
|
||||
|
||||
Two deliberate non-features of the SSH channel:
|
||||
|
||||
- **No remote filesystem mounts in flight.** FUSE/sshfs over a lossy mesh inherits NFS hang semantics — processes block uninterruptibly when the link drops. Mounts are fine on the bench and at the dock; in flight, data moves by pull, never by mount.
|
||||
- **No free-form remote execution.** Arbitrary drone-to-drone exec would mean one compromised unit owns the fleet. Instead, every permitted operation is an **SSH forced command** (`command="…"` in `authorized_keys`, one key pair per operation): the key *is* the API. Flexible like a CLI, auditable like an RPC, least-privilege by construction.
|
||||
|
||||
## Peer query standard
|
||||
|
||||
For everything that is not broadcast, drones (and mission services) query each other explicitly. The contract:
|
||||
|
||||
- **Schema-first:** all datasets share the versioned schemas from [03](03-data-platform.md); a query addresses `dataset + partition predicates + column projection + time range`.
|
||||
- **Recommended interface: GraphQL** over the mesh transport — one flexible, introspectable contract for "give me detections of class X since T from your flight" without inventing endpoints per use case. It also becomes the natural standard for the future multi-team integration surface.
|
||||
- **Alternative: Arrow Flight** — far more efficient for bulk columnar transfer, worth adopting on the ground (T3 warehouse serving) where result sets are large; overkill as the in-flight peer protocol.
|
||||
- Responses stream Parquet/Arrow batches, never JSON blobs, so the receiving side lands data straight into its own store.
|
||||
- **In flight — query over forced command:** a peer query is a DuckDB statement (from an allow-listed template set) executed via its dedicated SSH forced command, streaming the result back as Parquet/Arrow over the same multiplexed session. No server process, no new port, no new protocol — and the receiving side lands data straight into its own store.
|
||||
- **On the ground — GraphQL as the integration standard:** the warehouse exposes the same schemas through GraphQL for the multi-team surface, where flexibility and introspection matter more than grams and milliwatts. **Arrow Flight** is the upgrade path for heavy columnar serving from T3 if GraphQL result sizes become the bottleneck.
|
||||
- Responses are always columnar batches (Parquet/Arrow), never JSON blobs.
|
||||
|
||||
## Event-driven end to end
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Zero trust, fully air-gapped, everything provisioned before wheels-up.
|
||||
- **Network position grants nothing.** Being inside the mesh does not authorize anything; every connection authenticates mutually.
|
||||
- **Two layers of cryptography, by design:**
|
||||
1. **WireGuard mesh overlay** — every drone holds the public keys of every fleet member (distributed at provisioning). All swarm traffic runs inside these tunnels; a device without a provisioned key cannot even complete a handshake.
|
||||
2. **mTLS between services** — inside the overlay, service-to-service calls (query API, MinIO replication) present per-service certificates from the fleet PKI. Compromising the network layer alone still yields nothing readable.
|
||||
2. **Authenticated channels inside the overlay** — the bulk-sync and peer-query channel is SSH with **ed25519** keys and pinned `known_hosts` (no first-use prompts, no runtime key acceptance); every permitted operation is a separate key pair bound to a **forced command**, so a stolen key authorizes exactly one narrow action. Services that speak TLS (MinIO, ground APIs) present per-service certificates from the fleet PKI (mTLS). Compromising the network layer alone still yields nothing readable — and compromising one credential yields one operation, not a shell.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
|
||||
Reference in New Issue
Block a user