docs: WireGuard vs SSH, Flux GitOps, CONTRIBUTING with TDD-first
Document why WireGuard and SSH are complementary layers, add Flux boundaries for the ground segment, shared var/ data paths in docs/06, and a contributor guide with the required test-first workflow.
This commit is contained in:
@@ -36,6 +36,21 @@ graph TB
|
||||
- **RSSI is data.** Per-peer signal strength is captured into `telemetry` like any other sensor — it feeds relative-positioning estimates and post-flight link-quality analysis, and it is free.
|
||||
- Electromagnetic environment sensing (whatever the radio can observe) is likewise recorded — cheap in flight, valuable in replay.
|
||||
|
||||
## Why not SSH alone?
|
||||
|
||||
SSH and WireGuard solve different problems. SSH is the **authorization layer** for narrow, point-to-point operations (bulk sync, read-only SQL). WireGuard is the **membership and transport encryption layer** for everything on the mesh, including traffic SSH cannot carry.
|
||||
|
||||
| Concern | SSH alone | WireGuard + SSH (this design) |
|
||||
| --- | --- | --- |
|
||||
| Pose broadcast (UDP, ~5 Hz, compact frame) | Cannot carry subnet broadcast; would need a separate app protocol | Encrypted inside the mesh overlay; any peer on the tunnel can receive |
|
||||
| Broadcast confidentiality without WG | Payloads traverse the radio **in cleartext** — any receiver in range reads swarm positions | Only provisioned fleet members complete a handshake; outsiders see noise |
|
||||
| Lossy ad-hoc links | TCP head-of-line blocking; sessions stall and reconnect on packet loss | UDP transport survives loss and roaming; SSH sessions **over** WG are more stable than SSH directly on Wi-Fi |
|
||||
| Network membership | Any host that reaches the port gets an SSH banner — a probe surface | Devices without a provisioned key get **no handshake response**; the port is effectively silent |
|
||||
|
||||
**Rejected alternative:** encrypt each broadcast frame in application code with a fleet-wide AEAD key. That reinvents a crypto layer the OS already provides, still leaves TCP head-of-line blocking for peer queries, and does not answer the membership question (who is allowed on the mesh at all).
|
||||
|
||||
Keep both layers: WireGuard for *who is on the network and whether the wire is readable*; SSH forced commands for *what a peer is allowed to do once connected* ([04](04-swarm-sync.md)).
|
||||
|
||||
## Key lifecycle
|
||||
|
||||
| Phase | Action |
|
||||
|
||||
@@ -61,6 +61,29 @@ The environment engineers live in daily — and deliberately the first thing to
|
||||
- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
|
||||
- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).
|
||||
|
||||
## IaC boundaries
|
||||
|
||||
Infrastructure as code follows the same discipline as the data plane: each tool owns the layer it is actually good at, and nothing owns a layer it can't reach.
|
||||
|
||||
| Layer | Tool | Why |
|
||||
| --- | --- | --- |
|
||||
| Host preparation (drone bench provisioning, sim cluster creation) | **Ansible** ([`infra/ansible/`](../infra/ansible/)) | Idempotent, agentless, works over the same SSH channel the platform already trusts. Keys, forced commands, WireGuard, Compose bundles — all host-level state. |
|
||||
| Declared workloads (simulated fleet, ground warehouse) | **Terraform** ([`infra/terraform/`](../infra/terraform/)) | The ground segment is stationary, long-lived, and cluster-shaped — classic Terraform territory. Fleet size, offload schedule, and explorer endpoints are variables, not YAML edits. |
|
||||
| Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. |
|
||||
| On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). |
|
||||
|
||||
Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, T1) and **`var/t3/`** (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree.
|
||||
|
||||
Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle.
|
||||
|
||||
### k3d: the ground segment as an executable miniature
|
||||
|
||||
`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API.
|
||||
|
||||
### Next fidelity step: microVMs
|
||||
|
||||
The k3d fleet shares one kernel and one network namespace tree — good enough to exercise the data plane, not good enough to demo the *peer* plane (real SSH sessions between isolated machines, WireGuard handshakes, packet loss injection). The next step, when that fidelity is needed, is one **Firecracker microVM per drone**: a real kernel, a real network interface, and the actual `authorized_keys` forced-command path between virtual drones — on any KVM-capable workstation, still with no cloud dependency. The process-level simulator stays as the fast inner loop; microVMs become the pre-flight integration rig.
|
||||
|
||||
## Why identical layouts matter (the payoff)
|
||||
|
||||
| Operation | What it costs with one layout everywhere |
|
||||
|
||||
@@ -97,6 +97,39 @@ config:
|
||||
2. Docked drones fetch the manifest, verify digests/signatures ([05](05-network-security.md)), stage the new Compose bundle, and switch on the next boot cycle.
|
||||
3. **Never mid-flight.** Updates are a dock-only operation by construction — the update endpoint does not exist in the flight radio profile.
|
||||
|
||||
## GitOps on the ground segment
|
||||
|
||||
The fleet release manifest handles **drones** (atomic, digest-pinned, dock-only). The **ground k3s segment** uses a different tool: [Flux](https://fluxcd.io/) reconciles long-lived configuration from git — dashboard bundles, Prometheus rules, offload schedules, policy labels — without reprovisioning PVCs or re-running full Terraform applies for every label change.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph git [Git, on-prem]
|
||||
REPO["service repos"]
|
||||
GITOPS["infra/gitops/<br/>ground overlay"]
|
||||
end
|
||||
subgraph ground_k3s [Ground k3s]
|
||||
FLUX["Flux controllers"]
|
||||
WH["warehouse workloads<br/>(Terraform-provisioned)"]
|
||||
CFG["policy ConfigMaps<br/>dashboard bundles"]
|
||||
end
|
||||
subgraph drones [Fleet]
|
||||
MAN["fleet release manifest"]
|
||||
DOCK["dock: verify + apply"]
|
||||
end
|
||||
REPO --> CI
|
||||
GITOPS --> FLUX
|
||||
FLUX --> CFG
|
||||
MAN --> DOCK
|
||||
```
|
||||
|
||||
| Segment | Delivery mechanism | Reconciler in production? |
|
||||
| --- | --- | --- |
|
||||
| Drones | Fleet release manifest via registry mirror | **No** — no API server mid-flight; dock applies once |
|
||||
| Ground k3s (warehouse, dashboards, CI runners) | Terraform for shape + **Flux** for ongoing config | **Yes** — stationary cluster, wired network, git is the source of truth |
|
||||
| Dev simulation (k3d) | Ansible creates cluster; Terraform applies workloads; Flux CRs installed from [`infra/gitops/`](../infra/gitops/) | Yes (miniature of ground) |
|
||||
|
||||
Boundary: **Terraform provisions** (namespaces, PVCs, Deployments, CronJobs); **Flux reconciles** policy and observability overlays on top. Neither replaces the fleet manifest on the drone.
|
||||
|
||||
## Developer experience
|
||||
|
||||
- **Dev Containers** define the full toolchain (Python data tooling, DuckDB, compose, linters) — identical on any machine, onboarding in minutes.
|
||||
|
||||
Reference in New Issue
Block a user