swarm-house/docs/06-environments.md

# 06 — Environments

Three infrastructures, one data contract. The layout, schemas, and pipelines are identical everywhere; only scale and lifetime differ.

> **Decisions:** [ADR-0006 — IaC boundaries](adr/ADR-0006-iac-boundaries.md), [ADR-0008 — runtime data paths](adr/ADR-0008-runtime-data-paths.md).

```mermaid
graph LR
    subgraph dev [3 — Dev and simulation]
        SIM["virtual drone fleet<br/>(simulator, Compose)"]
        LAP["engineer laptop<br/>Dev Container"]
        SLICE["T4: data slices"]
    end
    subgraph fleet [1 — Fleet, in flight]
        D1["drone: Compose data plane<br/>T0 hot + T1 NVMe + T2 MinIO"]
        D2["drone …"]
    end
    subgraph ground [2 — Ground, on-prem]
        DOCK["base station docks"]
        DWH["T3 warehouse<br/>object store + Parquet/DuckDB"]
        K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
        TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
    end

    D1 <-.->|mesh| D2
    D1 -->|"offload: mirror + audit"| DOCK
    DOCK --> DWH
    DWH -->|"slices for development"| SLICE
    SIM -->|"same pipeline, synthetic flights"| SLICE
    DWH --> TRAIN
    TRAIN -->|"versioned weights"| K3S
```

## 1 — Fleet infrastructure (on the drones)

- Docker Compose under systemd; the full stack from [02 — Architecture](02-architecture.md).
- Storage floors T0–T2. Nothing here requires ground contact during a mission.
- Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.

## 2 — Ground infrastructure (on-prem)

The permanent installation. This **is** allowed to be a cluster — links are wired and stable here, so k3s earns its keep:

| Component | Runs on | Role |
| --- | --- | --- |
| **Warehouse (T3)** | Object store (MinIO or equivalent) + Parquet | Every flight of every drone; the system's long-term memory |
| **Base station docks** | Bare metal | Wired offload + integrity audit + drone provisioning ([03](03-data-platform.md), [05](05-network-security.md)) |
| **Registry mirror** | k3s | In-air-gap container registry + artifact storage; the only software source drones ever see |
| **CI runners** | k3s | Multi-arch builds, tests, scans ([11](11-cicd-delivery.md)) |
| **Simulation farm** | k3s | Scaled-up virtual swarm runs for regression and pre-mission validation |
| **Dashboards** | k3s | Observability stack ([07](07-observability.md)) + flight replay tooling |

Replay is the warehouse's flagship feature: because every broadcast, detection, and re-tasking trigger is preserved with nanosecond timestamps in one layout, any mission can be reconstructed step by step — for engineering analysis, for audit/traceability, and as training input.

## 3 — Dev & simulation infrastructure

The environment engineers live in daily — and deliberately the first thing to build (see [08 — Roadmap](08-roadmap.md)), because everything else is validated inside it.

- **Dev Containers** give every engineer the same toolchain in minutes; no snowflake laptops.
- **No debug API** ([04](04-swarm-sync.md)): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free.
- **The virtual drone fleet** ([`simulator/`](../simulator/)) runs the *identical* data plane: same writer, same partitioning, same sealing, same broadcast schema. `DRONE_COUNT=10 docker compose up` is a swarm on a laptop.
- **Scenarios are reproducible by seed:** route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
- **T4 data slices**: a thin tool pulls partition subsets from T3 (`flight=X, drone=Y, hour=Z`) for local work — layout-identical, so every query and pipeline runs unmodified.
- The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).

## IaC boundaries

Infrastructure as code follows the same discipline as the data plane: each tool owns the layer it is actually good at, and nothing owns a layer it can't reach.

| Layer | Tool | Why |
| --- | --- | --- |
| Host preparation (drone bench provisioning, sim cluster creation) | **Ansible** ([`infra/ansible/`](../infra/ansible/)) | Idempotent, agentless, works over the same SSH channel the platform already trusts. Keys, forced commands, WireGuard, Compose bundles — all host-level state. |
| Declared workloads (simulated fleet, ground warehouse) | **Terraform** ([`infra/terraform/`](../infra/terraform/)) | The ground segment is stationary, long-lived, and cluster-shaped — classic Terraform territory. Fleet size, offload schedule, and explorer endpoints are variables, not YAML edits. |
| Ongoing ground configuration | **Flux** ([`infra/gitops/`](../infra/gitops/)) | Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps. |
| On-board runtime | **Compose bundle from the fleet release manifest** | Terraform is deliberately *not* on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery ([11](11-cicd-delivery.md)). |

Shared local data paths (same layout everywhere): **`var/t1/`** (live lake, T1) and **`var/t3/`** (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree.

Two Terraform modules, two states, one deliberate split: [`sim-env`](../infra/terraform/sim-env/) (the disposable virtual fleet) and [`ground`](../infra/terraform/ground/) (the warehouse that must survive every `destroy` of the fleet). They never share a lifecycle.

### k3d: the ground segment as an executable miniature

`ansible-playbook infra/ansible/sim-cluster.yml` creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and `terraform apply` populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The whole diagram at the top of this page runs on one workstation, and the prototype's **live mode** renders real drone positions straight from the explorer's read-only SQL API.

### Next fidelity step: microVMs

The k3d fleet shares one kernel and one network namespace tree — good enough to exercise the data plane, not good enough to demo the *peer* plane (real SSH sessions between isolated machines, WireGuard handshakes, packet loss injection). The next step, when that fidelity is needed, is one **Firecracker microVM per drone**: a real kernel, a real network interface, and the actual `authorized_keys` forced-command path between virtual drones — on any KVM-capable workstation, still with no cloud dependency. The process-level simulator stays as the fast inner loop; microVMs become the pre-flight integration rig.

## Why identical layouts matter (the payoff)

| Operation | What it costs with one layout everywhere |
| --- | --- |
| Flight offload | `mc mirror` / `rsync` + audit — no transform step |
| Debugging a field issue | Pull the flight slice, replay locally, same queries |
| Validating a pipeline change | Run simulator, diff Parquet outputs |
| Training data prep | Read T3 directly, no export pipeline |
| Onboarding an engineer | One Dev Container, one `docker compose up` |