swarm-house/docs/06-environments.md

06 — Environments

Three infrastructures, one data contract. The layout, schemas, and pipelines are identical everywhere; only scale and lifetime differ.

Decisions: ADR-0006 — IaC boundaries, ADR-0008 — runtime data paths.

graph LR
    subgraph dev [3 — Dev and simulation]
        SIM["virtual drone fleet<br/>(simulator, Compose)"]
        LAP["engineer laptop<br/>Dev Container"]
        SLICE["[T4](00-glossary.md#t4--dev): data slices"]
    end
    subgraph fleet [1 — Fleet, in flight]
        D1["drone: Compose data plane<br/>[T0](00-glossary.md#t0--hot) hot + [T1](00-glossary.md#t1--warm) NVMe + [T2](00-glossary.md#t2--shared) MinIO"]
        D2["drone …"]
    end
    subgraph ground [2 — Ground, on-prem]
        DOCK["base station docks"]
        DWH["[T3](00-glossary.md#t3--warehouse) warehouse<br/>object store + Parquet/DuckDB"]
        K3S["k3s: CI runners, registry mirror,<br/>sim farm, dashboards"]
        TRAIN["model training (out of scope)<br/>reads T3, ships weights"]
    end

    D1 <-.->|mesh| D2
    D1 -->|"offload: mirror + audit"| DOCK
    DOCK --> DWH
    DWH -->|"slices for development"| SLICE
    SIM -->|"same pipeline, synthetic flights"| SLICE
    DWH --> TRAIN
    TRAIN -->|"versioned weights"| K3S

1 — Fleet infrastructure (on the drones)

• Docker Compose under systemd; the full stack from 02 — Architecture.
• Storage floors T0–T2. Nothing here requires ground contact during a mission.
• Receives new software only while docked, from the registry mirror, pinned by the fleet release manifest.

2 — Ground infrastructure (on-prem)

The permanent installation. This is allowed to be a cluster — links are wired and stable here, so k3s earns its keep:

Component	Runs on	Role
Warehouse (T3)	Object store (MinIO or equivalent) + Parquet	Every flight of every drone; the system's source of truth and long-term memory
Base station docks	Bare metal	Wired offload + integrity audit + drone provisioning (03, 05)
Registry mirror	k3s	In-air-gap container registry + artifact storage; the only software source drones ever see
CI runners	k3s	Multi-arch builds, tests, scans (11)
Simulation farm	k3s	Scaled-up virtual swarm runs for regression and pre-mission validation
Dashboards	k3s	Observability stack (07) + flight replay tooling

Replay is the warehouse's flagship feature: because every broadcast, detection, and re-tasking trigger is preserved with nanosecond timestamps in one layout, any mission can be reconstructed step by step — for engineering analysis, for audit/traceability, and as training input.

3 — Dev & simulation infrastructure

The environment engineers live in daily — and deliberately the first thing to build (see 08 — Roadmap), because everything else is validated inside it.

• Dev Containers give every engineer the same toolchain in minutes; no snowflake laptops.
• No debug API (04): bench access to a drone's data goes through the same read-only SQL-over-SSH wrapper the peers use in flight — same permissions, same statement gate, same columnar output. Debugging exercises the production path instead of a parallel one, so the query channel is regression-tested every working day for free.
• The virtual drone fleet (simulator/) runs the identical data plane: same writer, same partitioning, same sealing, same broadcast schema. DRONE_COUNT=10 docker compose up is a swarm on a laptop.
• Scenarios are reproducible by seed: route shapes, sensor noise, drone dropouts, link losses are all parameterized — a bug report is a seed + config, not a war story.
• T4 data slices: a thin tool pulls partition subsets from T3 (flight=X, drone=Y, hour=Z) for local work — layout-identical, so every query and pipeline runs unmodified.
• The same simulator images scale out on the ground k3s farm for CI regression runs: every merge request replays canonical scenarios and asserts on the resulting Parquet output (row counts, coverage, staleness budgets).

IaC boundaries

Infrastructure as code follows the same discipline as the data plane: each tool owns the layer it is actually good at, and nothing owns a layer it can't reach.

Layer	Tool	Why
Host preparation (drone bench provisioning, sim cluster creation)	Ansible (infra/ansible/)	Idempotent, agentless, works over the same SSH channel the platform already trusts. Keys, forced commands, WireGuard, Compose bundles — all host-level state.
Declared workloads (simulated fleet, ground warehouse)	Terraform (infra/terraform/)	The ground segment is stationary, long-lived, and cluster-shaped — classic Terraform territory. Fleet size, offload schedule, and explorer endpoints are variables, not YAML edits.
Ongoing ground configuration	Flux (infra/gitops/)	Policy labels, dashboard bundles, offload knobs — reconciled from git without reprovisioning PVCs. Drones stay on the fleet manifest, not GitOps.
On-board runtime	Compose bundle from the fleet release manifest	Terraform is deliberately not on the drone: there is no API server to reconcile against mid-flight, and the release manifest already gives atomic, versioned, rollback-able delivery (11).

Shared local data paths (same layout everywhere): var/t1/ (live lake, T1) and var/t3/ (warehouse, T3). Compose, k3d, and DuckDB on the host all read the same Parquet tree.

Two Terraform modules, two states, one deliberate split: sim-env (the disposable virtual fleet) and ground (the warehouse that must survive every destroy of the fleet). They never share a lifecycle.

k3d: the ground segment as an executable miniature

ansible-playbook infra/ansible/sim-cluster.yml creates a local k3d cluster (k3s in Docker — the same distribution as the real ground segment) and terraform apply populates it: virtual drones as StatefulSet replicas over a shared Parquet lake, the exporter/Prometheus/Grafana stack, the data-plane explorer, MinIO, and the T1→T3 offload CronJob. The sim CronJob uses cp -ru plus mc mirror as a shortcut; production dock offload uses rsync or mc mirror with checksum audit (03). The whole diagram at the top of this page runs on one workstation, and the prototype's live mode renders real drone positions straight from the explorer's read-only SQL API.

Next fidelity step: microVMs

The k3d fleet shares one kernel and one network namespace tree — good enough to exercise the data plane, not good enough to demo the peer plane (real SSH sessions between isolated machines, WireGuard handshakes, packet loss injection). The next step, when that fidelity is needed, is one Firecracker microVM per drone: a real kernel, a real network interface, and the actual authorized_keys forced-command path between virtual drones — on any KVM-capable workstation, still with no cloud dependency. The process-level simulator stays as the fast inner loop; microVMs become the pre-flight integration rig.

Why identical layouts matter (the payoff)

Operation	What it costs with one layout everywhere
Flight offload	rsync / mc mirror + audit — no transform step (03)
Debugging a field issue	Pull the flight slice, replay locally, same queries
Validating a pipeline change	Run simulator, diff Parquet outputs
Training data prep	Read T3 directly, no export pipeline
Onboarding an engineer	One Dev Container, one docker compose up