swarm-house/docs/07-observability.md

07 — Observability

The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.

Decision: ADR-0009 — bound observability lake scans to a flight window and cache.

Two kinds of signals, one storage

Signal	Examples	Where it goes
Hardware / environment telemetry	Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment	The regular telemetry dataset — it is sensor data (03)
Platform operational metrics	Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames	telemetry dataset, sensor=platform — same pipeline, zero extra moving parts on board

The deliberate trick: on board there is no separate metrics stack. A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window.

Logs

• Services log structured JSON to journald (Compose default); a lightweight shipper appends them into dataset=logs/flight=…/drone=…/service=… Parquet partitions on the same write path.
• Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles.

On the ground: the standard stack

Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs:

graph LR
    subgraph flight [In flight]
        PM["platform metrics<br/>= sensor stream"]
        LG["structured logs<br/>= Parquet partitions"]
    end
    subgraph ground [Ground k3s]
        DWH["T3 warehouse"]
        PR["Prometheus<br/>(live: docks, CI, sim farm)"]
        LK["Loki<br/>(live logs, dev plane)"]
        GF["Grafana<br/>dashboards + replay"]
        DK["DuckDB dashboards<br/>flight post-mortems"]
    end
    PM --> DWH
    LG --> DWH
    DWH --> DK
    PR --> GF
    LK --> GF
    DK --> GF

• Prometheus/Grafana/Loki cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
• Flight post-mortems query the warehouse directly with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.

Health questions this design answers cheaply

Question	Source
Did any drone approach its NVMe quota last mission?	sensor=platform quota gauges, one SQL query
How stale was peer state for drone N during the partition at minute 14?	Broadcast log in dataset=state (sent + received both recorded)
Which link degraded first, and was it distance or interference?	RSSI telemetry joined with pose distance
Is the new writer version flushing slower on ARM64?	CI simulation runs emit the same metrics; diff across fleet releases

A working miniature of this doctrine ships in simulator/: the monitoring Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and two provisioned Grafana dashboards:

Dashboard	Focus
Swarm Fleet — generated data	Row counts, detections, pose frames, Parquet bytes, battery, RSSI
Swarm Platform — CPU & scan health	Node CPU/memory (node-exporter), exporter/explorer scan durations, flight-window gauge

The exporter and explorer scan only the most recent flight partitions (METRIC_FLIGHT_WINDOW, default 5) and cache tree/views — otherwise CPU climbs as every pod restart appends a new flight= tree to the shared lake. Drones in k3d set KEEP_ALIVE=1 after sealing so they idle instead of exiting and spawning another flight.

Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.