# 07 — Observability

The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.

## Two kinds of signals, one storage

| Signal | Examples | Where it goes |
| --- | --- | --- |
| **Hardware / environment telemetry** | Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment | The regular `telemetry` dataset — it is sensor data ([03](03-data-platform.md)) |
| **Platform operational metrics** | Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames | `telemetry` dataset, `sensor=platform` — same pipeline, zero extra moving parts on board |

The deliberate trick: **on board there is no separate metrics stack.** A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window.

## Logs

- Services log structured JSON to journald (Compose default); a lightweight shipper appends them into `dataset=logs/flight=…/drone=…/service=…` Parquet partitions on the same write path.
- Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles.

## On the ground: the standard stack

Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs:

```mermaid
graph LR
    subgraph flight [In flight]
        PM["platform metrics<br/>= sensor stream"]
        LG["structured logs<br/>= Parquet partitions"]
    end
    subgraph ground [Ground k3s]
        DWH["T3 warehouse"]
        PR["Prometheus<br/>(live: docks, CI, sim farm)"]
        LK["Loki<br/>(live logs, dev plane)"]
        GF["Grafana<br/>dashboards + replay"]
        DK["DuckDB dashboards<br/>flight post-mortems"]
    end
    PM --> DWH
    LG --> DWH
    DWH --> DK
    PR --> GF
    LK --> GF
    DK --> GF
```

- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.

## Health questions this design answers cheaply

| Question | Source |
| --- | --- |
| Did any drone approach its NVMe quota last mission? | `sensor=platform` quota gauges, one SQL query |
| How stale was peer state for drone N during the partition at minute 14? | Broadcast log in `dataset=state` (sent + received both recorded) |
| Which link degraded first, and was it distance or interference? | RSSI telemetry joined with pose distance |
| Is the new writer version flushing slower on ARM64? | CI simulation runs emit the same metrics; diff across fleet releases |

A working miniature of this doctrine ships in [`simulator/`](../simulator/): the `monitoring` Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and two provisioned Grafana dashboards:

| Dashboard | Focus |
| --- | --- |
| **Swarm Fleet — generated data** | Row counts, detections, pose frames, Parquet bytes, battery, RSSI |
| **Swarm Platform — CPU & scan health** | Node CPU/memory (node-exporter), exporter/explorer scan durations, flight-window gauge |

The exporter and explorer **scan only the most recent flight partitions** (`METRIC_FLIGHT_WINDOW`, default 5) and cache tree/views — otherwise CPU climbs as every pod restart appends a new `flight=` tree to the shared lake. Drones in k3d set `KEEP_ALIVE=1` after sealing so they idle instead of exiting and spawning another flight.

Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.