# 07 — Observability

The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.

## Two kinds of signals, one storage

| Signal | Examples | Where it goes |
| --- | --- | --- |
| **Hardware / environment telemetry** | Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment | The regular `telemetry` dataset — it is sensor data ([03](03-data-platform.md)) |
| **Platform operational metrics** | Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames | `telemetry` dataset, `sensor=platform` — same pipeline, zero extra moving parts on board |

The deliberate trick: **on board there is no separate metrics stack.** A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window.

## Logs

- Services log structured JSON to journald (Compose default); a lightweight shipper appends them into `dataset=logs/flight=…/drone=…/service=…` Parquet partitions on the same write path.
- Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles.

## On the ground: the standard stack

Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs:

```mermaid
graph LR
    subgraph flight [In flight]
        PM["platform metrics<br/>= sensor stream"]
        LG["structured logs<br/>= Parquet partitions"]
    end
    subgraph ground [Ground k3s]
        DWH["T3 warehouse"]
        PR["Prometheus<br/>(live: docks, CI, sim farm)"]
        LK["Loki<br/>(live logs, dev plane)"]
        GF["Grafana<br/>dashboards + replay"]
        DK["DuckDB dashboards<br/>flight post-mortems"]
    end
    PM --> DWH
    LG --> DWH
    DWH --> DK
    PR --> GF
    LK --> GF
    DK --> GF
```

- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.

## Health questions this design answers cheaply

| Question | Source |
| --- | --- |
| Did any drone approach its NVMe quota last mission? | `sensor=platform` quota gauges, one SQL query |
| How stale was peer state for drone N during the partition at minute 14? | Broadcast log in `dataset=state` (sent + received both recorded) |
| Which link degraded first, and was it distance or interference? | RSSI telemetry joined with pose distance |
| Is the new writer version flushing slower on ARM64? | CI simulation runs emit the same metrics; diff across fleet releases |

Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.