eSlider
8f98a0c994
Monitoring compose profile: dependency-free Prometheus exporter that scans the Parquet output with DuckDB (rows, detections, pose frames, battery, RSSI, link distance, bytes on disk), Prometheus scrape config and a provisioned Grafana dashboard. Demonstrates the observability doctrine: fleet statistics derived from the data platform itself.
59 lines
3.6 KiB
Markdown
59 lines
3.6 KiB
Markdown
# 07 — Observability
|
|
|
|
The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.
|
|
|
|
## Two kinds of signals, one storage
|
|
|
|
| Signal | Examples | Where it goes |
|
|
| --- | --- | --- |
|
|
| **Hardware / environment telemetry** | Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment | The regular `telemetry` dataset — it is sensor data ([03](03-data-platform.md)) |
|
|
| **Platform operational metrics** | Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames | `telemetry` dataset, `sensor=platform` — same pipeline, zero extra moving parts on board |
|
|
|
|
The deliberate trick: **on board there is no separate metrics stack.** A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window.
|
|
|
|
## Logs
|
|
|
|
- Services log structured JSON to journald (Compose default); a lightweight shipper appends them into `dataset=logs/flight=…/drone=…/service=…` Parquet partitions on the same write path.
|
|
- Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles.
|
|
|
|
## On the ground: the standard stack
|
|
|
|
Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs:
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph flight [In flight]
|
|
PM["platform metrics<br/>= sensor stream"]
|
|
LG["structured logs<br/>= Parquet partitions"]
|
|
end
|
|
subgraph ground [Ground k3s]
|
|
DWH["T3 warehouse"]
|
|
PR["Prometheus<br/>(live: docks, CI, sim farm)"]
|
|
LK["Loki<br/>(live logs, dev plane)"]
|
|
GF["Grafana<br/>dashboards + replay"]
|
|
DK["DuckDB dashboards<br/>flight post-mortems"]
|
|
end
|
|
PM --> DWH
|
|
LG --> DWH
|
|
DWH --> DK
|
|
PR --> GF
|
|
LK --> GF
|
|
DK --> GF
|
|
```
|
|
|
|
- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
|
|
- **Flight post-mortems query the warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.
|
|
|
|
## Health questions this design answers cheaply
|
|
|
|
| Question | Source |
|
|
| --- | --- |
|
|
| Did any drone approach its NVMe quota last mission? | `sensor=platform` quota gauges, one SQL query |
|
|
| How stale was peer state for drone N during the partition at minute 14? | Broadcast log in `dataset=state` (sent + received both recorded) |
|
|
| Which link degraded first, and was it distance or interference? | RSSI telemetry joined with pose distance |
|
|
| Is the new writer version flushing slower on ARM64? | CI simulation runs emit the same metrics; diff across fleet releases |
|
|
|
|
A working miniature of this doctrine ships in [`simulator/`](../simulator/): the `monitoring` Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and a provisioned Grafana dashboard — fleet statistics derived from the data platform itself, with no agent on the (virtual) drones.
|
|
|
|
Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.
|