Files

T

eSlider 8f98a0c994 feat(simulator): Grafana monitoring over generated Parquet

Monitoring compose profile: dependency-free Prometheus exporter that
scans the Parquet output with DuckDB (rows, detections, pose frames,
battery, RSSI, link distance, bytes on disk), Prometheus scrape config
and a provisioned Grafana dashboard. Demonstrates the observability
doctrine: fleet statistics derived from the data platform itself.

2026-07-08 13:27:53 +01:00

3.6 KiB

Raw Blame History

07 — Observability

The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.

Two kinds of signals, one storage

Signal	Examples	Where it goes
Hardware / environment telemetry	Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment	The regular `telemetry` dataset — it is sensor data (03)
Platform operational metrics	Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames	`telemetry` dataset, `sensor=platform` — same pipeline, zero extra moving parts on board

The deliberate trick: on board there is no separate metrics stack. A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window.

Logs

Services log structured JSON to journald (Compose default); a lightweight shipper appends them into dataset=logs/flight=…/drone=…/service=… Parquet partitions on the same write path.
Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles.

On the ground: the standard stack

Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs:

graph LR
    subgraph flight [In flight]
        PM["platform metrics<br/>= sensor stream"]
        LG["structured logs<br/>= Parquet partitions"]
    end
    subgraph ground [Ground k3s]
        DWH["T3 warehouse"]
        PR["Prometheus<br/>(live: docks, CI, sim farm)"]
        LK["Loki<br/>(live logs, dev plane)"]
        GF["Grafana<br/>dashboards + replay"]
        DK["DuckDB dashboards<br/>flight post-mortems"]
    end
    PM --> DWH
    LG --> DWH
    DWH --> DK
    PR --> GF
    LK --> GF
    DK --> GF

Prometheus/Grafana/Loki cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
Flight post-mortems query the warehouse directly with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.

Health questions this design answers cheaply

Question	Source
Did any drone approach its NVMe quota last mission?	`sensor=platform` quota gauges, one SQL query
How stale was peer state for drone N during the partition at minute 14?	Broadcast log in `dataset=state` (sent + received both recorded)
Which link degraded first, and was it distance or interference?	RSSI telemetry joined with pose distance
Is the new writer version flushing slower on ARM64?	CI simulation runs emit the same metrics; diff across fleet releases

A working miniature of this doctrine ships in simulator/: the monitoring Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and a provisioned Grafana dashboard — fleet statistics derived from the data platform itself, with no agent on the (virtual) drones.

Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.

3.6 KiB Raw Blame History