eSlider
5f6fd8462c
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
68 lines
4.3 KiB
Markdown
68 lines
4.3 KiB
Markdown
# 07 — Observability
|
|
|
|
The platform watches itself with the same discipline it applies to sensor data — and largely through the same pipeline.
|
|
|
|
> **Decision:** [ADR-0009 — bound observability lake scans to a flight window and cache](adr/ADR-0009-bounded-lake-scans.md).
|
|
|
|
## Two kinds of signals, one storage
|
|
|
|
| Signal | Examples | Where it goes |
|
|
| --- | --- | --- |
|
|
| **Hardware / environment telemetry** | Temperatures, battery, barometer, accelerometer, LiDAR health, RSSI per peer, electromagnetic environment | The regular `telemetry` dataset — it is sensor data ([03](03-data-platform.md)) |
|
|
| **Platform operational metrics** | Ingest rates, writer flush latency, sealing backlog, NVMe quota pressure, broadcast staleness per peer, replication lag, dropped frames | `telemetry` dataset, `sensor=platform` — same pipeline, zero extra moving parts on board |
|
|
|
|
The deliberate trick: **on board there is no separate metrics stack.** A Prometheus server per drone would be dead weight; instead, platform metrics are just another sensor stream — compressed, partitioned, offloaded, and replayable like everything else. In-flight health decisions (quota pressure, degraded links) are made locally by the services themselves against their own T0 window.
|
|
|
|
## Logs
|
|
|
|
- Services log structured JSON to journald (Compose default); a lightweight shipper appends them into `dataset=logs/flight=…/drone=…/service=…` Parquet partitions on the same write path.
|
|
- Log volume is bounded by severity-based sampling in flight; full debug logging exists only in dev/bench profiles.
|
|
|
|
## On the ground: the standard stack
|
|
|
|
Ground infrastructure (k3s) runs the familiar trio, fed after offload and live during bench/simulation runs:
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph flight [In flight]
|
|
PM["platform metrics<br/>= sensor stream"]
|
|
LG["structured logs<br/>= Parquet partitions"]
|
|
end
|
|
subgraph ground [Ground k3s]
|
|
DWH["[T3](00-glossary.md#t3--warehouse) warehouse"]
|
|
PR["Prometheus<br/>(live: docks, CI, sim farm)"]
|
|
LK["Loki<br/>(live logs, dev plane)"]
|
|
GF["Grafana<br/>dashboards + replay"]
|
|
DK["DuckDB dashboards<br/>flight post-mortems"]
|
|
end
|
|
PM --> DWH
|
|
LG --> DWH
|
|
DWH --> DK
|
|
PR --> GF
|
|
LK --> GF
|
|
DK --> GF
|
|
```
|
|
|
|
- **Prometheus/Grafana/Loki** cover everything that is alive on the ground: docks, registry, CI runners, warehouse, simulation farm — plus drones on the bench through the dev plane.
|
|
- **Flight post-mortems query the [source of truth](00-glossary.md#source-of-truth) warehouse directly** with DuckDB: staleness budgets vs actuals, link quality vs distance (RSSI), sealing backlog vs ingest rate — per flight, per drone, per minute.
|
|
|
|
## Health questions this design answers cheaply
|
|
|
|
| Question | Source |
|
|
| --- | --- |
|
|
| Did any drone approach its NVMe quota last mission? | `sensor=platform` quota gauges, one SQL query |
|
|
| How stale was peer state for drone N during the partition at minute 14? | Broadcast log in `dataset=state` (sent + received both recorded) |
|
|
| Which link degraded first, and was it distance or interference? | RSSI telemetry joined with pose distance |
|
|
| Is the new writer version flushing slower on ARM64? | CI simulation runs emit the same metrics; diff across fleet releases |
|
|
|
|
A working miniature of this doctrine ships in [`simulator/`](../simulator/): the `monitoring` Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and two provisioned Grafana dashboards:
|
|
|
|
| Dashboard | Focus |
|
|
| --- | --- |
|
|
| **Swarm Fleet — generated data** | Row counts, detections, pose frames, Parquet bytes, battery, RSSI |
|
|
| **Swarm Platform — CPU & scan health** | Node CPU/memory (node-exporter), exporter/explorer scan durations, flight-window gauge |
|
|
|
|
The exporter and explorer **scan only the most recent flight partitions** (`METRIC_FLIGHT_WINDOW`, default 5) and cache tree/views — otherwise CPU climbs as every pod restart appends a new `flight=` tree to the shared lake. Drones in k3d set `KEEP_ALIVE=1` after sealing so they idle instead of exiting and spawning another flight.
|
|
|
|
Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.
|