eSlider/swarm-house
1
0
Fork 0
Code Issues 2 Pull Requests 3 Actions Packages Projects 1 Releases 13 Wiki Activity

perf: bound lake scans and add platform Grafana dashboard
CI & Release / Verify simulator (push) Successful in 11s
Details
CI & Release / Trivy scan (push) Successful in 9s
Details
CI & Release / Semantic Release (push) Successful in 5s
Details

Browse Source 
Explorer and exporter were scanning the full Parquet lake every 1–5s
(~2000 files, 200MB+), driving ~2.2 CPU cores. Limit metrics to the
last N flights, cache DuckDB views and the partition tree, slow live
polls to 2s, and keep drones alive after seal to stop restart churn.

Add node-exporter, scan-duration metrics, and a Swarm Platform Grafana
dashboard for node CPU/memory and scan health.
This commit is contained in:
eSlider
2026-07-08 21:19:53 +01:00
parent bcd956d11a
commit ea062af86a
11 changed files with 478 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/07-observability.md
+8 -1
View File
@@ -53,6 +53,13 @@ graph LR
| Which link degraded first, and was it distance or interference? | RSSI telemetry joined with pose distance |
| Is the new writer version flushing slower on ARM64? | CI simulation runs emit the same metrics; diff across fleet releases |
A working miniature of this doctrine ships in [`simulator/`](../simulator/): the `monitoring` Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and a provisioned Grafana dashboard — fleet statistics derived from the data platform itself, with no agent on the (virtual) drones.
A working miniature of this doctrine ships in [`simulator/`](../simulator/): the `monitoring` Compose profile runs a DuckDB-based exporter over the generated Parquet plus Prometheus and two provisioned Grafana dashboards:
| Dashboard | Focus |
| --- | --- |
| **Swarm Fleet — generated data** | Row counts, detections, pose frames, Parquet bytes, battery, RSSI |
| **Swarm Platform — CPU & scan health** | Node CPU/memory (node-exporter), exporter/explorer scan durations, flight-window gauge |
The exporter and explorer **scan only the most recent flight partitions** (`METRIC_FLIGHT_WINDOW`, default 5) and cache tree/views — otherwise CPU climbs as every pod restart appends a new `flight=` tree to the shared lake. Drones in k3d set `KEEP_ALIVE=1` after sealing so they idle instead of exiting and spawning another flight.
Alerting on the ground follows standard practice (Prometheus alert rules for infrastructure, CI gates for regression in simulated staleness/throughput budgets). In flight there is nobody to page — the platform's job is to degrade in the documented order and record everything for the post-mortem.