eSlider
d3ccd6f94e
Add docs/adr/ with a chronological ADR set reconstructed from the project history (no orchestrator, one storage format, derived-state sync, SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root image, runtime data paths, bounded lake scans), an index README, and a template. Distinguish repo-local ADRs from ecosystem-wide ASRs. Expand AGENTS.md with a decision-records section and a runtime/operations guide (service URLs, rebuild workflow, CPU budget knobs). Reference the ADRs from README, CONTRIBUTING, and the affected design docs, and add an open question for T1 partition pruning.
2.2 KiB
2.2 KiB
ADR-0009: Bound observability lake scans to a flight window and cache
Status
Accepted (2026-07-08)
Context
The k3d sim node was running at ~245% CPU. Three causes compounded:
- Explorer rebuilt a DuckDB view over the entire lake on every HTTP request (~2000 Parquet files, 200 MB+) — roughly 1.5 cores.
- Exporter scanned the full lake every 5 s — roughly 0.7 cores.
- Drones exited after sealing a flight; Kubernetes restarted each pod,
and every restart appended a fresh
flight=partition tree, so the lake grew without bound and every scan got more expensive.
The cost is inherent to scanning an append-only lake whose size grows with pod churn, not to any single slow query. Metrics and the live view only ever need recent flights.
Decision
Bound and cache the scans instead of reading the whole lake:
- Flight window. The exporter and explorer scan only the most recent
METRIC_FLIGHT_WINDOWflights (default 5) via a shared helper (simulator/monitoring/lake.py). Older partitions stay on disk but out of the hot path. - Caching. Explorer caches DuckDB views (
VIEW_REFRESH_S=30) and the partition tree (TREE_CACHE_S=15); the exporter scans everySCAN_INTERVAL_S=30instead of 5 s. - Stop restart churn. Drones set
KEEP_ALIVE=1and idle after sealing instead of exiting, so no new flight tree is created per restart. In k3d,IMU_HZ=20and resource limits cap per-drone load; the live prototype polls every 2 s. - Self-observability. Add node-exporter, a
swarm_exporter_scan_duration_secondsmetric, and a Swarm Platform Grafana dashboard (node CPU/memory + scan health) alongside the existing fleet dashboard.
Consequences
- Steady state dropped to explorer ~29 m, exporter ~16 m CPU; the k3d node to ~56% (from ~245%).
- Metrics reflect recent flights only; long-horizon analysis is a warehouse (T3) query, not an exporter concern — consistent with ADR-0003.
- Old flight partitions accumulate on disk; pruning them is a follow-up
(tracked in
../09-open-questions.md). - Observability design is documented in
../07-observability.md.