eSlider
d3ccd6f94e
Add docs/adr/ with a chronological ADR set reconstructed from the project history (no orchestrator, one storage format, derived-state sync, SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root image, runtime data paths, bounded lake scans), an index README, and a template. Distinguish repo-local ADRs from ecosystem-wide ASRs. Expand AGENTS.md with a decision-records section and a runtime/operations guide (service URLs, rebuild workflow, CPU budget knobs). Reference the ADRs from README, CONTRIBUTING, and the affected design docs, and add an open question for T1 partition pruning.
53 lines
2.2 KiB
Markdown
53 lines
2.2 KiB
Markdown
# ADR-0009: Bound observability lake scans to a flight window and cache
|
|
|
|
## Status
|
|
|
|
Accepted (2026-07-08)
|
|
|
|
## Context
|
|
|
|
The k3d sim node was running at ~245% CPU. Three causes compounded:
|
|
|
|
1. **Explorer** rebuilt a DuckDB view over the *entire* lake on every HTTP
|
|
request (~2000 Parquet files, 200 MB+) — roughly 1.5 cores.
|
|
2. **Exporter** scanned the full lake every 5 s — roughly 0.7 cores.
|
|
3. **Drones** exited after sealing a flight; Kubernetes restarted each pod,
|
|
and every restart appended a fresh `flight=` partition tree, so the lake grew
|
|
without bound and every scan got more expensive.
|
|
|
|
The cost is inherent to scanning an append-only lake whose size grows with pod
|
|
churn, not to any single slow query. Metrics and the live view only ever need
|
|
recent flights.
|
|
|
|
## Decision
|
|
|
|
Bound and cache the scans instead of reading the whole lake:
|
|
|
|
- **Flight window.** The exporter and explorer scan only the most recent
|
|
`METRIC_FLIGHT_WINDOW` flights (default 5) via a shared helper
|
|
(`simulator/monitoring/lake.py`). Older partitions stay on disk but out of the
|
|
hot path.
|
|
- **Caching.** Explorer caches DuckDB views (`VIEW_REFRESH_S=30`) and the
|
|
partition tree (`TREE_CACHE_S=15`); the exporter scans every
|
|
`SCAN_INTERVAL_S=30` instead of 5 s.
|
|
- **Stop restart churn.** Drones set `KEEP_ALIVE=1` and idle after sealing
|
|
instead of exiting, so no new flight tree is created per restart. In k3d,
|
|
`IMU_HZ=20` and resource limits cap per-drone load; the live prototype polls
|
|
every 2 s.
|
|
- **Self-observability.** Add node-exporter, a
|
|
`swarm_exporter_scan_duration_seconds` metric, and a **Swarm Platform**
|
|
Grafana dashboard (node CPU/memory + scan health) alongside the existing fleet
|
|
dashboard.
|
|
|
|
## Consequences
|
|
|
|
- Steady state dropped to explorer ~29 m, exporter ~16 m CPU; the k3d node to
|
|
~56% (from ~245%).
|
|
- Metrics reflect recent flights only; long-horizon analysis is a warehouse
|
|
(T3) query, not an exporter concern — consistent with
|
|
[ADR-0003](ADR-0003-sync-derived-state-only.md).
|
|
- Old flight partitions accumulate on disk; pruning them is a follow-up
|
|
(tracked in [`../09-open-questions.md`](../09-open-questions.md)).
|
|
- Observability design is documented in
|
|
[`../07-observability.md`](../07-observability.md).
|