docs: record architecture decisions (ADR-0001..0009) and operations knowledge
CI & Release / Verify simulator (push) Successful in 11s
CI & Release / Trivy scan (push) Successful in 8s
CI & Release / Semantic Release (push) Successful in 5s

Add docs/adr/ with a chronological ADR set reconstructed from the project
history (no orchestrator, one storage format, derived-state sync,
SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root
image, runtime data paths, bounded lake scans), an index README, and a
template. Distinguish repo-local ADRs from ecosystem-wide ASRs.

Expand AGENTS.md with a decision-records section and a runtime/operations
guide (service URLs, rebuild workflow, CPU budget knobs). Reference the
ADRs from README, CONTRIBUTING, and the affected design docs, and add an
open question for T1 partition pruning.
This commit is contained in:
2026-07-09 11:41:05 +01:00
parent ea062af86a
commit d3ccd6f94e
19 changed files with 488 additions and 0 deletions
+52
View File
@@ -0,0 +1,52 @@
# ADR-0009: Bound observability lake scans to a flight window and cache
## Status
Accepted (2026-07-08)
## Context
The k3d sim node was running at ~245% CPU. Three causes compounded:
1. **Explorer** rebuilt a DuckDB view over the *entire* lake on every HTTP
request (~2000 Parquet files, 200 MB+) — roughly 1.5 cores.
2. **Exporter** scanned the full lake every 5 s — roughly 0.7 cores.
3. **Drones** exited after sealing a flight; Kubernetes restarted each pod,
and every restart appended a fresh `flight=` partition tree, so the lake grew
without bound and every scan got more expensive.
The cost is inherent to scanning an append-only lake whose size grows with pod
churn, not to any single slow query. Metrics and the live view only ever need
recent flights.
## Decision
Bound and cache the scans instead of reading the whole lake:
- **Flight window.** The exporter and explorer scan only the most recent
`METRIC_FLIGHT_WINDOW` flights (default 5) via a shared helper
(`simulator/monitoring/lake.py`). Older partitions stay on disk but out of the
hot path.
- **Caching.** Explorer caches DuckDB views (`VIEW_REFRESH_S=30`) and the
partition tree (`TREE_CACHE_S=15`); the exporter scans every
`SCAN_INTERVAL_S=30` instead of 5 s.
- **Stop restart churn.** Drones set `KEEP_ALIVE=1` and idle after sealing
instead of exiting, so no new flight tree is created per restart. In k3d,
`IMU_HZ=20` and resource limits cap per-drone load; the live prototype polls
every 2 s.
- **Self-observability.** Add node-exporter, a
`swarm_exporter_scan_duration_seconds` metric, and a **Swarm Platform**
Grafana dashboard (node CPU/memory + scan health) alongside the existing fleet
dashboard.
## Consequences
- Steady state dropped to explorer ~29 m, exporter ~16 m CPU; the k3d node to
~56% (from ~245%).
- Metrics reflect recent flights only; long-horizon analysis is a warehouse
(T3) query, not an exporter concern — consistent with
[ADR-0003](ADR-0003-sync-derived-state-only.md).
- Old flight partitions accumulate on disk; pruning them is a follow-up
(tracked in [`../09-open-questions.md`](../09-open-questions.md)).
- Observability design is documented in
[`../07-observability.md`](../07-observability.md).