# ADR-0009: Bound observability lake scans to a flight window and cache

## Status

Accepted (2026-07-08)

## Context

The k3d sim node was running at ~245% CPU. Three causes compounded:

1. **Explorer** rebuilt a DuckDB view over the *entire* lake on every HTTP
   request (~2000 Parquet files, 200 MB+) — roughly 1.5 cores.
2. **Exporter** scanned the full lake every 5 s — roughly 0.7 cores.
3. **Drones** exited after sealing a flight; Kubernetes restarted each pod,
   and every restart appended a fresh `flight=` partition tree, so the lake grew
   without bound and every scan got more expensive.

The cost is inherent to scanning an append-only lake whose size grows with pod
churn, not to any single slow query. Metrics and the live view only ever need
recent flights.

## Decision

Bound and cache the scans instead of reading the whole lake:

- **Flight window.** The exporter and explorer scan only the most recent
  `METRIC_FLIGHT_WINDOW` flights (default 5) via a shared helper
  (`simulator/monitoring/lake.py`). Older partitions stay on disk but out of the
  hot path.
- **Caching.** Explorer caches DuckDB views (`VIEW_REFRESH_S=30`) and the
  partition tree (`TREE_CACHE_S=15`); the exporter scans every
  `SCAN_INTERVAL_S=30` instead of 5 s.
- **Stop restart churn.** Drones set `KEEP_ALIVE=1` and idle after sealing
  instead of exiting, so no new flight tree is created per restart. In k3d,
  `IMU_HZ=20` and resource limits cap per-drone load; the live prototype polls
  every 2 s.
- **Self-observability.** Add node-exporter, a
  `swarm_exporter_scan_duration_seconds` metric, and a **Swarm Platform**
  Grafana dashboard (node CPU/memory + scan health) alongside the existing fleet
  dashboard.

## Consequences

- Steady state dropped to explorer ~29 m, exporter ~16 m CPU; the k3d node to
  ~56% (from ~245%).
- Metrics reflect recent flights only; long-horizon analysis is a warehouse
  (T3) query, not an exporter concern — consistent with
  [ADR-0003](ADR-0003-sync-derived-state-only.md).
- Old flight partitions accumulate on disk; pruning them is a follow-up
  (tracked in [`../09-open-questions.md`](../09-open-questions.md)).
- Observability design is documented in
  [`../07-observability.md`](../07-observability.md).