eSlider
d3ccd6f94e
Add docs/adr/ with a chronological ADR set reconstructed from the project history (no orchestrator, one storage format, derived-state sync, SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root image, runtime data paths, bounded lake scans), an index README, and a template. Distinguish repo-local ADRs from ecosystem-wide ASRs. Expand AGENTS.md with a decision-records section and a runtime/operations guide (service URLs, rebuild workflow, CPU budget knobs). Reference the ADRs from README, CONTRIBUTING, and the affected design docs, and add an open question for T1 partition pruning.
1.3 KiB
1.3 KiB
ADR-0002: One storage format on every tier — Parquet + DuckDB, Hive layout
Status
Accepted (2026-07-08)
Context
Data lives in three places: on the drone's NVMe in flight (T1), in the ground warehouse after landing (T3), and in the dev/simulation environment. If each tier used a different format or layout, offload would need transformation code, schemas would drift, and a query proven on the bench would not run unchanged on flight data.
Decision
Use one storage format on every floor: columnar Parquet files in an
identical Hive-partitioned layout (dataset=…/flight=…/drone=…/…), queried
with DuckDB. Flight offload T1 → T3 is therefore a plain mirror
(copy/rsync of partition directories), not an ETL step.
Consequences
- Offload is a file operation, testable with a checksum; no format converters to maintain or version.
- The same SQL runs against T1, T3, and simulated data — the simulator
(
../../simulator/) generates the real layout, so tests exercise the production format. - Partitioning choices are load-bearing; changing them is a schema migration.
Detailed in
../03-data-platform.md. - DuckDB's single-file, no-server model fits an air-gapped drone with no room for a database daemon.