eSlider
d3ccd6f94e
Add docs/adr/ with a chronological ADR set reconstructed from the project history (no orchestrator, one storage format, derived-state sync, SQL-over-SSH contract, WireGuard beneath SSH, IaC boundaries, non-root image, runtime data paths, bounded lake scans), an index README, and a template. Distinguish repo-local ADRs from ecosystem-wide ASRs. Expand AGENTS.md with a decision-records section and a runtime/operations guide (service URLs, rebuild workflow, CPU budget knobs). Reference the ADRs from README, CONTRIBUTING, and the affected design docs, and add an open question for T1 partition pruning.
33 lines
1.3 KiB
Markdown
33 lines
1.3 KiB
Markdown
# ADR-0002: One storage format on every tier — Parquet + DuckDB, Hive layout
|
|
|
|
## Status
|
|
|
|
Accepted (2026-07-08)
|
|
|
|
## Context
|
|
|
|
Data lives in three places: on the drone's NVMe in flight (T1), in the ground
|
|
warehouse after landing (T3), and in the dev/simulation environment. If each
|
|
tier used a different format or layout, offload would need transformation code,
|
|
schemas would drift, and a query proven on the bench would not run unchanged on
|
|
flight data.
|
|
|
|
## Decision
|
|
|
|
Use **one storage format on every floor**: columnar **Parquet** files in an
|
|
identical **Hive-partitioned** layout (`dataset=…/flight=…/drone=…/…`), queried
|
|
with **DuckDB**. Flight offload T1 → T3 is therefore a plain **mirror**
|
|
(copy/rsync of partition directories), not an ETL step.
|
|
|
|
## Consequences
|
|
|
|
- Offload is a file operation, testable with a checksum; no format converters
|
|
to maintain or version.
|
|
- The same SQL runs against T1, T3, and simulated data — the simulator
|
|
([`../../simulator/`](../../simulator/)) generates the *real* layout, so tests
|
|
exercise the production format.
|
|
- Partitioning choices are load-bearing; changing them is a schema migration.
|
|
Detailed in [`../03-data-platform.md`](../03-data-platform.md).
|
|
- DuckDB's single-file, no-server model fits an air-gapped drone with no room
|
|
for a database daemon.
|