eSlider
5f6fd8462c
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
1.6 KiB
1.6 KiB
ADR-0002: One storage format on every tier — Parquet + DuckDB, Hive layout
Status
Accepted (2026-07-08)
Context
Data lives across storage floors T0–T4. Primary persistent stores are T1 (drone NVMe in flight) and T3 (ground warehouse after landing); T0 is the in-memory hot window, T2 optional on-board derived replication, T4 dev/sim slices. If each floor used a different format or layout, offload would need transformation code, schemas would drift, and a query proven on the bench would not run unchanged on flight data.
Decision
Use one storage format on every floor: columnar Parquet files in an
identical Hive-partitioned layout (dataset=…/flight=…/drone=…/…), queried
with DuckDB. Flight offload T1 → T3 is therefore a plain mirror
(copy/rsync of partition directories), not an ETL step.
Consequences
- Offload is a file operation, testable with a checksum; no format converters to maintain or version.
- The same SQL runs against T1, T3, and simulated data — the simulator
(
../../simulator/) generates the real layout, so tests exercise the production format. - Partitioning choices are load-bearing; changing them is a schema migration.
Detailed in
../03-data-platform.md. - DuckDB's single-file, no-server model fits an air-gapped drone with no room for a database daemon.