eSlider
5f6fd8462c
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
37 lines
1.6 KiB
Markdown
37 lines
1.6 KiB
Markdown
# ADR-0002: One storage format on every tier — Parquet + DuckDB, Hive layout
|
||
|
||
## Status
|
||
|
||
Accepted (2026-07-08)
|
||
|
||
## Context
|
||
|
||
Data lives across [storage floors](../00-glossary.md#t0--hot) T0–T4. Primary persistent
|
||
stores are [T1](../00-glossary.md#t1--warm) (drone NVMe in flight) and
|
||
[T3](../00-glossary.md#t3--warehouse) (ground warehouse after landing);
|
||
[T0](../00-glossary.md#t0--hot) is the in-memory hot window,
|
||
[T2](../00-glossary.md#t2--shared) optional on-board derived replication,
|
||
[T4](../00-glossary.md#t4--dev) dev/sim slices. If each floor used a different
|
||
format or layout, offload would need transformation code,
|
||
schemas would drift, and a query proven on the bench would not run unchanged on
|
||
flight data.
|
||
|
||
## Decision
|
||
|
||
Use **one storage format on every floor**: columnar **Parquet** files in an
|
||
identical **Hive-partitioned** layout (`dataset=…/flight=…/drone=…/…`), queried
|
||
with **DuckDB**. Flight offload T1 → T3 is therefore a plain **mirror**
|
||
(copy/rsync of partition directories), not an ETL step.
|
||
|
||
## Consequences
|
||
|
||
- Offload is a file operation, testable with a checksum; no format converters
|
||
to maintain or version.
|
||
- The same SQL runs against T1, T3, and simulated data — the simulator
|
||
([`../../simulator/`](../../simulator/)) generates the *real* layout, so tests
|
||
exercise the production format.
|
||
- Partitioning choices are load-bearing; changing them is a schema migration.
|
||
Detailed in [`../03-data-platform.md`](../03-data-platform.md).
|
||
- DuckDB's single-file, no-server model fits an air-gapped drone with no room
|
||
for a database daemon.
|