docs: audit links, glossary T0-T4, readability pass
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
This commit is contained in:
+15
-13
@@ -1,6 +1,6 @@
|
||||
# 03 — Data platform
|
||||
|
||||
The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every floor** of the system.
|
||||
The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with **one format and one layout on every [storage floor](00-glossary.md#t0--hot)** of the system.
|
||||
|
||||
## Datasets
|
||||
|
||||
@@ -14,7 +14,7 @@ Three logical datasets, distinct because their consumers and sync policies diffe
|
||||
|
||||
## Partitioning layout
|
||||
|
||||
Identity dimensions first, calendar time last, everything Hive-style `key=value`:
|
||||
Identity dimensions first, calendar time last, everything [Hive-style](00-glossary.md#data--storage) `key=value`:
|
||||
|
||||
```
|
||||
dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/
|
||||
@@ -29,7 +29,7 @@ Design decisions and their reasons:
|
||||
|
||||
- **`flight` before `drone`.** The dominant warehouse query is *replay of one flight across all drones*. With `flight` on top, a replay reads one subtree. (On board, `drone=` is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.)
|
||||
- **`sensor` before time.** Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both.
|
||||
- **Flight identifiers.** The canonical flight id is a **UUIDv7** (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
|
||||
- **Flight identifiers.** The canonical flight id is a [**UUIDv7**](00-glossary.md#data--storage) (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (`20260708T1100Z-a3f2`) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
|
||||
- **Timestamps: `int64` epoch nanoseconds.** Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time.
|
||||
- **No rounding in raw data.** Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied **only** to broadcast payloads, where every byte of air time counts ([04 — Swarm sync](04-swarm-sync.md)).
|
||||
|
||||
@@ -50,7 +50,7 @@ sequenceDiagram
|
||||
S->>W: rows (int64 ns, typed)
|
||||
W->>C: flush min_NN.parquet (small, ZSTD-fast)
|
||||
end
|
||||
Note over Z: window closes (hour sealed<br/>or flight ends)
|
||||
Note over Z: window closes (hour [sealed](00-glossary.md#data--storage)<br/>or flight ends)
|
||||
Z->>C: read all min_*.parquet
|
||||
Z->>P: write one compact file (ZSTD-high)
|
||||
Z->>C: remove blocks after verify
|
||||
@@ -63,26 +63,28 @@ sequenceDiagram
|
||||
|
||||
## Storage floors (tiers)
|
||||
|
||||
Same format on every floor; only volume, retention, and location change:
|
||||
Same [Parquet](00-glossary.md#data--storage) + [DuckDB](00-glossary.md#data--storage) format on every floor; only volume, retention, and location change. See [glossary](00-glossary.md#t0--hot) for the full T0–T4 definitions.
|
||||
|
||||
| Floor | Where | Contents | Retention |
|
||||
| --- | --- | --- | --- |
|
||||
| **T0 — hot** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
|
||||
| **T1 — warm** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
|
||||
| **T2 — shared** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission |
|
||||
| **T3 — warehouse** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; replay, analytics, model training | Years |
|
||||
| **T4 — dev** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
|
||||
| **[T0 — hot](00-glossary.md#t0--hot)** | RAM / DuckDB in-process | Sliding window of the last minutes; what mission logic queries in flight | Minutes |
|
||||
| **[T1 — warm](00-glossary.md#t1--warm)** | Drone NVMe | Full raw `telemetry` + `detections` + `state` of the current flight; never leaves the drone in flight | Current flight (+ quota-based headroom) |
|
||||
| **[T2 — shared](00-glossary.md#t2--shared)** | On-board MinIO bucket | Derived data only (`detections`, `state`), replicated opportunistically across the swarm | Current mission |
|
||||
| **[T3 — warehouse](00-glossary.md#t3--warehouse)** | Ground on-prem object store + Parquet lakehouse | Every flight of every drone, forever; [source of truth](00-glossary.md#source-of-truth) for replay, analytics, model training | Years |
|
||||
| **[T4 — dev](00-glossary.md#t4--dev)** | Engineer laptop / sim farm | Slices pulled from T3, or synthetic data from the simulator | Ephemeral |
|
||||
|
||||
## Flight offload: a mirror, not a migration
|
||||
|
||||
Because T1 and T3 share the identical layout, offload after landing is:
|
||||
Because [T1](00-glossary.md#t1--warm) and [T3](00-glossary.md#t3--warehouse) share the identical layout, [offload](00-glossary.md#offload) after landing is:
|
||||
|
||||
```bash
|
||||
# per drone, at the base station
|
||||
mc mirror drone-nvme/flights/ warehouse/flights/ # or rsync over the wired dock
|
||||
rsync -a drone-nvme/flights/ warehouse/flights/ # or mc mirror to object store
|
||||
duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…"
|
||||
```
|
||||
|
||||
For S3-only backends where rsync is awkward, [rclone](00-glossary.md#rclone) is an optional rsync-class alternative — not the default mesh path ([09 — Open questions](09-open-questions.md) §12).
|
||||
|
||||
The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job.
|
||||
|
||||
## Retention and quotas on board
|
||||
@@ -138,6 +140,6 @@ On the ground this powers replay overlays, coverage analysis (did ubiquitous sen
|
||||
|
||||
## Schema management
|
||||
|
||||
- Every dataset schema is versioned in-repo (`schemas/` — SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
|
||||
- Every dataset schema is versioned in-repo (planned `schemas/` directory — [roadmap stage 2](08-roadmap.md), [09 — Open questions](09-open-questions.md) §14; SemVer, one file per dataset version) and referenced by the fleet release manifest ([11](11-cicd-delivery.md)).
|
||||
- Parquet files carry the schema version in file metadata; readers select the matching deserializer.
|
||||
- Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release.
|
||||
|
||||
Reference in New Issue
Block a user