swarm-house/docs/03-data-platform.md

03 — Data platform

The heart of the proposal: how raw sensor streams become a compressed, queryable, offloadable store — with one format and one layout on every floor of the system.

Datasets

Three logical datasets, distinct because their consumers and sync policies differ:

Dataset	Contents	Rate	Leaves the drone in flight?
telemetry	Raw sensor rows: IMU, barometer, temperature, LiDAR, RSSI, power, magnetometer, …	Up to hundreds of Hz per sensor	Never
detections	Object-detection events from the video service: class, confidence, bounding box, ego-position at detection time	Event-shaped, bursty	Yes — derived, shareable
state	The drone's own pose stream: position, attitude, velocity, timestamp; plus the log of every broadcast sent/received	1–10 Hz effective	Yes — this is the shared state

Partitioning layout

Identity dimensions first, calendar time last, everything Hive-style key=value:

dataset=telemetry/flight=20260708T1100Z-a3f2/drone=dr-017/sensor=imu/year=2026/month=07/day=08/hour=11/
    current/min_00.parquet      ← hot write path (open window)
    current/min_01.parquet
    data.parquet                ← sealed once the hour closes
dataset=detections/flight=…/drone=…/year=…/month=…/day=…/hour=…/
dataset=state/flight=…/drone=…/year=…/month=…/day=…/hour=…/

Design decisions and their reasons:

• flight before drone. The dominant warehouse query is replay of one flight across all drones. With flight on top, a replay reads one subtree. (On board, drone= is constant and costs nothing — but keeping it means the on-board layout is byte-identical to the warehouse layout.)
• sensor before time. Each sensor has its own schema, so sensors need separate leaves anyway; and "one sensor over a period" is the most common analytical scan. Partition pruning handles both.
• Flight identifiers. The canonical flight id is a UUIDv7 (time-ordered, globally unique, embedded timestamp — sortable by design). Partition paths use a short human-readable time-sortable alias (20260708T1100Z-a3f2) so directory listings stay debuggable; the full UUIDv7 lives in the data and in the flight manifest.
• Timestamps: int64 epoch nanoseconds. Same 8 bytes as milliseconds, native resolution of ROS 2, sufficient for IMU-class rates, valid until year 2262. One time type everywhere — no unit confusion at merge time.
• No rounding in raw data. Coordinates and measurements are stored exactly as the sensor emits them (float64/float32 per sensor spec). ZSTD plus Parquet delta encoding makes rounding-for-size unnecessary. Quantization (millimeters as int32) is applied only to broadcast payloads, where every byte of air time counts (04 — Swarm sync).

Hot write path: current/ → sealed

Streaming many small rows straight into big Parquet files is impossible (Parquet is immutable); buffering a whole hour in RAM is unacceptable (crash = lost data). The proven pattern:

sequenceDiagram
    participant S as sensor-ingest
    participant W as parquet-writer
    participant C as current/ (open window)
    participant Z as sealer
    participant P as data.parquet (sealed)
    participant D as DuckDB

    loop every minute block
        S->>W: rows (int64 ns, typed)
        W->>C: flush min_NN.parquet (small, ZSTD-fast)
    end
    Note over Z: window closes (hour sealed<br/>or flight ends)
    Z->>C: read all min_*.parquet
    Z->>P: write one compact file (ZSTD-high)
    Z->>C: remove blocks after verify
    D->>P: read_parquet('…/**/*.parquet') — glob covers both states

• Crash tolerance is one minute-block, tunable per sensor rate.
• Readers never care about the sealing state: a single read_parquet glob matches both current/*.parquet and data.parquet.
• The sealer runs at low priority — flight-critical software always wins the CPU.

Storage floors (tiers)

Same format on every floor; only volume, retention, and location change:

Floor	Where	Contents	Retention
T0 — hot	RAM / DuckDB in-process	Sliding window of the last minutes; what mission logic queries in flight	Minutes
T1 — warm	Drone NVMe	Full raw telemetry + detections + state of the current flight; never leaves the drone in flight	Current flight (+ quota-based headroom)
T2 — shared	On-board MinIO bucket	Derived data only (detections, state), replicated opportunistically across the swarm	Current mission
T3 — warehouse	Ground on-prem object store + Parquet lakehouse	Every flight of every drone, forever; replay, analytics, model training	Years
T4 — dev	Engineer laptop / sim farm	Slices pulled from T3, or synthetic data from the simulator	Ephemeral

Flight offload: a mirror, not a migration

Because T1 and T3 share the identical layout, offload after landing is:

# per drone, at the base station
mc mirror drone-nvme/flights/ warehouse/flights/       # or rsync over the wired dock
duckdb -c "…integrity audit: row counts, time coverage, gap scan per partition…"

The audit compares expected vs actual coverage per partition (row counts, min/max timestamps, gap detection) before the drone's local copy is released for cleanup. Missing or corrupt partitions are re-pulled — a lazy gap-repair pass rather than a failed batch job.

Retention and quotas on board

• Per-dataset NVMe quotas enforced by the sealer (oldest sealed partitions of previous flights evicted first; the current flight is never evicted).
• If ingest pressure exceeds the write budget, degradation is explicit and ordered: reduce optional high-rate sensors' sampling first, never drop state or detections.

Query examples (DuckDB)

-- All detections of one flight, fused across the fleet (runs on T3)
SELECT drone, ts_ns, class, confidence, pos_x, pos_y, pos_z
FROM read_parquet('warehouse/dataset=detections/flight=20260708T1100Z-a3f2/**/*.parquet')
ORDER BY ts_ns;

-- IMU of one drone for one minute (runs identically on T1 or T3)
SELECT *
FROM read_parquet('…/dataset=telemetry/flight=…/drone=dr-017/sensor=imu/**/*.parquet')
WHERE ts_ns BETWEEN 1783941000000000000 AND 1783941060000000000;

-- Downsample for a dashboard: 1 Hz means over a 400 Hz stream
SELECT ts_ns // 1000000000 AS second, avg(accel_x), avg(accel_y), avg(accel_z)
FROM read_parquet('…/sensor=imu/**/*.parquet')
GROUP BY second ORDER BY second;

Schema management

• Every dataset schema is versioned in-repo (schemas/ — SemVer, one file per dataset version) and referenced by the fleet release manifest (11).
• Parquet files carry the schema version in file metadata; readers select the matching deserializer.
• Additive evolution only within a MAJOR version (new nullable columns); breaking changes bump MAJOR and require a manifest release.