swarm-house/simulator

Virtual drone fleet

A data generator that exercises the exact on-board pipeline described in 03 — Data platform: seeded kinematics along a patrol route, sensor streams at realistic rates, detection events, the hot current/ → sealed Parquet write path, and the compact UDP state broadcast between drones.

One process = one drone. Scaling the swarm is a Compose flag.

Run a swarm

# 5 drones (default), 2-minute flight, shared flight id
FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --build --scale drone=5

# 10 drones, 5-minute flight, 4x accelerated
DRONE_COUNT=10 DURATION_S=300 SPEEDUP=4 docker compose up --build --scale drone=10

Output lands in ../var/t1/ (the shared T1 lake — same path the k3d fleet and warehouse offload use). Override with SWARM_T1_DIR if needed:

var/t1/dataset=telemetry/flight=…/drone=…/sensor=imu/year=…/…/hour=…/data.parquet
var/t1/dataset=detections/flight=…/drone=…/…
var/t1/dataset=state/flight=…/drone=…/…        ← sent + received broadcasts

Historical flights after offload live under ../var/t3/ (T3 warehouse).

Run a single drone without Docker

pip install -r requirements.txt
DRONE_ID=dr-01 DURATION_S=30 SPEEDUP=10 DATA_DIR=./data python -m virtual_drone.main

Query the results

python - <<'EOF'
import duckdb
con = duckdb.connect()
print(con.sql("""
    SELECT drone, sensor, count(*) AS rows,
           to_timestamp(min(ts_ns)/1e9) AS first_row,
           to_timestamp(max(ts_ns)/1e9) AS last_row
    FROM read_parquet('data/dataset=telemetry/**/*.parquet', hive_partitioning=true)
    GROUP BY drone, sensor ORDER BY drone, sensor
"""))
print(con.sql("""
    SELECT drone, direction, count(*) AS frames, count(DISTINCT peer_id) AS peers
    FROM read_parquet('data/dataset=state/**/*.parquet', hive_partitioning=true)
    GROUP BY drone, direction ORDER BY drone, direction
"""))
EOF

Monitoring: Grafana over the generated data

The monitoring profile spins up a small metrics chain — a DuckDB-based exporter that scans the generated Parquet every few seconds, Prometheus, and a pre-provisioned Grafana dashboard:

docker compose --profile monitoring up -d          # exporter + prometheus + grafana
# generate some flights in parallel or beforehand:
FLIGHT_ID=$(date -u +%Y%m%dT%H%MZ)-sim docker compose up --scale drone=5

• Grafana: http://localhost:3000 (anonymous admin — demo only) → dashboard Swarm Fleet — generated data
• Prometheus: http://localhost:9090 · exporter: http://localhost:9105/metrics

Panels: telemetry rows by drone/sensor, detections by class, pose frames sent/received, battery per drone, RSSI and estimated distance per link, Parquet bytes/files on disk. The exporter is deliberately a demonstration of the observability doctrine from 07 — Observability: fleet statistics are derived from the data platform itself — no agent on the drone, just SQL over the same Parquet everyone else reads.

Data-plane explorer

The monitoring profile also starts a data-plane explorer at http://localhost:8088 — a single-page web view over the lake itself, complementing Grafana (which shows aggregates, not the structure):

• Partition tree, live: dataset → flight → drone → sensor → year/…/hour → file, with file counts and bytes rolled up at every level. You can watch current/ files appear and get sealed while a swarm is flying.
• Read-only SQL console with the datasets pre-registered as views (telemetry, detections, state) and one-click sample queries (fleet overview, battery timeline, detections by class, peer link quality, schema).

The console enforces the same statement gate as the peer query channel from 04 — Swarm sync: a single statement, SELECT/WITH/DESCRIBE/SUMMARIZE/SHOW only, write/config/extension keywords rejected. That is the point: exploring the data plane on the bench uses the same read-only SQL contract a peer drone uses in flight — the explorer is the "no debug API" principle made visible.

Reproducibility

Every run is deterministic per (SEED, DRONE_ID): same route jitter, same sensor noise, same detection sequence. A bug report is a seed and a config, not a description.

Knobs

Env	Default	Meaning
DRONE_ID	derived from hostname	Unique per container automatically under --scale
FLIGHT_ID	generated	Set explicitly so all drones share one flight partition
SEED	42	Determinism root
DURATION_S	120	Simulated flight length
SPEEDUP	1.0	Wall-clock acceleration
IMU_HZ	50	High-rate sensor load
STATE_HZ	5	Broadcast frequency
AREA_SIZE_M	400	Patrol square side