perf: bound lake scans and add platform Grafana dashboard
Explorer and exporter were scanning the full Parquet lake every 1–5s (~2000 files, 200MB+), driving ~2.2 CPU cores. Limit metrics to the last N flights, cache DuckDB views and the partition tree, slow live polls to 2s, and keep drones alive after seal to stop restart churn. Add node-exporter, scan-duration metrics, and a Swarm Platform Grafana dashboard for node CPU/memory and scan health.
This commit is contained in:
@@ -24,7 +24,8 @@ services:
|
||||
command: ["python", "monitoring/exporter.py"]
|
||||
environment:
|
||||
DATA_DIR: /data
|
||||
SCAN_INTERVAL_S: "5"
|
||||
SCAN_INTERVAL_S: "30"
|
||||
METRIC_FLIGHT_WINDOW: "5"
|
||||
volumes:
|
||||
- ${SWARM_T1_DIR:-../var/t1}:/data:ro
|
||||
- ./monitoring:/app/monitoring:ro
|
||||
@@ -37,6 +38,9 @@ services:
|
||||
command: ["python", "explorer/server.py"]
|
||||
environment:
|
||||
DATA_DIR: /data
|
||||
VIEW_REFRESH_S: "30"
|
||||
TREE_CACHE_S: "15"
|
||||
METRIC_FLIGHT_WINDOW: "5"
|
||||
volumes:
|
||||
- ${SWARM_T1_DIR:-../var/t1}:/data:ro
|
||||
- ./explorer:/app/explorer:ro
|
||||
|
||||
Reference in New Issue
Block a user