docs: audit links, glossary T0-T4, readability pass
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
CI & Release / Verify simulator (push) Skipped
CI & Release / Trivy scan (push) Skipped
CI & Release / Semantic Release (push) Skipped
CI & Release / Verify simulator (pull_request) Failing after 4s
CI & Release / Trivy scan (pull_request) Skipped
CI & Release / Semantic Release (pull_request) Skipped
Extend glossary with storage floors, source of truth, and linkable anchors. Fix broken links, pose frame drift, rsync/offload wording, and README tone. Add docs/improvement audit records and live demo links in design journey.
This commit is contained in:
@@ -1,7 +1,18 @@
|
||||
# Terraform — ground workloads
|
||||
|
||||
Declared workloads on the k3d simulation cluster. See [06 — IaC boundaries](../../docs/06-environments.md#iac-boundaries).
|
||||
|
||||
| Module | Tier | Purpose |
|
||||
| --- | --- | --- |
|
||||
| [`sim-env/`](sim-env/) | [T4](../../docs/00-glossary.md#t4--dev) dev/sim | Virtual fleet + observability on k3d |
|
||||
| [`ground/`](ground/) | [T3](../../docs/00-glossary.md#t3--warehouse) | Warehouse + [T1→T3](../../docs/03-data-platform.md#storage-floors-tiers) offload |
|
||||
|
||||
Two modules, two states. Fleet disposable; warehouse survives `destroy` of sim-env.
|
||||
|
||||
## Recovering from partial state
|
||||
|
||||
If resources were created outside Terraform (or state was lost), import them
|
||||
before `terraform apply`:
|
||||
before `terraform apply`. Run from the target module directory (`sim-env/` or `ground/`):
|
||||
|
||||
```bash
|
||||
# sim-env (namespace: swarm)
|
||||
|
||||
@@ -1,19 +1,14 @@
|
||||
# Terraform — ground warehouse (T3)
|
||||
# Terraform — ground warehouse ([T3](../../../docs/00-glossary.md#t3--warehouse))
|
||||
|
||||
The stationary half of the system: the historical warehouse that receives
|
||||
sealed partitions after every flight ([03 — Storage tiers](../../../docs/03-storage-design.md)).
|
||||
It is the longest-lived, most conventional infrastructure in the design —
|
||||
and therefore the most natural Terraform territory.
|
||||
Ground warehouse. Receives sealed partitions post-flight ([storage floors](../../../docs/03-data-platform.md#storage-floors-tiers)). Long-lived infra → Terraform.
|
||||
|
||||
Runs in the same k3d cluster as [`../sim-env`](../sim-env/) but in its own
|
||||
namespace with its own state: the fleet is disposable, the warehouse is not,
|
||||
and the two lifecycles must never share a `terraform destroy`.
|
||||
Runs in same k3d cluster as [`../sim-env`](../sim-env/) but separate namespace and state: fleet disposable, warehouse not — never share `terraform destroy`.
|
||||
|
||||
| Resource | Purpose |
|
||||
| --- | --- |
|
||||
| Deployment `minio` + PVC + NodePort 30901 | Object-store facade of the warehouse for downstream consumers (training pipelines, replay) |
|
||||
| CronJob `offload` | Post-flight offload: copies Hive partitions verbatim from the fleet lake (T1) into the warehouse (T3), then mirrors into MinIO |
|
||||
| Deployment `warehouse-explorer` + NodePort 30089 | Second explorer instance over the warehouse path — historical read-only SQL across **all** flights |
|
||||
| Deployment `minio` + PVC + NodePort 30901 | Object-store facade for downstream consumers |
|
||||
| CronJob `offload` | [T1](../../../docs/00-glossary.md#t1--warm) → [T3](../../../docs/00-glossary.md#t3--warehouse) offload: Hive partitions verbatim, then `mc mirror` into MinIO |
|
||||
| Deployment `warehouse-explorer` + NodePort 30089 | Read-only SQL across **all** offloaded flights |
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -23,16 +18,9 @@ terraform apply
|
||||
terraform output
|
||||
```
|
||||
|
||||
After a couple of offload runs, the warehouse explorer shows the same
|
||||
partition tree as the live lake but accumulated across flights — the T1
|
||||
layout **is** the T3 layout, which is the whole point: no transform step,
|
||||
no schema drift, DuckDB queries work identically on both ends.
|
||||
After offload runs, warehouse explorer shows same partition tree as live lake, accumulated across flights. [T1](../../../docs/00-glossary.md#t1--warm) layout **is** [T3](../../../docs/00-glossary.md#t3--warehouse) layout. No transform. No schema drift. Same DuckDB queries both ends.
|
||||
|
||||
## Simulation vs production
|
||||
|
||||
- The CronJob compresses "drone lands, docks, offloads, prunes" into a
|
||||
periodic rsync-style copy. Production offload is event-driven per docking
|
||||
and verifies checksums from the flight's partition manifest before pruning
|
||||
the on-board lake.
|
||||
- MinIO credentials here are throwaway defaults; production credentials are
|
||||
sealed per site and never in state or VCS.
|
||||
- Sim CronJob: `cp -ru` + `mc mirror` on a schedule — shortcut for "lands, offloads, prunes". Production: event-driven per dock, rsync or `mc mirror`, checksum audit before prune ([03 — Data platform](../../../docs/03-data-platform.md)).
|
||||
- MinIO credentials here are throwaway defaults; production credentials sealed per site, never in state or VCS.
|
||||
|
||||
@@ -1,17 +1,14 @@
|
||||
# Terraform — simulation environment (T4)
|
||||
# Terraform — simulation environment ([T4](../../../docs/00-glossary.md#t4--dev))
|
||||
|
||||
Declares the simulated fleet and its observability stack on the k3d cluster
|
||||
created by [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml).
|
||||
This module is the executable miniature of the dev/sim environment from
|
||||
[06 — Environments](../../../docs/06-environments.md).
|
||||
Sim fleet + observability on k3d. Cluster from [`../../ansible/sim-cluster.yml`](../../ansible/sim-cluster.yml). Executable [T4](../../../docs/00-glossary.md#t4--dev) miniature ([06 — Environments](../../../docs/06-environments.md)).
|
||||
|
||||
| Resource | Purpose |
|
||||
| --- | --- |
|
||||
| StatefulSet `drone` × `drone_count` | Virtual drones; stable pod names become `DRONE_ID`s; all write Hive-partitioned Parquet into the shared `/data` lake |
|
||||
| Deployment `exporter` | Prometheus exporter reading the lake with DuckDB |
|
||||
| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL console (also feeds the prototype's live mode) |
|
||||
| Deployment `prometheus` + NodePort 30990 | Scrapes the exporter |
|
||||
| Deployment `grafana` + NodePort 30300 | Same dashboard JSON as the Compose profile, provisioned from a ConfigMap |
|
||||
| StatefulSet `drone` × `drone_count` | Virtual drones; pod names → `DRONE_ID`; shared `/data` lake |
|
||||
| Deployment `exporter` | Prometheus exporter reading lake with DuckDB |
|
||||
| Deployment `explorer` + NodePort 30088 | Partition tree + read-only SQL (prototype live mode) |
|
||||
| Deployment `prometheus` + NodePort 30990 | Scrapes exporter |
|
||||
| Deployment `grafana` + NodePort 30300 | Dashboard JSON from ConfigMap |
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -22,7 +19,7 @@ terraform apply
|
||||
terraform output # URLs
|
||||
```
|
||||
|
||||
Scale the fleet without touching YAML:
|
||||
Scale without YAML edits:
|
||||
|
||||
```bash
|
||||
terraform apply -var drone_count=9 -var speedup=4
|
||||
@@ -30,11 +27,6 @@ terraform apply -var drone_count=9 -var speedup=4
|
||||
|
||||
## Notes
|
||||
|
||||
- `image_pull_policy = "Never"` — the simulator image is imported by the
|
||||
Ansible playbook (`k3d image import`); the cluster never pulls from a
|
||||
registry, mirroring the air-gap doctrine.
|
||||
- Pods discover their identity from the pod name (StatefulSet ordinal), so
|
||||
a flight survives `kubectl delete pod` the same way a drone survives a
|
||||
power cycle: new process, same identity, new flight ID.
|
||||
- State is local (`terraform.tfstate` in this directory) — the simulation
|
||||
cluster is disposable; nothing here is shared infrastructure.
|
||||
- `image_pull_policy = "Never"` — image imported by Ansible (`k3d image import`); no registry pull. Air-gap mirror.
|
||||
- Pod name → identity (StatefulSet ordinal). `kubectl delete pod` = power cycle: new process, same identity, new flight ID.
|
||||
- Local state only — cluster disposable.
|
||||
|
||||
Reference in New Issue
Block a user