diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index e704cb2..620f186 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -1,4 +1,5 @@ # Swarm House - Gitea Actions +# docs-verify: markdown links, glossary anchors, readability rules (every push/PR) # verify: compile check, virtual-drone smoke flight, SQL gate tests (every push/PR) # scan: Trivy vulnerability + misconfiguration scan # release: semantic version bump (conventional commits), tag, Gitea release @@ -17,6 +18,24 @@ on: branches: [main] jobs: + docs-verify: + name: Verify documentation + runs-on: ubuntu-latest + container: python:3.12-bookworm + steps: + - name: Checkout + run: | + git init . + git remote add origin https://${{ gitea.actor }}:${{ gitea.token }}@git.produktor.io/${{ gitea.repository }}.git + git fetch --depth=50 origin "${{ gitea.ref }}" + git checkout FETCH_HEAD + + - name: Install checker dependencies + run: pip install --quiet pyyaml + + - name: Documentation rules (links, glossary, readability) + run: python scripts/check_docs.py + verify: name: Verify simulator runs-on: ubuntu-latest @@ -85,7 +104,7 @@ jobs: name: Semantic Release runs-on: ubuntu-latest container: python:3.12-bookworm - needs: [verify, scan] + needs: [docs-verify, verify, scan] if: gitea.event_name == 'push' permissions: contents: write diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index db66de9..dd12c7b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,6 +17,7 @@ The CI pipeline enforces this: | Gate | When it runs | | --- | --- | +| `python scripts/check_docs.py` | Every push/PR (`docs-verify` job) — links, glossary anchors, readability rules ([ci-rules](docs/improvement/ci-rules.md)) | | `pytest tests/` | Every push/PR (`verify` job) | | `RUN pytest tests/` in the simulator Docker build | Every `docker build` — a red test blocks the image | | Smoke flight + DuckDB assertion | Every push/PR | diff --git a/docs/improvement/2026-07-10-audit.md b/docs/improvement/2026-07-10-audit.md index 7bc2550..9b888dc 100644 --- a/docs/improvement/2026-07-10-audit.md +++ b/docs/improvement/2026-07-10-audit.md @@ -16,7 +16,7 @@ Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify ### Links - `infra/terraform/ground/README.md` pointed at non-existent `03-storage-design.md` → `03-data-platform.md`. -- `11-cicd-delivery.md` bare `see 05` → proper markdown link. +- `11-cicd-delivery.md` bare cross-reference to doc 05 → proper markdown link. ### Factual drift (code vs docs) @@ -63,11 +63,7 @@ Manual pass over `docs/`, all READMEs, and glossary. Goal: correct links, unify ## Suggested CI checks (for issue #1) -```bash -# Relative .md link resolver (Python one-liner in audit script) -# Glossary anchor: every 00-glossary.md#... from docs/ must resolve -# Optional: markdown-spellcheck with US locale -``` +Implemented in [`scripts/check_docs.py`](../../scripts/check_docs.py) — see [ci-rules.md](ci-rules.md). ## Files touched diff --git a/docs/improvement/README.md b/docs/improvement/README.md index 5744f51..69596cc 100644 --- a/docs/improvement/README.md +++ b/docs/improvement/README.md @@ -6,9 +6,11 @@ Point-in-time audits and follow-up work for links, terminology, readability, and | --- | --- | --- | | [2026-07-10 audit](2026-07-10-audit.md) | 2026-07-10 | Full `docs/` + README pass; glossary T0–T4; live demo links | -## Future automation +## CI automation -Tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1): CI link checker, glossary anchor validation, terminology lint. +Enforced in Gitea Actions (`docs-verify` job). Rules live in [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml); see [ci-rules.md](ci-rules.md). + +Originally tracked in [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1). ## How to add a new report diff --git a/docs/improvement/ci-rules.md b/docs/improvement/ci-rules.md new file mode 100644 index 0000000..2c22722 --- /dev/null +++ b/docs/improvement/ci-rules.md @@ -0,0 +1,56 @@ +# CI documentation rules + +Automated checks for markdown in this repository. Enforced on every **pull request** and on **push to `main`** (including docs-only pushes that skip the simulator gate). + +Implementation: [`scripts/check_docs.py`](../../scripts/check_docs.py) reads [`scripts/docs_rules.yaml`](../../scripts/docs_rules.yaml). + +Workflow: [`.github/workflows/release.yaml`](../../.github/workflows/release.yaml) job **`Verify documentation`**. + +Tracked from [issue #1](https://git.produktor.io/eSlider/swarm-house/issues/1) and the [2026-07-10 audit](2026-07-10-audit.md). + +## Gates + +| Rule | Severity | What it checks | +| --- | --- | --- | +| Relative links | error | Every `[text](path)` and `[text](path#anchor)` to a local `.md` file resolves to an existing file | +| Fragment anchors | error | `#anchor` fragments match a `##` or `###` heading in the target file (GitHub-style slug) | +| Banned targets | error | Known-bad paths (e.g. deleted `03-storage-design.md`) are not linked | +| Glossary fragments | error | All `00-glossary.md#…` links use anchors that exist in the glossary | +| Section vs term links | warning | Links to glossary *section* headers (`#swarm--robotics`, …) are listed; prefer `###` term anchors where they exist | +| Bare cross-refs | error | Prose cross-references to numbered docs without a proper markdown link | + +Warnings are printed in the job log but do not fail CI. Errors exit non-zero and block merge. + +## PR workflow + +```mermaid +flowchart LR + PR[Pull request] --> DV[docs-verify] + PR --> V[verify simulator] + V --> S[Trivy scan] + DV --> OK{All required green?} + V --> OK + S --> OK + OK -->|yes| Merge[Mergeable] +``` + +| Event | `docs-verify` | `verify` + `scan` | +| --- | --- | --- | +| PR → `main` | always | always | +| Push → `main` (code paths) | always | runs | +| Push → `main` (docs/README/AGENTS only) | always | skipped (`paths-ignore`) | + +Docs-only changes therefore still get link and glossary validation on `main`; they do not trigger a semantic release. + +## Local run + +```bash +python scripts/check_docs.py +python scripts/check_docs.py --warnings-as-errors # treat section-link warnings as failures +``` + +## Changing rules + +1. Edit `scripts/docs_rules.yaml` (paths, banned targets, preferred anchors). +2. Adjust `scripts/check_docs.py` only when adding a new rule *type*. +3. Add a row to the table above and mention it in the audit/improvement record if the change is significant. diff --git a/scripts/check_docs.py b/scripts/check_docs.py new file mode 100644 index 0000000..b0f2c4a --- /dev/null +++ b/scripts/check_docs.py @@ -0,0 +1,219 @@ +#!/usr/bin/env python3 +"""Validate markdown links, glossary anchors, and readability rules.""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover - stdlib fallback for minimal envs + yaml = None # type: ignore[assignment] + +ROOT = Path(__file__).resolve().parents[1] +RULES_PATH = Path(__file__).with_name("docs_rules.yaml") + +LINK_RE = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)") +BARE_SEE_RE = re.compile(r"(? dict: + text = path.read_text(encoding="utf-8") + if yaml is not None: + return yaml.safe_load(text) + raise RuntimeError("PyYAML is required to load docs_rules.yaml") + + +def slug_heading(text: str) -> str: + """Anchor slug used in this repository's glossary links.""" + text = text.strip() + text = re.sub(r"\s&\s", " — ", text) + if " — " in text: + left, right = text.split(" — ", 1) + return f"{_slug_part(left)}--{_slug_part(right)}" + return _slug_part(text) + + +def _slug_part(text: str) -> str: + slug = text.strip().lower() + slug = re.sub(r"[^\w\s-]", "", slug) + slug = re.sub(r"\s+", "-", slug) + return slug + + +def collect_anchors(md_path: Path) -> set[str]: + anchors: set[str] = set() + for line in md_path.read_text(encoding="utf-8").splitlines(): + if line.startswith("##"): + anchors.add(slug_heading(line.lstrip("#").strip())) + return anchors + + +def expand_markdown_paths(patterns: list[str]) -> list[Path]: + files: set[Path] = set() + for pattern in patterns: + for path in ROOT.glob(pattern): + if path.is_file() and path.suffix == ".md": + files.add(path) + return sorted(files) + + +def split_link_target(raw: str) -> tuple[str, str | None]: + target = raw.strip() + if target.startswith("<") and target.endswith(">"): + target = target[1:-1] + if "#" in target: + path_part, fragment = target.split("#", 1) + return path_part, fragment or None + return target, None + + +def resolve_path(source: Path, link_path: str) -> Path | None: + if not link_path or link_path.startswith(SKIP_SCHEMES): + return None + candidate = (source.parent / link_path).resolve() + try: + candidate.relative_to(ROOT.resolve()) + except ValueError: + return None + return candidate + + +def is_glossary_link(link_path: str, rules: dict) -> bool: + glossary = rules["glossary"]["file"].replace("\\", "/") + normalized = link_path.replace("\\", "/") + return ( + normalized == glossary + or normalized.endswith("/" + glossary) + or normalized.endswith(glossary.split("/")[-1]) + ) + + +def check_file( + md_path: Path, + rules: dict, + anchor_cache: dict[Path, set[str]], +) -> tuple[list[str], list[str]]: + errors: list[str] = [] + warnings: list[str] = [] + rel = md_path.relative_to(ROOT) + text = md_path.read_text(encoding="utf-8") + + if rules.get("readability", {}).get("forbid_bare_see_refs"): + prose = re.sub(r"`[^`]*`", "", text) + for match in BARE_SEE_RE.finditer(prose): + errors.append( + f"{rel}:{_line_no(text, match.start())}: bare cross-ref " + f"'see {match.group(1)}' — use a markdown link" + ) + + for match in LINK_RE.finditer(text): + raw = match.group(1) + if any(raw.startswith(s) for s in SKIP_SCHEMES): + continue + + link_path, fragment = split_link_target(raw) + if not link_path.endswith(".md"): + continue + + for banned in rules.get("banned_link_targets", []): + if banned in link_path.replace("\\", "/"): + errors.append( + f"{rel}:{_line_no(text, match.start())}: banned link target " + f"'{link_path}'" + ) + + target = resolve_path(md_path, link_path) + if target is None: + errors.append( + f"{rel}:{_line_no(text, match.start())}: link escapes repo or " + f"is invalid: {raw}" + ) + continue + if not target.is_file(): + errors.append( + f"{rel}:{_line_no(text, match.start())}: broken link " + f"{link_path} -> {target.relative_to(ROOT)}" + ) + continue + + if fragment: + if target not in anchor_cache: + anchor_cache[target] = collect_anchors(target) + if fragment not in anchor_cache[target]: + errors.append( + f"{rel}:{_line_no(text, match.start())}: unknown anchor " + f"#{fragment} in {target.relative_to(ROOT)}" + ) + + if fragment and is_glossary_link(link_path, rules): + section_anchors = set(rules.get("glossary_section_anchors", [])) + if fragment in section_anchors: + warnings.append( + f"{rel}:{_line_no(text, match.start())}: glossary section " + f"link #{fragment} — prefer a ### term anchor where one exists" + ) + + return errors, warnings + + +def _line_no(text: str, index: int) -> int: + return text.count("\n", 0, index) + 1 + + +def run_checks(warnings_as_errors: bool = False) -> int: + rules = load_rules(ROOT / "scripts/docs_rules.yaml") + anchor_cache: dict[Path, set[str]] = {} + all_errors: list[str] = [] + all_warnings: list[str] = [] + + glossary_path = ROOT / rules["glossary"]["file"] + anchor_cache[glossary_path] = collect_anchors(glossary_path) + + for md_path in expand_markdown_paths(rules["markdown_paths"]): + errors, warnings = check_file(md_path, rules, anchor_cache) + all_errors.extend(errors) + all_warnings.extend(warnings) + + if all_warnings: + print("Documentation warnings:") + for msg in sorted(all_warnings): + print(f" warning: {msg}") + print() + + if all_errors: + print("Documentation errors:") + for msg in sorted(all_errors): + print(f" error: {msg}") + print(f"\n{len(all_errors)} error(s), {len(all_warnings)} warning(s)") + return 1 + + if warnings_as_errors and all_warnings: + print(f"\n{len(all_warnings)} warning(s) treated as errors") + return 1 + + print( + f"Documentation checks passed " + f"({len(expand_markdown_paths(rules['markdown_paths']))} files, " + f"{len(all_warnings)} warning(s))" + ) + return 0 + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--warnings-as-errors", + action="store_true", + help="Exit non-zero when section-link warnings are present", + ) + args = parser.parse_args() + sys.exit(run_checks(warnings_as_errors=args.warnings_as_errors)) + + +if __name__ == "__main__": + main() diff --git a/scripts/docs_rules.yaml b/scripts/docs_rules.yaml new file mode 100644 index 0000000..63c4233 --- /dev/null +++ b/scripts/docs_rules.yaml @@ -0,0 +1,47 @@ +# Documentation rules enforced by CI (scripts/check_docs.py). +# See docs/improvement/ci-rules.md for rationale and PR workflow. + +markdown_paths: + - docs/**/*.md + - README.md + - CONTRIBUTING.md + - AGENTS.md + - simulator/README.md + - prototype/README.md + - infra/**/README.md + +glossary: + file: docs/00-glossary.md + +# Relative link targets that must never reappear (known regressions). +banned_link_targets: + - 03-storage-design.md + +# Glossary links to these section headers are allowed but reported as warnings +# when a term-specific ### anchor exists (see preferred_term_anchors). +glossary_section_anchors: + - data--storage + - swarm--robotics + - infrastructure--delivery + +# First-use vocabulary should link to these ### anchors, not section headers. +preferred_term_anchors: + - t0--hot + - t1--warm + - t2--shared + - t3--warehouse + - t4--dev + - source-of-truth + - architecture-layer-1--ingestion + - architecture-layer-2--storage-and-transform + - architecture-layer-3--serving-and-sync + - pipeline-stage + - bulk-sync + - offload + - rclone + - adr + - asr + +readability: + # Fail when prose uses bare "see 05" instead of a markdown link. + forbid_bare_see_refs: true diff --git a/simulator/tests/test_check_docs.py b/simulator/tests/test_check_docs.py new file mode 100644 index 0000000..f079da3 --- /dev/null +++ b/simulator/tests/test_check_docs.py @@ -0,0 +1,73 @@ +"""Tests for scripts/check_docs.py — documentation CI rules.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +SCRIPTS = Path(__file__).resolve().parents[2] / "scripts" +sys.path.insert(0, str(SCRIPTS)) + +from check_docs import ( # noqa: E402 + collect_anchors, + run_checks, + slug_heading, + split_link_target, +) + +ROOT = Path(__file__).resolve().parents[2] +GLOSSARY = ROOT / "docs/00-glossary.md" + + +def test_slug_heading_matches_glossary_anchors() -> None: + assert slug_heading("T0 — hot") == "t0--hot" + assert slug_heading("T3 — warehouse") == "t3--warehouse" + assert slug_heading("Source of truth") == "source-of-truth" + assert slug_heading("Data & storage") == "data--storage" + + +def test_glossary_term_anchors_exist() -> None: + anchors = collect_anchors(GLOSSARY) + for term in ( + "t0--hot", + "t1--warm", + "t2--shared", + "t3--warehouse", + "t4--dev", + "source-of-truth", + "bulk-sync", + "offload", + ): + assert term in anchors, f"missing glossary anchor #{term}" + + +def test_split_link_target() -> None: + assert split_link_target("foo.md#bar") == ("foo.md", "bar") + assert split_link_target("foo.md") == ("foo.md", None) + + +def test_repo_docs_pass() -> None: + assert run_checks() == 0 + + +def test_broken_link_detected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + docs = tmp_path / "docs" + docs.mkdir() + (docs / "00-glossary.md").write_text("# Glossary\n\n## Data & storage\n") + (docs / "bad.md").write_text("See [missing](missing.md).\n") + rules = tmp_path / "scripts" + rules.mkdir() + (rules / "docs_rules.yaml").write_text( + "markdown_paths:\n - docs/**/*.md\n" + "glossary:\n file: docs/00-glossary.md\n" + "banned_link_targets: []\n" + "glossary_section_anchors: []\n" + "preferred_term_anchors: []\n" + "readability:\n forbid_bare_see_refs: false\n" + ) + checker = SCRIPTS / "check_docs.py" + monkeypatch.setattr("check_docs.ROOT", tmp_path) + monkeypatch.setattr("check_docs.RULES_PATH", rules / "docs_rules.yaml") + assert run_checks() == 1