diff --git a/PLAN.md b/PLAN.md index 5f66157..d086911 100644 --- a/PLAN.md +++ b/PLAN.md @@ -26,10 +26,10 @@ detective method: **a fact needs ≥2 independent sources or it is |---|----------|--------| | D1 | RAG corpus | ops stack (chat, onlyoffice, gitea/NPM, searchxng, observability, ai-bot, mcp-servers, `~/.ssh/config`) + portfolio. Exclude `office.dev` + jobs/applications. | | D2 | skill merging | integrate skills **in this project** `skills/`; skip gitea / brain-dependent skills. | -| D3 | web search | import `web-search`, retire local `searxng-ops`. Vendored here, no remote link. | +| D3 | web search | Vendored client; SearXNG URL is config. Optional Compose instance (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. | | D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. | | D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). | -| D6 | graph engine | **LadybugDB** (Kuzu successor, MIT, embedded, native FTS+vector+Cypher). Python binding for `bin/*`; Go shebang for golang tools. | +| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `internal/brain`); Python remains for index/write until the Go write path is safe. | | D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). | | D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. | | D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. | @@ -40,6 +40,8 @@ detective method: **a fact needs ≥2 independent sources or it is | D14 | tooling style | `bin/{subject}/{method}.go` shebang (e.g. `bin/brain/search.go`). Shared code in `internal/`. One root `go.mod` + `go.work`. No `bin/*/main.go`, no nested modules. | | D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. | | D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. | +| D17 | assertion gate | Fact-check every *claim* (facts → info → live sources → web), not every edit. Missing graph ≠ “does not exist”. | +| D18 | reasoner | Pluggable OpenAI-compatible URL. RAM: Qwen3.5-9B. Quality: Bonsai-27B or Qwen3.6-27B. No official Qwen3.6-9B. | ## Architecture @@ -51,9 +53,10 @@ detective method: **a fact needs ≥2 independent sources or it is bin/ facts/extract auto-pair 2 sources → lexicon yaml + graph facts/audit ["self"|"facts"|"info"|"stale"] 2-source + staleness gate - kb/index build FTS + HNSW from corpus - kb/search deduction: facts → info → web-search; --hop N + kb/index build FTS + HNSW from corpus (Python, for now) + brain/search.go deduction: facts → info → web-search kb/get kb/stats kb/eval + brain/serve.go HTTP API (internal/httpapi) md/import md/select md/tables md/gaps (mistune) brain/extract brain/audit brain/deduce (thinking wrapper) web/search (vendored) @@ -110,19 +113,18 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and indexing stay separate for crash safety. 4. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable - via `bin/kb/search`. + via `bin/brain/search.go`. ## CI/CD pipeline (D15) `.github/workflows/ci.yml`: -1. go vet + go test ./... (Go tools; root module) -2. `go test ./rank` in `bin/kbsearch` (cgo-free ranking + flag parser; nested module still needs ladybug for the rest) -3. `go test ./...` in `bin/chats` (Telegram + LinkedIn parsers; nested module) -4. python -m unittest discover (Py tools) -5. bin/facts/audit self (lexicon internal consistency) -6. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions) -7. md-docs build/lint if docs tooling arrives. +1. go vet + go test ./... (root module; packages without ladybug cgo) +2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser) +3. python -m unittest discover -s bin/tools (includes published-docs SoT) +4. bin/facts/audit self (lexicon internal consistency) +5. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions) +6. md-docs build/lint if docs tooling arrives. Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as `db/tech-poc`: contract first where there is an OpenAPI/message shape. diff --git a/README.md b/README.md index 6e41631..4acd857 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ graph TB AU["bin/facts/audit
confidence + staleness"] IDX["bin/kb/index
chunk + embed"] MD["bin/md/import
mistune leaves"] - SR["bin/kb/search
deduction + --hop"] + SR["bin/brain/search.go
deduction"] end subgraph store["Ladybug var/kb.lbug"] @@ -85,21 +85,23 @@ fact; conflicting sources or a single source → `hypothesis` → `(not confirme ## Deduction search ```bash -bin/kb/search "Matrix federation over HTTPS" # facts → info → web-search -bin/kb/search "what runs on arc-2" --hop 1 # walk graph edges -bin/kb/search "where is cs-lexicon" --json | yq '.' # YAML by default -bin/kb/get --body # full chunk on demand -bin/kb/stats # index health -bin/kb/eval # recall@5 gate +bin/brain/search.go "Matrix federation over HTTPS" # facts → info → web-search +bin/brain/search.go "onlyoffice postgres" --root facts +bin/brain/search.go "where is cs-lexicon" --json | yq '.' +bin/kb/get --body # full chunk on demand +bin/kb/stats # index health +bin/kb/eval # recall@5 gate ``` +`--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`. + Mail is a first-class corpus (retrievable through the same search): ```bash bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go) bin/mail/import --from-raw var/mail # JSON → markdown bin/mail/index_mail # rebuild brain incl. mail -bin/kb/search "Mietwagen Nürnberg invoice" # now answers from mail +bin/brain/search.go "invoice from last week" # same search over mail leafs ``` ## Storage @@ -117,10 +119,10 @@ bin/kb/search "Mietwagen Nürnberg invoice" # now a ## Tooling conventions -`bin/{subject}/{method}` — self-describing: shebang on line 1, usage comment -from line 2. bash + python primary; golang via the Go shebang when a compiled -helper is right. YAML default output, `--json` for machines. Everything that -touches network/db is read-only, throttled, cached. Tests gate every commit. +`bin/{subject}/{method}.go` — self-describing: shebang on line 1, usage comment +from line 2. Shared code in `internal/`. YAML default output, `--json` for +machines. Tests gate every commit. HTTP: `bin/brain/serve.go` (default search +binary `var/bin/brain-search`, not Python). ## Development @@ -136,7 +138,7 @@ Docker (optional, cached model + var volumes): ```bash docker compose run --rm brain index # (re)index corpus docker compose run --rm brain search "query" # one-shot query -docker compose run --rm brain serve # async Go HTTP server +docker compose run --rm brain serve # bin/brain/serve.go docker compose up brain-watch # auto re-index on change ``` diff --git a/bin/tools/test_published_docs.py b/bin/tools/test_published_docs.py new file mode 100644 index 0000000..5ded0b7 --- /dev/null +++ b/bin/tools/test_published_docs.py @@ -0,0 +1,48 @@ +"""Published docs must match live commands (Gitea SoT, brain/search, no fake --hop).""" +from __future__ import annotations + +import re +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] + + +class PublishedDocsTest(unittest.TestCase): + def test_readme_points_issues_at_gitea(self) -> None: + text = (ROOT / "README.md").read_text() + self.assertIn( + "https://git.produktor.io/eSlider/2dph/issues", + text, + "README must point issues at Gitea", + ) + + def test_plan_d15_names_gitea_origin(self) -> None: + text = (ROOT / "PLAN.md").read_text() + self.assertIn("D15", text) + self.assertIn("git.produktor.io/eSlider/2dph", text) + + def test_readme_primary_search_is_brain(self) -> None: + text = (ROOT / "README.md").read_text() + self.assertIn( + "bin/brain/search.go", + text, + "README deduction search must name bin/brain/search.go", + ) + + def test_docs_do_not_claim_hop_walks(self) -> None: + paths = [ + ROOT / "README.md", + ROOT / "docs" / "design.md", + ROOT / "skills" / "kb-search" / "SKILL.md", + ROOT / "skills" / "diataxis-docs" / "SKILL.md", + ] + # Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie. + # Honest "not implemented" notes must not match. + lie = re.compile(r"--hop (?:N|1).*(?:follow|walk)", re.I | re.S) + for path in paths: + text = path.read_text() + self.assertIsNone( + lie.search(text), + f"{path.relative_to(ROOT)} still claims --hop walks the graph", + ) diff --git a/docs/README.md b/docs/README.md index 9b1e72b..bc6923e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,4 +8,7 @@ Brain/ops/eSlider stack. Facts need proof or they are - [design](design.md) — schema, deduction model, sources - [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) — work board (origin) +Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`). `--hop` is +not a walk; the flag errors until File/FROM_FILE edges exist. + Published docs live here and mirror the project state. diff --git a/docs/design.md b/docs/design.md index 198022d..1de4020 100644 --- a/docs/design.md +++ b/docs/design.md @@ -17,14 +17,14 @@ as one consistent state. ## Deduction search ``` -bin/kb/search "question" +bin/brain/search.go "question" 1. facts root — confirmed answers only → return with evidence links 2. info root — supporting narrative → snippets, marked (not confirmed) 3. web-search — second independent source → upgrade hypothesis to confirmed ``` -`--hop N` follows graph edges (sibling leaves under a heading, owning file, -`related:` files, vector-neighbour leaves) — the deduction walk. +`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an +error; it is not a graph walk. ## Who / What / How / Where / When + evidence diff --git a/skills/diataxis-docs/SKILL.md b/skills/diataxis-docs/SKILL.md index 16bc675..d0cf77b 100644 --- a/skills/diataxis-docs/SKILL.md +++ b/skills/diataxis-docs/SKILL.md @@ -30,12 +30,11 @@ related: --- ``` -`bin/kb/index` reads this. `type` becomes a searchable column and `related` -becomes a graph edge: +`bin/kb/index` reads this. `type` becomes a searchable column. `related:` is +frontmatter for humans; graph hops from it are not implemented yet. ```bash -bin/kb/search "deploy" --type howto -bin/kb/search "Stecktafel" --hop 1 # follow links and related +bin/brain/search.go "deploy" ``` ## Audit checklist @@ -46,8 +45,7 @@ bin/kb/search "Stecktafel" --hop 1 # follow links and related The indexer chunks on H2, so split files also search better. 3. Is `status: archive` set on anything superseded? Archived files stay indexed but stop competing with current ones for a reader's attention. -4. Does every explanation link the reference it explains, and vice versa? That - link is what `--hop 1` walks. +4. Does every explanation link the reference it explains, and vice versa? ## Rule diff --git a/skills/kb-search/SKILL.md b/skills/kb-search/SKILL.md index 3800da3..9302de2 100644 --- a/skills/kb-search/SKILL.md +++ b/skills/kb-search/SKILL.md @@ -2,9 +2,10 @@ name: kb-search description: >- Deduction search over the 2dph brain (Ladybug graph: ops corpus, portfolio, - ssh hosts) with bin/kb/search instead of reading files or grepping repos. - Use whenever a question starts with "where is", "what runs on", "which file - describes", "who is", "how is X done", before opening any documentation. + ssh hosts) with bin/brain/search.go instead of reading files or grepping + repos. Use whenever a question starts with "where is", "what runs on", + "which file describes", "who is", "how is X done", before opening any + documentation. --- # kb-search — deduction over facts and info @@ -22,15 +23,17 @@ second independent source when local roots cannot confirm. An answer is `(not confirmed)`. ```bash -bin/kb/search "Matrix federation" # pointers + snippets, YAML -bin/kb/search "what runs on arc-2" --hop 1 # follow graph edges -bin/kb/search "onlyoffice postgres" --root facts # restrict to confirmed -bin/kb/search "where is cs-lexicon" --json | yq '.[].ref' -bin/kb/get --body # full chunk only when needed -bin/kb/stats # index health -bin/kb/eval # recall@5 >= 0.95 gate +bin/brain/search.go "Matrix federation" # pointers + snippets, YAML +bin/brain/search.go "onlyoffice postgres" --root facts # restrict to confirmed +bin/brain/search.go "where is cs-lexicon" --json | yq '.[].ref' +bin/kb/get --body # full chunk only when needed +bin/kb/stats # index health +bin/kb/eval # recall@5 >= 0.95 gate ``` +`bin/kb/search` is a deprecated wrapper. `--hop` errors (File/FROM_FILE edges +are not wired yet); do not treat it as a graph walk. + ## Rules - Search before you read. Never grep a repo for a concept the graph covers. @@ -38,8 +41,6 @@ bin/kb/eval # recall@5 >= 0.95 gate facts first, then info leafs clearly marked `(not confirmed)`. - If recall looks wrong, run `bin/kb/eval`; it gates control questions and should stay at or above 95% recall@5. -- `--hop N` follows sibling leaves, owning files, `related:` links and - vector-neighbours — that is the deduction walk, not random expansion. - Escalate to `web-search` (the `web-search` skill) as the independent second source when both local roots cannot confirm; never report an unconfirmed single-source local answer as fact. \ No newline at end of file diff --git a/skills/web-search/SKILL.md b/skills/web-search/SKILL.md index 9e32df1..0e6c707 100644 --- a/skills/web-search/SKILL.md +++ b/skills/web-search/SKILL.md @@ -18,9 +18,10 @@ bin/web/search "postgres partial index" --lang en --fresh year ## Web or knowledge base -`bin/kb/search` holds our own facts: the ops stack, portfolio, ssh hosts, the -lexicon. Go there first. Reach for `bin/web/search` when the answer is outside -our repos: upstream library behaviour, vendor documentation, public standards. +`bin/brain/search.go` holds our own facts: the ops stack, portfolio, ssh hosts, +the lexicon. Go there first. Reach for `bin/web/search` when the answer is +outside our repos: upstream library behaviour, vendor documentation, public +standards. Keep the two apart. A finding is stronger when the reader can see that one source was ours and one was not.