diff --git a/AGENTS.md b/AGENTS.md index f80c5bf..985370c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,7 +3,8 @@ Evidence-first brain over the ops/eSlider stack. Facts need proof or they are `(not confirmed)`. -Read first: [PLAN](PLAN.md) → [docs](docs/). +Read first: [PLAN](PLAN.md) → [docs](docs/) → [roadmap](docs/roadmap.md) +(epic [#16](https://git.produktor.io/eSlider/2dph/issues/16)). ## Method (detective, no fork) diff --git a/PLAN.md b/PLAN.md index fc73587..cb6e5e6 100644 --- a/PLAN.md +++ b/PLAN.md @@ -4,7 +4,9 @@ A brain that loves facts and deduction. Evidence-first knowledge graph + hybrid RAG over the operational Brain/ops/eSlider stack. Built like Sherlock Holmes: nothing is asserted unless it has proof. -Status: **in progress** — this file is the plan and the record of decisions. +Status: **in progress** — read path + MCP work; v1 goal is [epic #16](https://git.produktor.io/eSlider/2dph/issues/16) +(milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12)). +Gap: [docs/roadmap.md](docs/roadmap.md). ## What @@ -41,7 +43,7 @@ detective method: **a fact needs ≥2 independent sources or it is | D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. | | D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. | | D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. | -| D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is not shipped; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. | +| D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is compose profile `picoclaw`; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. Agent lever/loop: [#15](https://git.produktor.io/eSlider/2dph/issues/15). | | D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. | | D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). | | D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc` → `zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. | @@ -83,7 +85,8 @@ detective method: **a fact needs ≥2 independent sources or it is Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`. `Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`. -Edges: `RUNS / USES / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`. +Edges: `RUNS / USES / FROM_FILE / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`. +`FROM_FILE` / `HAS_VERSION` exist in schema; search `--hop` does not walk them yet ([#17](https://git.produktor.io/eSlider/2dph/issues/17)). Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, `where`, `when`, `source_rev`. @@ -108,9 +111,9 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, ## Open questions (v2) - OQ1: mutually-contradicting evidence — how to resolve (authority weighting, - temporal freshness, audit adjudication). -- OQ2: OCR pipeline for pdfs/images/docs — mostly solved: poppler pdftotext - fast-path for born-digital PDFs, docling fallback for the ~5% textless ones. + temporal freshness, audit adjudication). **v2**; does not block epic #16. +- OQ2: OCR — poppler `pdftotext` fast-path exists; scans still docling. + [#6](https://git.produktor.io/eSlider/2dph/issues/6) (v2, does not block #16). - OQ3: optional duckdb-md layer for `SELECT … FORMAT MARKDOWN` export/write-back. - OQ4: YAML-first storage for leafs — deferred: JSON is ~10x faster to serialize and unambiguous; YAML only where humans edit files. @@ -138,6 +141,7 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, 3. python -m unittest discover -s bin/tools (includes published-docs SoT) 4. `bin/facts/audit self` (lexicon internal consistency; `bin/facts/audit.go` is the D14 wrapper) 5. `bin/kb/eval` (recall@5 ≥ 0.95). Local SoT is `bin/brain/eval.go` via Zig CGO. + CI SoT switch: [#19](https://git.produktor.io/eSlider/2dph/issues/19). 6. `bin/cgo/zig go build -tags system_ladybug` (compile search with zig cc; fetches pinned zig+libs). Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as @@ -151,5 +155,23 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl 4. .venv: ladybug + model2vec + mistune 5. schema + tools with TDD (kb + md + facts + brain) 6. ~/.config/brain config -7. corpus extraction (facts/info) -8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit \ No newline at end of file +7. corpus extraction (facts/info) — **open**: [#18](https://git.produktor.io/eSlider/2dph/issues/18) +8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit + +## Gap to v1 (epic #16) + +Read path + MCP are in. The detective brain is not closed until the graph is +**writable incrementally** and search can **walk** it. Board: +[epic #16](https://git.produktor.io/eSlider/2dph/issues/16), +milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12). +Narrative: [docs/roadmap.md](docs/roadmap.md). + +| Order | Issue | Gap | +|-------|-------|-----| +| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | Write stays Python rebuild; `brain/add` / `POST /ingest` are hints. Ladybug 0.19 WAL corrupts if new leafs land while FTS/HNSW exist. | +| 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | `--hop` errors. `FROM_FILE` / `HAS_VERSION` are in schema; search does not walk them. | +| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. | +| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | Lever = 2dph fact-check. Loop = PicoClaw/MCP `search` → `get` → `audit`. Specify in-repo, not only live config. | +| 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. | + +Does **not** block epic close: [#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR, OQ1, OQ3, OQ4. \ No newline at end of file diff --git a/README.md b/README.md index b31a948..20a72ea 100644 --- a/README.md +++ b/README.md @@ -172,7 +172,9 @@ docker compose up brain-watch # auto re-index on change skills (`web-search`, `postgres`, …) that 2dph integrates - detective method — the two-source method -Work board (issues): [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues). +Work board (issues): [epic #16](https://git.produktor.io/eSlider/2dph/issues/16) +on [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues). PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph). -See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions. +See [PLAN.md](PLAN.md) for decisions, [docs/roadmap.md](docs/roadmap.md) for +the gap to v1, and v2 open questions. diff --git a/bin/tools/test_published_docs.py b/bin/tools/test_published_docs.py index 8d6ae3b..65d9d78 100644 --- a/bin/tools/test_published_docs.py +++ b/bin/tools/test_published_docs.py @@ -147,6 +147,7 @@ class PublishedDocsTest(unittest.TestCase): ROOT / "skills" / "diataxis-docs" / "SKILL.md", ROOT / "docs" / "runbook.md", ROOT / "docs" / "README.md", + ROOT / "docs" / "roadmap.md", ] # Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie. # Honest "not implemented" notes must not match. @@ -173,3 +174,19 @@ class PublishedDocsTest(unittest.TestCase): readme = (ROOT / "README.md").read_text() self.assertIn("docs/runbook.md", readme) self.assertNotIn("search.ops.io", readme) + + def test_v1_epic_is_named_in_docs(self) -> None: + plan = (ROOT / "PLAN.md").read_text() + self.assertIn("Gap to v1", plan) + self.assertIn("eSlider/2dph/issues/16", plan) + self.assertIn("eSlider/2dph/issues/17", plan) + self.assertIn("eSlider/2dph/milestone/12", plan) + road = (ROOT / "docs" / "roadmap.md").read_text() + self.assertIn("type: explanation", road) + self.assertIn("issues/16", road) + self.assertIn("issues/14", road) + index = (ROOT / "docs" / "README.md").read_text() + self.assertIn("roadmap.md", index) + self.assertIn("epic #16", index) + agents = (ROOT / "AGENTS.md").read_text() + self.assertIn("roadmap.md", agents) diff --git a/docs/README.md b/docs/README.md index 522821b..1c79412 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,6 +5,7 @@ related: - docs/runbook.md - docs/design.md - PLAN.md + - docs/roadmap.md --- # 2dph docs (Diataxis) @@ -16,6 +17,7 @@ Evidence-first knowledge graph. Facts need proof or they are |------|-----| | tutorial / howto | [runbook](runbook.md) — run anywhere (uv, Go, Docker) | | explanation | [design](design.md) — two roots, deduction, D17/D20/D18 | +| explanation | [roadmap](roadmap.md) — gap to v1 (epic #16) | | howto | [picoclaw](picoclaw.md) — MCP agent profile | | howto | [reasoner](reasoner.md) — CPU bake-off (D18) | | reference | [PLAN.md](../PLAN.md) — decisions D1–D21 | @@ -26,9 +28,11 @@ Python write sidecar, **D14** `bin/{subject}/{method}.go`, **D15** Gitea origin, Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go` — `/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is -not a walk; the flag errors until File/FROM_FILE edges exist. +not a walk; the flag errors. Schema has `FROM_FILE`; search does not +use it ([#17](https://git.produktor.io/eSlider/2dph/issues/17)). -Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues). +Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) +([epic #16](https://git.produktor.io/eSlider/2dph/issues/16)). PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph). Published docs live here and match live commands. diff --git a/docs/design.md b/docs/design.md index aef8d33..c16b85d 100644 --- a/docs/design.md +++ b/docs/design.md @@ -4,6 +4,7 @@ status: current related: - docs/README.md - docs/runbook.md + - docs/roadmap.md --- # Design — facts, info, deduction @@ -33,8 +34,9 @@ bin/brain/search.go "question" is not evidence of absence; `--no-web` / `--root` skip it) ``` -`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an -error; it is not a graph walk. +`--hop` is not implemented. `FROM_FILE` / `HAS_VERSION` exist in schema; +search does not walk them ([#17](https://git.produktor.io/eSlider/2dph/issues/17)). +The flag is an error; it is not a graph walk. ## Who / What / How / Where / When + evidence @@ -94,4 +96,7 @@ Pluggable OpenAI-compatible URL. RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU sidecar: compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `127.0.0.1:11435`). Bake-off: `bin/reasoner/bakeoff.go`. Weights stay out -of the 2dph image. See [docs/reasoner.md](reasoner.md). \ No newline at end of file +of the 2dph image. See [docs/reasoner.md](reasoner.md). + +Gap to v1 (write, hops, corpus, agent loop): [roadmap](roadmap.md), +[epic #16](https://git.produktor.io/eSlider/2dph/issues/16). \ No newline at end of file diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..6e0552d --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,67 @@ +--- +type: explanation +status: current +related: + - PLAN.md + - docs/design.md + - docs/runbook.md +--- + +# Gap to v1 — detective brain + +Goal: a brain that does not assert without proof. Search is deduction +(`facts` ≥2 sources → `info` → `web`). `confirmed` only from the facts root. + +**v1 is a living graph the agent can write and walk**, not “more RAG”. + +Epic: [Gitea #16](https://git.produktor.io/eSlider/2dph/issues/16). +Milestone: [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12). +Decisions: [PLAN.md](../PLAN.md). + +## In (do not reopen) + +Read path Go + Zig CGO (D21). HTTP + OpenAPI + MCP (D20). PicoClaw compose +profile + CPU reasoner (D18). Mail sync → import → rebuild. D14 shebangs. +Compose `api` (no CPython) / `index` (Python write). Issues #1–#5, #7–#13. + +`POST /ingest` is a rebuild **hint**. `add` is not implemented. + +## Blockers + +``` +question + │ + ├─ FTS + HNSW ← in + ├─ facts / info roots ← in + ├─ web (D17) ← in + ├─ Cypher hop ← #17 schema yes, search no + ├─ brain/add ACID ← #14 rebuild only + └─ facts+chats corpus ← #18 +``` + +1. **[#14](https://git.produktor.io/eSlider/2dph/issues/14) write** — + `bin/brain/index.go --rebuild` (Python `kblib`). No incremental + `brain/add`. Watch/mail/git cannot become facts “now”. +2. **[#17](https://git.produktor.io/eSlider/2dph/issues/17) hops** — + `Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person` + exists; `--hop` still errors. Without a walk, D9/D10 are paper. +3. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** — + rebuild loads repo markdown + mail as `info`. `facts/extract` pairing + and `bin/chats` are not indexed. WhatsApp is a stub. PII stays in `var/`. +4. **[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop** — + 2dph is the lever (`search` → `get` → `audit`). PicoClaw is the loop. + Document the contour in-repo; CPU turns need a large context window. +5. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** — + recall SoT should be `bin/brain/eval.go` via Zig, not Python `bin/kb/eval`. + +## Not v1 + +[#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR (OQ2), OQ1 +contradiction resolution, OQ3 duckdb-md export, OQ4 YAML-first leafs. + +## Close epic #16 when + +- facts+info can be written without a full rebuild for every leaf +- `--hop` stops erroring and runs a Cypher path from search hits +- ops pairing + chat import land as leafs on rebuild +- MCP tool order is documented and still gated by tests diff --git a/docs/runbook.md b/docs/runbook.md index 3658fc3..3f1087e 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -55,6 +55,7 @@ bin/brain/stats.go ``` `--hop` is not implemented. Empty web results are `throttled`, not absence. +Gap to v1: [roadmap](roadmap.md) / [epic #16](https://git.produktor.io/eSlider/2dph/issues/16). Ladybug 0.19: never `DROP INDEX` FTS/VECTOR (ghost catalog). Fresh indexes = delete `var/kb.lbug` then `--rebuild`. diff --git a/skills/brain/SKILL.md b/skills/brain/SKILL.md index d0afa3e..261cd05 100644 --- a/skills/brain/SKILL.md +++ b/skills/brain/SKILL.md @@ -31,8 +31,9 @@ bin/brain/stats.go # index health bin/brain/eval.go # recall@5 >= 0.95 gate (Go; Python bin/kb/eval is CI fallback) ``` -`bin/kb/search` is a deprecated wrapper. `--hop` errors (File/FROM_FILE edges -are not wired yet); do not treat it as a graph walk. +`bin/kb/search` is a deprecated wrapper. `--hop` errors (schema has +`FROM_FILE`; search does not walk it yet, [#17](https://git.produktor.io/eSlider/2dph/issues/17)); +do not treat it as a graph walk. ## Rules