diff --git a/AGENTS.md b/AGENTS.md index 7f76d15..7e05e37 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,7 @@ var/ kb.lbug, var/mail/*, caches (gitignored) bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox) bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only) -bin/brain/index.go --rebuild # rebuild brain incl. all mail (fresh DB) +bin/brain/index.go --rebuild --with-facts --with-chats ``` - `sync` (Go) downloads messages + attachments; Gmail uses paginated list + @@ -89,6 +89,7 @@ bin/kb/search "query" [--repo X] # deprecated wrapper → bin/b bin/brain/search.go "query" [--root facts|info] # deduction search → YAML bin/brain/search.go "query" --no-web # local graph only eval "$(bin/cgo/zig env)" # Zig cc + liblbug (not gcc) +bin/brain/index.go --rebuild [--with-mail] [--with-facts] [--with-chats] bin/brain/add.go --text T --root facts --source "a.md x b.md" # incremental write bin/brain/add.go --json # stdin leaf or {leafs:[...]} bin/brain/get.go [--body] [--json] # Go read; Python bin/kb/get CI fallback diff --git a/PLAN.md b/PLAN.md index e39fc9f..b9338e2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -159,13 +159,12 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl 4. .venv: ladybug + model2vec + mistune 5. schema + tools with TDD (kb + md + facts + brain) 6. ~/.config/brain config -7. corpus extraction (facts/info) — **open**: [#18](https://git.produktor.io/eSlider/2dph/issues/18) +7. corpus extraction (facts/info) — **in**: [#18](https://git.produktor.io/eSlider/2dph/issues/18) 8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit ## Gap to v1 (epic #16) -Read path + MCP are in. Incremental `brain/add` and `--hop` are in. Remaining: -facts+chats corpus on rebuild, and CI eval SoT. Board: +Remaining: CI eval SoT. Board: [epic #16](https://git.produktor.io/eSlider/2dph/issues/16), milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12). Narrative: [docs/roadmap.md](docs/roadmap.md). @@ -174,7 +173,7 @@ Narrative: [docs/roadmap.md](docs/roadmap.md). |-------|-------|-----| | 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in** — `bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. | | 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | **in** — `--hop N` walks `FROM_FILE` → `HAS_VERSION` → `AUTHORED` (max 3). | -| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. | +| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | **in** — `--with-facts` / `--facts-json` land `root=facts`; `--with-chats` indexes `var/chats/md`. WhatsApp sync is out of v1. | | 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search` → `get` → `audit`). | | 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. | diff --git a/README.md b/README.md index 5d88229..70ea26a 100644 --- a/README.md +++ b/README.md @@ -121,6 +121,7 @@ Mail is a first-class corpus (retrievable through the same search): bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go) bin/mail/import.go --from-raw var/mail # JSON → markdown bin/brain/add.go --text T --root facts --source "a.md x b.md" +bin/brain/index.go --rebuild --with-facts --with-chats # facts extract + chats md bin/brain/index.go --rebuild # rebuild brain (incl. mail) bin/brain/search.go "invoice from last week" # same search over mail leafs ``` diff --git a/bin/brain/index.go b/bin/brain/index.go index d896632..818f6e2 100755 --- a/bin/brain/index.go +++ b/bin/brain/index.go @@ -3,7 +3,7 @@ // // bin/brain/index.go - rebuild the Ladybug graph (Python write path). // -// ./bin/brain/index.go --rebuild +// ./bin/brain/index.go --rebuild --with-facts --with-chats // ./bin/brain/index.go --rebuild --with-mail // ./bin/brain/index.go --dry-run --with-mail // diff --git a/bin/chats/sync.go b/bin/chats/sync.go index 14eaac0..299116f 100755 --- a/bin/chats/sync.go +++ b/bin/chats/sync.go @@ -29,10 +29,11 @@ func main() { case "linkedin": os.Exit(chats.RunSyncLinkedIn(args)) case "whatsapp": - fmt.Fprintln(os.Stderr, "chats: WhatsApp not implemented yet") + fmt.Fprintln(os.Stderr, "chats: WhatsApp sync is out of v1") os.Exit(1) case "help", "-h", "--help": - fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`) + fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags] +WhatsApp sync is out of v1.`) return default: fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform) diff --git a/bin/kb/index b/bin/kb/index index 6705238..342e6f5 100755 --- a/bin/kb/index +++ b/bin/kb/index @@ -2,12 +2,15 @@ """kb/index - build the 2dph brain from markdown + factual leafs. bin/kb/index [--corpus DIR] [--rebuild] [--limit N] + bin/kb/index --rebuild --with-facts --with-chats bin/kb/index --json # emit stats as JSON Reads every .md under the corpus (default: repo root docs, skills, READMEs) as `info` leafs, embeds them with model2vec (potion-multilingual-128M), and writes them into var/kb.lbug with FTS + HNSW indexes. `facts` leafs come -from bin/facts/extract (docker x compose x ssh-config pairing). +from bin/facts/extract (docker × compose × ssh-config pairing) when +`--with-facts` is set. `--with-chats` indexes markdown under var/chats/md +(or a given dir) as info. WhatsApp sync stays out of v1. --rebuild drops the database file and indexes from scratch. Without it a run is idempotent (MERGE by (source,text) id). @@ -22,7 +25,7 @@ ROOT = Path(__file__).resolve().parents[2] sys.path.insert(0, str(ROOT / "bin" / "tools")) from kblib import ( # noqa: E402 - connect, ensure_indexes, init_schema, upsert_leaf, link_from_file, + add_leafs, connect, ensure_indexes, init_schema, upsert_leaf, link_from_file, open_readonly, stats, ) from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402 @@ -96,12 +99,65 @@ def embedder(): return lambda text: model.encode([text])[0].astype(float).tolist() +def index_fact_dicts(conn, facts: list[dict], embed_fn) -> int: + """Write extract-shaped dicts as root=facts leafs (2-source source field).""" + leafs = [] + for f in facts: + text = str(f.get("text") or "") + source = str(f.get("source") or "") + if not text or not source: + continue + leafs.append({ + "text": text, + "root": "facts", + "confidence": "confirmed", + "source": source, + "source_rev": f.get("source_rev") or "working-tree", + "how": f.get("how") or "facts/extract", + "loc": f.get("loc") or source, + "type": "fact", + "embedding": embed_fn(text) if text else None, + }) + return len(add_leafs(conn, leafs)) + + +def facts_from_extract() -> list[dict]: + import subprocess + proc = subprocess.run( + [sys.executable, str(ROOT / "bin" / "facts" / "extract"), "--json", "--dry-run"], + cwd=ROOT, + capture_output=True, + text=True, + check=False, + ) + if proc.returncode != 0: + print(f"kb/index: facts/extract failed: {proc.stderr}", file=sys.stderr) + return [] + try: + payload = json.loads(proc.stdout) + except json.JSONDecodeError: + print("kb/index: facts/extract produced non-JSON", file=sys.stderr) + return [] + return list(payload.get("facts") or []) + + def main(argv: list[str]) -> int: import argparse p = argparse.ArgumentParser(description="build the 2dph brain index") p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)") p.add_argument("--rebuild", action="store_true", help="fresh db + indexes") + p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)") + p.add_argument("--no-defaults", action="store_true", help="do not index repo README/docs/skills") p.add_argument("--with-mail", action="store_true", help="include var/mail message.md leafs") + p.add_argument("--with-facts", action="store_true", help="run facts/extract into root=facts") + p.add_argument("--facts-json", default="", help="JSON list (or {facts:[...]}) of fact dicts") + p.add_argument( + "--with-chats", + nargs="?", + const=str(ROOT / "var" / "chats" / "md"), + default="", + help="index chat markdown as info (default var/chats/md)", + ) p.add_argument("--since", default="", help="with --with-mail, only messages dated >= YYYY-MM-DD") p.add_argument("--dry-run", action="store_true", help="count leafs, write nothing") p.add_argument( @@ -115,44 +171,71 @@ def main(argv: list[str]) -> int: from kblib import DB_PATH, VAR - leafs = load_corpus(ROOT) + dbpath = Path(a.db) if a.db else DB_PATH + leafs: list[dict] = [] if a.no_defaults else load_corpus(ROOT) if a.corpus: for source in a.corpus: leafs.extend(load_corpus_glob(source)) + chat_n = 0 + if a.with_chats: + chats = load_corpus_glob(a.with_chats) + chat_n = len(chats) + leafs.extend(chats) mail_n = 0 if a.with_mail: mail = from_mail_root(ROOT / "var" / "mail", since=a.since) mail_n = len(mail) leafs.extend(mail) + facts: list[dict] = [] + if a.facts_json: + raw = Path(a.facts_json).read_text(encoding="utf-8") + payload = json.loads(raw) + facts = list(payload.get("facts") if isinstance(payload, dict) else payload) + if a.with_facts: + facts.extend(facts_from_extract()) + if a.dry_run: - msg = {"indexed": 0, "corpus_total": len(leafs), "mail_leafs": mail_n, "dry_run": True} + msg = { + "indexed": 0, + "corpus_total": len(leafs), + "mail_leafs": mail_n, + "chat_leafs": chat_n, + "facts_leafs": len(facts), + "dry_run": True, + } print(json.dumps(msg, indent=2) if a.json else - f"brain/index: {len(leafs)} leafs would be indexed (mail={mail_n})") + f"brain/index: {len(leafs)} info + {len(facts)} facts would be indexed") return 0 VAR.mkdir(exist_ok=True) - if a.rebuild and DB_PATH.exists(): - DB_PATH.unlink() + dbpath.parent.mkdir(parents=True, exist_ok=True) + if a.rebuild and dbpath.exists(): + dbpath.unlink() - db, conn = connect(DB_PATH, read_only=False) + db, conn = connect(dbpath, read_only=False) init_schema(conn) - # Never DROP FTS/VECTOR (ghost catalog). Write leafs, then ensure indexes - # unless --skip-indexes (seed facts first — MERGE under live FTS corrupts it). - # --rebuild already deleted kb.lbug above, so CREATE runs on a clean DB. embed = embedder() done, total = index_leafs(conn, leafs, embed, a.limit) + fact_n = index_fact_dicts(conn, facts, embed) if facts else 0 if not a.skip_indexes: ensure_indexes(conn) s = stats(conn) conn.close() db.close() - result = {"indexed": done, "corpus_total": total, **{k: v for k, v in s.items() if k in ("total", "by_root")}} + result = { + "indexed": done, + "corpus_total": total, + "facts_leafs": fact_n, + "chat_leafs": chat_n, + **{k: v for k, v in s.items() if k in ("total", "by_root")}, + } if a.skip_indexes: result["indexes"] = "skipped" - print(json.dumps(result, indent=2) if a.json else f"indexed {done}/{total} leafs; db total {s['total']}") + print(json.dumps(result, indent=2) if a.json else + f"indexed {done}/{total} info + {fact_n} facts; db total {s['total']}") return 0 diff --git a/bin/tools/test_index_adapters.py b/bin/tools/test_index_adapters.py index 638b226..09efd9b 100644 --- a/bin/tools/test_index_adapters.py +++ b/bin/tools/test_index_adapters.py @@ -39,3 +39,59 @@ class IndexAdapterTest(unittest.TestCase): self.assertTrue(msg.get("dry_run")) self.assertGreaterEqual(msg.get("corpus_total", 0), 1) self.assertFalse(lbug.exists(), "dry-run must not create a Ladybug file") + + def test_facts_json_and_chats_land_on_rebuild(self) -> None: + """Gitea #18: facts (2-source) + chats markdown become leafs on rebuild.""" + tmp = Path(tempfile.mkdtemp()) + dbpath = tmp / "kb.lbug" + chats = tmp / "chats" + chats.mkdir() + (chats / "alice.md").write_text( + "# Chat\n\n## Alice and Bob\n\nhello from chats fixture unique-chat-token\n", + encoding="utf-8", + ) + facts_path = tmp / "facts.json" + facts_path.write_text(json.dumps([{ + "text": "container 'brain' unique-fact-token is running and declared in compose.yaml", + "source": "docker ps x compose.yaml", + "loc": "compose.yaml:brain", + "how": "facts/extract", + }]), encoding="utf-8") + venv_py = ROOT / ".venv" / "bin" / "python" + py = str(venv_py) if venv_py.is_file() else sys.executable + proc = subprocess.run( + [ + py, str(ROOT / "bin" / "kb" / "index"), + "--rebuild", "--db", str(dbpath), "--no-defaults", + "--with-chats", str(chats), + "--facts-json", str(facts_path), + "--json", + ], + cwd=ROOT, + capture_output=True, + text=True, + env=os.environ.copy(), + check=False, + ) + self.assertEqual(proc.returncode, 0, proc.stderr) + msg = json.loads(proc.stdout) + self.assertGreaterEqual(msg.get("facts_leafs", 0), 1) + self.assertGreaterEqual(msg.get("chat_leafs", 0), 1) + self.assertTrue(dbpath.exists()) + sys.path.insert(0, str(ROOT / "bin" / "tools")) + import kblib + db, conn = kblib.connect(dbpath, read_only=True) + try: + stats = kblib.stats(conn) + self.assertGreaterEqual(stats["by_root"].get("facts", 0), 1) + fts = kblib.query_fts(conn, "unique-chat-token", 5) + self.assertTrue(fts, "chats markdown must be FTS-searchable") + fact_hits = kblib.query_fts(conn, "unique-fact-token", 5) + self.assertTrue(any(h.get("root") == "facts" for h in fact_hits)) + src = conn.execute( + "MATCH (l:Leaf {root:'facts'}) RETURN l.source" + ).get_all() + self.assertTrue(any(" x " in str(r[0]) for r in src)) + finally: + conn.close() + db.close() diff --git a/docs/chat-import-plan.md b/docs/chat-import-plan.md index 91875e0..3f574b1 100644 --- a/docs/chat-import-plan.md +++ b/docs/chat-import-plan.md @@ -21,4 +21,5 @@ OO_CLI (default: $HOME/go/bin/oo) ./bin/chats/apply.go --dry-run ``` -JSONL → markdown only. Brain ingest is `bin/brain/index.go` (not a `chats index`). +JSONL → markdown only. Brain ingest is `bin/brain/index.go --with-chats` +(default `var/chats/md`). WhatsApp sync is out of v1. diff --git a/docs/roadmap.md b/docs/roadmap.md index 1dd6c5e..c4eced3 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -26,8 +26,8 @@ Compose `api` (no CPython) / `index` (Python write). Issues #1–#5, #7–#13. [#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop. [#14](https://git.produktor.io/eSlider/2dph/issues/14) `bin/brain/add.go` / `POST /ingest` (Python `kblib.add_leafs`; no Go upsert port). -[#17](https://git.produktor.io/eSlider/2dph/issues/17) `--hop N` walks -FROM_FILE / HAS_VERSION / AUTHORED. +[#18](https://git.produktor.io/eSlider/2dph/issues/18) `--with-facts` / +`--with-chats` on rebuild (WhatsApp out of v1). ## Blockers @@ -39,13 +39,10 @@ question ├─ web (D17) ← in ├─ brain/add ACID ← in ├─ Cypher hop ← in - └─ facts+chats corpus ← #18 + └─ facts+chats corpus ← in ``` -1. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** — - rebuild loads repo markdown + mail as `info`. `facts/extract` pairing - and `bin/chats` are not indexed. WhatsApp is a stub. PII stays in `var/`. -2. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** — +1. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** — recall SoT should be `bin/brain/eval.go` via Zig, not Python `bin/kb/eval`. ## Not v1 @@ -55,6 +52,5 @@ contradiction resolution, OQ3 duckdb-md export, OQ4 YAML-first leafs. ## Close epic #16 when -- ops pairing + chat import land as leafs on rebuild - MCP tool order is documented and still gated by tests - CI recall SoT is `bin/brain/eval.go` via Zig diff --git a/docs/runbook.md b/docs/runbook.md index 39c97f3..cff9227 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -49,7 +49,7 @@ corpus rebuild remains `bin/brain/index.go --rebuild` (Compose profile ```bash bin/brain/add.go --text "arc-1 runs Matrix" --root facts --source "compose.yml x docker ps" -bin/brain/index.go --rebuild +bin/brain/index.go --rebuild --with-facts --with-chats bin/brain/search.go "LadybugDB vector index" # facts → info → web (D17) bin/brain/search.go "upstream flag" --no-web bin/brain/get.go --body