diff --git a/AGENTS.md b/AGENTS.md index 985370c..7f76d15 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,7 +40,7 @@ PLAN.md decisions + execution + open questions docs/ published docs skills/ in-project agent skills (vendored, no external links) bin/ self-describing tools bin/{subject}/{method}.go (shebang) -bin/brain/ search.go serve.go index.go get.go stats.go eval.go watch.go +bin/brain/ search.go serve.go index.go add.go get.go stats.go eval.go watch.go bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats bin/mail/ sync.go import.go (index_mail → brain/index.go) bin/markdown/ import.go (H2 leaf split; Python bin/md/import fallback) @@ -74,9 +74,9 @@ bin/brain/index.go --rebuild # rebuil `pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to docling (isolated subprocess — its native onnx can segfault the parent). Conversion never touches the brain DB (crash safety). -- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Ladybug - corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector - indexes exist; a fresh DB with indexes created last is the only safe path. +- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Bulk + rebuild still deletes `var/kb.lbug` and creates FTS/HNSW last. Single-leaf + write is `bin/brain/add.go` (safe while indexes exist; do not DROP INDEX). Keep conversion + indexing separate so a conversion crash can't leave the DB mid-transaction. @@ -89,6 +89,8 @@ bin/kb/search "query" [--repo X] # deprecated wrapper → bin/b bin/brain/search.go "query" [--root facts|info] # deduction search → YAML bin/brain/search.go "query" --no-web # local graph only eval "$(bin/cgo/zig env)" # Zig cc + liblbug (not gcc) +bin/brain/add.go --text T --root facts --source "a.md x b.md" # incremental write +bin/brain/add.go --json # stdin leaf or {leafs:[...]} bin/brain/get.go [--body] [--json] # Go read; Python bin/kb/get CI fallback bin/brain/stats.go [--json] bin/brain/eval.go [--json] # recall@5; questions in internal/brain/rank diff --git a/PLAN.md b/PLAN.md index cb6e5e6..0b867ac 100644 --- a/PLAN.md +++ b/PLAN.md @@ -31,9 +31,9 @@ detective method: **a fact needs ≥2 independent sources or it is | D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. | | D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. | | D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). | -| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Index/write stays Python (`compose --profile index`) until the Go write path is safe. | +| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Incremental write is Python `bin/kb/add` (`bin/brain/add.go`). Bulk rebuild stays `compose --profile index` until the Go write path is safe. | | D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). | -| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. | +| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. 2-source auto-pair docker ps × compose × ssh-config × docs. | | D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. | | D10 | versioning | everything is a leaf with `sha256 + observed_at + source_rev`; `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`. Stale = `source_rev` < git HEAD. | | D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. | @@ -45,7 +45,7 @@ detective method: **a fact needs ≥2 independent sources or it is | D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. | | D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is compose profile `picoclaw`; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. Agent lever/loop: [#15](https://git.produktor.io/eSlider/2dph/issues/15). | | D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. | -| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). | +| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`/`ingest`). | | D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc` → `zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. | ## Architecture @@ -57,8 +57,10 @@ detective method: **a fact needs ≥2 independent sources or it is skills/ in-project skills (web-search, postgres, brain, picoclaw, diataxis-docs) bin/ facts/extract.go audit.go crm.go # D14 shebang; Python implementation - kb/index Python write path (called by bin/brain/index.go) + kb/index Python bulk write (called by bin/brain/index.go) + kb/add Python incremental write (called by bin/brain/add.go) brain/index.go rebuild FTS + HNSW (incl. --with-mail) + brain/add.go incremental leaf write (no rebuild) brain/get.go stats.go eval.go # Go read (cgo); Python bin/kb/* CI fallback brain/watch.go brain/search.go deduction: facts → info → web-search @@ -160,18 +162,19 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl ## Gap to v1 (epic #16) -Read path + MCP are in. The detective brain is not closed until the graph is -**writable incrementally** and search can **walk** it. Board: +Read path + MCP are in. Incremental `brain/add` is in. The detective brain is +not closed until search can **walk** the graph and the facts+chats corpus +lands on rebuild. Board: [epic #16](https://git.produktor.io/eSlider/2dph/issues/16), milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12). Narrative: [docs/roadmap.md](docs/roadmap.md). | Order | Issue | Gap | |-------|-------|-----| -| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | Write stays Python rebuild; `brain/add` / `POST /ingest` are hints. Ladybug 0.19 WAL corrupts if new leafs land while FTS/HNSW exist. | +| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in** — `bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. | | 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | `--hop` errors. `FROM_FILE` / `HAS_VERSION` are in schema; search does not walk them. | | 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. | -| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | Lever = 2dph fact-check. Loop = PicoClaw/MCP `search` → `get` → `audit`. Specify in-repo, not only live config. | +| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search` → `get` → `audit`). | | 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. | Does **not** block epic close: [#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR, OQ1, OQ3, OQ4. \ No newline at end of file diff --git a/README.md b/README.md index 20a72ea..659ae98 100644 --- a/README.md +++ b/README.md @@ -120,6 +120,7 @@ Mail is a first-class corpus (retrievable through the same search): ```bash bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go) bin/mail/import.go --from-raw var/mail # JSON → markdown +bin/brain/add.go --text T --root facts --source "a.md x b.md" bin/brain/index.go --rebuild # rebuild brain (incl. mail) bin/brain/search.go "invoice from last week" # same search over mail leafs ``` @@ -128,8 +129,9 @@ bin/brain/search.go "invoice from last week" # same s - **LadybugDB** — single `var/kb.lbug`, Cypher + HNSW + BM25, embedded. Read tools (`get` / `stats` / `eval`) are Go + Zig CGO (`bin/cgo/zcc`). - Python fallbacks stay for CI until the runner fetches Zig. Write is - Compose profile `index` (`bin/brain/index.go`). + Python fallbacks stay for CI until the runner fetches Zig. Incremental + write is `bin/brain/add.go` (Python `kblib.add_leafs`). Bulk rebuild is + Compose profile `index` (`bin/brain/index.go --rebuild`). - **model2vec** — `potion-multilingual-128M` (256-dim), CPU, no Ollama runtime dependency. - facts and info split by `root` but written in the same transaction. diff --git a/bin/brain/add.go b/bin/brain/add.go new file mode 100755 index 0000000..23e817f --- /dev/null +++ b/bin/brain/add.go @@ -0,0 +1,21 @@ +//usr/bin/env go run -tags=brain_add "$0" "$@"; exit +//go:build brain_add +// +// bin/brain/add.go - incremental leaf write (Python kblib, no rebuild). +// +// ./bin/brain/add.go --text T --root facts --source "a.md x b.md" +// ./bin/brain/add.go --json +// +// D6: write stays Python. Does not delete var/kb.lbug. +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/kb/add", os.Args[1:])) +} diff --git a/bin/brain/index.go b/bin/brain/index.go index d58e2f4..d896632 100755 --- a/bin/brain/index.go +++ b/bin/brain/index.go @@ -7,8 +7,8 @@ // ./bin/brain/index.go --rebuild --with-mail // ./bin/brain/index.go --dry-run --with-mail // -// v1 write is always a rebuild when mail is included (live FTS/HNSW + bulk -// insert corrupts Ladybug 0.19 WAL). `add` is v2. +// v1 write: bin/brain/add.go for one/few leafs (indexes may already exist). +// Bulk mail/corpus still --rebuild (fresh file, indexes last). // NOTE: never run `gofmt -w` on this file — it breaks the shebang. package main diff --git a/bin/kb/add b/bin/kb/add new file mode 100755 index 0000000..14aa0d8 --- /dev/null +++ b/bin/kb/add @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +"""kb/add - incremental leaf write (no rebuild). + + bin/kb/add --text T --root facts|info --source S + bin/kb/add --json # stdin: one object or {"leafs":[...]} + bin/kb/add --db PATH --json + +Writes facts+info in one Ladybug transaction. Does not delete kb.lbug. +Embedding is used when provided; otherwise model2vec encodes the text. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / "bin" / "tools")) + +from kblib import ( # noqa: E402 + EMBED_DIM, + add_leafs, + connect, + ensure_indexes, + init_schema, +) + + +def _as_leafs(payload: object) -> list[dict]: + if isinstance(payload, list): + return [dict(x) for x in payload] + if isinstance(payload, dict): + if "leafs" in payload: + return [dict(x) for x in payload["leafs"]] + return [dict(payload)] + raise ValueError("json must be an object, a list, or {leafs:[...]}") + + +def _embed_missing(leafs: list[dict]) -> None: + missing = [lf for lf in leafs if not lf.get("embedding")] + if not missing: + return + from model2vec import StaticModel + + model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M") + for lf in missing: + text = str(lf.get("text") or "") + vec = model.encode([text])[0].astype(float).tolist() + if len(vec) != EMBED_DIM: + vec = (vec + [0.0] * EMBED_DIM)[:EMBED_DIM] + lf["embedding"] = vec + + +def main(argv: list[str]) -> int: + import argparse + + p = argparse.ArgumentParser(description="add leafs without rebuilding the brain") + p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)") + p.add_argument("--json", action="store_true", help="read leaf JSON from stdin") + p.add_argument("--text", default="", help="leaf text") + p.add_argument("--root", default="info", choices=("facts", "info")) + p.add_argument("--source", default="") + p.add_argument("--confidence", default="confirmed") + p.add_argument("--source-rev", default="working-tree") + p.add_argument("--how", default="brain/add") + p.add_argument("--loc", default="") + p.add_argument("--type", default="reference", dest="type_") + args = p.parse_args(argv) + + if args.json: + raw = sys.stdin.read() + if not raw.strip(): + print("kb/add: empty stdin", file=sys.stderr) + return 2 + leafs = _as_leafs(json.loads(raw)) + else: + if not args.text or not args.source: + print("kb/add: --text and --source are required (or --json)", file=sys.stderr) + return 2 + leafs = [{ + "text": args.text, + "root": args.root, + "source": args.source, + "confidence": args.confidence, + "source_rev": args.source_rev, + "how": args.how, + "loc": args.loc or args.source, + "type": args.type_, + }] + + for lf in leafs: + if not lf.get("text") or not lf.get("source"): + print("kb/add: each leaf needs text and source", file=sys.stderr) + return 2 + + _embed_missing(leafs) + + from kblib import DB_PATH, VAR + + dbpath = Path(args.db) if args.db else DB_PATH + dbpath.parent.mkdir(parents=True, exist_ok=True) + VAR.mkdir(exist_ok=True) + db, conn = connect(dbpath, read_only=False) + init_schema(conn) + ids = add_leafs(conn, leafs) + ensure_indexes(conn) + conn.close() + db.close() + print(json.dumps({"mode": "add", "ids": ids, "db": str(dbpath)})) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bin/tools/kblib.py b/bin/tools/kblib.py index 028ebc1..5841739 100644 --- a/bin/tools/kblib.py +++ b/bin/tools/kblib.py @@ -4,7 +4,7 @@ Single embedded graph `var/kb.lbug`. Two roots: facts (assertions backed by >=2 independent sources) and info (narrative leafs). Hybrid retrieval: BM25 (FTS extension) + HNSW cosine (VECTOR extension) + Cypher graph hops. -All access is read-only unless `--rebuild` is passed to kb/index. +All access is read-only unless `--rebuild` (kb/index) or `kb/add`. """ from __future__ import annotations @@ -118,6 +118,50 @@ def upsert_leaf(conn: ladybug.Connection, *, text: str, root: str, confidence: s return lid +def add_leafs(conn: ladybug.Connection, leafs: list[dict]) -> list[str]: + """Write facts+info leafs in one transaction. Safe while FTS/HNSW exist. + + Each leaf dict: text, source, optional root/confidence/source_rev/how/loc/type/embedding. + Does not delete the database file. Measured on Ladybug 0.19: MERGE of new + ids (and updates) stays FTS+HNSW queryable; DROP INDEX is the fatal path. + """ + if not leafs: + return [] + started = False + try: + conn.execute("BEGIN TRANSACTION") + started = True + except Exception: + started = False + ids: list[str] = [] + try: + for lf in leafs: + ids.append( + upsert_leaf( + conn, + text=str(lf["text"]), + root=str(lf.get("root") or ROOT_INFO), + confidence=str(lf.get("confidence") or CONF_CONFIRMED), + source=str(lf["source"]), + source_rev=str(lf.get("source_rev") or "working-tree"), + how=str(lf.get("how") or "brain/add"), + loc=str(lf.get("loc") or lf.get("source") or ""), + type_=str(lf.get("type") or lf.get("type_") or "reference"), + embedding=lf.get("embedding"), + ) + ) + if started: + conn.execute("COMMIT") + except Exception: + if started: + try: + conn.execute("ROLLBACK") + except Exception: + pass + raise + return ids + + def leaf_index_names(conn: ladybug.Connection) -> set[str]: """Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'}).""" rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all() diff --git a/bin/tools/test_bin_layout.py b/bin/tools/test_bin_layout.py index b1f50b4..88691b0 100644 --- a/bin/tools/test_bin_layout.py +++ b/bin/tools/test_bin_layout.py @@ -75,9 +75,20 @@ class BinLayoutTest(unittest.TestCase): ) def test_brain_methods_are_shebangs(self) -> None: - for method in ("index.go", "get.go", "stats.go", "eval.go", "watch.go"): + for method in ("index.go", "add.go", "get.go", "stats.go", "eval.go", "watch.go"): self._assert_shebang(f"bin/brain/{method}") + def test_brain_add_is_python_write_not_rebuild(self) -> None: + self._assert_shebang("bin/brain/add.go") + text = (ROOT / "bin" / "brain" / "add.go").read_text() + self.assertIn("cmdbin.ExecFile", text) + self.assertIn("bin/kb/add", text) + self.assertNotIn("--rebuild", text) + py = (ROOT / "bin" / "kb" / "add").read_text() + self.assertIn("add_leafs", py) + self.assertIn("--json", py) + self.assertNotIn("unlink", py.lower()) + def test_brain_get_stats_eval_are_not_python_exec(self) -> None: for method in ("get.go", "stats.go", "eval.go"): text = (ROOT / "bin" / "brain" / method).read_text() diff --git a/bin/tools/test_kb_add.py b/bin/tools/test_kb_add.py new file mode 100644 index 0000000..2e32276 --- /dev/null +++ b/bin/tools/test_kb_add.py @@ -0,0 +1,69 @@ +"""Incremental add writes leafs without deleting kb.lbug.""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] + + +class KbAddCLITest(unittest.TestCase): + def test_json_add_does_not_delete_db(self) -> None: + tmp = Path(tempfile.mkdtemp()) + dbpath = tmp / "kb.lbug" + py = sys.executable + venv_py = ROOT / ".venv" / "bin" / "python" + if venv_py.is_file(): + py = str(venv_py) + payload = { + "text": "cli zebra leaf", + "root": "info", + "source": "cli-test", + "confidence": "confirmed", + "how": "test", + "loc": str(tmp), + "type": "reference", + "embedding": [0.0] * 256, + } + payload["embedding"][0] = 0.3 + proc = subprocess.run( + [py, str(ROOT / "bin" / "kb" / "add"), "--db", str(dbpath), "--json"], + cwd=ROOT, + input=json.dumps(payload), + capture_output=True, + text=True, + env=os.environ.copy(), + check=False, + ) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertTrue(dbpath.exists(), "add must create the db, not skip write") + out = json.loads(proc.stdout) + self.assertEqual(out.get("mode"), "add") + self.assertEqual(len(out.get("ids") or []), 1) + again = subprocess.run( + [py, str(ROOT / "bin" / "kb" / "add"), "--db", str(dbpath), "--json"], + cwd=ROOT, + input=json.dumps({ + **payload, + "text": "second moose leaf", + "source": "cli-test-2", + }), + capture_output=True, + text=True, + env=os.environ.copy(), + check=False, + ) + self.assertEqual(again.returncode, 0, again.stderr) + self.assertTrue(dbpath.exists()) + second = json.loads(again.stdout) + self.assertEqual(len(second.get("ids") or []), 1) + self.assertNotEqual(out["ids"][0], second["ids"][0]) + + +if __name__ == "__main__": + unittest.main() diff --git a/bin/tools/test_kblib.py b/bin/tools/test_kblib.py index 7e5e28c..b9e1c60 100644 --- a/bin/tools/test_kblib.py +++ b/bin/tools/test_kblib.py @@ -71,6 +71,69 @@ class KblibTest(unittest.TestCase): self.assertTrue(hits) self.assertIn("Leaf_vec", kblib.leaf_index_names(self.conn)) + def test_add_after_indexes_keeps_fts_queryable(self): + """Incremental add after FTS+HNSW must find the new leaf on both indexes.""" + kblib.upsert_leaf(self.conn, text="seed fox leaf", root="info", + confidence="confirmed", source="s", source_rev="r1", + how="test", loc="/tmp", type_="reference", + embedding=make_emb(0.1)) + kblib.ensure_indexes(self.conn) + ids = kblib.add_leafs(self.conn, [{ + "text": "added zebra after index", + "root": "facts", + "confidence": "confirmed", + "source": "a.md x b.md", + "source_rev": "r1", + "how": "test", + "loc": "/tmp", + "type": "fact", + "embedding": make_emb(0.9), + }]) + self.assertEqual(len(ids), 1) + fts = kblib.query_fts(self.conn, "zebra", 5) + self.assertTrue(fts) + self.assertIn("zebra", fts[0]["text"]) + self.assertEqual(fts[0]["root"], "facts") + vec = kblib.query_vector(self.conn, make_emb(0.9), 5) + self.assertTrue(any("zebra" in h["text"] for h in vec)) + fox = kblib.query_fts(self.conn, "fox", 5) + self.assertTrue(fox) + self.assertIn("fox", fox[0]["text"]) + + def test_add_facts_and_info_one_transaction(self): + """D12: facts and info land in the same transaction.""" + kblib.ensure_indexes(self.conn) + ids = kblib.add_leafs(self.conn, [ + { + "text": "tx fact leaf two-source", + "root": "facts", + "confidence": "confirmed", + "source": "compose.yml x docker ps", + "source_rev": "r1", + "how": "test", + "loc": "/tmp", + "type": "fact", + "embedding": make_emb(0.4), + }, + { + "text": "tx info narrative", + "root": "info", + "confidence": "confirmed", + "source": "note.md", + "source_rev": "r1", + "how": "test", + "loc": "/tmp", + "type": "reference", + "embedding": make_emb(0.5), + }, + ]) + self.assertEqual(len(ids), 2) + stats = kblib.stats(self.conn) + self.assertEqual(stats["by_root"].get("facts"), 1) + self.assertEqual(stats["by_root"].get("info"), 1) + self.assertTrue(kblib.query_fts(self.conn, "two-source", 5)) + self.assertTrue(kblib.query_fts(self.conn, "narrative", 5)) + def test_drop_vector_then_create_raises_clear_error(self): """DROP INDEX leaves ghost catalog; create_fts_and_vector must raise.""" kblib.upsert_leaf(self.conn, text="seed", root="info", diff --git a/docs/design.md b/docs/design.md index c16b85d..96b51f1 100644 --- a/docs/design.md +++ b/docs/design.md @@ -80,14 +80,16 @@ Conflicting pairings (≥2 yes vs ≥2 no) = hypothesis (OQ1 → v2 resolution). They do not exec Python. Control questions for recall@5 live in `internal/brain/rank` so CI can test the table without libladybug. Python `bin/kb/{get,stats,eval}` remain for GitHub Actions until the runner -fetches Zig + libs (`bin/cgo/zig`). Index/write is still `bin/kb/index` +fetches Zig + libs (`bin/cgo/zig`). Incremental write is `bin/kb/add` +(`bin/brain/add.go`). Bulk index/write is still `bin/kb/index` (`docker compose --profile index`). ## Agent API (D20) `bin/brain/serve.go` exposes the same `internal/httpapi.Ops` table as OpenAPI (`GET /openapi.json`) and MCP (`POST /mcp` JSON-RPC `tools/list` + -`tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`. +`tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`, +`ingest` (add a leaf; omit body for the CLI hint). Agents should use these endpoints instead of shebang CLIs. ## Reasoner (D18) @@ -98,5 +100,5 @@ CPU sidecar: compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `127.0.0.1:11435`). Bake-off: `bin/reasoner/bakeoff.go`. Weights stay out of the 2dph image. See [docs/reasoner.md](reasoner.md). -Gap to v1 (write, hops, corpus, agent loop): [roadmap](roadmap.md), +Gap to v1 (hops, corpus, CI eval): [roadmap](roadmap.md), [epic #16](https://git.produktor.io/eSlider/2dph/issues/16). \ No newline at end of file diff --git a/docs/roadmap.md b/docs/roadmap.md index 6e0552d..623549c 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -23,8 +23,9 @@ Decisions: [PLAN.md](../PLAN.md). Read path Go + Zig CGO (D21). HTTP + OpenAPI + MCP (D20). PicoClaw compose profile + CPU reasoner (D18). Mail sync → import → rebuild. D14 shebangs. Compose `api` (no CPython) / `index` (Python write). Issues #1–#5, #7–#13. - -`POST /ingest` is a rebuild **hint**. `add` is not implemented. +[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop. +[#14](https://git.produktor.io/eSlider/2dph/issues/14) `bin/brain/add.go` / +`POST /ingest` (Python `kblib.add_leafs`; no Go upsert port). ## Blockers @@ -34,24 +35,18 @@ question ├─ FTS + HNSW ← in ├─ facts / info roots ← in ├─ web (D17) ← in + ├─ brain/add ACID ← in ├─ Cypher hop ← #17 schema yes, search no - ├─ brain/add ACID ← #14 rebuild only └─ facts+chats corpus ← #18 ``` -1. **[#14](https://git.produktor.io/eSlider/2dph/issues/14) write** — - `bin/brain/index.go --rebuild` (Python `kblib`). No incremental - `brain/add`. Watch/mail/git cannot become facts “now”. -2. **[#17](https://git.produktor.io/eSlider/2dph/issues/17) hops** — +1. **[#17](https://git.produktor.io/eSlider/2dph/issues/17) hops** — `Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person` exists; `--hop` still errors. Without a walk, D9/D10 are paper. -3. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** — +2. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** — rebuild loads repo markdown + mail as `info`. `facts/extract` pairing and `bin/chats` are not indexed. WhatsApp is a stub. PII stays in `var/`. -4. **[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop** — - 2dph is the lever (`search` → `get` → `audit`). PicoClaw is the loop. - Document the contour in-repo; CPU turns need a large context window. -5. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** — +3. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** — recall SoT should be `bin/brain/eval.go` via Zig, not Python `bin/kb/eval`. ## Not v1 @@ -61,7 +56,6 @@ contradiction resolution, OQ3 duckdb-md export, OQ4 YAML-first leafs. ## Close epic #16 when -- facts+info can be written without a full rebuild for every leaf - `--hop` stops erroring and runs a Cypher path from search hits - ops pairing + chat import land as leafs on rebuild - MCP tool order is documented and still gated by tests diff --git a/docs/runbook.md b/docs/runbook.md index 3f1087e..ff34c4a 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -43,10 +43,12 @@ That binds `127.0.0.1:8888`. JSON format must stay enabled. ## Index then search -Write path is Compose profile `index` (Python Ladybug rebuild) until -`brain/add` is v2. The operator command is `bin/brain/index.go`. +Write path is `bin/brain/add.go` for a leaf (or `POST /ingest`). Bulk +corpus rebuild remains `bin/brain/index.go --rebuild` (Compose profile +`index`). Do not DROP INDEX on Ladybug 0.19. ```bash +bin/brain/add.go --text "arc-1 runs Matrix" --root facts --source "compose.yml x docker ps" bin/brain/index.go --rebuild bin/brain/search.go "LadybugDB vector index" # facts → info → web (D17) bin/brain/search.go "upstream flag" --no-web diff --git a/internal/brain/http.go b/internal/brain/http.go index c65a5dc..1bd5311 100644 --- a/internal/brain/http.go +++ b/internal/brain/http.go @@ -7,6 +7,8 @@ import ( "context" "encoding/json" "fmt" + "os/exec" + "path/filepath" "github.com/eSlider/2dph/internal/brain/rank" ) @@ -137,12 +139,22 @@ func (HTTP) Audit(context.Context) ([]byte, error) { return json.Marshal(map[string]any{"status": "ok", "by_confidence": rows}) } -func (HTTP) Ingest(context.Context) ([]byte, error) { - return json.Marshal(map[string]any{ - "mode": "rebuild", - "command": "bin/brain/index.go --rebuild", - "add": "v2", - }) +func (HTTP) Ingest(ctx context.Context, body []byte) ([]byte, error) { + if len(bytes.TrimSpace(body)) == 0 { + return json.Marshal(map[string]any{ + "mode": "add", + "command": "bin/brain/add.go", + "rebuild": "bin/brain/index.go --rebuild", + }) + } + cmd := exec.CommandContext(ctx, filepath.Join(repoRoot(), "bin", "kb", "add"), "--json") + cmd.Stdin = bytes.NewReader(body) + cmd.Dir = repoRoot() + out, err := cmd.Output() + if err != nil { + return nil, fmt.Errorf("add: %w", err) + } + return out, nil } func asInt(v any) int64 { diff --git a/internal/httpapi/mcp.go b/internal/httpapi/mcp.go index dbd6d16..3f39668 100644 --- a/internal/httpapi/mcp.go +++ b/internal/httpapi/mcp.go @@ -144,7 +144,15 @@ func (s *Server) mcpCall(r *http.Request, params json.RawMessage) (any, error) { return nil, fmt.Errorf("cancelled") } defer s.release() - body, err = s.api.Ingest(r.Context()) + var payload []byte + text := strings.TrimSpace(fmt.Sprint(p.Arguments["text"])) + if text != "" && text != "" { + payload, err = json.Marshal(p.Arguments) + if err != nil { + return nil, err + } + } + body, err = s.api.Ingest(r.Context(), payload) default: return nil, fmt.Errorf("unknown tool %s", p.Name) } diff --git a/internal/httpapi/server.go b/internal/httpapi/server.go index 061dce5..8dea1c3 100644 --- a/internal/httpapi/server.go +++ b/internal/httpapi/server.go @@ -11,6 +11,7 @@ import ( "context" "encoding/json" "errors" + "io" "log" "net/http" "os" @@ -27,7 +28,7 @@ type API interface { Get(ctx context.Context, id string, body bool) ([]byte, error) Stats(ctx context.Context) ([]byte, error) Audit(ctx context.Context) ([]byte, error) - Ingest(ctx context.Context) ([]byte, error) + Ingest(ctx context.Context, body []byte) ([]byte, error) } type Server struct { @@ -59,7 +60,7 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { case PathAudit: s.handleJSON(w, r, s.api.Audit) case PathIngest: - s.handleJSON(w, r, s.api.Ingest) + s.handleIngest(w, r) case PathOpenAPI: s.handleOpenAPI(w, r) case PathMCP: @@ -116,6 +117,24 @@ func (s *Server) handleJSON(w http.ResponseWriter, r *http.Request, fn func(cont writeAPI(w, body, err) } +func (s *Server) handleIngest(w http.ResponseWriter, r *http.Request) { + var raw []byte + if r.Method == http.MethodPost { + b, err := io.ReadAll(io.LimitReader(r.Body, 1<<20)) + if err != nil { + writeJSON(w, http.StatusBadRequest, map[string]any{"error": "read body"}) + return + } + raw = b + } + if !s.acquire(w, r) { + return + } + defer s.release() + body, err := s.api.Ingest(r.Context(), raw) + writeAPI(w, body, err) +} + func (s *Server) tryAcquire(r *http.Request) bool { return s.acquire(nopWriter{}, r) } @@ -191,11 +210,30 @@ func (ExecSearcher) Get(context.Context, string, bool) ([]byte, error) { } func (ExecSearcher) Stats(context.Context) ([]byte, error) { return nil, errUnimplemented } func (ExecSearcher) Audit(context.Context) ([]byte, error) { return nil, errUnimplemented } -func (ExecSearcher) Ingest(context.Context) ([]byte, error) { - return json.Marshal(map[string]any{ - "mode": "rebuild", - "command": "bin/brain/index.go --rebuild", - }) +func (b ExecSearcher) Ingest(ctx context.Context, body []byte) ([]byte, error) { + if len(strings.TrimSpace(string(body))) == 0 { + return json.Marshal(map[string]any{ + "mode": "add", + "command": "bin/brain/add.go", + "rebuild": "bin/brain/index.go --rebuild", + }) + } + root := os.Getenv("KB_ROOT") + if root == "" { + root = "." + } + cmd := exec.CommandContext(ctx, filepath.Join(root, "bin/kb/add"), "--json") + cmd.Stdin = strings.NewReader(string(body)) + cmd.Dir = root + out, err := cmd.Output() + if err != nil { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) { + return nil, errors.New("add failed: " + strings.TrimSpace(string(exitErr.Stderr))) + } + return nil, err + } + return out, nil } func defaultSearchCmd(root string) string { diff --git a/internal/httpapi/server_test.go b/internal/httpapi/server_test.go index 2ea3462..01d8482 100644 --- a/internal/httpapi/server_test.go +++ b/internal/httpapi/server_test.go @@ -1,6 +1,7 @@ package httpapi import ( + "bytes" "context" "encoding/json" "net/http" @@ -64,8 +65,11 @@ func (f *fakeSearcher) Audit(context.Context) ([]byte, error) { return []byte(`{"status":"ok"}`), nil } -func (f *fakeSearcher) Ingest(context.Context) ([]byte, error) { - return []byte(`{"mode":"rebuild","command":"bin/brain/index.go --rebuild"}`), nil +func (f *fakeSearcher) Ingest(_ context.Context, body []byte) ([]byte, error) { + if len(bytes.TrimSpace(body)) == 0 { + return []byte(`{"mode":"add","command":"bin/brain/add.go"}`), nil + } + return []byte(`{"mode":"add","ids":["fake-leaf"]}`), nil } func (f *fakeSearcher) count() int { @@ -193,6 +197,27 @@ func TestStatsAuditIngest(t *testing.T) { } } +func TestIngestIsAddNotRebuildHint(t *testing.T) { + h := NewServer(&fakeSearcher{}, 1) + code, body := get(t, h, "/ingest") + if code != http.StatusOK { + t.Fatalf("GET /ingest code = %d body=%s", code, body) + } + if strings.Contains(string(body), `"add":"v2"`) || strings.Contains(string(body), "write is v2") { + t.Fatalf("GET /ingest still a v2 hint: %s", body) + } + if !strings.Contains(string(body), "bin/brain/add.go") { + t.Fatalf("GET /ingest should name add.go: %s", body) + } + code, body = postJSON(t, h, "/ingest", `{"text":"hello","root":"info","source":"t"}`) + if code != http.StatusOK { + t.Fatalf("POST /ingest code = %d body=%s", code, body) + } + if !strings.Contains(string(body), "fake-leaf") { + t.Fatalf("POST /ingest should add: %s", body) + } +} + func TestHTTPPackageDoesNotExecPython(t *testing.T) { raw, err := os.ReadFile("server.go") if err != nil { diff --git a/internal/httpapi/spec.go b/internal/httpapi/spec.go index 37d4a7d..f20bdff 100644 --- a/internal/httpapi/spec.go +++ b/internal/httpapi/spec.go @@ -47,7 +47,15 @@ var Ops = []Op{ }, {Path: PathStats, Method: "get", ID: "stats", Summary: "index health", MCP: true}, {Path: PathAudit, Method: "get", ID: "audit", Summary: "facts confidence histogram", MCP: true}, - {Path: PathIngest, Method: "get", ID: "ingest", Summary: "rebuild hint (write is v2)", MCP: true}, + { + Path: PathIngest, Method: "post", ID: "ingest", Summary: "add a leaf without rebuild", + MCP: true, + Params: []Param{ + {Name: "text", In: "query", Type: "string", Description: "leaf text (omit for CLI hint)"}, + {Name: "root", In: "query", Type: "string", Description: "facts or info (default info)"}, + {Name: "source", In: "query", Type: "string", Description: "evidence pointer; facts need two sources"}, + }, + }, {Path: PathOpenAPI, Method: "get", ID: "openapi", Summary: "OpenAPI 3 document for this server"}, } diff --git a/internal/httpapi/spec_test.go b/internal/httpapi/spec_test.go index 07286db..1884245 100644 --- a/internal/httpapi/spec_test.go +++ b/internal/httpapi/spec_test.go @@ -38,11 +38,24 @@ func TestMCPToolsMatchOpenAPIPaths(t *testing.T) { t.Fatalf("MCP tool %s has no OpenAPI path %s", tool.Name, path) } } - for _, need := range []string{"search", "get", "stats", "audit"} { + for _, need := range []string{"search", "get", "stats", "audit", "ingest"} { if !names[need] { t.Fatalf("MCP tools missing %s: %v", need, names) } } + var ingest MCPTool + for _, tool := range tools { + if tool.Name == "ingest" { + ingest = tool + break + } + } + if strings.Contains(ingest.Description, "v2") { + t.Fatalf("ingest still a v2 hint: %s", ingest.Description) + } + if !strings.Contains(ingest.Description, "add") { + t.Fatalf("ingest should describe add: %s", ingest.Description) + } } func TestOpenAPIHTTP(t *testing.T) { diff --git a/skills/brain/SKILL.md b/skills/brain/SKILL.md index 261cd05..f15eac8 100644 --- a/skills/brain/SKILL.md +++ b/skills/brain/SKILL.md @@ -26,6 +26,7 @@ second independent source when local roots cannot confirm. An answer is bin/brain/search.go "Matrix federation" # pointers + snippets, YAML bin/brain/search.go "onlyoffice postgres" --root facts # restrict to confirmed bin/brain/search.go "where is cs-lexicon" --json | yq '.[].ref' +bin/brain/add.go --text T --root facts --source "a.md x b.md" bin/brain/get.go --body # full chunk only when needed bin/brain/stats.go # index health bin/brain/eval.go # recall@5 >= 0.95 gate (Go; Python bin/kb/eval is CI fallback) diff --git a/skills/brain/tools.md b/skills/brain/tools.md index fd66429..033cd3e 100644 --- a/skills/brain/tools.md +++ b/skills/brain/tools.md @@ -8,4 +8,4 @@ Serve: `bin/brain/serve.go` (`GET /openapi.json`, `POST /mcp`). - `get` — read one leaf by id - `stats` — index health - `audit` — facts confidence histogram -- `ingest` — rebuild hint (write is v2) +- `ingest` — add a leaf without rebuild