diff --git a/AGENTS.md b/AGENTS.md index d57865b..6382595 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,11 +36,13 @@ PLAN.md decisions + execution + open questions docs/ published docs skills/ in-project agent skills (vendored, no external links) bin/ self-describing tools bin/{subject}/{method}.go (shebang) -bin/brain/ search.go, serve.go; libs in internal/brain and internal/httpapi +bin/brain/ search.go serve.go index.go get.go stats.go eval.go watch.go bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats +bin/mail/ sync.go import.go (index_mail → brain/index.go) +bin/markdown/ import.go (mistune leafs) +bin/postgres/ query.go (read-only YAML) internal/ shared Go (brain/rank is cgo-free; chats parsers too) -bin/watch/ corpus watcher (internal via bin/brain/watch later) -bin/mail/ mail pipeline: sync (Go), import (md), index_mail (rebuild) +bin/watch/ corpus watcher (used by bin/brain/watch.go) bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch) bin/docker-entrypoint container entrypoint (brain index|search|serve|watch) compose.yaml docker composition (root level, not docker/) @@ -54,8 +56,8 @@ var/ kb.lbug, var/mail/*, caches (gitignored) ```bash bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox) -bin/mail/import --from-raw var/mail # message.json → message.md (convert only) -bin/mail/index_mail # rebuild brain incl. all mail (fresh DB) +bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only) +bin/brain/index.go --rebuild # rebuild brain incl. all mail (fresh DB) ``` - `sync` (Go) downloads messages + attachments; Gmail uses paginated list + @@ -64,7 +66,7 @@ bin/mail/index_mail # rebuil `pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to docling (isolated subprocess — its native onnx can segfault the parent). Conversion never touches the brain DB (crash safety). -- `index_mail` always rebuilds from scratch (repo corpus + mail). Ladybug +- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Ladybug corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector indexes exist; a fresh DB with indexes created last is the only safe path. Keep conversion + indexing separate so a conversion crash can't leave the @@ -77,6 +79,9 @@ bin/facts/audit ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate bin/facts/crm [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT) bin/kb/search "query" [--repo X] # deprecated wrapper → bin/brain/search.go bin/brain/search.go "query" [--root facts|info] # deduction search → YAML +bin/brain/get.go [--body] +bin/markdown/import.go [dir] # mistune leaves → YAML +bin/postgres/query.go --profile onlyoffice -c 'SELECT 1' bin/md/tables # what the graph holds → YAML bin/brain/deduce "question" # thinking wrapper ``` diff --git a/PLAN.md b/PLAN.md index d30673d..7a107ef 100644 --- a/PLAN.md +++ b/PLAN.md @@ -53,13 +53,17 @@ detective method: **a fact needs ≥2 independent sources or it is bin/ facts/extract auto-pair 2 sources → lexicon yaml + graph facts/audit ["self"|"facts"|"info"|"stale"] 2-source + staleness gate - kb/index build FTS + HNSW from corpus (Python, for now) + kb/index Python write path (called by bin/brain/index.go) + brain/index.go rebuild FTS + HNSW (incl. --with-mail) + brain/get.go stats.go eval.go watch.go brain/search.go deduction: facts → info → web-search - kb/get kb/stats kb/eval brain/serve.go HTTP API (internal/httpapi) + mail/import.go JSON → markdown (no brain write) + markdown/import.go mistune leaves + postgres/query.go read-only YAML (wraps bin/db/psql-yq) chats/sync.go import.go facts.go apply.go (libs in internal/chats; no chats index) - md/import md/select md/tables md/gaps (mistune) + md/import (deprecated; bin/markdown/import.go) brain/extract brain/audit brain/deduce (thinking wrapper) web/search (vendored) db/psql-yq (vendored) @@ -108,12 +112,13 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, 1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download. Gmail attachments key off `body.attachmentId`, not MIME `partId`. -2. `bin/mail/import --from-raw` — message.json → message.md; PDFs via +2. `bin/mail/import.go --from-raw` — message.json → message.md; PDFs via `pdftotext -layout` (~15ms) with docling subprocess fallback; ICS sidecars Latin-1→UTF-8 normalized. -3. `bin/mail/index_mail` — fresh rebuild (repo corpus + mail) because ladybug +3. `bin/brain/index.go --rebuild` — fresh rebuild (repo corpus + mail) because ladybug corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and - indexing stay separate for crash safety. + indexing stay separate for crash safety. `bin/mail/index_mail` is a + deprecation shim. 4. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable via `bin/brain/search.go`. @@ -125,7 +130,7 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, 2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser) 3. python -m unittest discover -s bin/tools (includes published-docs SoT) 4. bin/facts/audit self (lexicon internal consistency) -5. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions) +5. bin/brain/eval.go (recall@5 ≥ 0.95, gates index regressions) 6. md-docs build/lint if docs tooling arrives. Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as diff --git a/README.md b/README.md index 4acd857..c668225 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,8 @@ graph TB subgraph dph["2dph tools"] EX["bin/facts/extract
2-source pairing"] AU["bin/facts/audit
confidence + staleness"] - IDX["bin/kb/index
chunk + embed"] - MD["bin/md/import
mistune leaves"] + IDX["bin/brain/index.go
chunk + embed"] + MD["bin/markdown/import.go
mistune leaves"] SR["bin/brain/search.go
deduction"] end @@ -88,9 +88,9 @@ fact; conflicting sources or a single source → `hypothesis` → `(not confirme bin/brain/search.go "Matrix federation over HTTPS" # facts → info → web-search bin/brain/search.go "onlyoffice postgres" --root facts bin/brain/search.go "where is cs-lexicon" --json | yq '.' -bin/kb/get --body # full chunk on demand -bin/kb/stats # index health -bin/kb/eval # recall@5 gate +bin/brain/get.go --body # full chunk on demand +bin/brain/stats.go # index health +bin/brain/eval.go # recall@5 gate ``` `--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`. @@ -99,8 +99,8 @@ Mail is a first-class corpus (retrievable through the same search): ```bash bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go) -bin/mail/import --from-raw var/mail # JSON → markdown -bin/mail/index_mail # rebuild brain incl. mail +bin/mail/import.go --from-raw var/mail # JSON → markdown +bin/brain/index.go --rebuild # rebuild brain (incl. mail) bin/brain/search.go "invoice from last week" # same search over mail leafs ``` @@ -111,7 +111,7 @@ bin/brain/search.go "invoice from last week" # same s readers. **Never `DROP INDEX` FTS/VECTOR** on Ladybug 0.19: DROP leaves ghost catalog tables (`_0_Leaf_vec_UPPER`) so recreate fails while `SHOW_INDEXES` omits HNSW. Fresh indexes = delete `var/kb.lbug` + - `bin/kb/index --rebuild`. Use `ensure_indexes()` after upserts. + `bin/brain/index.go --rebuild`. Use `ensure_indexes()` after upserts. - **model2vec** — `potion-multilingual-128M` static embeddings (256-dim), CPU-fast, deterministic, no Ollama runtime dependency. - facts and info split semantically by `root` column but written inside the diff --git a/bin/brain/doc.go b/bin/brain/doc.go index a132c92..a90cf4b 100644 --- a/bin/brain/doc.go +++ b/bin/brain/doc.go @@ -1,3 +1,3 @@ -// Commands in this directory are shebang mains (search.go). -// search.go is behind the system_ladybug build tag (cgo). +// Commands in this directory are shebang mains (search.go, serve.go, index.go, +// get.go, stats.go, eval.go, watch.go), each behind an exclusive build tag. package main diff --git a/bin/brain/eval.go b/bin/brain/eval.go new file mode 100755 index 0000000..821c367 --- /dev/null +++ b/bin/brain/eval.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=brain_eval "$0" "$@"; exit +//go:build brain_eval +// +// bin/brain/eval.go - recall@5 gate. +// +// ./bin/brain/eval.go +// ./bin/brain/eval.go --json +// +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/kb/eval", os.Args[1:])) +} diff --git a/bin/brain/get.go b/bin/brain/get.go new file mode 100755 index 0000000..75784fc --- /dev/null +++ b/bin/brain/get.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=brain_get "$0" "$@"; exit +//go:build brain_get +// +// bin/brain/get.go - read one leaf by id. +// +// ./bin/brain/get.go +// ./bin/brain/get.go --body +// +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/kb/get", os.Args[1:])) +} diff --git a/bin/brain/index.go b/bin/brain/index.go new file mode 100755 index 0000000..d58e2f4 --- /dev/null +++ b/bin/brain/index.go @@ -0,0 +1,24 @@ +//usr/bin/env go run -tags=brain_index "$0" "$@"; exit +//go:build brain_index +// +// bin/brain/index.go - rebuild the Ladybug graph (Python write path). +// +// ./bin/brain/index.go --rebuild +// ./bin/brain/index.go --rebuild --with-mail +// ./bin/brain/index.go --dry-run --with-mail +// +// v1 write is always a rebuild when mail is included (live FTS/HNSW + bulk +// insert corrupts Ladybug 0.19 WAL). `add` is v2. +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + args := append([]string{"--with-mail"}, os.Args[1:]...) + os.Exit(cmdbin.ExecFile("bin/kb/index", args)) +} diff --git a/bin/brain/stats.go b/bin/brain/stats.go new file mode 100755 index 0000000..73e1868 --- /dev/null +++ b/bin/brain/stats.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=brain_stats "$0" "$@"; exit +//go:build brain_stats +// +// bin/brain/stats.go - index health. +// +// ./bin/brain/stats.go +// ./bin/brain/stats.go --json +// +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/kb/stats", os.Args[1:])) +} diff --git a/bin/brain/watch.go b/bin/brain/watch.go new file mode 100755 index 0000000..e09f321 --- /dev/null +++ b/bin/brain/watch.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=brain_watch "$0" "$@"; exit +//go:build brain_watch +// +// bin/brain/watch.go - re-index when corpus files change. +// +// ./bin/brain/watch.go [dir...] +// KB_WATCH_INTERVAL=15 ./bin/brain/watch.go +// +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/bin/watch" +) + +func main() { + watch.Run(os.Args[1:]) +} diff --git a/bin/docker-entrypoint b/bin/docker-entrypoint index 941aa02..adce335 100755 --- a/bin/docker-entrypoint +++ b/bin/docker-entrypoint @@ -2,10 +2,10 @@ # bin/docker-entrypoint - run 2dph tools inside the container. # # brain shell (default) -# brain search bin/kb/search -# brain index bin/kb/index -# brain watch watchdog re-indexer (bin/kb/watch) -# brain serve async Go HTTP server (bin/serve) +# brain search bin/brain/search.go +# brain index bin/kb/index --with-mail +# brain watch compiled /app/bin/watch (bin/brain/watch.go) +# brain serve compiled /app/bin/serve (bin/brain/serve.go) # brain extract bin/facts/extract (docker×compose pairing) # brain audit bin/facts/audit # @@ -18,7 +18,7 @@ shift || true case "$CMD" in shell) exec bash ;; search) exec "$KB_PY" /app/bin/kb/search "$@" ;; - index) exec "$KB_PY" /app/bin/kb/index "$@" ;; + index) exec "$KB_PY" /app/bin/kb/index --with-mail "$@" ;; watch) exec /app/bin/watch "$@" ;; serve) exec /app/bin/serve "$@" ;; extract) exec "$KB_PY" /app/bin/facts/extract "$@" ;; diff --git a/bin/kb/index b/bin/kb/index index c818134..29e214f 100755 --- a/bin/kb/index +++ b/bin/kb/index @@ -26,6 +26,7 @@ from kblib import ( # noqa: E402 open_readonly, stats, ) from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402 +from mailleafs import from_mail_root # noqa: E402 CORPUS_DEFAULTS = ["README.md", "PLAN.md", "AGENTS.md", "docs", "skills"] @@ -99,6 +100,9 @@ def main(argv: list[str]) -> int: p = argparse.ArgumentParser(description="build the 2dph brain index") p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)") p.add_argument("--rebuild", action="store_true", help="fresh db + indexes") + p.add_argument("--with-mail", action="store_true", help="include var/mail message.md leafs") + p.add_argument("--since", default="", help="with --with-mail, only messages dated >= YYYY-MM-DD") + p.add_argument("--dry-run", action="store_true", help="count leafs, write nothing") p.add_argument( "--skip-indexes", action="store_true", @@ -109,14 +113,26 @@ def main(argv: list[str]) -> int: a = p.parse_args(argv) from kblib import DB_PATH, VAR - VAR.mkdir(exist_ok=True) - if a.rebuild and DB_PATH.exists(): - DB_PATH.unlink() leafs = load_corpus(ROOT) if a.corpus: for source in a.corpus: leafs.extend(load_corpus_glob(source)) + mail_n = 0 + if a.with_mail: + mail = from_mail_root(ROOT / "var" / "mail", since=a.since) + mail_n = len(mail) + leafs.extend(mail) + + if a.dry_run: + msg = {"indexed": 0, "corpus_total": len(leafs), "mail_leafs": mail_n, "dry_run": True} + print(json.dumps(msg, indent=2) if a.json else + f"brain/index: {len(leafs)} leafs would be indexed (mail={mail_n})") + return 0 + + VAR.mkdir(exist_ok=True) + if a.rebuild and DB_PATH.exists(): + DB_PATH.unlink() db, conn = connect(DB_PATH, read_only=False) init_schema(conn) diff --git a/bin/kb/watch.go b/bin/kb/watch.go index 261b3ad..38891cf 100755 --- a/bin/kb/watch.go +++ b/bin/kb/watch.go @@ -1,5 +1,5 @@ //usr/bin/env go run "$0" "$@"; exit -// bin/kb/watch.go - re-index the 2dph brain when corpus files change. +// bin/kb/watch.go — deprecated. Use bin/brain/watch.go. // // Usage: // diff --git a/bin/mail/import b/bin/mail/import index 044834c..a153746 100755 --- a/bin/mail/import +++ b/bin/mail/import @@ -15,8 +15,8 @@ Writes one directory per message: var/mail/{folder}/{message_id}/ attachments/ raw attachment files (zips unpacked to _unpacked/) attachments/*.md converted attachment content -Indexing is a separate step (bin/mail/index_mail): conversion can crash in -native docling and must not leave the brain DB mid-transaction. +Indexing is a separate step (`bin/brain/index.go --rebuild`): conversion can +crash in native docling and must not leave the brain DB mid-transaction. Requires ONLYOFFICE_URL/USER/PASS in .env (or env). Idempotent: a message already present (message.md exists) is skipped unless --force. diff --git a/bin/mail/import.go b/bin/mail/import.go new file mode 100755 index 0000000..44374cb --- /dev/null +++ b/bin/mail/import.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=mail_import "$0" "$@"; exit +//go:build mail_import +// +// bin/mail/import.go - message.json → markdown (no brain write). +// +// ./bin/mail/import.go --from-raw var/mail +// +// Indexing is bin/brain/index.go --rebuild, not this command. +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/mail/import", os.Args[1:])) +} diff --git a/bin/mail/index_mail b/bin/mail/index_mail index 45bfb0b..a2a20c0 100755 --- a/bin/mail/index_mail +++ b/bin/mail/index_mail @@ -1,135 +1,26 @@ #!/usr/bin/env python3 -"""mail/index_mail - rebuild the brain with every markdown under var/mail. +"""mail/index_mail — deprecated. Use bin/brain/index.go --rebuild --with-mail. -Ladybug corrupts its WAL when brand-new leafs are bulk-inserted while the -FTS/VECTOR indexes already exist, so indexing ALWAYS runs as a fresh rebuild -(repo corpus + var/mail), matching the proven-safe `kb/index --rebuild` path. -Conversion and indexing stay separate: conversion can crash in native docling -and must not leave the brain DB mid-transaction. - - bin/mail/index_mail rebuild the index incl. all mail - bin/mail/index_mail --dry-run count without writing - bin/mail/index_mail --limit N cap messages included - bin/mail/index_mail --since D only messages dated >= D (YYYY-MM-DD) +Ladybug corrupts its WAL on bulk-insert into an already-indexed DB, so this +shim always rebuilds (repo corpus + var/mail). Conversion stays in mail/import. """ from __future__ import annotations -import argparse -import json +import os import sys from pathlib import Path ROOT = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(ROOT / "bin" / "tools")) - -from kblib import DB_PATH, VAR, connect, ensure_indexes, init_schema, stats, upsert_leaf # noqa: E402 -from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402 - - -def msg_date(md: Path) -> str: - j = md.parent / "message.json" - try: - d = json.loads(j.read_text(encoding="utf-8")) - return (d.get("receivedDate") or d.get("receivedAt") or "")[:10] - except Exception: - return "" - - -def mail_leafs(limit: int, since: str, repo: str = "ooMail") -> list[dict]: - root = ROOT / "var" / "mail" - mds = sorted(root.rglob("message.md")) - if since: - mds = [m for m in mds if msg_date(m) >= since] - if limit: - mds = mds[:limit] - leafs: list[dict] = [] - for md in mds: - files = [md] + sorted((md.parent / "attachments").glob("*.md")) - for f in files: - if not f.exists(): - continue - for lf in to_all(read_markdown(f), f, repo=repo): - lf["source"] = f"ooMail:{md.parent.name}:{f.name}" - lf["how"] = "mail/import" - leafs.append(lf) - return leafs def main(argv: list[str]) -> int: - p = argparse.ArgumentParser(description="rebuild the brain incl. all mail") - p.add_argument("--dry-run", action="store_true", help="count only, write nothing") - p.add_argument("--limit", type=int, default=0, help="cap messages included") - p.add_argument("--since", default="", help="only messages dated >= YYYY-MM-DD") - p.add_argument("--json", action="store_true") - a = p.parse_args(argv) - - mail = mail_leafs(a.limit, a.since) - if a.dry_run: - print(f"mail/index_mail: {len(mail)} mail leafs would be indexed") - return 0 - - # Fresh rebuild: delete DB, index repo corpus + mail, create indexes once - # at the end. Never insert into an already-indexed DB (WAL corruption). - VAR.mkdir(exist_ok=True) - if DB_PATH.exists(): - DB_PATH.unlink() - - corpus = _load_corpus() - leafs = corpus + mail - - db, conn = connect(DB_PATH, read_only=False) - init_schema(conn) - embed = _embedder() - done, total = _index_leafs(conn, leafs, embed) - ensure_indexes(conn) - s = stats(conn) - conn.close() - db.close() - - result = {"indexed": done, "corpus_total": total, "mail_leafs": len(mail), - **{k: v for k, v in s.items() if k in ("total", "by_root")}} - print(json.dumps(result, indent=2) if a.json else - f"mail/index_mail: indexed {done}/{total} leafs (mail={len(mail)}); db total {s['total']}") - return 0 - - -CORPUS_DEFAULTS = ["README.md", "PLAN.md", "AGENTS.md", "docs", "skills"] - - -def _load_corpus() -> list[dict]: - files: list[Path] = [] - for entry in CORPUS_DEFAULTS: - p = ROOT / entry - if p.is_file(): - files.append(p) - elif p.is_dir(): - files.extend(walk_markdown(p)) - leafs: list[dict] = [] - for path in files: - try: - leafs.extend(to_all(read_markdown(path), path, repo="eSlider/2dph")) - except OSError as e: - print(f"mail/index_mail: skip {path}: {e}", file=sys.stderr) - return leafs - - -def _index_leafs(conn, leafs: list[dict], embed_fn) -> tuple[int, int]: - count = 0 - for lf in leafs: - query = f"{lf['heading']}\n\n{lf['text']}" - emb = embed_fn(lf["text"]) if lf["text"] else None - upsert_leaf(conn, text=query, root="info", confidence="confirmed", - source=lf["source"], source_rev="mail" if lf.get("how") == "mail/import" else "working-tree", - how=lf.get("how", "kb/index"), loc=lf["source"], type_=lf.get("type", "reference"), - embedding=emb) - count += 1 - return count, len(leafs) - - -def _embedder(): - from model2vec import StaticModel - model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M") - return lambda text: model.encode([text])[0].astype(float).tolist() + print( + "bin/mail/index_mail is deprecated; use bin/brain/index.go --rebuild --with-mail", + file=sys.stderr, + ) + index = ROOT / "bin" / "kb" / "index" + os.execv(sys.executable, [sys.executable, str(index), "--rebuild", "--with-mail", *argv]) + return 1 if __name__ == "__main__": diff --git a/bin/mail/sync.go b/bin/mail/sync.go index b6d7f05..f03d989 100755 --- a/bin/mail/sync.go +++ b/bin/mail/sync.go @@ -6,7 +6,7 @@ // ./bin/mail/sync.go --dry-run // // Writes raw message.json + attachments under var/mail///; run -// bin/mail/import --from-raw afterwards to convert everything to markdown. +// bin/mail/import.go --from-raw afterwards to convert everything to markdown. // // Shebang trick: first line is a Go `//` comment; the real code lives in the // importable package (module path, never a relative import). diff --git a/bin/markdown/doc.go b/bin/markdown/doc.go new file mode 100644 index 0000000..7946de6 --- /dev/null +++ b/bin/markdown/doc.go @@ -0,0 +1,3 @@ +// Commands in this directory are shebang mains (import.go), tagged so +// `go build ./bin/markdown` does not see two mains. +package main diff --git a/bin/markdown/import.go b/bin/markdown/import.go new file mode 100755 index 0000000..546c562 --- /dev/null +++ b/bin/markdown/import.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=markdown_import "$0" "$@"; exit +//go:build markdown_import +// +// bin/markdown/import.go - split markdown into leafs (mistune). +// +// ./bin/markdown/import.go [dir] +// ./bin/markdown/import.go --files a.md,b.md --json +// +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/md/import", os.Args[1:])) +} diff --git a/bin/postgres/doc.go b/bin/postgres/doc.go new file mode 100644 index 0000000..1ff198d --- /dev/null +++ b/bin/postgres/doc.go @@ -0,0 +1,2 @@ +// Commands in this directory are shebang mains (query.go). +package main diff --git a/bin/postgres/query.go b/bin/postgres/query.go new file mode 100755 index 0000000..9dcc526 --- /dev/null +++ b/bin/postgres/query.go @@ -0,0 +1,20 @@ +//usr/bin/env go run -tags=postgres_query "$0" "$@"; exit +//go:build postgres_query +// +// bin/postgres/query.go - read-only Postgres as YAML. +// +// ./bin/postgres/query.go --profile onlyoffice -c 'SELECT 1' +// +// Profiles: $HOME/.config/brain/db-profiles.yml (credentials stay out of git). +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "os" + + "github.com/eSlider/2dph/internal/cmdbin" +) + +func main() { + os.Exit(cmdbin.ExecFile("bin/db/psql-yq", os.Args[1:])) +} diff --git a/bin/tools/kblib.py b/bin/tools/kblib.py index 9d894b9..028ebc1 100644 --- a/bin/tools/kblib.py +++ b/bin/tools/kblib.py @@ -135,7 +135,7 @@ def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None `force=True` is accepted for API compatibility but does **not** drop. Fresh indexes require deleting `var/kb.lbug` and rebuilding - (`bin/kb/index --rebuild`). + (`bin/brain/index.go --rebuild`). """ del force # API compat; DROP is unsafe — see docstring names = leaf_index_names(conn) @@ -145,7 +145,7 @@ def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None except Exception as e: raise RuntimeError( "CREATE_FTS_INDEX failed (often ghost catalog after DROP INDEX). " - "Delete var/kb.lbug and run bin/kb/index --rebuild. " + "Delete var/kb.lbug and run bin/brain/index.go --rebuild. " f"Cause: {e}" ) from e if "Leaf_vec" not in names: @@ -158,7 +158,7 @@ def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None raise RuntimeError( "CREATE_VECTOR_INDEX failed (often ghost catalog after DROP INDEX " "Leaf.Leaf_vec → `_0_Leaf_vec_UPPER already exists in catalog`). " - "Delete var/kb.lbug and run bin/kb/index --rebuild. " + "Delete var/kb.lbug and run bin/brain/index.go --rebuild. " f"Cause: {e}" ) from e names = leaf_index_names(conn) @@ -237,6 +237,6 @@ def stats(conn: ladybug.Connection) -> dict: def open_readonly() -> tuple[ladybug.Database, ladybug.Connection]: if not DB_PATH.exists(): - raise FileNotFoundError(f"{DB_PATH} missing - run bin/kb/index first") + raise FileNotFoundError(f"{DB_PATH} missing - run bin/brain/index.go --rebuild first") db, conn = connect(read_only=True) return db, conn \ No newline at end of file diff --git a/bin/tools/mailleafs.py b/bin/tools/mailleafs.py new file mode 100644 index 0000000..2243632 --- /dev/null +++ b/bin/tools/mailleafs.py @@ -0,0 +1,37 @@ +"""Mail markdown under var/mail → info leafs. Conversion stays off the brain DB.""" +from __future__ import annotations + +import json +from pathlib import Path + +from mdleaves import read_markdown, to_all + + +def msg_date(md: Path) -> str: + j = md.parent / "message.json" + try: + d = json.loads(j.read_text(encoding="utf-8")) + return (d.get("receivedDate") or d.get("receivedAt") or "")[:10] + except (OSError, json.JSONDecodeError, TypeError): + return "" + + +def from_mail_root(root: Path, limit: int = 0, since: str = "", repo: str = "ooMail") -> list[dict]: + if not root.is_dir(): + return [] + mds = sorted(root.rglob("message.md")) + if since: + mds = [m for m in mds if msg_date(m) >= since] + if limit: + mds = mds[:limit] + leafs: list[dict] = [] + for md in mds: + files = [md] + sorted((md.parent / "attachments").glob("*.md")) + for f in files: + if not f.exists(): + continue + for lf in to_all(read_markdown(f), f, repo=repo): + lf["source"] = f"ooMail:{md.parent.name}:{f.name}" + lf["how"] = "mail/import" + leafs.append(lf) + return leafs diff --git a/bin/tools/test_bin_layout.py b/bin/tools/test_bin_layout.py index 203aa9a..4f3deff 100644 --- a/bin/tools/test_bin_layout.py +++ b/bin/tools/test_bin_layout.py @@ -63,3 +63,31 @@ class BinLayoutTest(unittest.TestCase): (ROOT / "bin" / "chats" / "linkedin.go").exists(), "parser must not stay under bin/chats as a second main", ) + + def _assert_shebang(self, rel: str) -> None: + p = ROOT / rel + self.assertTrue(p.is_file(), f"missing {rel}") + first = p.read_text().splitlines()[0] + self.assertTrue( + first.startswith("//usr/bin/env go run"), + f"{rel} shebang, got {first!r}", + ) + + def test_brain_methods_are_shebangs(self) -> None: + for method in ("index.go", "get.go", "stats.go", "eval.go", "watch.go"): + self._assert_shebang(f"bin/brain/{method}") + + def test_mail_import_is_shebang_not_brain_write(self) -> None: + self._assert_shebang("bin/mail/import.go") + index_mail = (ROOT / "bin" / "mail" / "index_mail").read_text() + self.assertIn( + "bin/brain/index.go", + index_mail, + "index_mail must point at bin/brain/index.go", + ) + + def test_markdown_import_is_shebang(self) -> None: + self._assert_shebang("bin/markdown/import.go") + + def test_postgres_query_is_shebang(self) -> None: + self._assert_shebang("bin/postgres/query.go") diff --git a/bin/tools/test_mailleafs.py b/bin/tools/test_mailleafs.py new file mode 100644 index 0000000..a2adb78 --- /dev/null +++ b/bin/tools/test_mailleafs.py @@ -0,0 +1,46 @@ +"""Mail markdown → leafs (no Ladybug). Brain index --with-mail uses this.""" +from __future__ import annotations + +import json +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +import mailleafs # noqa: E402 + + +class MailLeafsTest(unittest.TestCase): + def test_message_md_becomes_info_leaf(self) -> None: + root = Path(tempfile.mkdtemp()) + msg = root / "inbox" / "alice-1" + msg.mkdir(parents=True) + (msg / "message.json").write_text( + json.dumps({"receivedDate": "2026-01-15T10:00:00Z", "subject": "Hello"}), + encoding="utf-8", + ) + (msg / "message.md").write_text( + "---\nroot: info\n---\n\n# Hello\n\nFrom Alice to Bob.\n", + encoding="utf-8", + ) + leafs = mailleafs.from_mail_root(root) + self.assertEqual(len(leafs), 1) + self.assertIn("Alice", leafs[0]["text"]) + self.assertTrue(leafs[0]["source"].startswith("ooMail:")) + self.assertEqual(leafs[0]["how"], "mail/import") + + def test_since_filters_by_message_json_date(self) -> None: + root = Path(tempfile.mkdtemp()) + for name, day in (("old", "2025-01-01"), ("new", "2026-06-01")): + d = root / "inbox" / name + d.mkdir(parents=True) + (d / "message.json").write_text( + json.dumps({"receivedDate": f"{day}T00:00:00Z"}), + encoding="utf-8", + ) + (d / "message.md").write_text(f"# {name}\n\nbody\n", encoding="utf-8") + leafs = mailleafs.from_mail_root(root, since="2026-01-01") + self.assertEqual(len(leafs), 1) + self.assertIn("new", leafs[0]["text"]) diff --git a/bin/tools/test_published_docs.py b/bin/tools/test_published_docs.py index 5ded0b7..0facabe 100644 --- a/bin/tools/test_published_docs.py +++ b/bin/tools/test_published_docs.py @@ -30,6 +30,15 @@ class PublishedDocsTest(unittest.TestCase): "README deduction search must name bin/brain/search.go", ) + def test_readme_index_is_brain_not_index_mail(self) -> None: + text = (ROOT / "README.md").read_text() + self.assertIn("bin/brain/index.go", text) + self.assertNotIn( + "bin/mail/index_mail", + text, + "mail index is a brain write; README must name bin/brain/index.go", + ) + def test_docs_do_not_claim_hop_walks(self) -> None: paths = [ ROOT / "README.md", diff --git a/bin/watch/watch.go b/bin/watch/watch.go index c7669c0..38fddc3 100644 --- a/bin/watch/watch.go +++ b/bin/watch/watch.go @@ -1,4 +1,4 @@ -// Package watch polls corpus directories for changes and re-runs bin/kb/index. +// Package watch polls corpus directories for changes and re-runs brain/index. // // Port of the former bin/kb-watch bash script to an importable, testable Go // package. Polls file mtimes (no inotify deps); cheap and reliable. @@ -18,8 +18,8 @@ import ( type Options struct { Dirs []string Interval time.Duration - // IndexCmd is the kb/index command template. %s is replaced by the repo - // root (from KB_ROOT). Defaults to `python3 /bin/kb/index`. + // IndexCmd is the index command template. %s is replaced by the repo + // root (from KB_ROOT). Defaults to `python3 /bin/kb/index --with-mail`. IndexCmd string } @@ -67,7 +67,7 @@ func fromEnv(args []string) Options { if pys == "" { pys = "python3" } - opts.IndexCmd = pys + " /bin/kb/index" + opts.IndexCmd = pys + " /bin/kb/index --with-mail" return opts } diff --git a/bin/watch/watch_test.go b/bin/watch/watch_test.go index 2292def..4fdbeac 100644 --- a/bin/watch/watch_test.go +++ b/bin/watch/watch_test.go @@ -3,6 +3,7 @@ package watch import ( "os" "path/filepath" + "strings" "testing" "time" ) @@ -43,7 +44,10 @@ func TestFromEnvDefaults(t *testing.T) { if opts.Interval != 30*time.Second { t.Fatalf("default interval = %s, want 30s", opts.Interval) } - if opts.IndexCmd == "" { - t.Fatal("default index cmd is empty") + if !strings.Contains(opts.IndexCmd, "kb/index") { + t.Fatalf("default index cmd = %q, want kb/index", opts.IndexCmd) + } + if !strings.Contains(opts.IndexCmd, "--with-mail") { + t.Fatalf("default index cmd must include --with-mail, got %q", opts.IndexCmd) } } diff --git a/internal/cmdbin/exec.go b/internal/cmdbin/exec.go new file mode 100644 index 0000000..de69acb --- /dev/null +++ b/internal/cmdbin/exec.go @@ -0,0 +1,54 @@ +package cmdbin + +import ( + "errors" + "os" + "os/exec" + "path/filepath" + "strings" +) + +// Root is the 2dph checkout (KB_ROOT, or walk up for .git / var). +func Root() string { + if v := os.Getenv("KB_ROOT"); v != "" { + return v + } + wd, err := os.Getwd() + if err != nil { + return "." + } + for i := 0; i < 10; i++ { + if _, err := os.Stat(filepath.Join(wd, ".git")); err == nil { + return wd + } + if _, err := os.Stat(filepath.Join(wd, "var")); err == nil { + return wd + } + parent := filepath.Dir(wd) + if parent == wd { + break + } + wd = parent + } + return "." +} + +// ExecFile runs repo-relative path (python/bash shebang scripts) with stdio. +func ExecFile(rel string, args []string) int { + path := filepath.Join(Root(), filepath.FromSlash(rel)) + cmd := exec.Command(path, args...) + cmd.Stdin = os.Stdin + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + cmd.Dir = Root() + if err := cmd.Run(); err != nil { + if ee, ok := err.(*exec.ExitError); ok { + return ee.ExitCode() + } + if errors.Is(err, os.ErrNotExist) || strings.Contains(err.Error(), "no such file") { + return 127 + } + return 1 + } + return 0 +} diff --git a/internal/cmdbin/exec_test.go b/internal/cmdbin/exec_test.go new file mode 100644 index 0000000..feeea76 --- /dev/null +++ b/internal/cmdbin/exec_test.go @@ -0,0 +1,34 @@ +package cmdbin + +import ( + "os" + "path/filepath" + "testing" +) + +func TestRootHonorsKBROOT(t *testing.T) { + dir := t.TempDir() + t.Setenv("KB_ROOT", dir) + if got := Root(); got != dir { + t.Fatalf("Root() = %q, want %q", got, dir) + } +} + +func TestExecFileMissingIs127(t *testing.T) { + t.Setenv("KB_ROOT", t.TempDir()) + if code := ExecFile("no/such-tool", nil); code != 127 { + t.Fatalf("exit = %d, want 127", code) + } +} + +func TestExecFileRuns(t *testing.T) { + root := t.TempDir() + script := filepath.Join(root, "echo.sh") + if err := os.WriteFile(script, []byte("#!/bin/sh\nexit 3\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("KB_ROOT", root) + if code := ExecFile("echo.sh", nil); code != 3 { + t.Fatalf("exit = %d, want 3", code) + } +} diff --git a/skills/diataxis-docs/SKILL.md b/skills/diataxis-docs/SKILL.md index d0cf77b..b18e767 100644 --- a/skills/diataxis-docs/SKILL.md +++ b/skills/diataxis-docs/SKILL.md @@ -30,7 +30,7 @@ related: --- ``` -`bin/kb/index` reads this. `type` becomes a searchable column. `related:` is +`bin/brain/index.go` reads this. `type` becomes a searchable column. `related:` is frontmatter for humans; graph hops from it are not implemented yet. ```bash diff --git a/skills/kb-search/SKILL.md b/skills/kb-search/SKILL.md index 9302de2..8aa1a5f 100644 --- a/skills/kb-search/SKILL.md +++ b/skills/kb-search/SKILL.md @@ -26,9 +26,9 @@ second independent source when local roots cannot confirm. An answer is bin/brain/search.go "Matrix federation" # pointers + snippets, YAML bin/brain/search.go "onlyoffice postgres" --root facts # restrict to confirmed bin/brain/search.go "where is cs-lexicon" --json | yq '.[].ref' -bin/kb/get --body # full chunk only when needed -bin/kb/stats # index health -bin/kb/eval # recall@5 >= 0.95 gate +bin/brain/get.go --body # full chunk only when needed +bin/brain/stats.go # index health +bin/brain/eval.go # recall@5 >= 0.95 gate ``` `bin/kb/search` is a deprecated wrapper. `--hop` errors (File/FROM_FILE edges @@ -39,7 +39,7 @@ are not wired yet); do not treat it as a graph walk. - Search before you read. Never grep a repo for a concept the graph covers. - `--root facts` returns only confirmed evidence-linked answers. Default shows facts first, then info leafs clearly marked `(not confirmed)`. -- If recall looks wrong, run `bin/kb/eval`; it gates control questions and +- If recall looks wrong, run `bin/brain/eval.go`; it gates control questions and should stay at or above 95% recall@5. - Escalate to `web-search` (the `web-search` skill) as the independent second source when both local roots cannot confirm; never report an unconfirmed