feat: search --hop walks FROM_FILE to Person.
Parser no longer errors; hop 1 returns File, hop 3 reaches Person. Rebuild writes Leaf-[:FROM_FILE]->File so the walk is not empty on a fresh index (Gitea #17).
This commit is contained in:
@@ -88,7 +88,9 @@ detective method: **a fact needs ≥2 independent sources or it is
|
|||||||
Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`.
|
Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`.
|
||||||
`Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`.
|
`Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`.
|
||||||
Edges: `RUNS / USES / FROM_FILE / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
|
Edges: `RUNS / USES / FROM_FILE / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
|
||||||
`FROM_FILE` / `HAS_VERSION` exist in schema; search `--hop` does not walk them yet ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
|
`FROM_FILE` / `HAS_VERSION` / `AUTHORED`: `bin/brain/search.go --hop N` walks
|
||||||
|
them from each hit (1=File, 2=Commit, 3=Person). Rebuild writes
|
||||||
|
`Leaf-[:FROM_FILE]->File`; git import writes the rest.
|
||||||
|
|
||||||
Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
|
Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
|
||||||
`where`, `when`, `source_rev`.
|
`where`, `when`, `source_rev`.
|
||||||
@@ -162,9 +164,8 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
|
|||||||
|
|
||||||
## Gap to v1 (epic #16)
|
## Gap to v1 (epic #16)
|
||||||
|
|
||||||
Read path + MCP are in. Incremental `brain/add` is in. The detective brain is
|
Read path + MCP are in. Incremental `brain/add` and `--hop` are in. Remaining:
|
||||||
not closed until search can **walk** the graph and the facts+chats corpus
|
facts+chats corpus on rebuild, and CI eval SoT. Board:
|
||||||
lands on rebuild. Board:
|
|
||||||
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16),
|
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16),
|
||||||
milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
|
milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
|
||||||
Narrative: [docs/roadmap.md](docs/roadmap.md).
|
Narrative: [docs/roadmap.md](docs/roadmap.md).
|
||||||
@@ -172,7 +173,7 @@ Narrative: [docs/roadmap.md](docs/roadmap.md).
|
|||||||
| Order | Issue | Gap |
|
| Order | Issue | Gap |
|
||||||
|-------|-------|-----|
|
|-------|-------|-----|
|
||||||
| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in** — `bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. |
|
| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in** — `bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. |
|
||||||
| 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | `--hop` errors. `FROM_FILE` / `HAS_VERSION` are in schema; search does not walk them. |
|
| 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | **in** — `--hop N` walks `FROM_FILE` → `HAS_VERSION` → `AUTHORED` (max 3). |
|
||||||
| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. |
|
| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. |
|
||||||
| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search` → `get` → `audit`). |
|
| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search` → `get` → `audit`). |
|
||||||
| 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. |
|
| 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. |
|
||||||
|
|||||||
@@ -96,7 +96,7 @@ bin/brain/stats.go # index health
|
|||||||
bin/brain/eval.go # recall@5 gate
|
bin/brain/eval.go # recall@5 gate
|
||||||
```
|
```
|
||||||
|
|
||||||
`--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
|
`--hop N` walks File/Commit/Person from each hit (max 3). `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
|
||||||
|
|
||||||
Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary):
|
Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary):
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -3,7 +3,7 @@
|
|||||||
//
|
//
|
||||||
// bin/brain/search.go - deduction search over the 2dph brain.
|
// bin/brain/search.go - deduction search over the 2dph brain.
|
||||||
//
|
//
|
||||||
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--json] [--no-web]
|
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--hop N] [--json] [--no-web]
|
||||||
// ./bin/brain/search.go serve [port]
|
// ./bin/brain/search.go serve [port]
|
||||||
// ./bin/brain/search.go --list-model
|
// ./bin/brain/search.go --list-model
|
||||||
//
|
//
|
||||||
|
|||||||
+3
-2
@@ -22,7 +22,7 @@ ROOT = Path(__file__).resolve().parents[2]
|
|||||||
sys.path.insert(0, str(ROOT / "bin" / "tools"))
|
sys.path.insert(0, str(ROOT / "bin" / "tools"))
|
||||||
|
|
||||||
from kblib import ( # noqa: E402
|
from kblib import ( # noqa: E402
|
||||||
connect, ensure_indexes, init_schema, upsert_leaf,
|
connect, ensure_indexes, init_schema, upsert_leaf, link_from_file,
|
||||||
open_readonly, stats,
|
open_readonly, stats,
|
||||||
)
|
)
|
||||||
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
|
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
|
||||||
@@ -81,10 +81,11 @@ def index_leafs(conn, leafs: list[dict], embed_fn, limit: int) -> tuple[int, int
|
|||||||
for lf in leafs[:limit] if limit else leafs:
|
for lf in leafs[:limit] if limit else leafs:
|
||||||
query = f"{lf['heading']}\n\n{lf['text']}"
|
query = f"{lf['heading']}\n\n{lf['text']}"
|
||||||
emb = embed_fn(lf["text"]) if lf["text"] else None
|
emb = embed_fn(lf["text"]) if lf["text"] else None
|
||||||
upsert_leaf(conn, text=query, root="info", confidence="confirmed",
|
lid = upsert_leaf(conn, text=query, root="info", confidence="confirmed",
|
||||||
source=lf["source"], source_rev="working-tree",
|
source=lf["source"], source_rev="working-tree",
|
||||||
how="kb/index", loc=lf["source"], type_=lf.get("type", "reference"),
|
how="kb/index", loc=lf["source"], type_=lf.get("type", "reference"),
|
||||||
embedding=emb)
|
embedding=emb)
|
||||||
|
link_from_file(conn, lid, lf["source"], repo=str(lf.get("repo") or ""))
|
||||||
count += 1
|
count += 1
|
||||||
return count, len(leafs)
|
return count, len(leafs)
|
||||||
|
|
||||||
|
|||||||
@@ -162,6 +162,53 @@ def add_leafs(conn: ladybug.Connection, leafs: list[dict]) -> list[str]:
|
|||||||
return ids
|
return ids
|
||||||
|
|
||||||
|
|
||||||
|
def file_id(repo: str, path: str) -> str:
|
||||||
|
"""Stable File.id matching gitimport (`repo:path`)."""
|
||||||
|
return f"{repo}:{path}" if repo else path
|
||||||
|
|
||||||
|
|
||||||
|
def link_from_file(conn: ladybug.Connection, leaf_id: str, path: str,
|
||||||
|
repo: str = "", mtime: str = "") -> str:
|
||||||
|
"""MERGE File and Leaf-[:FROM_FILE]->File so --hop 1 can walk."""
|
||||||
|
fid = file_id(repo, path)
|
||||||
|
conn.execute(
|
||||||
|
"MERGE (f:File {id:$id}) SET f.path=$path, f.repo=$repo, f.mtime=$mtime",
|
||||||
|
parameters={"id": fid, "path": path, "repo": repo, "mtime": mtime},
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"MATCH (l:Leaf {id:$lid}), (f:File {id:$fid}) "
|
||||||
|
"MERGE (l)-[:FROM_FILE]->(f)",
|
||||||
|
parameters={"lid": leaf_id, "fid": fid},
|
||||||
|
)
|
||||||
|
return fid
|
||||||
|
|
||||||
|
|
||||||
|
HOP_STMTS = {
|
||||||
|
1: "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File) RETURN f.id, f.path, 1",
|
||||||
|
2: ("MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit) "
|
||||||
|
"RETURN c.id, c.subject, 2"),
|
||||||
|
3: ("MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit)"
|
||||||
|
"-[:AUTHORED]->(p:Person) RETURN p.id, p.name, 3"),
|
||||||
|
}
|
||||||
|
HOP_LABELS = {1: "File", 2: "Commit", 3: "Person"}
|
||||||
|
|
||||||
|
|
||||||
|
def hop_walk(conn: ladybug.Connection, leaf_id: str, n: int) -> list[dict]:
|
||||||
|
"""Walk Leaf → File → Commit → Person up to n hops (max 3)."""
|
||||||
|
depth = min(max(int(n), 0), 3)
|
||||||
|
out: list[dict] = []
|
||||||
|
for d in range(1, depth + 1):
|
||||||
|
rows = conn.execute(HOP_STMTS[d], parameters={"id": leaf_id}).get_all()
|
||||||
|
for row in rows:
|
||||||
|
out.append({
|
||||||
|
"id": row[0],
|
||||||
|
"label": HOP_LABELS[d],
|
||||||
|
"name": row[1],
|
||||||
|
"depth": int(row[2]),
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def leaf_index_names(conn: ladybug.Connection) -> set[str]:
|
def leaf_index_names(conn: ladybug.Connection) -> set[str]:
|
||||||
"""Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'})."""
|
"""Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'})."""
|
||||||
rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()
|
rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()
|
||||||
|
|||||||
@@ -161,6 +161,39 @@ class KblibTest(unittest.TestCase):
|
|||||||
self.assertEqual(stats["total"], 2)
|
self.assertEqual(stats["total"], 2)
|
||||||
self.assertEqual(stats["by_root"], {"facts": 1, "info": 1})
|
self.assertEqual(stats["by_root"], {"facts": 1, "info": 1})
|
||||||
|
|
||||||
|
def test_hop_1_returns_file_hop_3_reaches_person(self):
|
||||||
|
"""--hop walks FROM_FILE / HAS_VERSION / AUTHORED (Gitea #17)."""
|
||||||
|
import gitimport
|
||||||
|
|
||||||
|
lid = kblib.upsert_leaf(
|
||||||
|
self.conn, text="readme hop fixture", root="info",
|
||||||
|
confidence="confirmed", source="README.md", source_rev="r1",
|
||||||
|
how="test", loc="README.md", type_="reference",
|
||||||
|
embedding=make_emb(0.3),
|
||||||
|
)
|
||||||
|
kblib.link_from_file(self.conn, lid, "README.md", repo="sample-repo")
|
||||||
|
gitimport.index_commits(self.conn, [gitimport.Commit(
|
||||||
|
sha="a1b2c3d",
|
||||||
|
author="Ada Lovelace",
|
||||||
|
email="ada@example.com",
|
||||||
|
date="2026-08-10T12:00:00Z",
|
||||||
|
subject="feat: first commit",
|
||||||
|
files=["README.md"],
|
||||||
|
)], "sample-repo")
|
||||||
|
hop1 = kblib.hop_walk(self.conn, lid, 1)
|
||||||
|
self.assertEqual(len(hop1), 1)
|
||||||
|
self.assertEqual(hop1[0]["label"], "File")
|
||||||
|
self.assertEqual(hop1[0]["name"], "README.md")
|
||||||
|
self.assertEqual(hop1[0]["depth"], 1)
|
||||||
|
hop3 = kblib.hop_walk(self.conn, lid, 3)
|
||||||
|
labels = {n["label"] for n in hop3}
|
||||||
|
self.assertIn("File", labels)
|
||||||
|
self.assertIn("Commit", labels)
|
||||||
|
self.assertIn("Person", labels)
|
||||||
|
person = [n for n in hop3 if n["label"] == "Person"][0]
|
||||||
|
self.assertEqual(person["name"], "Ada Lovelace")
|
||||||
|
self.assertEqual(person["depth"], 3)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
"""Published docs must match live commands (Gitea SoT, brain/search, no fake --hop)."""
|
"""Published docs must match live commands (Gitea SoT, brain/search)."""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import re
|
|
||||||
import unittest
|
import unittest
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -139,24 +138,21 @@ class PublishedDocsTest(unittest.TestCase):
|
|||||||
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
|
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
|
||||||
self.assertIn("`web` block", skill)
|
self.assertIn("`web` block", skill)
|
||||||
|
|
||||||
def test_docs_do_not_claim_hop_walks(self) -> None:
|
def test_docs_say_hop_walks_from_file(self) -> None:
|
||||||
paths = [
|
paths = [
|
||||||
ROOT / "README.md",
|
ROOT / "README.md",
|
||||||
ROOT / "docs" / "design.md",
|
ROOT / "docs" / "design.md",
|
||||||
ROOT / "skills" / "brain" / "SKILL.md",
|
ROOT / "skills" / "brain" / "SKILL.md",
|
||||||
ROOT / "skills" / "diataxis-docs" / "SKILL.md",
|
|
||||||
ROOT / "docs" / "runbook.md",
|
ROOT / "docs" / "runbook.md",
|
||||||
ROOT / "docs" / "README.md",
|
ROOT / "docs" / "README.md",
|
||||||
ROOT / "docs" / "roadmap.md",
|
|
||||||
]
|
]
|
||||||
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
|
|
||||||
# Honest "not implemented" notes must not match.
|
|
||||||
lie = re.compile(r"--hop (?:N|1).*(?:follow|walk)", re.I | re.S)
|
|
||||||
for path in paths:
|
for path in paths:
|
||||||
text = path.read_text()
|
text = path.read_text()
|
||||||
self.assertIsNone(
|
self.assertIn("--hop", text, f"{path.relative_to(ROOT)} must document --hop")
|
||||||
lie.search(text),
|
self.assertNotIn(
|
||||||
f"{path.relative_to(ROOT)} still claims --hop walks the graph",
|
"not implemented",
|
||||||
|
text.lower(),
|
||||||
|
f"{path.relative_to(ROOT)} still says hop is not implemented",
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_docs_are_portable_diataxis(self) -> None:
|
def test_docs_are_portable_diataxis(self) -> None:
|
||||||
|
|||||||
+3
-3
@@ -27,9 +27,9 @@ Python write sidecar, **D14** `bin/{subject}/{method}.go`, **D15** Gitea origin,
|
|||||||
**D17** assertion gate (facts → info → web), **D18** pluggable reasoner.
|
**D17** assertion gate (facts → info → web), **D18** pluggable reasoner.
|
||||||
|
|
||||||
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go` —
|
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go` —
|
||||||
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is
|
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop N` walks
|
||||||
not a walk; the flag errors. Schema has `FROM_FILE`; search does not
|
`FROM_FILE` → Commit → Person from each hit (max 3). Rebuild writes
|
||||||
use it ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
|
File edges ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
|
||||||
|
|
||||||
Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues)
|
Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues)
|
||||||
([epic #16](https://git.produktor.io/eSlider/2dph/issues/16)).
|
([epic #16](https://git.produktor.io/eSlider/2dph/issues/16)).
|
||||||
|
|||||||
+3
-3
@@ -34,9 +34,9 @@ bin/brain/search.go "question"
|
|||||||
is not evidence of absence; `--no-web` / `--root` skip it)
|
is not evidence of absence; `--no-web` / `--root` skip it)
|
||||||
```
|
```
|
||||||
|
|
||||||
`--hop` is not implemented. `FROM_FILE` / `HAS_VERSION` exist in schema;
|
`--hop N` walks `Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`
|
||||||
search does not walk them ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
|
from each hit (1=File, 2=Commit, 3=Person). Rebuild writes FROM_FILE;
|
||||||
The flag is an error; it is not a graph walk.
|
git import writes HAS_VERSION/AUTHORED ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
|
||||||
|
|
||||||
## Who / What / How / Where / When + evidence
|
## Who / What / How / Where / When + evidence
|
||||||
|
|
||||||
|
|||||||
+6
-7
@@ -26,6 +26,8 @@ Compose `api` (no CPython) / `index` (Python write). Issues #1–#5, #7–#13.
|
|||||||
[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop.
|
[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop.
|
||||||
[#14](https://git.produktor.io/eSlider/2dph/issues/14) `bin/brain/add.go` /
|
[#14](https://git.produktor.io/eSlider/2dph/issues/14) `bin/brain/add.go` /
|
||||||
`POST /ingest` (Python `kblib.add_leafs`; no Go upsert port).
|
`POST /ingest` (Python `kblib.add_leafs`; no Go upsert port).
|
||||||
|
[#17](https://git.produktor.io/eSlider/2dph/issues/17) `--hop N` walks
|
||||||
|
FROM_FILE / HAS_VERSION / AUTHORED.
|
||||||
|
|
||||||
## Blockers
|
## Blockers
|
||||||
|
|
||||||
@@ -36,17 +38,14 @@ question
|
|||||||
├─ facts / info roots ← in
|
├─ facts / info roots ← in
|
||||||
├─ web (D17) ← in
|
├─ web (D17) ← in
|
||||||
├─ brain/add ACID ← in
|
├─ brain/add ACID ← in
|
||||||
├─ Cypher hop ← #17 schema yes, search no
|
├─ Cypher hop ← in
|
||||||
└─ facts+chats corpus ← #18
|
└─ facts+chats corpus ← #18
|
||||||
```
|
```
|
||||||
|
|
||||||
1. **[#17](https://git.produktor.io/eSlider/2dph/issues/17) hops** —
|
1. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** —
|
||||||
`Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`
|
|
||||||
exists; `--hop` still errors. Without a walk, D9/D10 are paper.
|
|
||||||
2. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** —
|
|
||||||
rebuild loads repo markdown + mail as `info`. `facts/extract` pairing
|
rebuild loads repo markdown + mail as `info`. `facts/extract` pairing
|
||||||
and `bin/chats` are not indexed. WhatsApp is a stub. PII stays in `var/`.
|
and `bin/chats` are not indexed. WhatsApp is a stub. PII stays in `var/`.
|
||||||
3. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** —
|
2. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** —
|
||||||
recall SoT should be `bin/brain/eval.go` via Zig, not Python `bin/kb/eval`.
|
recall SoT should be `bin/brain/eval.go` via Zig, not Python `bin/kb/eval`.
|
||||||
|
|
||||||
## Not v1
|
## Not v1
|
||||||
@@ -56,6 +55,6 @@ contradiction resolution, OQ3 duckdb-md export, OQ4 YAML-first leafs.
|
|||||||
|
|
||||||
## Close epic #16 when
|
## Close epic #16 when
|
||||||
|
|
||||||
- `--hop` stops erroring and runs a Cypher path from search hits
|
|
||||||
- ops pairing + chat import land as leafs on rebuild
|
- ops pairing + chat import land as leafs on rebuild
|
||||||
- MCP tool order is documented and still gated by tests
|
- MCP tool order is documented and still gated by tests
|
||||||
|
- CI recall SoT is `bin/brain/eval.go` via Zig
|
||||||
|
|||||||
+1
-1
@@ -56,7 +56,7 @@ bin/brain/get.go <id> --body
|
|||||||
bin/brain/stats.go
|
bin/brain/stats.go
|
||||||
```
|
```
|
||||||
|
|
||||||
`--hop` is not implemented. Empty web results are `throttled`, not absence.
|
`--hop N` walks File → Commit → Person from each hit. Empty web results are `throttled`, not absence.
|
||||||
Gap to v1: [roadmap](roadmap.md) / [epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
|
Gap to v1: [roadmap](roadmap.md) / [epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
|
||||||
|
|
||||||
Ladybug 0.19: never `DROP INDEX` FTS/VECTOR (ghost catalog). Fresh indexes =
|
Ladybug 0.19: never `DROP INDEX` FTS/VECTOR (ghost catalog). Fresh indexes =
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--json] [--no-web]
|
const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--hop N] [--json] [--no-web]
|
||||||
bin/brain/search.go serve [port]
|
bin/brain/search.go serve [port]
|
||||||
bin/brain/search.go --list-model`
|
bin/brain/search.go --list-model`
|
||||||
|
|
||||||
@@ -15,6 +15,7 @@ type Options struct {
|
|||||||
Root string
|
Root string
|
||||||
Repo string
|
Repo string
|
||||||
Limit int
|
Limit int
|
||||||
|
Hop int
|
||||||
JSONOut bool
|
JSONOut bool
|
||||||
ListModel bool
|
ListModel bool
|
||||||
NoWeb bool
|
NoWeb bool
|
||||||
@@ -22,8 +23,6 @@ type Options struct {
|
|||||||
|
|
||||||
// ParseArgs reads flags. Unknown flags are an error: silently dropping them
|
// ParseArgs reads flags. Unknown flags are an error: silently dropping them
|
||||||
// meant `--hop 1` vanished and its argument `1` was appended to the query.
|
// meant `--hop 1` vanished and its argument `1` was appended to the query.
|
||||||
// --hop is recognised so it cannot be swallowed; it is not implemented until
|
|
||||||
// File/FROM_FILE edges exist.
|
|
||||||
func ParseArgs(args []string) (Options, error) {
|
func ParseArgs(args []string) (Options, error) {
|
||||||
opt := Options{Limit: 20}
|
opt := Options{Limit: 20}
|
||||||
var queryArgs []string
|
var queryArgs []string
|
||||||
@@ -52,7 +51,15 @@ func ParseArgs(args []string) (Options, error) {
|
|||||||
}
|
}
|
||||||
opt.Limit = n
|
opt.Limit = n
|
||||||
case "--hop":
|
case "--hop":
|
||||||
return opt, fmt.Errorf("--hop is not implemented yet (needs File/FROM_FILE edges)")
|
i++
|
||||||
|
n, err := strconv.Atoi(args[i])
|
||||||
|
if err != nil || n < 1 {
|
||||||
|
return opt, fmt.Errorf("--hop must be a positive integer, got %q", args[i])
|
||||||
|
}
|
||||||
|
if n > 3 {
|
||||||
|
return opt, fmt.Errorf("--hop max is 3 (File → Commit → Person)")
|
||||||
|
}
|
||||||
|
opt.Hop = n
|
||||||
case "--json":
|
case "--json":
|
||||||
opt.JSONOut = true
|
opt.JSONOut = true
|
||||||
case "--no-web":
|
case "--no-web":
|
||||||
|
|||||||
@@ -7,3 +7,30 @@ const FTSStmt = "CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " +
|
|||||||
|
|
||||||
const VecStmt = "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
|
const VecStmt = "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
|
||||||
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n"
|
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n"
|
||||||
|
|
||||||
|
// HopStmt is the Cypher walk from a search hit. Depth 1 = File, 2 = Commit, 3 = Person.
|
||||||
|
func HopStmt(depth int) string {
|
||||||
|
switch depth {
|
||||||
|
case 1:
|
||||||
|
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File) RETURN f.id, f.path, 1"
|
||||||
|
case 2:
|
||||||
|
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit) RETURN c.id, c.subject, 2"
|
||||||
|
case 3:
|
||||||
|
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit)-[:AUTHORED]->(p:Person) RETURN p.id, p.name, 3"
|
||||||
|
default:
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func HopLabel(depth int) string {
|
||||||
|
switch depth {
|
||||||
|
case 1:
|
||||||
|
return "File"
|
||||||
|
case 2:
|
||||||
|
return "Commit"
|
||||||
|
case 3:
|
||||||
|
return "Person"
|
||||||
|
default:
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -7,6 +7,13 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
type HopNode struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Label string `json:"label"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
Depth int `json:"depth"`
|
||||||
|
}
|
||||||
|
|
||||||
// Hit is one search result, mirroring the python script's dict shape.
|
// Hit is one search result, mirroring the python script's dict shape.
|
||||||
type Hit struct {
|
type Hit struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
@@ -15,6 +22,7 @@ type Hit struct {
|
|||||||
Source string `json:"-"`
|
Source string `json:"-"`
|
||||||
Score float64 `json:"score"`
|
Score float64 `json:"score"`
|
||||||
Snippet string `json:"snippet,omitempty"`
|
Snippet string `json:"snippet,omitempty"`
|
||||||
|
Hops []HopNode `json:"hops,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// rrfK dampens the contribution of low ranks; same constant as kblib.py.
|
// rrfK dampens the contribution of low ranks; same constant as kblib.py.
|
||||||
|
|||||||
@@ -91,15 +91,41 @@ func TestHybridKeepsVectorScoreForSharedHit(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The old parser dropped unknown flags and appended their arguments to the
|
// The old parser dropped unknown flags and appended their arguments to the
|
||||||
// query, so `search "q" --hop 1` searched for "q 1". --hop is not implemented
|
// query, so `search "q" --hop 1` searched for "q 1". --hop must stay a flag.
|
||||||
// here (needs File edges); it must still fail closed instead of changing q.
|
|
||||||
func TestParseHopIsNotSwallowedIntoTheQuery(t *testing.T) {
|
func TestParseHopIsNotSwallowedIntoTheQuery(t *testing.T) {
|
||||||
_, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"})
|
opt, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"})
|
||||||
if err == nil {
|
if err != nil {
|
||||||
t.Fatal("expected --hop to error (not implemented), not be swallowed")
|
t.Fatalf("unexpected error: %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(err.Error(), "--hop") {
|
if opt.Query != "what runs on arc-2" {
|
||||||
t.Fatalf("error should name --hop, got %v", err)
|
t.Fatalf("query swallowed hop arg: %q", opt.Query)
|
||||||
|
}
|
||||||
|
if opt.Hop != 1 {
|
||||||
|
t.Fatalf("hop = %d, want 1", opt.Hop)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseHopMaxIsThree(t *testing.T) {
|
||||||
|
if _, err := ParseArgs([]string{"q", "--hop", "4"}); err == nil {
|
||||||
|
t.Fatal("expected --hop 4 to error")
|
||||||
|
}
|
||||||
|
opt, err := ParseArgs([]string{"q", "--hop", "3"})
|
||||||
|
if err != nil || opt.Hop != 3 {
|
||||||
|
t.Fatalf("hop 3: %+v err=%v", opt, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHopStmtWalksFromFile(t *testing.T) {
|
||||||
|
s := HopStmt(1)
|
||||||
|
if !strings.Contains(s, "FROM_FILE") || !strings.Contains(s, "File") {
|
||||||
|
t.Fatalf("hop 1 must walk FROM_FILE, got %q", s)
|
||||||
|
}
|
||||||
|
s3 := HopStmt(3)
|
||||||
|
if !strings.Contains(s3, "HAS_VERSION") || !strings.Contains(s3, "AUTHORED") || !strings.Contains(s3, "Person") {
|
||||||
|
t.Fatalf("hop 3 must reach Person, got %q", s3)
|
||||||
|
}
|
||||||
|
if HopLabel(1) != "File" || HopLabel(3) != "Person" {
|
||||||
|
t.Fatal("hop labels")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -56,6 +56,12 @@ func runSearch(args []string) int {
|
|||||||
fmt.Fprintf(os.Stderr, "search: %v\n", err)
|
fmt.Fprintf(os.Stderr, "search: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
if opt.Hop > 0 {
|
||||||
|
if err := attachHops(hits, opt.Hop); err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "hop: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
results := hits
|
results := hits
|
||||||
for i := range results {
|
for i := range results {
|
||||||
@@ -108,6 +114,44 @@ func searchHits(query, root, repo string, limit int) ([]Hit, error) {
|
|||||||
return rank.RankAndFilter(fts, vec, root, repo, limit), nil
|
return rank.RankAndFilter(fts, vec, root, repo, limit), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func attachHops(hits []Hit, n int) error {
|
||||||
|
if conn == nil {
|
||||||
|
return fmt.Errorf("brain not open")
|
||||||
|
}
|
||||||
|
for i := range hits {
|
||||||
|
var hops []rank.HopNode
|
||||||
|
for d := 1; d <= n; d++ {
|
||||||
|
stmt, err := conn.Prepare(rank.HopStmt(d))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
res, err := conn.Execute(stmt, map[string]any{"id": hits[i].ID})
|
||||||
|
stmt.Close()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for res.HasNext() {
|
||||||
|
row, err := res.Next()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
vals, err := row.GetAsSlice()
|
||||||
|
if err != nil || len(vals) < 3 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
hops = append(hops, rank.HopNode{
|
||||||
|
ID: fmt.Sprint(vals[0]),
|
||||||
|
Label: rank.HopLabel(d),
|
||||||
|
Name: fmt.Sprint(vals[1]),
|
||||||
|
Depth: int(asInt(vals[2])),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hits[i].Hops = hops
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func b2i(err error) int {
|
func b2i(err error) int {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 1
|
return 1
|
||||||
@@ -188,6 +232,7 @@ type jsonHit struct {
|
|||||||
Root string `json:"root"`
|
Root string `json:"root"`
|
||||||
Score float64 `json:"score"`
|
Score float64 `json:"score"`
|
||||||
Snippet string `json:"snippet,omitempty"`
|
Snippet string `json:"snippet,omitempty"`
|
||||||
|
Hops []rank.HopNode `json:"hops,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *jsonOut {
|
func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *jsonOut {
|
||||||
@@ -199,6 +244,7 @@ func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *js
|
|||||||
Root: h.Root,
|
Root: h.Root,
|
||||||
Score: h.Score,
|
Score: h.Score,
|
||||||
Snippet: h.Snippet,
|
Snippet: h.Snippet,
|
||||||
|
Hops: h.Hops,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return &jsonOut{
|
return &jsonOut{
|
||||||
@@ -222,6 +268,18 @@ func resultsToDicts(hits []Hit) []any {
|
|||||||
if h.Snippet != "" {
|
if h.Snippet != "" {
|
||||||
d = append(d, KV{"snippet", h.Snippet})
|
d = append(d, KV{"snippet", h.Snippet})
|
||||||
}
|
}
|
||||||
|
if len(h.Hops) > 0 {
|
||||||
|
nodes := make([]any, len(h.Hops))
|
||||||
|
for j, n := range h.Hops {
|
||||||
|
nodes[j] = Dict{
|
||||||
|
{"id", n.ID},
|
||||||
|
{"label", n.Label},
|
||||||
|
{"name", n.Name},
|
||||||
|
{"depth", n.Depth},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
d = append(d, KV{"hops", nodes})
|
||||||
|
}
|
||||||
out[i] = d
|
out[i] = d
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
|
|||||||
@@ -32,9 +32,8 @@ bin/brain/stats.go # index health
|
|||||||
bin/brain/eval.go # recall@5 >= 0.95 gate (Go; Python bin/kb/eval is CI fallback)
|
bin/brain/eval.go # recall@5 >= 0.95 gate (Go; Python bin/kb/eval is CI fallback)
|
||||||
```
|
```
|
||||||
|
|
||||||
`bin/kb/search` is a deprecated wrapper. `--hop` errors (schema has
|
`bin/kb/search` is a deprecated wrapper. `--hop N` walks
|
||||||
`FROM_FILE`; search does not walk it yet, [#17](https://git.produktor.io/eSlider/2dph/issues/17));
|
`FROM_FILE` / `HAS_VERSION` / `AUTHORED` from each hit (1=File, 3=Person).
|
||||||
do not treat it as a graph walk.
|
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user