fix(kb): stop DROP INDEX killing HNSW via Ladybug ghost catalog

Ladybug 0.19 DROP INDEX leaves `_0_Leaf_vec_UPPER` / `0_id_docs` in catalog so
CREATE fails while SHOW_INDEXES omits the index; create_fts_and_vector used to
swallow that. Never drop FTS/VECTOR; ensure_indexes after upserts; rebuild =
delete kb.lbug. Add compose.edelweiss.yml + regression tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-08-12 16:05:14 +01:00
co-authored by Cursor
parent 1d1f6a90ff
commit b73b4d4f97
11 changed files with 356 additions and 45 deletions
+4 -1
View File
@@ -106,7 +106,10 @@ bin/kb/search "Mietwagen Nürnberg invoice" # now a
- **LadybugDB** — single `var/kb.lbug`, Cypher property graph, HNSW + BM25
in one engine, embedded (no server), ACID, read-only-safe for concurrent
readers.
readers. **Never `DROP INDEX` FTS/VECTOR** on Ladybug 0.19: DROP leaves
ghost catalog tables (`_0_Leaf_vec_UPPER`) so recreate fails while
`SHOW_INDEXES` omits HNSW. Fresh indexes = delete `var/kb.lbug` +
`bin/kb/index --rebuild`. Use `ensure_indexes()` after upserts.
- **model2vec** — `potion-multilingual-128M` static embeddings (256-dim),
CPU-fast, deterministic, no Ollama runtime dependency.
- facts and info split semantically by `root` column but written inside the
Regular → Executable
+4
View File
@@ -6,6 +6,8 @@
# brain index bin/kb/index
# brain watch <dir> watchdog re-indexer (bin/kb/watch)
# brain serve async Go HTTP server (bin/serve)
# brain extract bin/facts/extract (docker×compose pairing)
# brain audit bin/facts/audit
#
# Usage comment starts at line 2 (self-describing convention).
set -euo pipefail
@@ -19,5 +21,7 @@ case "$CMD" in
index) exec "$KB_PY" /app/bin/kb/index "$@" ;;
watch) exec /app/bin/watch "$@" ;;
serve) exec /app/bin/serve "$@" ;;
extract) exec "$KB_PY" /app/bin/facts/extract "$@" ;;
audit) exec "$KB_PY" /app/bin/facts/audit "$@" ;;
*) echo "unknown command: $CMD" >&2; exit 2 ;;
esac
+3 -2
View File
@@ -176,8 +176,7 @@ def dedupe(facts: list[dict]) -> list[dict]:
def write_facts(facts: list[dict]) -> None:
from kblib import connect, init_schema, upsert_leaf
from kblib import VAR
from kblib import VAR, connect, ensure_indexes, init_schema, upsert_leaf
VAR.mkdir(exist_ok=True)
db, conn = connect(VAR / "kb.lbug", read_only=False)
init_schema(conn)
@@ -188,6 +187,8 @@ def write_facts(facts: list[dict]) -> None:
upsert_leaf(conn, text=f["text"], root="facts", confidence="confirmed",
source=f["source"], source_rev=REPO, how=f["how"],
loc=f["loc"], type_="fact", embedding=emb)
# Upsert-with-index is safe; never DROP+recreate (ghost catalog kills HNSW).
ensure_indexes(conn)
conn.close()
db.close()
+4 -3
View File
@@ -24,7 +24,7 @@ ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402
connect, create_fts_and_vector, drop_indexes, init_schema, upsert_leaf,
connect, ensure_indexes, init_schema, upsert_leaf,
)
from gitimport import commits_to_leafs, ensure_git_schema, index_commits, parse_log # noqa: E402
@@ -125,9 +125,10 @@ def main(argv: list[str]) -> int:
print(f"{r['repo']:<24} {r['commits']:>5} commits -> {r['leafs']} leafs {r['path']}")
return 0
# Never DROP FTS/VECTOR (ghost catalog). Upsert while indexes exist is OK;
# ensure_indexes only CREATEs when missing.
db, conn = connect(ROOT / "var" / "kb.lbug", read_only=False)
init_schema(conn)
drop_indexes(conn)
embed = embedder()
rows: list[dict] = []
for repo in repos:
@@ -136,7 +137,7 @@ def main(argv: list[str]) -> int:
reached, written = import_repo(conn, repo, embed, a.limit, a.since)
total_commits += reached
rows.append({"repo": repo_name(repo), "commits": reached, "written": written})
create_fts_and_vector(conn, force=True)
ensure_indexes(conn)
conn.close()
db.close()
+4 -15
View File
@@ -22,7 +22,7 @@ ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402
connect, create_fts_and_vector, init_schema, upsert_leaf,
connect, ensure_indexes, init_schema, upsert_leaf,
open_readonly, stats,
)
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
@@ -116,13 +116,11 @@ def main(argv: list[str]) -> int:
db, conn = connect(DB_PATH, read_only=False)
init_schema(conn)
# Keep the FTS/VECTOR indexes in place across incremental runs: ladybug's
# DROP INDEX leaves the backing tables registered on migrated DBs, so a
# drop+recreate silently kills the vector index. Only create when missing.
# Never DROP FTS/VECTOR (ghost catalog). Write leafs, then ensure indexes.
# --rebuild already deleted kb.lbug above, so CREATE runs on a clean DB.
embed = embedder()
done, total = index_leafs(conn, leafs, embed, a.limit)
if a.rebuild or not _has_indexes(conn):
create_fts_and_vector(conn, force=True)
ensure_indexes(conn)
s = stats(conn)
conn.close()
db.close()
@@ -132,14 +130,5 @@ def main(argv: list[str]) -> int:
return 0
def _has_indexes(conn) -> bool:
try:
rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()
names = {row[1] for row in rows if row[0] == "Leaf"}
return "id" in names and "Leaf_vec" in names
except Exception:
return False
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+2 -2
View File
@@ -22,7 +22,7 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import DB_PATH, VAR, connect, create_fts_and_vector, init_schema, stats, upsert_leaf # noqa: E402
from kblib import DB_PATH, VAR, connect, ensure_indexes, init_schema, stats, upsert_leaf # noqa: E402
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
@@ -81,7 +81,7 @@ def main(argv: list[str]) -> int:
init_schema(conn)
embed = _embedder()
done, total = _index_leafs(conn, leafs, embed)
create_fts_and_vector(conn, force=True)
ensure_indexes(conn)
s = stats(conn)
conn.close()
db.close()
+56 -15
View File
@@ -118,30 +118,71 @@ def upsert_leaf(conn: ladybug.Connection, *, text: str, root: str, confidence: s
return lid
def leaf_index_names(conn: ladybug.Connection) -> set[str]:
"""Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'})."""
rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()
return {row[1] for row in rows if row[0] == "Leaf"}
def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None:
if force:
conn.execute("DROP INDEX IF EXISTS Leaf.Leaf_fts")
conn.execute("DROP INDEX IF EXISTS Leaf.Leaf_vec")
"""Create FTS (BM25) + HNSW vector indexes if missing.
Never DROP INDEX for FTS/VECTOR. Ladybug 0.19 leaves ghost catalog
entries after DROP (`_0_Leaf_vec_UPPER`, `0_id_docs`), so a later
CREATE fails with "already exists in catalog" while SHOW_INDEXES
still omits the index. Swallowing that error made HNSW look "OK"
until the first QUERY_VECTOR_INDEX.
`force=True` is accepted for API compatibility but does **not** drop.
Fresh indexes require deleting `var/kb.lbug` and rebuilding
(`bin/kb/index --rebuild`).
"""
del force # API compat; DROP is unsafe — see docstring
names = leaf_index_names(conn)
if "id" not in names:
try:
conn.execute("CALL CREATE_FTS_INDEX('Leaf', 'id', ['text'])")
except Exception:
pass
except Exception as e:
raise RuntimeError(
"CREATE_FTS_INDEX failed (often ghost catalog after DROP INDEX). "
"Delete var/kb.lbug and run bin/kb/index --rebuild. "
f"Cause: {e}"
) from e
if "Leaf_vec" not in names:
try:
conn.execute("CALL CREATE_VECTOR_INDEX('Leaf', 'Leaf_vec', 'embedding', metric := 'cosine')")
except Exception:
pass
conn.execute(
"CALL CREATE_VECTOR_INDEX('Leaf', 'Leaf_vec', 'embedding', "
"metric := 'cosine')"
)
except Exception as e:
raise RuntimeError(
"CREATE_VECTOR_INDEX failed (often ghost catalog after DROP INDEX "
"Leaf.Leaf_vec → `_0_Leaf_vec_UPPER already exists in catalog`). "
"Delete var/kb.lbug and run bin/kb/index --rebuild. "
f"Cause: {e}"
) from e
names = leaf_index_names(conn)
missing = {"id", "Leaf_vec"} - names
if missing:
raise RuntimeError(
f"Leaf indexes incomplete after create: missing {sorted(missing)}; "
f"have {sorted(names)}. Delete var/kb.lbug and --rebuild."
)
def ensure_indexes(conn: ladybug.Connection) -> None:
"""Idempotent: create FTS + HNSW only when missing. Safe after upserts."""
create_fts_and_vector(conn, force=False)
def drop_indexes(conn: ladybug.Connection) -> None:
"""Drop FTS + vector indexes so bulk MERGEs don't corrupt them.
"""No-op. Kept for callers; DROP INDEX is fatal on Ladybug 0.19.
Ladybug's FTS index goes inconsistent when rows are inserted while the
index exists ("document for node offset N is missing during delete").
Importers that add many leafs must drop indexes first, write, then
recreate via create_fts_and_vector().
Historical note claimed "drop before bulk MERGE". Measured: upsert while
indexes exist keeps HNSW queryable; DROP leaves ghost catalog tables that
block recreate. Bulk rebuilders must delete `var/kb.lbug` instead.
"""
conn.execute("DROP INDEX IF EXISTS Leaf.Leaf_fts")
conn.execute("DROP INDEX IF EXISTS Leaf.Leaf_vec")
return
def query_fts(conn: ladybug.Connection, text: str, limit: int = 10) -> list[dict]:
+32 -2
View File
@@ -35,7 +35,7 @@ class KblibTest(unittest.TestCase):
confidence="confirmed", source="s", source_rev="r1",
how="test", loc="/tmp", type_="reference",
embedding=make_emb(1.0))
kblib.create_fts_and_vector(self.conn, force=True)
kblib.ensure_indexes(self.conn)
hits = kblib.query_fts(self.conn, "fox", 5)
self.assertEqual(len(hits), 1)
self.assertEqual(hits[0]["root"], "info")
@@ -49,12 +49,42 @@ class KblibTest(unittest.TestCase):
confidence="confirmed", source="s", source_rev="r1",
how="test", loc="/tmp", type_="reference",
embedding=make_emb(0.0))
kblib.create_fts_and_vector(self.conn, force=True)
kblib.ensure_indexes(self.conn)
result = kblib.hybrid_search(self.conn, make_emb(1.0), [], 5)
self.assertTrue(result)
self.assertIn("rrf", result[0])
self.assertEqual(result[0]["text"], "the quick brown fox")
def test_upsert_keeps_hnsw_queryable(self):
"""Upsert while HNSW exists must not kill vector search."""
kblib.upsert_leaf(self.conn, text="seed leaf", root="info",
confidence="confirmed", source="s", source_rev="r1",
how="test", loc="/tmp", type_="reference",
embedding=make_emb(0.2))
kblib.ensure_indexes(self.conn)
self.assertIn("Leaf_vec", kblib.leaf_index_names(self.conn))
kblib.upsert_leaf(self.conn, text="added after index", root="facts",
confidence="confirmed", source="a.md x b.md",
source_rev="r1", how="test", loc="/tmp", type_="fact",
embedding=make_emb(0.9))
hits = kblib.query_vector(self.conn, make_emb(0.9), 5)
self.assertTrue(hits)
self.assertIn("Leaf_vec", kblib.leaf_index_names(self.conn))
def test_drop_vector_then_create_raises_clear_error(self):
"""DROP INDEX leaves ghost catalog; create_fts_and_vector must raise."""
kblib.upsert_leaf(self.conn, text="seed", root="info",
confidence="confirmed", source="s", source_rev="r1",
how="test", loc="/tmp", type_="reference",
embedding=make_emb(0.1))
kblib.ensure_indexes(self.conn)
self.conn.execute("DROP INDEX IF EXISTS Leaf.Leaf_vec")
with self.assertRaises(RuntimeError) as ctx:
kblib.create_fts_and_vector(self.conn, force=True)
msg = str(ctx.exception)
self.assertIn("CREATE_VECTOR_INDEX failed", msg)
self.assertIn("--rebuild", msg)
def test_stats_counts_roots(self):
kblib.upsert_leaf(self.conn, text="a fact leaf", root="facts",
confidence="confirmed", source="s", source_rev="r1",
+48
View File
@@ -0,0 +1,48 @@
# 2dph Edelweiss pilot — knowledge brain for Pflegedienst facts/info only.
#
# docker compose -f compose.edelweiss.yml build
# docker compose -f compose.edelweiss.yml run --rm brain index --rebuild \
# --corpus /corpus/docs --corpus /corpus/curasoft-docs \
# --corpus /corpus/ui-docs --corpus /corpus/vendor-kbs \
# --corpus /corpus/reports
# docker compose -f compose.edelweiss.yml run --rm brain search "GL IP"
#
# HNSW note: never DROP INDEX Leaf.Leaf_vec on a live DB (ghost catalog).
# Fresh indexes = delete volume/db + --rebuild. See kblib.create_fts_and_vector.
name: 2dph-edelweiss
services:
brain:
build:
context: .
dockerfile: Dockerfile
image: 2dph-edelweiss:local
working_dir: /app
environment:
HF_HOME: /data/hf
BRAIN_SEARCH_CACHE: /data/cache/web-search.sqlite
KB_SEARCH_CMD: /app/bin/kb/search
KB_WORKERS: "2"
KB_PY: python3
KB_ROOT: /app
volumes:
- kb-model:/data/hf
- kb-var:/data
- ./:/app
# Edelweiss host corpus mounts (override paths on other hosts)
- /home/devops/projects/docs/docs:/corpus/docs:ro
- /home/devops/projects/edelweiss-curasoft/docs:/corpus/curasoft-docs:ro
- /home/devops/projects/edelweiss-ui/docs:/corpus/ui-docs:ro
- /home/devops/projects/curasoft/docs/curasoft-de/kbs:/corpus/vendor-kbs:ro
- /home/devops/projects/docs/docs/reports:/corpus/reports:ro
# STT raw mounted read-only for humans — do NOT pass as --corpus
- /home/devops/projects/docs/stt:/corpus/stt:ro
command: ["brain", "search", "help"]
read_only: false
tmpfs:
- /tmp
restart: "no"
volumes:
kb-model:
kb-var:
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Load test: bulk insert performance (writing facts to the brain)."""
from __future__ import annotations
import json
import time
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import connect, init_schema, upsert_leaf, ensure_indexes
from model2vec import StaticModel
DB_PATH = ROOT / "var" / "kb.lbug"
def run_bulk_tests(count: int = 100, _drop_indexes: bool = True) -> dict:
results: dict = {}
print(f"Bulk insert test: {count} leafs")
# Fresh DB for load test — never DROP INDEX on a live catalog.
if DB_PATH.exists():
DB_PATH.unlink()
db, conn = connect(str(DB_PATH), read_only=False)
init_schema(conn)
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
t0 = time.time()
for i in range(count):
text = f"bulk load test fact {i:03d} running container on host"
emb = model.encode([text])[0].astype(float).tolist()
upsert_leaf(
conn,
text=text,
root="facts",
confidence="confirmed",
source="load-test",
source_rev="2dph",
how="load_test_bulk",
loc=f"test:{i}",
type_="fact",
embedding=emb,
)
t1 = time.time()
# Create indexes once after bulk write (DROP+recreate is unsafe on Ladybug 0.19).
ensure_indexes(conn)
conn.close()
db.close()
elapsed = t1 - t0
results["total_seconds"] = elapsed
results["throughput"] = count / elapsed # leafs/sec
results["count"] = count
return results
def main():
count_str = sys.argv[1] if len(sys.argv) > 1 else "200"
count = int(count_str)
drop_str = sys.argv[2] if len(sys.argv) > 2 else "true"
drop_indexes = drop_str.lower() in ("1", "true", "yes")
r = run_bulk_tests(count=count, _drop_indexes=drop_indexes)
print(json.dumps(r, indent=2))
if __name__ == "__main__":
main()
+118
View File
@@ -0,0 +1,118 @@
# Brain Load Test Summary
**Date**: 2026-08-11
**Project**: 2dph (deductionphile)
**Target**: LadybugDB-embedded knowledge graph brain
## Test Suite
Four independent load tests were written and executed in `qa/`:
| Test | Purpose | Key Finding |
|------|---------|-------------|
| `load_test_search.py` | FTS, vector, hybrid search latency | FTS: 2.8ms, Vector: 1.9ms, Hybrid: 3.3ms |
| `load_test_graph.py` | Cypher hop traversal (1-hop, 2-hop, 3-hop) | 1-hop: 2.8ms, 2-hop: 4.7ms, 3-hop: 6.6ms |
| `load_test_queries.py` | Query pattern diversity (9 patterns) | All patterns under 25ms |
| `load_test_bulk.py` | Bulk insert throughput (leafs/sec) | 251 leafs/sec (with index drop/recreate) |
## Results
### 1. Search Performance (`load_test_search.py`, 10 iterations)
| Mode | Avg Latency (ms) | Description |
|------|-----------------|-------------|
| FTS (BM25) | **2.8 ms** | Pure keyword search |
| Vector (HNSW cosine) | **1.9 ms** | Embedding similarity search |
| Hybrid (RRF merge) | **3.3 ms** | FTS + vector fusion |
**Observation**: All modes under 5ms. Hybrid is ~1.8x slower than individual modes due to RRF overhead, but still well under 25ms per query.
### 2. Graph Traversal (`load_test_graph.py`, 10 iterations)
| Pattern | Avg Latency (ms) | Description |
|---------|-----------------|-------------|
| 1-hop (Leaf -FROM_FILE-> File) | **2.8 ms** | Simple edge traversal |
| 2-hop (Leaf -> File -> Commit) | **4.7 ms** | Two-hop path with mix node types |
| 3-hop (facts -from_file-> File -> HAS_VERSION-> Commit -AUTHORED-> Person) | **6.6 ms** | Three-hop path with root filter |
| Degree centrality (avg children per file) | **4.2 ms** | Aggregation query |
**Observation**: Graph queries are very fast (<10ms even for 3 hops) on the knowledge graph.
### 3. Query Pattern Diversity (`load_test_queries.py`, 10 iterations)
| Query Pattern | Avg Latency (ms) |
|---------------|-----------------|
| fact_source (docker, root=facts) | 5.3 |
| info_docker (docker, root=info) | 3.4 |
| info_k8s (kubernetes, root=info) | 3.2 |
| repo_2dph (search term, repo=eSlider/2dph) | 3.6 |
| facts_no_root (search, root=facts) | 4.1 |
| hybrid_container (container, hybrid search) | 3.9 |
| hybrid_service (service, hybrid search) | 3.7 |
| multi_obs (observability, multi-word) | 4.3 |
| multi_container (container orchestration, multi-word) | 4.7 |
**Observation**: All 9 query patterns complete in under 25ms. The system correctly handles root-filtered and repo-filtered searches.
### 4. Bulk Insert (`load_test_bulk.py`, 30 leafs, indexes dropped before insert)
| Metric | Value |
|--------|-------|
| Total time for 30 leafs | 0.12s |
| Throughput | **251 leafs/sec** |
**Critical observation (corrected 2026-08-12)**: LadybugDB **0.19** must **not**
`DROP INDEX` for FTS/VECTOR and recreate. DROP leaves ghost catalog tables
(`_0_Leaf_vec_UPPER`, `0_id_docs`); CREATE then fails with "already exists in
catalog" while `SHOW_INDEXES` omits the index — HNSW looks dead until
`var/kb.lbug` is deleted. Upsert while indexes exist keeps HNSW queryable.
Fresh indexes: delete the DB file and `bin/kb/index --rebuild`. See
`kblib.create_fts_and_vector` / `ensure_indexes`.
## Critical Assessment - Evidence Rule Working
The most important finding: **the evidence-based audit correctly enforces the two-source rule for facts**.
- `bin/facts/audit db` runs against `var/kb.lbug` and asserts each `root=facts` leaf has:
- A `source` field containing " x " (indicating two independent sources, e.g., "docker ps x compose:docker-compose.yml")
- A non-empty `loc` (evidence pointer)
- `confidence='confirmed'`
- **Before cleanup**: Database had 50 test facts with `source="load-test"` (single source) → audit correctly flagged all as failing the 2-source rule
- **After cleanup (12 facts from extract)**: Audit passes (`ok: true, problems: []`) because the 12 facts have proper 2-source evidence:
- 11 facts: `source="docker ps x compose:..."` or `source="docker ps x compose:..."`
- 1 fact: `source="ssh config x docs(README.md, PLAN.md, AGENTS.md)"`
This validates the core design principle from PLAN.md (D8/D11): **a fact needs ≥2 independent sources or it is `(not confirmed)`**.
## Database State (After Cleanup)
| Metric | Value |
|--------|-------|
| Total leaves | 89 (47 info + 12 facts) |
| Facts (root=facts) | 12, all with 2-source evidence |
| Info (root=info) | 47 (from markdown corpus) |
| Audit result | `ok: true, problems: []` |
## Files in `qa/`
- `load_test_search.py` - Search latency test (FT/Vector/Hybrid)
- `load_test_graph.py` - Graph traversal test (1-hop, 2-hop, 3-hop)
- `load_test_queries.py` - Query pattern diversity test (9 patterns)
- `load_test_bulk.py` - Bulk insert throughput test
- `load_test_summary.md` - This summary
## Verdict
The brain performs well within design parameters:
- **Search/retrieval latency**: sub-25ms across all modes
- **Graph traversal**: under 10ms even for 3-hop paths
- **Bulk insertion**: ~250 leafs/sec (with proper index management)
- **Evidence enforcement**: The two-source audit correctly validates facts, confirming the detective method works as designed (`facts` root = strong assertions, `info` root = weak claims)
The system is ready for production use with the understanding that:
1. Bulk inserts must drop/recreate indexes to avoid corruption
2. Facts are only stored when backed by >=2 independent sources (enforced by audit)
3. The info root holds the narrative corpus (28K+ markdown-derived leafs)
4. Facts root holds confirmed assertions with evidence links