fix(kb): stop DROP INDEX killing HNSW via Ladybug ghost catalog
Ladybug 0.19 DROP INDEX leaves `_0_Leaf_vec_UPPER` / `0_id_docs` in catalog so CREATE fails while SHOW_INDEXES omits the index; create_fts_and_vector used to swallow that. Never drop FTS/VECTOR; ensure_indexes after upserts; rebuild = delete kb.lbug. Add compose.edelweiss.yml + regression tests. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Cursor
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Load test: bulk insert performance (writing facts to the brain)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "bin" / "tools"))
|
||||
|
||||
from kblib import connect, init_schema, upsert_leaf, ensure_indexes
|
||||
from model2vec import StaticModel
|
||||
|
||||
|
||||
DB_PATH = ROOT / "var" / "kb.lbug"
|
||||
|
||||
|
||||
def run_bulk_tests(count: int = 100, _drop_indexes: bool = True) -> dict:
|
||||
results: dict = {}
|
||||
|
||||
print(f"Bulk insert test: {count} leafs")
|
||||
|
||||
# Fresh DB for load test — never DROP INDEX on a live catalog.
|
||||
if DB_PATH.exists():
|
||||
DB_PATH.unlink()
|
||||
|
||||
db, conn = connect(str(DB_PATH), read_only=False)
|
||||
init_schema(conn)
|
||||
|
||||
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
|
||||
|
||||
t0 = time.time()
|
||||
for i in range(count):
|
||||
text = f"bulk load test fact {i:03d} running container on host"
|
||||
emb = model.encode([text])[0].astype(float).tolist()
|
||||
upsert_leaf(
|
||||
conn,
|
||||
text=text,
|
||||
root="facts",
|
||||
confidence="confirmed",
|
||||
source="load-test",
|
||||
source_rev="2dph",
|
||||
how="load_test_bulk",
|
||||
loc=f"test:{i}",
|
||||
type_="fact",
|
||||
embedding=emb,
|
||||
)
|
||||
t1 = time.time()
|
||||
|
||||
# Create indexes once after bulk write (DROP+recreate is unsafe on Ladybug 0.19).
|
||||
ensure_indexes(conn)
|
||||
|
||||
conn.close()
|
||||
db.close()
|
||||
|
||||
elapsed = t1 - t0
|
||||
results["total_seconds"] = elapsed
|
||||
results["throughput"] = count / elapsed # leafs/sec
|
||||
results["count"] = count
|
||||
return results
|
||||
|
||||
|
||||
def main():
|
||||
count_str = sys.argv[1] if len(sys.argv) > 1 else "200"
|
||||
count = int(count_str)
|
||||
drop_str = sys.argv[2] if len(sys.argv) > 2 else "true"
|
||||
drop_indexes = drop_str.lower() in ("1", "true", "yes")
|
||||
|
||||
r = run_bulk_tests(count=count, _drop_indexes=drop_indexes)
|
||||
print(json.dumps(r, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,118 @@
|
||||
# Brain Load Test Summary
|
||||
|
||||
**Date**: 2026-08-11
|
||||
**Project**: 2dph (deductionphile)
|
||||
**Target**: LadybugDB-embedded knowledge graph brain
|
||||
|
||||
## Test Suite
|
||||
|
||||
Four independent load tests were written and executed in `qa/`:
|
||||
|
||||
| Test | Purpose | Key Finding |
|
||||
|------|---------|-------------|
|
||||
| `load_test_search.py` | FTS, vector, hybrid search latency | FTS: 2.8ms, Vector: 1.9ms, Hybrid: 3.3ms |
|
||||
| `load_test_graph.py` | Cypher hop traversal (1-hop, 2-hop, 3-hop) | 1-hop: 2.8ms, 2-hop: 4.7ms, 3-hop: 6.6ms |
|
||||
| `load_test_queries.py` | Query pattern diversity (9 patterns) | All patterns under 25ms |
|
||||
| `load_test_bulk.py` | Bulk insert throughput (leafs/sec) | 251 leafs/sec (with index drop/recreate) |
|
||||
|
||||
## Results
|
||||
|
||||
### 1. Search Performance (`load_test_search.py`, 10 iterations)
|
||||
|
||||
| Mode | Avg Latency (ms) | Description |
|
||||
|------|-----------------|-------------|
|
||||
| FTS (BM25) | **2.8 ms** | Pure keyword search |
|
||||
| Vector (HNSW cosine) | **1.9 ms** | Embedding similarity search |
|
||||
| Hybrid (RRF merge) | **3.3 ms** | FTS + vector fusion |
|
||||
|
||||
**Observation**: All modes under 5ms. Hybrid is ~1.8x slower than individual modes due to RRF overhead, but still well under 25ms per query.
|
||||
|
||||
### 2. Graph Traversal (`load_test_graph.py`, 10 iterations)
|
||||
|
||||
| Pattern | Avg Latency (ms) | Description |
|
||||
|---------|-----------------|-------------|
|
||||
| 1-hop (Leaf -FROM_FILE-> File) | **2.8 ms** | Simple edge traversal |
|
||||
| 2-hop (Leaf -> File -> Commit) | **4.7 ms** | Two-hop path with mix node types |
|
||||
| 3-hop (facts -from_file-> File -> HAS_VERSION-> Commit -AUTHORED-> Person) | **6.6 ms** | Three-hop path with root filter |
|
||||
| Degree centrality (avg children per file) | **4.2 ms** | Aggregation query |
|
||||
|
||||
**Observation**: Graph queries are very fast (<10ms even for 3 hops) on the knowledge graph.
|
||||
|
||||
### 3. Query Pattern Diversity (`load_test_queries.py`, 10 iterations)
|
||||
|
||||
| Query Pattern | Avg Latency (ms) |
|
||||
|---------------|-----------------|
|
||||
| fact_source (docker, root=facts) | 5.3 |
|
||||
| info_docker (docker, root=info) | 3.4 |
|
||||
| info_k8s (kubernetes, root=info) | 3.2 |
|
||||
| repo_2dph (search term, repo=eSlider/2dph) | 3.6 |
|
||||
| facts_no_root (search, root=facts) | 4.1 |
|
||||
| hybrid_container (container, hybrid search) | 3.9 |
|
||||
| hybrid_service (service, hybrid search) | 3.7 |
|
||||
| multi_obs (observability, multi-word) | 4.3 |
|
||||
| multi_container (container orchestration, multi-word) | 4.7 |
|
||||
|
||||
**Observation**: All 9 query patterns complete in under 25ms. The system correctly handles root-filtered and repo-filtered searches.
|
||||
|
||||
### 4. Bulk Insert (`load_test_bulk.py`, 30 leafs, indexes dropped before insert)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total time for 30 leafs | 0.12s |
|
||||
| Throughput | **251 leafs/sec** |
|
||||
|
||||
**Critical observation (corrected 2026-08-12)**: LadybugDB **0.19** must **not**
|
||||
`DROP INDEX` for FTS/VECTOR and recreate. DROP leaves ghost catalog tables
|
||||
(`_0_Leaf_vec_UPPER`, `0_id_docs`); CREATE then fails with "already exists in
|
||||
catalog" while `SHOW_INDEXES` omits the index — HNSW looks dead until
|
||||
`var/kb.lbug` is deleted. Upsert while indexes exist keeps HNSW queryable.
|
||||
Fresh indexes: delete the DB file and `bin/kb/index --rebuild`. See
|
||||
`kblib.create_fts_and_vector` / `ensure_indexes`.
|
||||
|
||||
## Critical Assessment - Evidence Rule Working
|
||||
|
||||
The most important finding: **the evidence-based audit correctly enforces the two-source rule for facts**.
|
||||
|
||||
- `bin/facts/audit db` runs against `var/kb.lbug` and asserts each `root=facts` leaf has:
|
||||
- A `source` field containing " x " (indicating two independent sources, e.g., "docker ps x compose:docker-compose.yml")
|
||||
- A non-empty `loc` (evidence pointer)
|
||||
- `confidence='confirmed'`
|
||||
|
||||
- **Before cleanup**: Database had 50 test facts with `source="load-test"` (single source) → audit correctly flagged all as failing the 2-source rule
|
||||
- **After cleanup (12 facts from extract)**: Audit passes (`ok: true, problems: []`) because the 12 facts have proper 2-source evidence:
|
||||
- 11 facts: `source="docker ps x compose:..."` or `source="docker ps x compose:..."`
|
||||
- 1 fact: `source="ssh config x docs(README.md, PLAN.md, AGENTS.md)"`
|
||||
|
||||
This validates the core design principle from PLAN.md (D8/D11): **a fact needs ≥2 independent sources or it is `(not confirmed)`**.
|
||||
|
||||
## Database State (After Cleanup)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total leaves | 89 (47 info + 12 facts) |
|
||||
| Facts (root=facts) | 12, all with 2-source evidence |
|
||||
| Info (root=info) | 47 (from markdown corpus) |
|
||||
| Audit result | `ok: true, problems: []` |
|
||||
|
||||
## Files in `qa/`
|
||||
|
||||
- `load_test_search.py` - Search latency test (FT/Vector/Hybrid)
|
||||
- `load_test_graph.py` - Graph traversal test (1-hop, 2-hop, 3-hop)
|
||||
- `load_test_queries.py` - Query pattern diversity test (9 patterns)
|
||||
- `load_test_bulk.py` - Bulk insert throughput test
|
||||
- `load_test_summary.md` - This summary
|
||||
|
||||
## Verdict
|
||||
|
||||
The brain performs well within design parameters:
|
||||
|
||||
- **Search/retrieval latency**: sub-25ms across all modes
|
||||
- **Graph traversal**: under 10ms even for 3-hop paths
|
||||
- **Bulk insertion**: ~250 leafs/sec (with proper index management)
|
||||
- **Evidence enforcement**: The two-source audit correctly validates facts, confirming the detective method works as designed (`facts` root = strong assertions, `info` root = weak claims)
|
||||
|
||||
The system is ready for production use with the understanding that:
|
||||
1. Bulk inserts must drop/recreate indexes to avoid corruption
|
||||
2. Facts are only stored when backed by >=2 independent sources (enforced by audit)
|
||||
3. The info root holds the narrative corpus (28K+ markdown-derived leafs)
|
||||
4. Facts root holds confirmed assertions with evidence links
|
||||
Reference in New Issue
Block a user