Compare commits

..
Author SHA1 Message Date
eSliderandGitHub 15d59054ff docs: delete agent-cost; rename kb-search skill to brain (#14)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
* docs: delete agent-cost; rename kb-search skill to brain.

bin/agents/cost does not exist. CI unittest now fails if a SKILL.md names a
missing bin/ path.

* test: gate SKILL.md bin/ paths; name the brain skill brain.

Follow-up to the agent-cost delete: unittest fails if a skill names a missing
tool. Frontmatter name is brain, not kb-search.
2026-08-13 17:55:41 +01:00
eSliderandGitHub 3f30052ea8 feat: in-process HTTP search; /get /stats /audit /ingest. (#13)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
bin/brain/serve.go (ladybug tags) calls internal/brain instead of exec.
HTTP tests inject a fake API so CI stays cgo-free. ExecSearcher remains
the fallback when the binary is built without system_ladybug.
2026-08-13 17:52:15 +01:00
eSliderandGitHub c1ee920b0a feat: brain/index.go shebang; mail import is not a brain write (D14). (#12)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Commands live at bin/brain/{index,get,stats,eval,watch}.go and
bin/mail/import.go, bin/markdown/import.go, bin/postgres/query.go.
Python remains the Ladybug write worker. index_mail is a deprecation
shim that rebuilds via --with-mail.
2026-08-13 17:46:25 +01:00
eSliderandGitHub cec0161ff6 refactor: chats method shebangs; drop chats index (D14). (#11)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Parsers and commands live in internal/chats. bin/chats/{sync,import,facts,apply}.go
are tagged shebang mains. Brain ingest is not a chats command.
2026-08-13 17:31:03 +01:00
eSliderandGitHub 46310f8773 docs: name bin/brain/search.go; --hop is not a graph walk. (#10)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Published docs and skills still taught bin/kb/search --hop 1. Search lives
at bin/brain/search.go; --hop errors until File edges exist. A unittest
gates the SoT so the lie cannot return.
2026-08-13 17:23:40 +01:00
eSliderandGitHub 7e0f3c9e06 feat: bin/brain/serve.go; search backend is Go not Python (#9)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
* feat(brain): HTTP serve from bin/brain/serve.go, default Go search binary.

Move the HTTP package to internal/httpapi. Default backend is
var/bin/brain-search, not Python. bin/serve.go stays as a deprecation shim.

* feat(httpapi): default search backend is var/bin/brain-search.

bin/brain/serve.go is the command; bin/serve.go stays as a tagged
deprecation shim. Tests fail if the default path still names Python.
2026-08-13 14:32:21 +01:00
eSliderandGitHub f14025304e refactor: one Go module; brain search in bin/brain + internal/brain. (#8)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Collapse nested kbsearch/chats go.mod into the root module. Ranking stays
cgo-free under internal/brain/rank so CI does not need ladybug. bin/kb/search
is a deprecation wrapper that still sets CGO and builds the binary.
2026-08-13 14:26:54 +01:00
eSliderandGitHub dd6d7e9395 docs: point issues at Gitea origin (D15). (#7)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
GitHub stays the public clone for PRs and Actions. Work board is
https://git.produktor.io/eSlider/2dph/issues.
2026-08-13 14:11:00 +01:00
eSliderandGitHub 68d478224f feat(chats): parse LinkedIn MCP v4.22 inbox/conversation blobs. (#6)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
get_inbox/get_conversation return a sections+references envelope, not a
message list. Parser is covered by synthetic Alice/Bob fixtures; CI now
runs the nested bin/chats tests. Session check no longer launches Chromium.
2026-08-13 12:25:51 +01:00
eSliderandGitHub 117f3c2cfd fix(kbsearch): rank FTS correctly, filter before -n, start the daemon. (#5)
Go search took worst BM25 hits (ORDER BY score), cut to -n before --root,
and never called ensureDaemon. Ranking and flag parsing move to a cgo-free
package so CI can fail those regressions without ladybug. --hop errors
instead of being swallowed into the query.
2026-08-13 12:19:26 +01:00
eSliderandGitHub 140d86a4b9 Add Gmail --query to mail/sync (default in:inbox) (#4)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
* Add --query to Gmail mail/sync instead of always listing in:inbox.

Callers keep the search string; default remains in:inbox.

* Document Gmail --query on the mail/sync pipeline.

* test(mail): assert Gmail --query reaches ListIDs, not only the CLI flag.

ParseCLI coverage left a hole: an empty query still has to become in:inbox
and a custom q has to be the string the client lists with.
2026-08-13 12:19:22 +01:00
eSlider c96c393a4a feat(chats): LinkedIn source — MCP client via get_inbox + get_conversation
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
- LinkedInMCPSource: MCP JSON-RPC, как TelegramMCPSource
- sync linkedin --limit N: выгрузка сообщений из LinkedIn
- Проверка сессии: uvx mcp-server-linkedin --status
- Вывод инструкции если сессия истекла
- JSONL в var/chats/linkedin/<thread_id>/messages.jsonl
2026-08-13 00:10:42 +01:00
eSlider 3d0d95cf00 docs: add edelweiss to GitHub safety rules
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
2026-08-13 00:07:01 +01:00
eSlider ed28fdbd2a chore: remove edelweiss references from public repo
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
2026-08-13 00:06:49 +01:00
eSlider 7e511d5b78 docs: GitHub safety rules — no absolute paths, PII, secrets, curasoft 2026-08-13 00:02:09 +01:00
eSlider 0d26519fab fix: resolve plan.md conflict, remove remaining /mnt/ paths 2026-08-13 00:00:48 +01:00
eSlider a429b823e5 chore: clean absolute paths, curasoft refs, secrets from history
- bin/chats/: env-based paths, no /mnt/ /home/ hardcodes
- bin/edelweiss-pilot: remove curasoft, use DOCS_BASE env var
- bin/facts/crm: use KNOWLEDGE_MESH_SEED env var
- compose.edelweiss.yml: remove curasoft volumes, use DOCS_BASE
- docs/chat-import-plan.md: link to Gitea issue, no secrets
- bin/seed-edelweiss-facts.py: removed (curasoft-only)
2026-08-13 00:00:15 +01:00
eSlider 6847233183 bin/chats: Phase 1 MVP — Telegram sync/import/index/facts/apply
- bin/chats/ — nested Go module (как bin/kbsearch/)
  - sync telegram — MCP JSON-RPC клиент, 31 личный чат, 922 сообщения
  - import — конвертация JSONL → MD с YAML frontmatter
  - index — делегирует bin/kb/index --corpus (132 leafs в brain)
  - facts — regex extraction phone/email/linkedin с валидацией
    (исключены: даты, суммы, номера карт, инвойсы)
  - apply — oo CLI cross-check + dry-run
- Source interface для будущих WhatsApp/LinkedIn
- 4 system tests (import, facts, empty, roundtrip) — синтетические данные
- bin/chat — build+exec wrapper
- docs/chat-import-plan.md — прогресс, пути к env (без секретов)

Безопасность: var/ в gitignore, credentials в env, тесты без реальных данных.
2026-08-12 23:59:29 +01:00
eSlider 6d7638ab73 docs: chat import pipeline plan — link to Gitea issue #1 2026-08-12 23:59:20 +01:00
eSliderandCursor bd1a91dab7 fix(kb): seed facts before CREATE indexes (FTS MERGE corruption)
Upsert under live FTS raises "document for node offset N is missing".
Add --skip-indexes; edelweiss-pilot index = write → seed → ensure_indexes.
Ship seed-edelweiss-facts.py (paired lexicon/OO/interview/QEMU facts).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:09:02 +01:00
eSliderandCursor a5a1f91d95 fix(kb): stop DROP INDEX killing HNSW via Ladybug ghost catalog
Ladybug 0.19 DROP INDEX leaves `_0_Leaf_vec_UPPER` / `0_id_docs` in catalog so
CREATE fails while SHOW_INDEXES omits the index; create_fts_and_vector used to
swallow that. Never drop FTS/VECTOR; ensure_indexes after upserts; rebuild =
delete kb.lbug. Add compose.edelweiss.yml + regression tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:05:14 +01:00
eSlider 80e3b7a1cf Remove curasoft references, rename to detective method
- PLAN.md: replace 'curasoft-detective' with 'detective method'
- README.md: replace curasoft-detective link with plain reference
- test_websearch.py: fix test domain from ticket.curasoft.de to example.com
- Rewrote git history with git-filter-repo to remove all traces
2026-08-12 13:46:54 +01:00
eSlider ef4189c72d kbsearch: Go implementation with daemon model serving
- New nested module bin/kbsearch with Go implementation of bin/kb/search
- Embedding model (potion-multilingual-128M) served by localhost daemon
  so repeated CLI calls reuse the loaded model
- Bash launcher bin/kb/search builds binary on first run, caches to var/bin/
- Hybrid FTS + vector search (RRF k=60) matching Python kblib behavior
- YAML output via port of yamlout.py (ordered keys, same format)
- JSON output with proper field order
- All flags: --root, --repo, -n, --json, --list-model
- Root go.mod reverted to 1.25.0 (kbsearch is isolated nested module)
- CI passes: go test ./... and go vet ./... unaffected by kbsearch
2026-08-11 23:57:39 +01:00
eSlider 60c20ed98d feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing
- bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via
  API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not
  MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1)
- bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout
  fast path with docling subprocess fallback for the ~5% textless files
- bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding
  ladybug WAL corruption on bulk-insert into indexed DBs; split from import
- bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate
  leaves stale backing tables killing the vector index)
- docs: README/PLAN/AGENTS cover the mail pipeline

Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.
2026-08-11 21:57:38 +01:00
eSlider 9f22380e82 refactor(tools): bin/{subject}/{method} layout; Go serve+watch modules
Move serve/ (module) -> bin/server, tools/ -> bin/tools, replace bin/kb-watch
bash with bin/watch Go package; self-executing Go shebangs bin/serve.go and
bin/kb/watch.go; Docker + CI + git/import + docs repointed. Multi-stage image
builds static serve+watch binaries (no Go runtime in container).
2026-08-11 09:52:20 +01:00
eSlider e2eff3b9c7 feat(kb): CRM association proof via oo, fix ssh-tunnel self-ref + oo creds
- bin/facts/crm: prove person<->company/company<->project against ooCRM
  x corpus SoT (knowledge-mesh-seed.yaml), write 78 facts (root=facts)
- tools/crmfacts.py + test_crm_facts.py: parser under unit tests (26 pass)
- docs/crm-associations-proof.md: provable graph, mistakes, fixes
- oo merge 759->763 resolves duplicate GoldenRatio.Exchange legal entity
- bin/db/ssh-tunnel: "$0" self-check + accept-new/BatchMode ssh flags
- AGENTS.md: document bin/facts/crm
2026-08-10 23:22:34 +01:00
15 changed files with 418 additions and 135 deletions
+4 -4
View File
@@ -29,7 +29,7 @@ detective method: **a fact needs ≥2 independent sources or it is
| D3 | web search | Vendored client; SearXNG URL is config. Optional Compose instance (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. |
| D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. |
| D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). |
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `internal/brain`); Python remains for index/write until the Go write path is safe. |
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`); Python remains for index/write until the Go write path is safe. |
| D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). |
| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. |
| D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. |
@@ -49,7 +49,7 @@ detective method: **a fact needs ≥2 independent sources or it is
2dph/
PLAN.md / AGENTS.md
docs/ published docs (this conversation → docs/ as md)
skills/ in-project skills (web-search, db-yaml, kb-search, agent-cost, diataxis-docs, …)
skills/ in-project skills (web-search, db-yaml, brain, diataxis-docs)
bin/
facts/extract auto-pair 2 sources → lexicon yaml + graph
facts/audit ["self"|"facts"|"info"|"stale"] 2-source + staleness gate
@@ -57,7 +57,7 @@ detective method: **a fact needs ≥2 independent sources or it is
brain/index.go rebuild FTS + HNSW (incl. --with-mail)
brain/get.go stats.go eval.go watch.go
brain/search.go deduction: facts → info → web-search
brain/serve.go HTTP API (internal/httpapi)
brain/serve.go HTTP API in-process (internal/httpapi + internal/brain)
mail/import.go JSON → markdown (no brain write)
markdown/import.go mistune leaves
postgres/query.go read-only YAML (wraps bin/db/psql-yq)
@@ -140,7 +140,7 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
1. scaffold repo (:done after this file + AGENTS.md + .gitignore + ci)
2. gh repo create eSlider/2dph --private + initial commit + CI
3. vendored skill integration (web-search, db-yaml, kb-search, agent-cost, diataxis-docs) — no remote links
3. vendored skill integration (web-search, db-yaml, brain, diataxis-docs) — no remote links
4. .venv: ladybug + model2vec + mistune
5. schema + tools with TDD (kb + md + facts + brain)
6. ~/.config/brain config
+2 -2
View File
@@ -121,8 +121,8 @@ bin/brain/search.go "invoice from last week" # same s
`bin/{subject}/{method}.go` — self-describing: shebang on line 1, usage comment
from line 2. Shared code in `internal/`. YAML default output, `--json` for
machines. Tests gate every commit. HTTP: `bin/brain/serve.go` (default search
binary `var/bin/brain-search`, not Python).
machines. Tests gate every commit. HTTP: `bin/brain/serve.go` calls
`internal/brain` in-process (`/health` `/search` `/get` `/stats` `/audit` `/ingest`).
## Development
+11 -6
View File
@@ -1,18 +1,20 @@
//usr/bin/env go run -tags=brain_serve "$0" "$@"; exit
//go:build brain_serve
//usr/bin/env go run -tags=brain_serve,system_ladybug "$0" "$@"; exit
//go:build brain_serve && cgo && system_ladybug
//
// bin/brain/serve.go - HTTP API for the 2dph brain.
// bin/brain/serve.go - HTTP API (in-process ladybug search).
//
// KB_ROOT=/path/to/2dph ./bin/brain/serve.go
// KB_SEARCH_CMD=... KB_WORKERS=4 KB_PORT=8630 ./bin/brain/serve.go
// KB_WORKERS=4 KB_PORT=8630 ./bin/brain/serve.go
//
// Default search backend is var/bin/brain-search (Go), not Python.
// Needs CGO + libladybug (same as bin/brain/search.go).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"log"
"os"
"github.com/eSlider/2dph/internal/brain"
"github.com/eSlider/2dph/internal/httpapi"
)
@@ -22,5 +24,8 @@ func main() {
os.Setenv("KB_ROOT", wd)
}
}
httpapi.Run()
if err := brain.Ready(); err != nil {
log.Fatal(err)
}
httpapi.Run(brain.HTTP{})
}
+20
View File
@@ -0,0 +1,20 @@
//go:build brain_serve && !system_ladybug
//
// Fallback serve when ladybug cgo is not in the build (CI / tags=brain_serve).
// Production shebang is serve.go (in-process).
package main
import (
"os"
"github.com/eSlider/2dph/internal/httpapi"
)
func main() {
if os.Getenv("KB_ROOT") == "" {
if wd, err := os.Getwd(); err == nil {
os.Setenv("KB_ROOT", wd)
}
}
httpapi.Run(nil)
}
+1 -1
View File
@@ -18,5 +18,5 @@ func main() {
os.Setenv("KB_ROOT", wd)
}
}
httpapi.Run()
httpapi.Run(nil)
}
+1 -1
View File
@@ -43,7 +43,7 @@ class PublishedDocsTest(unittest.TestCase):
paths = [
ROOT / "README.md",
ROOT / "docs" / "design.md",
ROOT / "skills" / "kb-search" / "SKILL.md",
ROOT / "skills" / "brain" / "SKILL.md",
ROOT / "skills" / "diataxis-docs" / "SKILL.md",
]
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
+33
View File
@@ -0,0 +1,33 @@
"""Every bin/ path named in skills/ must exist on disk."""
from __future__ import annotations
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
BIN_PATH = re.compile(r"\b(bin/[A-Za-z0-9_./-]+)")
class SkillsBinPathsTest(unittest.TestCase):
def test_agent_cost_skill_is_gone(self) -> None:
self.assertFalse(
(ROOT / "skills" / "agent-cost").exists(),
"skills/agent-cost documents bin/agents/cost which does not exist",
)
def test_brain_skill_replaces_kb_search(self) -> None:
self.assertTrue((ROOT / "skills" / "brain" / "SKILL.md").is_file())
self.assertFalse((ROOT / "skills" / "kb-search").exists())
def test_skill_bin_paths_exist(self) -> None:
missing: list[str] = []
for skill in sorted((ROOT / "skills").rglob("SKILL.md")):
text = skill.read_text()
for match in BIN_PATH.findall(text):
rel = match.rstrip("`'.,")
if rel.endswith(".go") or Path(rel).suffix == "" or Path(rel).suffix in {".go", ".py"}:
p = ROOT / rel
if not p.exists():
missing.append(f"{skill.relative_to(ROOT)}: {rel}")
self.assertEqual(missing, [], "SKILL.md names bin/ paths that do not exist")
+2 -1
View File
@@ -8,7 +8,8 @@ Brain/ops/eSlider stack. Facts need proof or they are
- [design](design.md) — schema, deduction model, sources
- [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) — work board (origin)
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`). `--hop` is
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is
not a walk; the flag errors until File/FROM_FILE edges exist.
Published docs live here and mirror the project state.
-33
View File
@@ -1,33 +0,0 @@
# CRM association proof (oo CLI ↔ corpus)
Proven with `oo` (eslider/go-onlyoffice) against the OnlyOffice portal
(`office.produktor.io`). Portal CRM is the SSOT for company ↔ person ↔
project associations; the corpus SoT (`eslider/cv/projects/knowledge-mesh-seed.yaml`)
is the second, independent source. Facts that can be backed by both are
written to the brain under `root=facts` by `bin/facts/crm`.
## What was verified
- Logical counts (portal MySQL): 1300 contacts = 897 persons + 404 companies,
198 projects, 998 deals, 939 project↔contact links.
- Every client company linked to a project has ≥1 person underneath.
- Every person `company_id` resolves to an existing company.
- Corpus org list (9) maps 1:1 onto CRM companies:
ProProdukt SL / produktor.io, Dyvenia, Immowelt AG, WhereGroup,
Keynote SIGOS, D2S/SYSTEMS, GRID, Pack und Cup, Markets Platform.
- 78 person↔company association facts written to the brain
(`how=crm-crosscheck`, `type=association`). Recall@5 in `bin/kb/eval` = 1.0.
## Mistakes found
| # | Mistake | Fix |
|---|---------|-----|
| 1 | Duplicate legal entity `GoldenRatio.Exchange` (contact 759) vs `Golden Ratio Exchange` (763); 3 deals (211, 287, 559) were linked to 759 | `oo contacts merge 759 763` — 763 kept, 759 removed, deal links re-pointed to 763 |
| 2 | `env/`-wide: OnlyOffice creds file used wrong UX (user `eslider`, password with `$2` suffix) making `oo` auth fail | `.env` fixed to `eslider@gmail.com` + clean password; `.env` stays gitignored |
## Gates after fix
- `uv run python -m unittest discover -s bin/tools -t .` → 26 tests OK
- `bin/facts/audit self` + `bin/facts/audit db` → ok
- `bin/kb/eval` → recall@5 = 1.0
- `go test ./...` (bin/server + bin/watch) → ok
+154
View File
@@ -0,0 +1,154 @@
//go:build cgo && system_ladybug
package brain
import (
"bytes"
"context"
"encoding/json"
"fmt"
)
// Ready opens the Ladybug file for the life of the serve process.
func Ready() error {
return openBrain()
}
// HTTP is the in-process API used by bin/brain/serve.go.
type HTTP struct{}
func (HTTP) Search(_ context.Context, query string, limit int) ([]byte, error) {
hits, err := searchHits(query, "", "", limit)
if err != nil {
return nil, err
}
for i := range hits {
if hits[i].Text != "" {
runes := []rune(hits[i].Text)
if len(runes) > 280 {
runes = runes[:280]
}
hits[i].Snippet = string(runes)
}
}
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false)
if err := enc.Encode(toJSONOut(hits, query, "")); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func (HTTP) Get(_ context.Context, id string, body bool) ([]byte, error) {
if conn == nil {
return nil, fmt.Errorf("brain not open")
}
stmt, err := conn.Prepare(
"MATCH (l:Leaf {id:$id}) RETURN l.id, l.text, l.root, l.confidence, l.source, l.type",
)
if err != nil {
return nil, err
}
defer stmt.Close()
res, err := conn.Execute(stmt, map[string]any{"id": id})
if err != nil {
return nil, err
}
if !res.HasNext() {
return nil, fmt.Errorf("no leaf %s", id)
}
row, err := res.Next()
if err != nil {
return nil, err
}
vals, err := row.GetAsSlice()
if err != nil || len(vals) < 6 {
return nil, fmt.Errorf("leaf row")
}
out := map[string]any{
"id": fmt.Sprint(vals[0]),
"root": fmt.Sprint(vals[2]),
"confidence": fmt.Sprint(vals[3]),
"source": fmt.Sprint(vals[4]),
"type": fmt.Sprint(vals[5]),
}
if body {
out["text"] = fmt.Sprint(vals[1])
}
return json.Marshal(out)
}
func (HTTP) Stats(context.Context) ([]byte, error) {
if conn == nil {
return nil, fmt.Errorf("brain not open")
}
res, err := conn.Query("MATCH (l:Leaf) RETURN l.root, count(*)")
if err != nil {
return nil, err
}
byRoot := map[string]int{}
total := 0
for res.HasNext() {
row, err := res.Next()
if err != nil {
return nil, err
}
vals, err := row.GetAsSlice()
if err != nil || len(vals) < 2 {
continue
}
n := int(asInt(vals[1]))
byRoot[fmt.Sprint(vals[0])] = n
total += n
}
return json.Marshal(map[string]any{"total": total, "by_root": byRoot, "db": dbPath()})
}
func (HTTP) Audit(context.Context) ([]byte, error) {
if conn == nil {
return nil, fmt.Errorf("brain not open")
}
res, err := conn.Query("MATCH (l:Leaf) RETURN l.root, l.confidence, count(*)")
if err != nil {
return nil, err
}
var rows []map[string]any
for res.HasNext() {
row, err := res.Next()
if err != nil {
return nil, err
}
vals, err := row.GetAsSlice()
if err != nil || len(vals) < 3 {
continue
}
rows = append(rows, map[string]any{
"root": fmt.Sprint(vals[0]),
"confidence": fmt.Sprint(vals[1]),
"count": asInt(vals[2]),
})
}
return json.Marshal(map[string]any{"status": "ok", "by_confidence": rows})
}
func (HTTP) Ingest(context.Context) ([]byte, error) {
return json.Marshal(map[string]any{
"mode": "rebuild",
"command": "bin/brain/index.go --rebuild",
"add": "v2",
})
}
func asInt(v any) int64 {
switch n := v.(type) {
case int64:
return n
case int:
return int64(n)
case float64:
return int64(n)
default:
return 0
}
}
+19 -15
View File
@@ -51,25 +51,13 @@ func runSearch(args []string) int {
}
defer closeBrain()
emb, err := embedQuery(query)
hits, err := searchHits(query, root, repo, limit)
if err != nil {
fmt.Fprintf(os.Stderr, "embed: %v\n", err)
fmt.Fprintf(os.Stderr, "search: %v\n", err)
return 1
}
fts, err := queryFTS(query, limit*3)
if err != nil {
fmt.Fprintf(os.Stderr, "fts: %v\n", err)
return 1
}
var vec []Hit
if vec, err = queryVector(emb, limit*3); err != nil {
fmt.Fprintf(os.Stderr, "vec: %v\n", err)
}
results := rank.RankAndFilter(fts, vec, root, repo, limit)
results := hits
for i := range results {
if results[i].Text != "" {
runes := []rune(results[i].Text)
@@ -97,6 +85,22 @@ func runSearch(args []string) int {
return 0
}
func searchHits(query, root, repo string, limit int) ([]Hit, error) {
emb, err := embedQuery(query)
if err != nil {
return nil, fmt.Errorf("embed: %w", err)
}
fts, err := queryFTS(query, limit*3)
if err != nil {
return nil, fmt.Errorf("fts: %w", err)
}
var vec []Hit
if vec, err = queryVector(emb, limit*3); err != nil {
fmt.Fprintf(os.Stderr, "vec: %v\n", err)
}
return rank.RankAndFilter(fts, vec, root, repo, limit), nil
}
func b2i(err error) int {
if err != nil {
return 1
+107 -36
View File
@@ -1,10 +1,10 @@
// Package server serves the 2dph brain over HTTP.
// Package httpapi serves the 2dph brain over HTTP.
//
// Async by design: every request runs on its own goroutine, and CPU-heavy
// searches are serialized through a bounded worker pool (a counting
// semaphore) so N requests can't spawn N search processes at once.
// searches are serialized through a bounded worker pool so N requests can't
// spawn N backends at once.
//
// Used by bin/brain/serve.go.
// Used by bin/brain/serve.go. Tests inject a fake API (no exec, no ladybug).
package httpapi
import (
@@ -21,30 +21,45 @@ import (
"time"
)
type Searcher interface {
// API is the in-process brain surface. Production serve.go wires internal/brain.
type API interface {
Search(ctx context.Context, query string, limit int) ([]byte, error)
Get(ctx context.Context, id string, body bool) ([]byte, error)
Stats(ctx context.Context) ([]byte, error)
Audit(ctx context.Context) ([]byte, error)
Ingest(ctx context.Context) ([]byte, error)
}
type Server struct {
searcher Searcher
api API
semaphore chan struct{}
}
const defaultPort = 8630
func NewServer(searcher Searcher, workers int) http.Handler {
var errUnimplemented = errors.New("not implemented")
func NewServer(api API, workers int) http.Handler {
return &Server{
searcher: searcher,
api: api,
semaphore: make(chan struct{}, workers),
}
}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/health":
switch r.URL.Path {
case "/health":
writeJSON(w, http.StatusOK, map[string]any{"status": "ok"})
case r.URL.Path == "/search":
case "/search":
s.handleSearch(w, r)
case "/get":
s.handleGet(w, r)
case "/stats":
s.handleJSON(w, r, s.api.Stats)
case "/audit":
s.handleJSON(w, r, s.api.Audit)
case "/ingest":
s.handleJSON(w, r, s.api.Ingest)
default:
writeJSON(w, http.StatusNotFound, map[string]any{"error": "not found"})
}
@@ -65,19 +80,56 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
}
limit = n
}
// Worker pool: block until a slot frees, so burst concurrency still
// bounds memory (no unbounded python processes).
select {
case s.semaphore <- struct{}{}:
defer func() { <-s.semaphore }()
case <-r.Context().Done():
if !s.acquire(w, r) {
return
}
defer s.release()
body, err := s.api.Search(r.Context(), q, limit)
writeAPI(w, body, err)
}
body, err := s.searcher.Search(r.Context(), q, limit)
func (s *Server) handleGet(w http.ResponseWriter, r *http.Request) {
id := strings.TrimSpace(r.URL.Query().Get("id"))
if id == "" {
writeJSON(w, http.StatusBadRequest, map[string]any{"error": "id required"})
return
}
body := r.URL.Query().Get("body") == "1" || r.URL.Query().Get("body") == "true"
if !s.acquire(w, r) {
return
}
defer s.release()
out, err := s.api.Get(r.Context(), id, body)
writeAPI(w, out, err)
}
func (s *Server) handleJSON(w http.ResponseWriter, r *http.Request, fn func(context.Context) ([]byte, error)) {
if !s.acquire(w, r) {
return
}
defer s.release()
body, err := fn(r.Context())
writeAPI(w, body, err)
}
func (s *Server) acquire(w http.ResponseWriter, r *http.Request) bool {
select {
case s.semaphore <- struct{}{}:
return true
case <-r.Context().Done():
return false
}
}
func (s *Server) release() { <-s.semaphore }
func writeAPI(w http.ResponseWriter, body []byte, err error) {
if err != nil {
writeJSON(w, http.StatusGatewayTimeout, map[string]any{"error": err.Error()})
code := http.StatusBadGateway
if errors.Is(err, errUnimplemented) {
code = http.StatusNotImplemented
}
writeJSON(w, code, map[string]any{"error": err.Error()})
return
}
writeRaw(w, http.StatusOK, body)
@@ -95,17 +147,20 @@ func writeRaw(w http.ResponseWriter, code int, body []byte) {
w.Write(body)
}
// brainSearcher shells out to the Go brain-search binary (not Python).
// A single search is bounded and short-lived; the worker pool keeps at most N live.
type brainSearcher struct {
cmdPath string
timeout time.Duration
// ExecSearcher shells out to var/bin/brain-search. Fallback when the serve
// binary is built without ladybug cgo (CI / tags=brain_serve only).
type ExecSearcher struct {
CmdPath string
Timeout time.Duration
}
func (b *brainSearcher) Search(ctx context.Context, query string, limit int) ([]byte, error) {
ctx, cancel := context.WithTimeout(ctx, b.timeout)
func (b ExecSearcher) Search(ctx context.Context, query string, limit int) ([]byte, error) {
if b.Timeout == 0 {
b.Timeout = 60 * time.Second
}
ctx, cancel := context.WithTimeout(ctx, b.Timeout)
defer cancel()
cmd := exec.CommandContext(ctx, b.cmdPath, "--json", "-n", strconv.Itoa(limit), query)
cmd := exec.CommandContext(ctx, b.CmdPath, "--json", "-n", strconv.Itoa(limit), query)
out, err := cmd.Output()
if err != nil {
var exitErr *exec.ExitError
@@ -117,6 +172,18 @@ func (b *brainSearcher) Search(ctx context.Context, query string, limit int) ([]
return out, nil
}
func (ExecSearcher) Get(context.Context, string, bool) ([]byte, error) {
return nil, errUnimplemented
}
func (ExecSearcher) Stats(context.Context) ([]byte, error) { return nil, errUnimplemented }
func (ExecSearcher) Audit(context.Context) ([]byte, error) { return nil, errUnimplemented }
func (ExecSearcher) Ingest(context.Context) ([]byte, error) {
return json.Marshal(map[string]any{
"mode": "rebuild",
"command": "bin/brain/index.go --rebuild",
})
}
func defaultSearchCmd(root string) string {
if env := os.Getenv("KB_SEARCH_CMD"); env != "" {
return env
@@ -124,11 +191,7 @@ func defaultSearchCmd(root string) string {
return filepath.Join(root, "var", "bin", "brain-search")
}
// Run starts the HTTP server. Reads env: KB_SEARCH_CMD (default
// $KB_ROOT/var/bin/brain-search), KB_WORKERS (default 4), KB_PORT (default 8630).
func Run() {
root := os.Getenv("KB_ROOT")
searchPath := defaultSearchCmd(root)
func workersAndPort() (int, int) {
workers := 4
if raw := os.Getenv("KB_WORKERS"); raw != "" {
if n, err := strconv.Atoi(raw); err == nil && n > 0 {
@@ -141,11 +204,19 @@ func Run() {
port = n
}
}
return workers, port
}
searcher := &brainSearcher{cmdPath: searchPath, timeout: 60 * time.Second}
handler := NewServer(searcher, workers)
// Run starts the HTTP server with an injected API (in-process brain, or ExecSearcher).
func Run(api API) {
if api == nil {
root := os.Getenv("KB_ROOT")
api = ExecSearcher{CmdPath: defaultSearchCmd(root), Timeout: 60 * time.Second}
}
workers, port := workersAndPort()
handler := NewServer(api, workers)
addr := "127.0.0.1:" + strconv.Itoa(port)
log.Printf("serve: %s (workers=%d cmd=%s)", addr, workers, searchPath)
log.Printf("serve: %s (workers=%d)", addr, workers)
if err := http.ListenAndServe(addr, handler); err != nil {
log.Fatal(err)
}
+62
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync"
"sync/atomic"
@@ -47,6 +48,26 @@ func (f *fakeSearcher) Search(ctx context.Context, query string, limit int) ([]b
return []byte(`{"query":"` + query + `","count":0,"results":[]}`), nil
}
func (f *fakeSearcher) Get(_ context.Context, id string, body bool) ([]byte, error) {
out := map[string]any{"id": id, "root": "info"}
if body {
out["text"] = "fake body"
}
return json.Marshal(out)
}
func (f *fakeSearcher) Stats(context.Context) ([]byte, error) {
return []byte(`{"total":0,"by_root":{}}`), nil
}
func (f *fakeSearcher) Audit(context.Context) ([]byte, error) {
return []byte(`{"status":"ok"}`), nil
}
func (f *fakeSearcher) Ingest(context.Context) ([]byte, error) {
return []byte(`{"mode":"rebuild","command":"bin/brain/index.go --rebuild"}`), nil
}
func (f *fakeSearcher) count() int {
f.mu.Lock()
defer f.mu.Unlock()
@@ -142,6 +163,47 @@ func TestSearchRejectsBadLimit(t *testing.T) {
}
}
func TestGetLeaf(t *testing.T) {
fs := &fakeSearcher{callback: func(q string, limit int) ([]byte, error) {
return []byte(`{}`), nil
}}
h := NewServer(fs, 1)
if code, _ := get(t, h, "/get"); code != http.StatusBadRequest {
t.Fatalf("missing id code = %d, want 400", code)
}
code, body := get(t, h, "/get?id=leaf-1&body=1")
if code != http.StatusOK {
t.Fatalf("get code = %d, want 200 body=%s", code, body)
}
if !strings.Contains(string(body), "leaf-1") {
t.Fatalf("get body %s missing id", body)
}
}
func TestStatsAuditIngest(t *testing.T) {
h := NewServer(&fakeSearcher{}, 1)
for _, path := range []string{"/stats", "/audit", "/ingest"} {
code, body := get(t, h, path)
if code != http.StatusOK {
t.Fatalf("%s code = %d, want 200 (%s)", path, code, body)
}
if !json.Valid(body) {
t.Fatalf("%s body not json: %s", path, body)
}
}
}
func TestHTTPPackageDoesNotExecPython(t *testing.T) {
raw, err := os.ReadFile("server.go")
if err != nil {
t.Fatal(err)
}
lower := strings.ToLower(string(raw))
if strings.Contains(lower, "python3") || strings.Contains(lower, "bin/kb/search") {
t.Fatal("httpapi must not exec Python or bin/kb/search")
}
}
func TestDefaultSearchCmdIsBrainNotPython(t *testing.T) {
t.Setenv("KB_SEARCH_CMD", "")
cmd := defaultSearchCmd("/repo")
-34
View File
@@ -1,34 +0,0 @@
---
name: agent-cost
description: >-
Measure what an agent session actually costs in tokens using bin/agents/cost.
Use before and after changing documentation, skills or context layout, and when
a session feels unexpectedly expensive.
---
# agent-cost
```bash
bin/agents/cost # every project, YAML
bin/agents/cost --repo 2dph # only sessions whose cwd matches
bin/agents/cost --json | jq .cursor.by_tool
bin/agents/cost --snapshot after-x --repo 2dph # append a row to docs/CONTEXT-BUDGET.md
```
Reads local session storage from OpenCode and Cursor transcripts. Reports the
always-loaded baseline, cache hit/miss/thrash, and which tools moved the most
bytes.
## How to read it
- **Baseline** is what every single message pays for: `AGENTS.md` plus anything
eagerly linked from it. Keep it small; it multiplies by message count.
- **Cache thrash** matters more than raw size. Editing a file that sits early in
the context invalidates the prompt cache for the whole session.
- **by_tool bytes** shows where the real spend is. Usually it is unfiltered
command output, not documentation.
## Rule
Measure before and after. A claim that something "reduces tokens" without a
before and an after number is an opinion, not a result.
@@ -1,5 +1,5 @@
---
name: kb-search
name: brain
description: >-
Deduction search over the 2dph brain (Ladybug graph: ops corpus, portfolio,
ssh hosts) with bin/brain/search.go instead of reading files or grepping
@@ -8,7 +8,7 @@ description: >-
documentation.
---
# kb-search — deduction over facts and info
# brain — deduction over facts and info
One embedded Ladybug graph (`var/kb.lbug`, read-only when queried) holding two
roots: