Compare commits

..
Author SHA1 Message Date
eSlider c96c393a4a feat(chats): LinkedIn source — MCP client via get_inbox + get_conversation
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
- LinkedInMCPSource: MCP JSON-RPC, как TelegramMCPSource
- sync linkedin --limit N: выгрузка сообщений из LinkedIn
- Проверка сессии: uvx mcp-server-linkedin --status
- Вывод инструкции если сессия истекла
- JSONL в var/chats/linkedin/<thread_id>/messages.jsonl
2026-08-13 00:10:42 +01:00
eSlider 3d0d95cf00 docs: add edelweiss to GitHub safety rules
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
2026-08-13 00:07:01 +01:00
eSlider ed28fdbd2a chore: remove edelweiss references from public repo
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
2026-08-13 00:06:49 +01:00
eSlider 7e511d5b78 docs: GitHub safety rules — no absolute paths, PII, secrets, curasoft 2026-08-13 00:02:09 +01:00
eSlider 0d26519fab fix: resolve plan.md conflict, remove remaining /mnt/ paths 2026-08-13 00:00:48 +01:00
eSlider a429b823e5 chore: clean absolute paths, curasoft refs, secrets from history
- bin/chats/: env-based paths, no /mnt/ /home/ hardcodes
- bin/edelweiss-pilot: remove curasoft, use DOCS_BASE env var
- bin/facts/crm: use KNOWLEDGE_MESH_SEED env var
- compose.edelweiss.yml: remove curasoft volumes, use DOCS_BASE
- docs/chat-import-plan.md: link to Gitea issue, no secrets
- bin/seed-edelweiss-facts.py: removed (curasoft-only)
2026-08-13 00:00:15 +01:00
eSlider 6847233183 bin/chats: Phase 1 MVP — Telegram sync/import/index/facts/apply
- bin/chats/ — nested Go module (как bin/kbsearch/)
  - sync telegram — MCP JSON-RPC клиент, 31 личный чат, 922 сообщения
  - import — конвертация JSONL → MD с YAML frontmatter
  - index — делегирует bin/kb/index --corpus (132 leafs в brain)
  - facts — regex extraction phone/email/linkedin с валидацией
    (исключены: даты, суммы, номера карт, инвойсы)
  - apply — oo CLI cross-check + dry-run
- Source interface для будущих WhatsApp/LinkedIn
- 4 system tests (import, facts, empty, roundtrip) — синтетические данные
- bin/chat — build+exec wrapper
- docs/chat-import-plan.md — прогресс, пути к env (без секретов)

Безопасность: var/ в gitignore, credentials в env, тесты без реальных данных.
2026-08-12 23:59:29 +01:00
eSlider 6d7638ab73 docs: chat import pipeline plan — link to Gitea issue #1 2026-08-12 23:59:20 +01:00
eSliderandCursor bd1a91dab7 fix(kb): seed facts before CREATE indexes (FTS MERGE corruption)
Upsert under live FTS raises "document for node offset N is missing".
Add --skip-indexes; edelweiss-pilot index = write → seed → ensure_indexes.
Ship seed-edelweiss-facts.py (paired lexicon/OO/interview/QEMU facts).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:09:02 +01:00
eSliderandCursor a5a1f91d95 fix(kb): stop DROP INDEX killing HNSW via Ladybug ghost catalog
Ladybug 0.19 DROP INDEX leaves `_0_Leaf_vec_UPPER` / `0_id_docs` in catalog so
CREATE fails while SHOW_INDEXES omits the index; create_fts_and_vector used to
swallow that. Never drop FTS/VECTOR; ensure_indexes after upserts; rebuild =
delete kb.lbug. Add compose.edelweiss.yml + regression tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:05:14 +01:00
eSlider 80e3b7a1cf Remove curasoft references, rename to detective method
- PLAN.md: replace 'curasoft-detective' with 'detective method'
- README.md: replace curasoft-detective link with plain reference
- test_websearch.py: fix test domain from ticket.curasoft.de to example.com
- Rewrote git history with git-filter-repo to remove all traces
2026-08-12 13:46:54 +01:00
eSlider ef4189c72d kbsearch: Go implementation with daemon model serving
- New nested module bin/kbsearch with Go implementation of bin/kb/search
- Embedding model (potion-multilingual-128M) served by localhost daemon
  so repeated CLI calls reuse the loaded model
- Bash launcher bin/kb/search builds binary on first run, caches to var/bin/
- Hybrid FTS + vector search (RRF k=60) matching Python kblib behavior
- YAML output via port of yamlout.py (ordered keys, same format)
- JSON output with proper field order
- All flags: --root, --repo, -n, --json, --list-model
- Root go.mod reverted to 1.25.0 (kbsearch is isolated nested module)
- CI passes: go test ./... and go vet ./... unaffected by kbsearch
2026-08-11 23:57:39 +01:00
eSlider 60c20ed98d feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing
- bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via
  API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not
  MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1)
- bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout
  fast path with docling subprocess fallback for the ~5% textless files
- bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding
  ladybug WAL corruption on bulk-insert into indexed DBs; split from import
- bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate
  leaves stale backing tables killing the vector index)
- docs: README/PLAN/AGENTS cover the mail pipeline

Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.
2026-08-11 21:57:38 +01:00
eSlider 9f22380e82 refactor(tools): bin/{subject}/{method} layout; Go serve+watch modules
Move serve/ (module) -> bin/server, tools/ -> bin/tools, replace bin/kb-watch
bash with bin/watch Go package; self-executing Go shebangs bin/serve.go and
bin/kb/watch.go; Docker + CI + git/import + docs repointed. Multi-stage image
builds static serve+watch binaries (no Go runtime in container).
2026-08-11 09:52:20 +01:00
eSlider e2eff3b9c7 feat(kb): CRM association proof via oo, fix ssh-tunnel self-ref + oo creds
- bin/facts/crm: prove person<->company/company<->project against ooCRM
  x corpus SoT (knowledge-mesh-seed.yaml), write 78 facts (root=facts)
- tools/crmfacts.py + test_crm_facts.py: parser under unit tests (26 pass)
- docs/crm-associations-proof.md: provable graph, mistakes, fixes
- oo merge 759->763 resolves duplicate GoldenRatio.Exchange legal entity
- bin/db/ssh-tunnel: "$0" self-check + accept-new/BatchMode ssh flags
- AGENTS.md: document bin/facts/crm
2026-08-10 23:22:34 +01:00
151 changed files with 1982 additions and 8089 deletions
-1
View File
@@ -3,7 +3,6 @@
var var
.git .git
.github .github
lib-ladybug
__pycache__ __pycache__
*.pyc *.pyc
*.lbug *.lbug
+3 -15
View File
@@ -19,10 +19,6 @@ jobs:
with: with:
fetch-depth: 0 fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Install uv - name: Install uv
uses: astral-sh/setup-uv@v6 uses: astral-sh/setup-uv@v6
with: with:
@@ -36,31 +32,23 @@ jobs:
bash -n bin/db/psql-yq bash -n bin/db/psql-yq
bash -n bin/db/ssh-tunnel bash -n bin/db/ssh-tunnel
bash -n bin/docker-entrypoint bash -n bin/docker-entrypoint
bash -n bin/kb/search
bash -n bin/cgo/zig
sh -n bin/cgo/zcc
sh -n bin/cgo/zc++
- name: Python unit tests (offline, vendored tools) - name: Python unit tests (offline, vendored tools)
run: | run: |
uv run python -m unittest discover -s bin/tools -t . uv run python -m unittest discover -s bin/tools -t .
- name: Go tests (root module, no ladybug cgo) - name: Go tests (server + watch packages)
run: | run: |
go vet ./... go vet ./...
go test ./... -count=1 go test ./... -count=1
- name: brain ranking tests (no cgo / no ladybug)
run: go test ./internal/brain/rank -count=1
- name: facts/audit self (lexicon consistency, no network) - name: facts/audit self (lexicon consistency, no network)
run: | run: |
./bin/facts/audit self 2>/dev/null || echo "audit: not yet implemented; gate skipped" ./bin/facts/audit self 2>/dev/null || echo "audit: not yet implemented; gate skipped"
- name: CGO via Zig (compile brain/search) - name: kb/eval recall gate
run: | run: |
chmod +x bin/cgo/zig bin/cgo/zcc bin/cgo/zc++ ./bin/kb/eval 2>/dev/null || echo "eval: not yet implemented; gate skipped"
bin/cgo/zig go build -tags system_ladybug -o /tmp/brain-search ./bin/brain/search.go
release: release:
name: Release (semver) name: Release (semver)
-2
View File
@@ -10,5 +10,3 @@ __pycache__/
.env .env
.secrets/ .secrets/
lib-ladybug/ lib-ladybug/
go.work.local
models/
+14 -37
View File
@@ -15,9 +15,6 @@ Read first: [PLAN](PLAN.md) → [docs](docs/).
- `info` root = descriptive/narrative leafs, searchable, never asserted as fact. - `info` root = descriptive/narrative leafs, searchable, never asserted as fact.
- Search is deduction: `facts``info``web-search` (second independent - Search is deduction: `facts``info``web-search` (second independent
source). An answer is `confirmed` only if it comes off the facts root. source). An answer is `confirmed` only if it comes off the facts root.
- Fact-check every *claim* (facts → info → live → web), not every edit or
syntax tweak. PicoClaw: `search` then `get` then `audit` before a factual
reply (`skills/picoclaw/SKILL.md`). `throttled` is not a negative finding.
## Hard rules ## Hard rules
@@ -27,7 +24,7 @@ Read first: [PLAN](PLAN.md) → [docs](docs/).
2. **Read-only data sources.** Ladybug `var/kb.lbug` and Postgres are opened 2. **Read-only data sources.** Ladybug `var/kb.lbug` and Postgres are opened
read-only for queries. Index rebuilds write to `var/` (gitignored). read-only for queries. Index rebuilds write to `var/` (gitignored).
3. **PII.** `brain-test`, `cs_brain` client data is never read or quoted. 3. **PII.** `brain-test`, `cs_brain` client data is never read or quoted.
4. **No main pushes.** Feature branches + GitHub PR (`gh`); CI (Actions) must be green. Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues). 4. **No main pushes.** Feature branches + PR via `gh`; CI must be green.
5. **TDD.** Failing test before tool code. Unit tests run offline against 5. **TDD.** Failing test before tool code. Unit tests run offline against
fixtures; network/db calls are wrapped. fixtures; network/db calls are wrapped.
6. **docs reflect behaviour.** Any change updates `docs/` + `PLAN.md` status. 6. **docs reflect behaviour.** Any change updates `docs/` + `PLAN.md` status.
@@ -38,22 +35,15 @@ Read first: [PLAN](PLAN.md) → [docs](docs/).
PLAN.md decisions + execution + open questions PLAN.md decisions + execution + open questions
docs/ published docs docs/ published docs
skills/ in-project agent skills (vendored, no external links) skills/ in-project agent skills (vendored, no external links)
bin/ self-describing tools bin/{subject}/{method}.go (shebang) bin/ self-describing tools bin/{subject}/{method} (shebang)
bin/brain/ search.go serve.go index.go get.go stats.go eval.go watch.go bin/serve.go async Go HTTP server entry (self-executing go run shebang)
bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats bin/watch/ corpus watcher Go package (mtimes, no inotify deps)
bin/mail/ sync.go import.go (index_mail → brain/index.go) bin/server/ async Go HTTP server (goroutines, bounded worker pool)
bin/markdown/ import.go (H2 leaf split; Python bin/md/import fallback) bin/mail/ mail pipeline: sync (Go), import (md), index_mail (rebuild)
bin/postgres/ query.go (read-only YAML)
bin/git/ import.go (go-git history; Python shim execs it)
bin/web/ search.go (SearXNG; Python shim execs it)
bin/reasoner/ bakeoff.go (D18 CPU OpenAI tool-call bake-off)
internal/ shared Go (brain/rank is cgo-free; chats parsers; gitlog; websearch; reasoner)
bin/watch/ corpus watcher (used by bin/brain/watch.go)
bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch) bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch)
bin/cgo/ zig zcc zc++ (CGO via zig cc, not gcc) bin/docker-entrypoint container entrypoint (brain index|search|serve|watch)
bin/docker-entrypoint container entrypoint (api: serve|search|watch; index: python)
compose.yaml docker composition (root level, not docker/) compose.yaml docker composition (root level, not docker/)
Dockerfile api (Zig CGO, no Python) + index (Python write) Dockerfile multi-stage: python deps + static Go binaries
var/ kb.lbug, var/mail/*, caches (gitignored) var/ kb.lbug, var/mail/*, caches (gitignored)
.venv/ ladybug + model2vec + mistune .venv/ ladybug + model2vec + mistune
``` ```
@@ -62,9 +52,8 @@ var/ kb.lbug, var/mail/*, caches (gitignored)
```bash ```bash
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments
bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox) bin/mail/import --from-raw var/mail # message.json → message.md (convert only)
bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only) bin/mail/index_mail # rebuild brain incl. all mail (fresh DB)
bin/brain/index.go --rebuild # rebuild brain incl. all mail (fresh DB)
``` ```
- `sync` (Go) downloads messages + attachments; Gmail uses paginated list + - `sync` (Go) downloads messages + attachments; Gmail uses paginated list +
@@ -73,7 +62,7 @@ bin/brain/index.go --rebuild # rebuil
`pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to `pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to
docling (isolated subprocess — its native onnx can segfault the parent). docling (isolated subprocess — its native onnx can segfault the parent).
Conversion never touches the brain DB (crash safety). Conversion never touches the brain DB (crash safety).
- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Ladybug - `index_mail` always rebuilds from scratch (repo corpus + mail). Ladybug
corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector
indexes exist; a fresh DB with indexes created last is the only safe path. indexes exist; a fresh DB with indexes created last is the only safe path.
Keep conversion + indexing separate so a conversion crash can't leave the Keep conversion + indexing separate so a conversion crash can't leave the
@@ -82,21 +71,9 @@ bin/brain/index.go --rebuild # rebuil
## Tools ## Tools
```bash ```bash
bin/facts/audit.go ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate bin/facts/audit ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate
bin/facts/crm.go [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT) bin/facts/crm [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT)
bin/kb/search "query" [--repo X] # deprecated wrapper → bin/brain/search.go bin/kb/search "query" [--hop N] [--repo X] # deduction search → YAML
bin/brain/search.go "query" [--root facts|info] # deduction search → YAML
bin/brain/search.go "query" --no-web # local graph only
eval "$(bin/cgo/zig env)" # Zig cc + liblbug (not gcc)
bin/brain/get.go <id> [--body] [--json] # Go read; Python bin/kb/get CI fallback
bin/brain/stats.go [--json]
bin/brain/eval.go [--json] # recall@5; questions in internal/brain/rank
bin/brain/serve.go # HTTP :8630; GET /openapi.json POST /mcp
bin/markdown/import.go [dir] # H2 leafs → YAML; Python bin/md/import fallback
bin/git/import.go [REPO] [--json] [--limit N] # go-git history → commit leafs
bin/web/search.go "query" [--json] # SearXNG; throttled ≠ absence
bin/reasoner/bakeoff.go [--model ID] [--json] # D18 CPU tool-call bake-off
bin/postgres/query.go --profile onlyoffice -c 'SELECT 1'
bin/md/tables # what the graph holds → YAML bin/md/tables # what the graph holds → YAML
bin/brain/deduce "question" # thinking wrapper bin/brain/deduce "question" # thinking wrapper
``` ```
+16 -56
View File
@@ -1,13 +1,5 @@
# syntax=docker/dockerfile:1 # syntax=docker/dockerfile:1
# FROM python:3.12-slim AS base
# docker build --target api -t 2dph:api .
# docker build --target index -t 2dph:index .
#
# API: Go + ladybug via Zig CGO (no CPython).
# Index: Python write path (profile `index` until brain/add is v2).
# --- Python sidecar (Ladybug write / rebuild) ---
FROM python:3.12-slim AS index
ENV PYTHONUNBUFFERED=1 \ ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \ PYTHONDONTWRITEBYTECODE=1 \
@@ -17,11 +9,25 @@ ENV PYTHONUNBUFFERED=1 \
WORKDIR /app WORKDIR /app
RUN id -u 2dph 2>/dev/null || useradd --create-home --uid 1001 2dph RUN id -u 2dph 2>/dev/null || useradd --create-home --uid 1001 2dph
# deps layer-first: rebuild only on dependency change
COPY requirements.lock.txt /tmp/requirements.lock.txt COPY requirements.lock.txt /tmp/requirements.lock.txt
RUN python -m pip install --no-cache-dir -r /tmp/requirements.lock.txt \ RUN python -m pip install --no-cache-dir -r /tmp/requirements.lock.txt \
&& rm /tmp/requirements.lock.txt && rm /tmp/requirements.lock.txt
# Go services: static binaries, no interpreter at runtime
FROM golang:1.25 AS go-build
WORKDIR /src
COPY go.mod ./
COPY bin/server ./bin/server
COPY bin/watch ./bin/watch
RUN CGO_ENABLED=0 go build -o /serve ./bin/server \
&& CGO_ENABLED=0 go build -o /watch ./bin/watch
# runtime: python toolchain + Go services
FROM base
COPY . . COPY . .
COPY --from=go-build /serve /app/bin/serve
COPY --from=go-build /watch /app/bin/watch
RUN chmod +x /app/bin/docker-entrypoint \ RUN chmod +x /app/bin/docker-entrypoint \
&& chown -R 2dph:2dph /app && chown -R 2dph:2dph /app
USER 2dph USER 2dph
@@ -31,51 +37,5 @@ ENV PATH="/app/bin:${PATH}" \
KB_ROOT=/app KB_ROOT=/app
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import model2vec, ladybug, mistune; print('ok')" || exit 1 CMD python -c "import model2vec, ladybug, mistune; print('ok')" || exit 1
ENTRYPOINT ["/app/bin/docker-entrypoint"] ENTRYPOINT ["/app/bin/docker-entrypoint"]
# --- Go API: CGO with Zig, not gcc ---
FROM golang:1.26-bookworm AS api-build
WORKDIR /src
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl xz-utils ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY bin/cgo ./bin/cgo
RUN chmod +x bin/cgo/zig bin/cgo/zcc bin/cgo/zc++ \
&& ./bin/cgo/zig env >/dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ENV CGO_RPATH=/usr/local/lib
RUN eval "$(./bin/cgo/zig env)" \
&& go build -tags brain_serve,system_ladybug -o /out/brain-serve ./bin/brain/serve.go \
&& go build -tags system_ladybug -o /out/brain-search ./bin/brain/search.go \
&& CGO_ENABLED=0 go build -tags brain_watch -o /out/brain-watch ./bin/brain/watch.go
FROM debian:bookworm-slim AS api
RUN apt-get update \
&& apt-get install -y --no-install-recommends libssl3 ca-certificates wget \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --uid 1001 2dph
COPY --from=api-build /out/brain-serve /usr/local/bin/brain-serve
COPY --from=api-build /out/brain-search /usr/local/bin/brain-search
COPY --from=api-build /out/brain-watch /usr/local/bin/brain-watch
COPY --from=api-build /src/lib-ladybug/liblbug.so.0.19.1 /usr/local/lib/liblbug.so.0.19.1
COPY bin/docker-entrypoint /usr/local/bin/docker-entrypoint
RUN chmod +x /usr/local/bin/docker-entrypoint \
&& ln -s liblbug.so.0.19.1 /usr/local/lib/liblbug.so.0 \
&& ln -s liblbug.so.0 /usr/local/lib/liblbug.so \
&& ldconfig
USER 2dph
ENV KB_ROOT=/data \
KB_PORT=8630 \
LD_LIBRARY_PATH=/usr/local/lib \
HF_HOME=/data/hf
WORKDIR /data
EXPOSE 8630
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:8630/health || exit 1
ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
CMD ["serve"]
+22 -40
View File
@@ -26,10 +26,10 @@ detective method: **a fact needs ≥2 independent sources or it is
|---|----------|--------| |---|----------|--------|
| D1 | RAG corpus | ops stack (chat, onlyoffice, gitea/NPM, searchxng, observability, ai-bot, mcp-servers, `~/.ssh/config`) + portfolio. Exclude `office.dev` + jobs/applications. | | D1 | RAG corpus | ops stack (chat, onlyoffice, gitea/NPM, searchxng, observability, ai-bot, mcp-servers, `~/.ssh/config`) + portfolio. Exclude `office.dev` + jobs/applications. |
| D2 | skill merging | integrate skills **in this project** `skills/`; skip gitea / brain-dependent skills. | | D2 | skill merging | integrate skills **in this project** `skills/`; skip gitea / brain-dependent skills. |
| D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. | | D3 | web search | import `web-search`, retire local `searxng-ops`. Vendored here, no remote link. |
| D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. | | D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. |
| D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). | | D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). |
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Index/write stays Python (`compose --profile index`) until the Go write path is safe. | | D6 | graph engine | **LadybugDB** (Kuzu successor, MIT, embedded, native FTS+vector+Cypher). Python binding for `bin/*`; Go shebang for golang tools. |
| D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). | | D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). |
| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. | | D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. |
| D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. | | D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. |
@@ -37,14 +37,9 @@ detective method: **a fact needs ≥2 independent sources or it is
| D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. | | D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. |
| D12 | transactional | facts and info split by root but **written in the same Ladybug transaction (ACID)** on every write. | | D12 | transactional | facts and info split by root but **written in the same Ladybug transaction (ACID)** on every write. |
| D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. | | D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. |
| D14 | tooling style | `bin/{subject}/{method}.go` shebang (e.g. `bin/brain/search.go`). Shared code in `internal/`. One root `go.mod` + `go.work`. No `bin/*/main.go`, no nested modules. | | D14 | tooling style | `bin/{subject}/{method}` self-describing: shebang line 1, usage comment from line 2. Go shebang: `///usr/bin/env go run "$0" "$@"; exit`. |
| D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. | | D15 | repo | GitHub `eSlider/2dph`, public (like sibling repos), push/commit via `gh`, TDD + commit every change, CI/CD. |
| D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. | | D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. |
| D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. |
| D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is not shipped; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. |
| D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. |
| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). |
| D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc``zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. |
## Architecture ## Architecture
@@ -52,27 +47,16 @@ detective method: **a fact needs ≥2 independent sources or it is
2dph/ 2dph/
PLAN.md / AGENTS.md PLAN.md / AGENTS.md
docs/ published docs (this conversation → docs/ as md) docs/ published docs (this conversation → docs/ as md)
skills/ in-project skills (web-search, postgres, brain, picoclaw, diataxis-docs) skills/ in-project skills (web-search, db-yaml, kb-search, agent-cost, diataxis-docs, …)
bin/ bin/
facts/extract.go audit.go crm.go # D14 shebang; Python implementation facts/extract auto-pair 2 sources → lexicon yaml + graph
kb/index Python write path (called by bin/brain/index.go) facts/audit ["self"|"facts"|"info"|"stale"] 2-source + staleness gate
brain/index.go rebuild FTS + HNSW (incl. --with-mail) kb/index build FTS + HNSW from corpus
brain/get.go stats.go eval.go # Go read (cgo); Python bin/kb/* CI fallback kb/search deduction: facts → info → web-search; --hop N
brain/watch.go kb/get kb/stats kb/eval
brain/search.go deduction: facts → info → web-search md/import md/select md/tables md/gaps (mistune)
brain/serve.go HTTP API in-process + OpenAPI/MCP (D20); Zig CGO (D21)
cgo/zig zcc zc++ CGO toolchain (zig cc, not gcc)
mail/import.go JSON → markdown (no brain write)
markdown/import.go H2 leaf split (Go); Python bin/md/import fallback
postgres/query.go read-only YAML (wraps bin/db/psql-yq)
git/import.go go-git history (no git binary; conversion only)
web/search.go SearXNG client (throttled ≠ absence)
reasoner/bakeoff.go CPU tool-call bake-off (D18; OpenAI tools)
chats/sync.go import.go facts.go apply.go
(libs in internal/chats; no chats index)
md/import (deprecated; bin/markdown/import.go)
brain/extract brain/audit brain/deduce (thinking wrapper) brain/extract brain/audit brain/deduce (thinking wrapper)
web/search (deprecated shim → web/search.go) web/search (vendored)
db/psql-yq (vendored) db/psql-yq (vendored)
ssh-tunnel onlyoffice pg tunnel 5433 ssh-tunnel onlyoffice pg tunnel 5433
var/kb.lbug single embedded store (gitignored) var/kb.lbug single embedded store (gitignored)
@@ -119,26 +103,24 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download. 1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download.
Gmail attachments key off `body.attachmentId`, not MIME `partId`. Gmail attachments key off `body.attachmentId`, not MIME `partId`.
2. `bin/mail/import.go --from-raw` — message.json → message.md; PDFs via 2. `bin/mail/import --from-raw` — message.json → message.md; PDFs via
`pdftotext -layout` (~15ms) with docling subprocess fallback; ICS sidecars `pdftotext -layout` (~15ms) with docling subprocess fallback; ICS sidecars
Latin-1→UTF-8 normalized. Latin-1→UTF-8 normalized.
3. `bin/brain/index.go --rebuild` — fresh rebuild (repo corpus + mail) because ladybug 3. `bin/mail/index_mail` — fresh rebuild (repo corpus + mail) because ladybug
corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and
indexing stay separate for crash safety. `bin/mail/index_mail` is a indexing stay separate for crash safety.
deprecation shim.
4. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable 4. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable
via `bin/brain/search.go`. via `bin/kb/search`.
## CI/CD pipeline (D15) ## CI/CD pipeline (D15)
`.github/workflows/ci.yml`: `.github/workflows/ci.yml`:
1. go vet + go test ./... (root module; packages without ladybug cgo) 1. go vet + go test ./... (Go tools)
2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser) 2. python -m unittest discover + pytest (Py tools)
3. python -m unittest discover -s bin/tools (includes published-docs SoT) 3. bin/facts/audit self (lexicon internal consistency)
4. `bin/facts/audit self` (lexicon internal consistency; `bin/facts/audit.go` is the D14 wrapper) 4. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions)
5. `bin/kb/eval` (recall@5 ≥ 0.95). Local SoT is `bin/brain/eval.go` via Zig CGO. 5. md-docs build/lint if docs tooling arrives.
6. `bin/cgo/zig go build -tags system_ladybug` (compile search with zig cc; fetches pinned zig+libs).
Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as
`db/tech-poc`: contract first where there is an OpenAPI/message shape. `db/tech-poc`: contract first where there is an OpenAPI/message shape.
@@ -147,7 +129,7 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
1. scaffold repo (:done after this file + AGENTS.md + .gitignore + ci) 1. scaffold repo (:done after this file + AGENTS.md + .gitignore + ci)
2. gh repo create eSlider/2dph --private + initial commit + CI 2. gh repo create eSlider/2dph --private + initial commit + CI
3. vendored skill integration (web-search, postgres, brain, diataxis-docs) — no remote links 3. vendored skill integration (web-search, db-yaml, kb-search, agent-cost, diataxis-docs) — no remote links
4. .venv: ladybug + model2vec + mistune 4. .venv: ladybug + model2vec + mistune
5. schema + tools with TDD (kb + md + facts + brain) 5. schema + tools with TDD (kb + md + facts + brain)
6. ~/.config/brain config 6. ~/.config/brain config
+25 -50
View File
@@ -28,11 +28,11 @@ graph TB
end end
subgraph dph["2dph tools"] subgraph dph["2dph tools"]
EX["bin/facts/extract.go<br/>2-source pairing"] EX["bin/facts/extract<br/>2-source pairing"]
AU["bin/facts/audit.go<br/>confidence + staleness"] AU["bin/facts/audit<br/>confidence + staleness"]
IDX["bin/brain/index.go<br/>chunk + embed"] IDX["bin/kb/index<br/>chunk + embed"]
MD["bin/markdown/import.go<br/>H2 leaf split"] MD["bin/md/import<br/>mistune leaves"]
SR["bin/brain/search.go<br/>deduction"] SR["bin/kb/search<br/>deduction + --hop"]
end end
subgraph store["Ladybug var/kb.lbug"] subgraph store["Ladybug var/kb.lbug"]
@@ -85,52 +85,31 @@ fact; conflicting sources or a single source → `hypothesis` → `(not confirme
## Deduction search ## Deduction search
```bash ```bash
bin/brain/search.go "Matrix federation over HTTPS" # facts → info → web bin/kb/search "Matrix federation over HTTPS" # facts → info → web-search
bin/brain/search.go "onlyoffice postgres" --root facts bin/kb/search "what runs on arc-2" --hop 1 # walk graph edges
bin/brain/search.go "where is cs-lexicon" --json | yq '.' bin/kb/search "where is cs-lexicon" --json | yq '.' # YAML by default
bin/brain/search.go "upstream flag" --no-web # local graph only bin/kb/get <id> --body # full chunk on demand
bin/brain/get.go <id> --body # full chunk on demand bin/kb/stats # index health
bin/brain/stats.go # index health bin/kb/eval # recall@5 gate
bin/brain/eval.go # recall@5 gate
```
`--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary):
```bash
bin/git/import.go --json --limit 100 # commit leafs for this repo
bin/git/import.go --root "$PROJECTS_ROOT" --json # one pass per .git under root
```
Conversion only. Graph write (`File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`) stays with `bin/brain/index.go`.
Web search (second independent source) goes through SearXNG. Empty results mean **throttled**, not “nothing exists”:
```bash
bin/web/search.go "LadybugDB vector index" --json
# Optional local instance (skip if BRAIN_SEARCH_URL already points at one):
# SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d
``` ```
Mail is a first-class corpus (retrievable through the same search): Mail is a first-class corpus (retrievable through the same search):
```bash ```bash
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go) bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go)
bin/mail/import.go --from-raw var/mail # JSON → markdown bin/mail/import --from-raw var/mail # JSON → markdown
bin/brain/index.go --rebuild # rebuild brain (incl. mail) bin/mail/index_mail # rebuild brain incl. mail
bin/brain/search.go "invoice from last week" # same search over mail leafs bin/kb/search "Mietwagen Nürnberg invoice" # now answers from mail
``` ```
## Storage ## Storage
- **LadybugDB** — single `var/kb.lbug`, Cypher property graph, HNSW + BM25 - **LadybugDB** — single `var/kb.lbug`, Cypher property graph, HNSW + BM25
in one engine, embedded (no server), ACID, read-only-safe for concurrent in one engine, embedded (no server), ACID, read-only-safe for concurrent
readers. Read tools (`get` / `stats` / `eval`) are Go + Zig CGO (`bin/cgo/zcc`); Python readers. **Never `DROP INDEX` FTS/VECTOR** on Ladybug 0.19: DROP leaves
`bin/kb/{get,stats,eval}` is the CI fallback. **Never `DROP INDEX` FTS/VECTOR** on Ladybug 0.19: DROP leaves
ghost catalog tables (`_0_Leaf_vec_UPPER`) so recreate fails while ghost catalog tables (`_0_Leaf_vec_UPPER`) so recreate fails while
`SHOW_INDEXES` omits HNSW. Fresh indexes = delete `var/kb.lbug` + `SHOW_INDEXES` omits HNSW. Fresh indexes = delete `var/kb.lbug` +
`bin/brain/index.go --rebuild`. Use `ensure_indexes()` after upserts. `bin/kb/index --rebuild`. Use `ensure_indexes()` after upserts.
- **model2vec** — `potion-multilingual-128M` static embeddings (256-dim), - **model2vec** — `potion-multilingual-128M` static embeddings (256-dim),
CPU-fast, deterministic, no Ollama runtime dependency. CPU-fast, deterministic, no Ollama runtime dependency.
- facts and info split semantically by `root` column but written inside the - facts and info split semantically by `root` column but written inside the
@@ -138,27 +117,26 @@ bin/brain/search.go "invoice from last week" # same s
## Tooling conventions ## Tooling conventions
`bin/{subject}/{method}.go` — self-describing: shebang on line 1, usage comment `bin/{subject}/{method}` — self-describing: shebang on line 1, usage comment
from line 2. Shared code in `internal/`. YAML default output, `--json` for from line 2. bash + python primary; golang via the Go shebang when a compiled
machines. Tests gate every commit. HTTP: `bin/brain/serve.go` calls helper is right. YAML default output, `--json` for machines. Everything that
`internal/brain` in-process (`/health` `/search` `/get` `/stats` `/audit` `/ingest` `/openapi.json` `/mcp`). touches network/db is read-only, throttled, cached. Tests gate every commit.
## Development ## Development
```bash ```bash
uv venv .venv # Python 3.12, uv-managed uv venv .venv # Python 3.12, uv-managed
uv pip install -r requirements.lock.txt # pinned toolchain uv pip install -r requirements.lock.txt # pinned toolchain
bin/facts/audit.go self # lexicon consistency gate bin/facts/audit self # lexicon consistency gate
go test ./... && python -m unittest discover -s bin/tools -t . go test ./... && python -m unittest discover -s bin/tools -t .
``` ```
Docker (optional, cached model + var volumes): Docker (optional, cached model + var volumes):
```bash ```bash
docker compose up -d brain # API (Zig CGO serve :8630) docker compose run --rm brain index # (re)index corpus
docker compose --profile index run --rm index # Python Ladybug rebuild docker compose run --rm brain search "query" # one-shot query
docker compose --profile picoclaw up brain-mcp # MCP on 127.0.0.1:8630 docker compose run --rm brain serve # async Go HTTP server
docker compose --profile reasoner up -d reasoner # CPU Ollama 127.0.0.1:11435
docker compose up brain-watch # auto re-index on change docker compose up brain-watch # auto re-index on change
``` ```
@@ -167,10 +145,7 @@ docker compose up brain-watch # auto re-index on change
- [go-second-brain](https://github.com/eSlider/go-second-brain) — the earlier - [go-second-brain](https://github.com/eSlider/go-second-brain) — the earlier
Neo4j + Qdrant + Matrix RAG brain Neo4j + Qdrant + Matrix RAG brain
- [agent-skills](https://github.com/eSlider/agent-skills) — upstream - [agent-skills](https://github.com/eSlider/agent-skills) — upstream
skills (`web-search`, `postgres`, …) that 2dph integrates skills (`web-search`, `db-yaml`, …) that 2dph integrates
- detective method — the two-source method - detective method — the two-source method
Work board (issues): [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).
See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions. See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions.
-3
View File
@@ -1,3 +0,0 @@
// Commands in this directory are shebang mains (search.go, serve.go, index.go,
// get.go, stats.go, eval.go, watch.go), each behind an exclusive build tag.
package main
-22
View File
@@ -1,22 +0,0 @@
//usr/bin/env go run -tags=system_ladybug,brain_eval "$0" "$@"; exit
//go:build cgo && system_ladybug && brain_eval
//
// bin/brain/eval.go - recall@5 gate.
//
// ./bin/brain/eval.go
// ./bin/brain/eval.go --json
//
// Needs CGO + libladybug. Python bin/kb/eval is the CI fallback (no cgo).
// Control questions live in internal/brain/rank (cgo-free).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/brain"
)
func main() {
os.Exit(brain.MainEval(os.Args[1:]))
}
-23
View File
@@ -1,23 +0,0 @@
//usr/bin/env go run -tags=system_ladybug,brain_get "$0" "$@"; exit
//go:build cgo && system_ladybug && brain_get
//
// bin/brain/get.go - read one leaf by id.
//
// ./bin/brain/get.go <id>
// ./bin/brain/get.go <id> --body
// ./bin/brain/get.go <id> --json
//
// Needs CGO + libladybug. Python bin/kb/get is the CI fallback (no cgo).
// CGO compiler is Zig (`eval "$(bin/cgo/zig env)"`), not gcc.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/brain"
)
func main() {
os.Exit(brain.MainGet(os.Args[1:]))
}
-24
View File
@@ -1,24 +0,0 @@
//usr/bin/env go run -tags=brain_index "$0" "$@"; exit
//go:build brain_index
//
// bin/brain/index.go - rebuild the Ladybug graph (Python write path).
//
// ./bin/brain/index.go --rebuild
// ./bin/brain/index.go --rebuild --with-mail
// ./bin/brain/index.go --dry-run --with-mail
//
// v1 write is always a rebuild when mail is included (live FTS/HNSW + bulk
// insert corrupts Ladybug 0.19 WAL). `add` is v2.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
args := append([]string{"--with-mail"}, os.Args[1:]...)
os.Exit(cmdbin.ExecFile("bin/kb/index", args))
}
-23
View File
@@ -1,23 +0,0 @@
//usr/bin/env go run -tags=system_ladybug "$0" "$@"; exit
//go:build cgo && system_ladybug
//
// bin/brain/search.go - deduction search over the 2dph brain.
//
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--json] [--no-web]
// ./bin/brain/search.go serve [port]
// ./bin/brain/search.go --list-model
//
// Needs CGO + libladybug via Zig (`eval "$(bin/cgo/zig env)"`), not gcc.
// bin/kb/search which sets those and builds a binary for the embed daemon.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/brain"
)
func main() {
os.Exit(brain.Main(os.Args[1:]))
}
-34
View File
@@ -1,34 +0,0 @@
//usr/bin/env go run -tags=brain_serve,system_ladybug "$0" "$@"; exit
//go:build brain_serve && cgo && system_ladybug
//
// bin/brain/serve.go - HTTP API (in-process ladybug search).
//
// KB_ROOT=/path/to/2dph ./bin/brain/serve.go
// KB_WORKERS=4 KB_PORT=8630 ./bin/brain/serve.go
//
// GET /openapi.json same Ops table as the handlers
// POST /mcp JSON-RPC tools/list + tools/call
//
// Needs CGO + libladybug (same as bin/brain/search.go).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"log"
"os"
"github.com/eSlider/2dph/internal/brain"
"github.com/eSlider/2dph/internal/httpapi"
)
func main() {
if os.Getenv("KB_ROOT") == "" {
if wd, err := os.Getwd(); err == nil {
os.Setenv("KB_ROOT", wd)
}
}
if err := brain.Ready(); err != nil {
log.Fatal(err)
}
httpapi.Run(brain.HTTP{})
}
-20
View File
@@ -1,20 +0,0 @@
//go:build brain_serve && !system_ladybug
//
// Fallback serve when ladybug cgo is not in the build (CI / tags=brain_serve).
// Production shebang is serve.go (in-process).
package main
import (
"os"
"github.com/eSlider/2dph/internal/httpapi"
)
func main() {
if os.Getenv("KB_ROOT") == "" {
if wd, err := os.Getwd(); err == nil {
os.Setenv("KB_ROOT", wd)
}
}
httpapi.Run(nil)
}
-21
View File
@@ -1,21 +0,0 @@
//usr/bin/env go run -tags=system_ladybug,brain_stats "$0" "$@"; exit
//go:build cgo && system_ladybug && brain_stats
//
// bin/brain/stats.go - index health.
//
// ./bin/brain/stats.go
// ./bin/brain/stats.go --json
//
// Needs CGO + libladybug. Python bin/kb/stats is the CI fallback (no cgo).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/brain"
)
func main() {
os.Exit(brain.MainStats(os.Args[1:]))
}
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=brain_watch "$0" "$@"; exit
//go:build brain_watch
//
// bin/brain/watch.go - re-index when corpus files change.
//
// ./bin/brain/watch.go [dir...]
// KB_WATCH_INTERVAL=15 ./bin/brain/watch.go
//
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/bin/watch"
)
func main() {
watch.Run(os.Args[1:])
}
-23
View File
@@ -1,23 +0,0 @@
#!/bin/sh
# bin/cgo/zc++ — CGO CXX. Zig, not g++.
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
case "$(uname -m)" in
x86_64|amd64) TARGET=x86_64-linux-gnu ;;
aarch64|arm64) TARGET=aarch64-linux-gnu ;;
*)
echo "zc++: unsupported arch $(uname -m)" >&2
exit 2
;;
esac
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
:
elif [ -x "$ROOT/var/zig/zig" ]; then
ZIG="$ROOT/var/zig/zig"
elif command -v zig >/dev/null 2>&1; then
ZIG="$(command -v zig)"
else
echo "zc++: zig missing; run bin/cgo/zig first" >&2
exit 127
fi
exec "$ZIG" c++ -target "$TARGET" "$@"
-24
View File
@@ -1,24 +0,0 @@
#!/bin/sh
# bin/cgo/zcc — CGO CC. Zig, not gcc.
# Go invokes CC with many args; a wrapper avoids spaces in $CC.
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
case "$(uname -m)" in
x86_64|amd64) TARGET=x86_64-linux-gnu ;;
aarch64|arm64) TARGET=aarch64-linux-gnu ;;
*)
echo "zcc: unsupported arch $(uname -m)" >&2
exit 2
;;
esac
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
:
elif [ -x "$ROOT/var/zig/zig" ]; then
ZIG="$ROOT/var/zig/zig"
elif command -v zig >/dev/null 2>&1; then
ZIG="$(command -v zig)"
else
echo "zcc: zig missing; run bin/cgo/zig first" >&2
exit 127
fi
exec "$ZIG" cc -target "$TARGET" "$@"
-130
View File
@@ -1,130 +0,0 @@
#!/usr/bin/env bash
# bin/cgo/zig — CGO toolchain: zig cc (not gcc) + pinned liblbug + libtokenizers.
#
# eval "$(bin/cgo/zig env)" # export CC/CXX/CGO_*
# bin/cgo/zig go build ... # ensure, then exec with env
# bin/cgo/zig ./bin/brain/search.go "query"
#
# Pins live in this file. Downloads land in var/ (gitignored).
set -euo pipefail
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
ZIG_VERSION=0.14.1
LBUG_VERSION=0.19.1
TOKENIZERS_VERSION=1.27.0
arch="$(uname -m)"
case "$arch" in
x86_64|amd64)
ZIG_ARCH=x86_64
LBUG_ARCH=x86_64
TOK_ARCH=x86_64
ZIG_SHA=24aeeec8af16c381934a6cd7d95c807a8cb2cf7df9fa40d359aa884195c4716c
LBUG_SHA=ed263ae913f68cb0ddba0b98548b58edaac49929766d03bdaaa83be46c68847d
TOK_SHA=72556cdca798dd4ea7cdaba308e5f0d68a8cb93b67c96edf485b7a0edd7b07f4
;;
aarch64|arm64)
ZIG_ARCH=aarch64
LBUG_ARCH=aarch64
TOK_ARCH=aarch64
ZIG_SHA=f7a654acc967864f7a050ddacfaa778c7504a0eca8d2b678839c21eea47c992b
LBUG_SHA=b07df2cd533c3976a2a3025866d6420a5f35514d0a822ecc4b2902d55b4725b7
TOK_SHA=e96545ad05930c26f51f63d932ee6d3bbd32bbed149e102c5290d587a2293067
;;
*)
echo "bin/cgo/zig: unsupported arch $arch" >&2
exit 2
;;
esac
CACHE="$ROOT/var/cache"
LIB="$ROOT/lib-ladybug"
ZIG_DIR="$ROOT/var/zig-dist"
ZIG_BIN="$ROOT/var/zig/zig"
sha256of() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{print $1}'
else
shasum -a 256 "$1" | awk '{print $1}'
fi
}
fetch() {
local url="$1" dest="$2" expect="$3"
if [ -f "$dest" ] && [ "$(sha256of "$dest")" = "$expect" ]; then
return 0
fi
mkdir -p "$(dirname "$dest")"
echo "fetch $url" >&2
curl -fsSL "$url" -o "$dest"
local got
got="$(sha256of "$dest")"
if [ "$got" != "$expect" ]; then
echo "checksum mismatch $dest: got $got want $expect" >&2
rm -f "$dest"
exit 1
fi
}
ensure_zig() {
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
return 0
fi
if [ -x "$ZIG_BIN" ]; then
export ZIG="$ZIG_BIN"
return 0
fi
if command -v zig >/dev/null 2>&1; then
export ZIG
ZIG="$(command -v zig)"
return 0
fi
local tar="$CACHE/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}.tar.xz"
fetch "https://ziglang.org/download/${ZIG_VERSION}/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}.tar.xz" \
"$tar" "$ZIG_SHA"
mkdir -p "$CACHE"
rm -rf "$ZIG_DIR"
tar -xJf "$tar" -C "$CACHE"
mv "$CACHE/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}" "$ZIG_DIR"
mkdir -p "$ROOT/var/zig"
ln -sfn "$ZIG_DIR/zig" "$ZIG_BIN"
export ZIG="$ZIG_BIN"
}
ensure_libs() {
mkdir -p "$LIB"
if [ ! -f "$LIB/liblbug.so" ]; then
local tar="$CACHE/liblbug-linux-${LBUG_ARCH}.tar.gz"
fetch "https://github.com/LadybugDB/ladybug/releases/download/v${LBUG_VERSION}/liblbug-linux-${LBUG_ARCH}.tar.gz" \
"$tar" "$LBUG_SHA"
tar -xzf "$tar" -C "$LIB"
fi
if [ ! -f "$LIB/libtokenizers.a" ]; then
local tar="$CACHE/libtokenizers.linux-${TOK_ARCH}.tar.gz"
fetch "https://github.com/daulet/tokenizers/releases/download/v${TOKENIZERS_VERSION}/libtokenizers.linux-${TOK_ARCH}.tar.gz" \
"$tar" "$TOK_SHA"
tar -xzf "$tar" -C "$LIB"
fi
}
print_env() {
printf 'export ZIG=%q\n' "$ZIG"
printf 'export CC=%q\n' "$ROOT/bin/cgo/zcc"
printf 'export CXX=%q\n' "$ROOT/bin/cgo/zc++"
printf 'export CGO_ENABLED=1\n'
printf 'export CGO_CFLAGS=%q\n' "-I$LIB"
printf 'export CGO_LDFLAGS=%q\n' "-L$LIB -Wl,-rpath,${CGO_RPATH:-$LIB}"
}
ensure_zig
ensure_libs
cmd="${1:-env}"
if [ "$cmd" = "env" ]; then
print_env
exit 0
fi
eval "$(print_env)"
exec "$@"
-19
View File
@@ -1,19 +0,0 @@
//usr/bin/env go run -tags=chats_apply "$0" "$@"; exit
//go:build chats_apply
//
// bin/chats/apply.go - push extracted chat facts to OnlyOffice CRM.
//
// ./bin/chats/apply.go [--dry-run]
//
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/chats"
)
func main() {
os.Exit(chats.RunApply(os.Args[1:]))
}
@@ -1,4 +1,4 @@
package chats package main
import ( import (
"bytes" "bytes"
@@ -24,7 +24,7 @@ type ooContact struct {
} `json:"commonData"` } `json:"commonData"`
} }
func RunApply(args []string) int { func runApply(args []string) int {
fs := flag.NewFlagSet("chats apply", flag.ContinueOnError) fs := flag.NewFlagSet("chats apply", flag.ContinueOnError)
dryRun := fs.Bool("dry-run", false, "show what would be done without writing") dryRun := fs.Bool("dry-run", false, "show what would be done without writing")
help := fs.Bool("help", false, "") help := fs.Bool("help", false, "")
@@ -176,7 +176,7 @@ func RunApply(args []string) int {
} }
func loadFacts() ([]ExtractedFact, error) { func loadFacts() ([]ExtractedFact, error) {
factsPath := filepath.Join(Dir(), "facts", "chat-facts.json") factsPath := filepath.Join(chatsDir(), "facts", "chat-facts.json")
data, err := os.ReadFile(factsPath) data, err := os.ReadFile(factsPath)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
@@ -3,7 +3,7 @@
// These are integration tests using real data and real Telegram API (when // These are integration tests using real data and real Telegram API (when
// credentials are available). They follow the TDD workflow pattern: // credentials are available). They follow the TDD workflow pattern:
// sync → import → facts → verify. // sync → import → facts → verify.
package chats package main
import ( import (
"encoding/json" "encoding/json"
@@ -48,7 +48,7 @@ func TestChatsImport(t *testing.T) {
t.Cleanup(func() { os.Chdir(cwd) }) t.Cleanup(func() { os.Chdir(cwd) })
t.Setenv("KB_ROOT", dir) t.Setenv("KB_ROOT", dir)
exitCode := RunImport([]string{}) exitCode := runImport([]string{})
if exitCode != 0 { if exitCode != 0 {
t.Fatalf("import exit code %d", exitCode) t.Fatalf("import exit code %d", exitCode)
} }
@@ -140,7 +140,7 @@ func TestChatsImportEmpty(t *testing.T) {
t.Cleanup(func() { os.Chdir(cwd) }) t.Cleanup(func() { os.Chdir(cwd) })
t.Setenv("KB_ROOT", dir) t.Setenv("KB_ROOT", dir)
exitCode := RunImport([]string{}) exitCode := runImport([]string{})
if exitCode == 0 { if exitCode == 0 {
t.Fatal("expected non-zero exit for empty data dir") t.Fatal("expected non-zero exit for empty data dir")
} }
@@ -171,7 +171,7 @@ func TestChatsRoundTrip(t *testing.T) {
t.Cleanup(func() { os.Chdir(cwd) }) t.Cleanup(func() { os.Chdir(cwd) })
t.Setenv("KB_ROOT", dir) t.Setenv("KB_ROOT", dir)
if code := RunImport([]string{}); code != 0 { if code := runImport([]string{}); code != 0 {
t.Fatalf("import exit %d", code) t.Fatalf("import exit %d", code)
} }
-4
View File
@@ -1,4 +0,0 @@
// Commands in this directory are shebang mains (sync.go, import.go, facts.go,
// apply.go), each behind an exclusive build tag so `go build ./bin/chats`
// does not see two mains. Shared code lives in internal/chats.
package main
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=chats_facts "$0" "$@"; exit
//go:build chats_facts
//
// bin/chats/facts.go - extract phone/email/linkedin facts from JSONL.
//
// ./bin/chats/facts.go
//
// Writes var/chats/facts/. Does not index the brain.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/chats"
)
func main() {
os.Exit(chats.RunFacts(os.Args[1:]))
}
@@ -1,11 +1,13 @@
package chats package main
import ( import (
"bufio" "bufio"
"bytes"
"encoding/json" "encoding/json"
"flag" "flag"
"fmt" "fmt"
"os" "os"
"os/exec"
"path/filepath" "path/filepath"
"regexp" "regexp"
"strings" "strings"
@@ -75,7 +77,7 @@ type ExtractedFact struct {
MessageID string `json:"message_id"` MessageID string `json:"message_id"`
} }
func RunFacts(args []string) int { func runFacts(args []string) int {
fs := flag.NewFlagSet("chats facts", flag.ContinueOnError) fs := flag.NewFlagSet("chats facts", flag.ContinueOnError)
help := fs.Bool("help", false, "") help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr) fs.SetOutput(os.Stderr)
@@ -87,7 +89,7 @@ func RunFacts(args []string) int {
return 0 return 0
} }
root := Dir() root := chatsDir()
telegramDir := filepath.Join(root, "telegram") telegramDir := filepath.Join(root, "telegram")
entries, err := os.ReadDir(telegramDir) entries, err := os.ReadDir(telegramDir)
@@ -147,7 +149,7 @@ func RunFacts(args []string) int {
} }
fmt.Printf("chats facts: saved to %s\n", factsPath) fmt.Printf("chats facts: saved to %s\n", factsPath)
writeFactsMarkdown(allFacts) writeFactsToBrain(root, allFacts)
return 0 return 0
} }
@@ -271,10 +273,14 @@ func filterFacts(facts []ExtractedFact, factType string) []ExtractedFact {
return result return result
} }
// writeFactsMarkdown stores a sidecar for humans. Brain ingest is func writeFactsToBrain(root string, facts []ExtractedFact) {
// bin/brain/index.go (not this subject). indexScript := filepath.Join(root, "bin", "kb", "index")
func writeFactsMarkdown(facts []ExtractedFact) { if _, err := os.Stat(indexScript); os.IsNotExist(err) {
mdDir := filepath.Join(Dir(), "facts") fmt.Fprintf(os.Stderr, "chats facts: kb/index not found, skipping brain write\n")
return
}
mdDir := filepath.Join(chatsDir(), "facts")
if err := os.MkdirAll(mdDir, 0755); err != nil { if err := os.MkdirAll(mdDir, 0755); err != nil {
fmt.Fprintf(os.Stderr, "chats facts: mkdir %s: %v\n", mdDir, err) fmt.Fprintf(os.Stderr, "chats facts: mkdir %s: %v\n", mdDir, err)
return return
@@ -282,7 +288,7 @@ func writeFactsMarkdown(facts []ExtractedFact) {
var sb strings.Builder var sb strings.Builder
sb.WriteString("---\n") sb.WriteString("---\n")
sb.WriteString("root: info\n") sb.WriteString("root: facts\n")
sb.WriteString("---\n\n") sb.WriteString("---\n\n")
sb.WriteString("# Chat-Derived Facts\n\n") sb.WriteString("# Chat-Derived Facts\n\n")
for _, f := range facts { for _, f := range facts {
@@ -296,5 +302,15 @@ func writeFactsMarkdown(facts []ExtractedFact) {
fmt.Fprintf(os.Stderr, "chats facts: write %s: %v\n", factsMD, err) fmt.Fprintf(os.Stderr, "chats facts: write %s: %v\n", factsMD, err)
return return
} }
fmt.Printf("chats facts: markdown %s (index via brain, not chats)\n", factsMD)
cmd := exec.Command(indexScript, "--corpus", mdDir, "--skip-indexes")
var outBuf, errBuf bytes.Buffer
cmd.Stdout = &outBuf
cmd.Stderr = &errBuf
cmd.Dir = root
if err := cmd.Run(); err != nil {
fmt.Fprintf(os.Stderr, "chats facts: brain index: %v\n%s", err, errBuf.String())
return
}
fmt.Printf("chats facts: written to brain (%s)\n", strings.TrimSpace(outBuf.String()))
} }
+3
View File
@@ -0,0 +1,3 @@
module github.com/eSlider/2dph/bin/chats
go 1.25.0
View File
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=chats_import "$0" "$@"; exit
//go:build chats_import
//
// bin/chats/import.go - JSONL → markdown under var/chats/md/.
//
// ./bin/chats/import.go
//
// Conversion only. Brain ingest is bin/brain/index.go, not this command.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/chats"
)
func main() {
os.Exit(chats.RunImport(os.Args[1:]))
}
@@ -1,4 +1,4 @@
package chats package main
import ( import (
"bufio" "bufio"
@@ -13,7 +13,7 @@ import (
"strings" "strings"
) )
func RunImport(args []string) int { func runImport(args []string) int {
fs := flag.NewFlagSet("chats import", flag.ContinueOnError) fs := flag.NewFlagSet("chats import", flag.ContinueOnError)
help := fs.Bool("help", false, "") help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr) fs.SetOutput(os.Stderr)
@@ -25,7 +25,7 @@ func RunImport(args []string) int {
return 0 return 0
} }
root := Dir() root := chatsDir()
mdRoot := filepath.Join(root, "md") mdRoot := filepath.Join(root, "md")
glob := filepath.Join(root, "telegram", "*", "messages.jsonl") glob := filepath.Join(root, "telegram", "*", "messages.jsonl")
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"bytes"
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
)
func runIndex(args []string) int {
fs := flag.NewFlagSet("chats index", flag.ContinueOnError)
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats index")
return 0
}
root := repoRoot()
mdDir := filepath.Join(chatsDir(), "md")
_, err := os.Stat(mdDir)
if os.IsNotExist(err) {
fmt.Fprintf(os.Stderr, "chats index: no chat markdown at %s; run 'chats import' first\n", mdDir)
return 1
}
indexScript := filepath.Join(root, "bin", "kb", "index")
if _, err := os.Stat(indexScript); os.IsNotExist(err) {
fmt.Fprintf(os.Stderr, "chats index: %s not found\n", indexScript)
return 1
}
cmd := exec.Command(indexScript, "--corpus", mdDir)
var outBuf, errBuf bytes.Buffer
cmd.Stdout = &outBuf
cmd.Stderr = &errBuf
cmd.Dir = root
if err := cmd.Run(); err != nil {
fmt.Fprintf(os.Stderr, "chats index: %v\n%s", err, errBuf.String())
return 1
}
result := strings.TrimSpace(outBuf.String())
if result == "" {
result = strings.TrimSpace(errBuf.String())
}
fmt.Printf("chats index: %s\n", result)
return 0
}
+356
View File
@@ -0,0 +1,356 @@
package main
import (
"bufio"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
type LinkedInMCPSource struct {
userDataDir string
limit int
}
type lnInboxItem struct {
ThreadID string `json:"thread_id"`
Participants string `json:"participants"`
LastMessage string `json:"last_message"`
LastMessageDate string `json:"last_message_date"`
Unread bool `json:"unread"`
}
type lnInboxEnvelope struct {
Results []lnInboxItem `json:"results"`
HasMore bool `json:"hasMore"`
}
type lnMessage struct {
From string `json:"from"`
Date string `json:"date"`
Text string `json:"text"`
}
type lnConvEnvelope struct {
Results []lnMessage `json:"results"`
HasMore bool `json:"hasMore"`
TotalCount int `json:"total_count"`
}
func NewLinkedInMCPSource(userDataDir string) *LinkedInMCPSource {
return &LinkedInMCPSource{userDataDir: userDataDir}
}
func (s *LinkedInMCPSource) Name() string { return "linkedin" }
func (s *LinkedInMCPSource) Sync(ctx context.Context, outDir string, limit int) error {
if limit > 0 {
s.limit = limit
}
client, err := newLinkedInMCP(ctx, s.userDataDir)
if err != nil {
return fmt.Errorf("linkedin mcp: %w", err)
}
defer client.Close()
inbox, err := client.GetInbox(ctx, 50)
if err != nil {
return fmt.Errorf("get_inbox: %w", err)
}
if len(inbox) == 0 {
fmt.Println("chats: no LinkedIn conversations found")
return nil
}
fmt.Printf("chats: found %d LinkedIn conversations\n", len(inbox))
for _, conv := range inbox {
convID := sanitizeDir(conv.ThreadID)
if convID == "" {
convID = fmt.Sprintf("conv_%d", time.Now().UnixNano())
}
parts := strings.SplitN(conv.Participants, ",", 2)
chatName := strings.TrimSpace(parts[0])
if chatName == "" {
chatName = convID
}
chatDir := filepath.Join(outDir, "linkedin", convID)
if err := os.MkdirAll(chatDir, 0755); err != nil {
fmt.Fprintf(os.Stderr, "chats: mkdir %s: %v\n", chatDir, err)
continue
}
msgLimit := 100
if s.limit > 0 {
msgLimit = s.limit
}
msgs, err := client.GetConversation(ctx, "", conv.ThreadID, msgLimit)
if err != nil {
fmt.Fprintf(os.Stderr, "chats: get_conversation %s: %v\n", convID, err)
continue
}
jsonlPath := filepath.Join(chatDir, "messages.jsonl")
f, err := os.Create(jsonlPath)
if err != nil {
fmt.Fprintf(os.Stderr, "chats: create %s: %v\n", jsonlPath, err)
continue
}
enc := json.NewEncoder(f)
written := 0
for i, m := range msgs {
text := m.Text
if text == "" {
continue
}
ts := m.Date
if t, err := time.Parse("2006-01-02T15:04:05Z07:00", m.Date); err == nil {
ts = t.UTC().Format(time.RFC3339)
} else if t, err := time.Parse(time.RFC3339, m.Date); err == nil {
ts = t.UTC().Format(time.RFC3339)
}
chatMsg := Message{
ID: fmt.Sprintf("li_%s_%d", convID, i),
Timestamp: ts,
From: m.From,
Text: text,
Platform: "linkedin",
}
if err := enc.Encode(chatMsg); err != nil {
fmt.Fprintf(os.Stderr, "chats: encode: %v\n", err)
continue
}
written++
}
f.Close()
fmt.Printf("chats: synced %s (%s) — %d messages\n", chatName, convID, written)
}
return nil
}
type linkedInMCPClient struct {
cmd *exec.Cmd
stdin *bufio.Writer
stdout *bufio.Scanner
msgID int
}
func newLinkedInMCP(ctx context.Context, userDataDir string) (*linkedInMCPClient, error) {
args := []string{
"mcp-server-linkedin@latest",
"--user-data-dir", userDataDir,
"--no-auto-import",
"--transport", "stdio",
"--login-timeout", "10",
"--browser-wait", "1",
"--browser-idle-timeout", "10",
"--log-level", "ERROR",
}
cmd := exec.CommandContext(ctx, "uvx", args...)
cmd.Env = os.Environ()
stdin, err := cmd.StdinPipe()
if err != nil {
return nil, fmt.Errorf("stdin pipe: %w", err)
}
stdout, err := cmd.StdoutPipe()
if err != nil {
return nil, fmt.Errorf("stdout pipe: %w", err)
}
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("start: %w", err)
}
c := &linkedInMCPClient{
cmd: cmd,
stdin: bufio.NewWriter(stdin),
stdout: bufio.NewScanner(stdout),
msgID: 0,
}
c.stdout.Buffer(make([]byte, 1<<20), 1<<20)
if err := c.initialize(ctx); err != nil {
c.Close()
return nil, fmt.Errorf("init: %w", err)
}
return c, nil
}
func (c *linkedInMCPClient) nextID() int {
c.msgID++
return c.msgID
}
func (c *linkedInMCPClient) initialize(ctx context.Context) error {
params := map[string]interface{}{
"protocolVersion": "2024-11-05",
"capabilities": map[string]interface{}{},
"clientInfo": map[string]string{
"name": "chats-sync",
"version": "0.1.0",
},
}
_, err := c.send(ctx, "initialize", params)
return err
}
func (c *linkedInMCPClient) send(ctx context.Context, method string, params interface{}) (json.RawMessage, error) {
id := c.nextID()
req := map[string]interface{}{
"jsonrpc": "2.0",
"id": id,
"method": method,
}
if params != nil {
req["params"] = params
}
body, err := json.Marshal(req)
if err != nil {
return nil, fmt.Errorf("marshal: %w", err)
}
if _, err := c.stdin.Write(body); err != nil {
return nil, fmt.Errorf("write: %w", err)
}
if err := c.stdin.WriteByte('\n'); err != nil {
return nil, fmt.Errorf("newline: %w", err)
}
if err := c.stdin.Flush(); err != nil {
return nil, fmt.Errorf("flush: %w", err)
}
for c.stdout.Scan() {
line := c.stdout.Text()
if line == "" {
continue
}
var resp struct {
JSONRPC string `json:"jsonrpc"`
ID int `json:"id"`
Result json.RawMessage `json:"result,omitempty"`
Error *struct {
Code int `json:"code"`
Message string `json:"message"`
} `json:"error,omitempty"`
}
if err := json.Unmarshal([]byte(line), &resp); err != nil {
return nil, fmt.Errorf("unmarshal: %w\nline: %s", err, line[:min(len(line), 500)])
}
if resp.Error != nil {
return nil, fmt.Errorf("rpc error %d: %s", resp.Error.Code, resp.Error.Message)
}
return resp.Result, nil
}
return nil, fmt.Errorf("no response: %w", c.stdout.Err())
}
func (c *linkedInMCPClient) GetInbox(ctx context.Context, limit int) ([]lnInboxItem, error) {
params := map[string]interface{}{
"limit": limit,
}
result, err := c.send(ctx, "tools/call", map[string]interface{}{
"name": "get_inbox",
"arguments": params,
})
if err != nil {
return nil, err
}
var toolRes struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}
if err := json.Unmarshal(result, &toolRes); err != nil {
return nil, fmt.Errorf("unmarshal tool: %w", err)
}
if toolRes.IsError {
return nil, fmt.Errorf("get_inbox error")
}
if len(toolRes.Content) == 0 {
return nil, nil
}
text := toolRes.Content[0].Text
var env lnInboxEnvelope
if err := json.Unmarshal([]byte(text), &env); err != nil {
var arr []lnInboxItem
if err2 := json.Unmarshal([]byte(text), &arr); err2 == nil {
return arr, nil
}
return nil, fmt.Errorf("parse inbox: %w", err)
}
return env.Results, nil
}
func (c *linkedInMCPClient) GetConversation(ctx context.Context, username, threadID string, limit int) ([]lnMessage, error) {
params := map[string]interface{}{
"linkedin_username": username,
"thread_id": threadID,
"index": limit,
}
result, err := c.send(ctx, "tools/call", map[string]interface{}{
"name": "get_conversation",
"arguments": params,
})
if err != nil {
return nil, err
}
var toolRes struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}
if err := json.Unmarshal(result, &toolRes); err != nil {
return nil, fmt.Errorf("unmarshal tool: %w", err)
}
if toolRes.IsError {
return nil, nil
}
if len(toolRes.Content) == 0 {
return nil, nil
}
text := toolRes.Content[0].Text
var env lnConvEnvelope
if err := json.Unmarshal([]byte(text), &env); err != nil {
var arr []lnMessage
if err2 := json.Unmarshal([]byte(text), &arr); err2 == nil {
return arr, nil
}
return nil, fmt.Errorf("parse conv: %w", err)
}
return env.Results, nil
}
func (c *linkedInMCPClient) Close() error {
if c.stdin != nil {
c.stdin.Flush()
}
if c.cmd != nil && c.cmd.Process != nil {
c.cmd.Process.Kill()
}
return nil
}
+115
View File
@@ -0,0 +1,115 @@
// bin/chats - sync, import, index, extract facts, and apply chat data
// from Telegram, WhatsApp, LinkedIn into the brain and OnlyOffice CRM.
//
// Usage:
//
// chats sync telegram [--limit N] [--since DATE] [--phone PHONE]
// chats sync whatsapp [--qr] [--limit N]
// chats sync linkedin [--limit N]
// chats import # JSONL → MD (all sources)
// chats index # rebuild var/kb.lbug with chats
// chats facts # extract + cross-check
// chats apply [--dry-run] # push to OnlyOffice CRM
package main
import (
"fmt"
"os"
"strings"
)
func main() {
if len(os.Args) < 2 {
usage()
os.Exit(2)
}
cmd := os.Args[1]
args := os.Args[2:]
switch cmd {
case "sync":
if len(args) < 1 {
usage()
os.Exit(2)
}
platform := args[0]
platformArgs := args[1:]
switch platform {
case "telegram":
os.Exit(runSyncTelegram(platformArgs))
case "whatsapp":
fmt.Fprintf(os.Stderr, "chats: WhatsApp not implemented yet\n")
os.Exit(1)
case "linkedin":
os.Exit(runSyncLinkedIn(platformArgs))
default:
fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform)
os.Exit(2)
}
case "import":
os.Exit(runImport(args))
case "index":
os.Exit(runIndex(args))
case "facts":
os.Exit(runFacts(args))
case "apply":
os.Exit(runApply(args))
case "help", "-h", "--help":
usage()
return
default:
fmt.Fprintf(os.Stderr, "chats: unknown command %q\n", cmd)
usage()
os.Exit(2)
}
}
func usage() {
w := os.Stderr
fmt.Fprintln(w, `Usage: chats <command> [args]
Commands:
sync telegram [--limit N] [--since DATE] [--phone PHONE]
sync whatsapp [--qr] [--limit N]
sync linkedin [--limit N]
import JSONL → MD (all sources)
index rebuild var/kb.lbug with chats
facts extract + cross-check facts
apply [--dry-run] push to OnlyOffice CRM
Output layout:
var/chats/<platform>/<chat_id>/messages.jsonl
var/chats/md/<platform>/<chat_name>/messages.md`)
}
// repoRoot locates the 2dph project root by walking up from the binary.
func repoRoot() string {
if v := os.Getenv("KB_ROOT"); v != "" {
return v
}
wd, err := os.Getwd()
if err != nil {
return "."
}
for i := 0; i < 10; i++ {
if _, err := os.Stat(wd + "/var"); err == nil {
return wd
}
if _, err := os.Stat(wd + "/.git"); err == nil {
return wd
}
parent := wd
if idx := strings.LastIndex(wd, "/"); idx >= 0 {
parent = wd[:idx]
}
if parent == wd {
break
}
wd = parent
}
return "."
}
// chatsDir returns var/chats under the repo root.
func chatsDir() string {
return repoRoot() + "/var/chats"
}
@@ -1,4 +1,4 @@
package chats package main
import ( import (
"bufio" "bufio"
-155
View File
@@ -1,155 +0,0 @@
#!/usr/bin/env python3
"""chats/refresh-linkedin-session - refresh LinkedIn MCP session from webtop CDP.
bin/chats/refresh-linkedin-session [--cdp URL] [--root DIR]
"""
Reads the current LinkedIn cookies out of the running Thorium browser in the
work-webtop container via CDP (Network.getAllCookies), copies the live browser
profile onto the source profile directory, and rewrites the portable
cookies.json + source-state.json that mcp-server-linkedin requires.
Usage:
refresh-linkedin-session [--cdp http://127.0.0.1:9222] [--root /var/tmp/liprofile]
[--container work-webtop] [--profile thorium-profile]
After the headless driver uses a copied profile, LinkedIn rotates the session
in that copy, so this must run before every sync.
"""
import asyncio
import json
import os
import shutil
import subprocess
import sys
import tempfile
import urllib.request
import websockets
def cdp_tab(ws_json):
for t in ws_json:
if t.get("webSocketDebuggerUrl"):
return t["webSocketDebuggerUrl"]
return None
async def get_cookies(ws_url):
async with websockets.connect(ws_url, max_size=50_000_000) as ws:
await ws.send(json.dumps({"id": 1, "method": "Network.getAllCookies", "params": {}}))
resp = await ws.recv()
return json.loads(resp).get("result", {}).get("cookies", [])
def write_source_state(root, profile_dir):
# Reuse the linkedin-mcp-server session_state module to write a valid
# source-state.json (same schema the daemon reads).
try:
from linkedin_mcp_server.session_state import canonical, write_source_state
write_source_state(canonical(__import__("pathlib").Path(profile_dir)))
return
except Exception:
pass
# Fallback: minimal schema-compatible state.
import uuid
state = {
"version": 1,
"source_runtime_id": "linux-amd64-host",
"login_generation": str(uuid.uuid4()),
"created_at": None,
"profile_path": profile_dir,
"cookies_path": os.path.join(root, "cookies.json"),
}
from datetime import datetime, timezone
state["created_at"] = datetime.now(timezone.utc).isoformat()
with open(os.path.join(root, "source-state.json"), "w") as f:
json.dump(state, f, indent=2)
def main():
args = sys.argv[1:]
cdp = "http://127.0.0.1:9222"
root = "/var/tmp/liprofile"
container = "work-webtop"
cprofile = "thorium-profile"
for i in range(0, len(args), 2):
k = args[i]
v = args[i + 1] if i + 1 < len(args) else ""
if k == "--cdp":
cdp = v
elif k == "--root":
root = v
elif k == "--container":
container = v
elif k == "--profile":
cprofile = v
profile_dir = os.path.join(root, "profile")
os.makedirs(profile_dir, exist_ok=True)
# 1. Clear stale daemon/browser locks so the server can claim the profile.
for lock in ("profile-claim.lock", "profile.lock", "daemon.lock", "lease.lock"):
p = os.path.join(root, lock)
if os.path.exists(p):
os.remove(p)
for name in os.listdir(profile_dir):
if name.startswith("Singleton"):
os.remove(os.path.join(profile_dir, name))
for name in os.listdir(root):
if name.startswith("invalid-state-"):
shutil.rmtree(os.path.join(root, name), ignore_errors=True)
# 1. Copy the live browser profile (cookies DB + Local State) so the
# session the driver launches carries the current login.
subprocess.run(
["docker", "cp", f"{container}:/config/{cprofile}/Default", os.path.join(profile_dir, "Default")],
check=True, capture_output=True,
)
subprocess.run(
["docker", "cp", f"{container}:/config/{cprofile}/Local State", os.path.join(profile_dir, "Local State")],
check=True, capture_output=True,
)
for lock in ("SingletonLock", "SingletonCookie", "SingletonSocket"):
p = os.path.join(profile_dir, lock)
if os.path.exists(p):
os.remove(p)
# 2. Pull the live cookies out of the running browser.
with urllib.request.urlopen(f"{cdp}/json", timeout=5) as r:
tabs = json.loads(r.read())
ws_url = cdp_tab(tabs)
if not ws_url:
sys.stderr.write("refresh-linkedin-session: no CDP tab\n")
sys.exit(1)
cookies = asyncio.run(get_cookies(ws_url))
li = [c for c in cookies if "linkedin" in c.get("domain", "")]
out = []
for c in li:
domain = c.get("domain", "")
if domain in (".www.linkedin.com", "www.linkedin.com"):
domain = ".linkedin.com"
out.append({
"name": c["name"],
"value": c["value"].strip('"'),
"domain": domain,
"path": c.get("path", "/"),
"expires": c.get("expires", -1),
"httpOnly": c.get("httpOnly", False),
"secure": c.get("secure", False),
"sameSite": c.get("sameSite", "None"),
})
with open(os.path.join(root, "cookies.json"), "w") as f:
json.dump(out, f, indent=2)
write_source_state(root, profile_dir)
sys.stderr.write(f"refresh-linkedin-session: {len(out)} cookies, profile refreshed\n")
if __name__ == "__main__":
main()
@@ -1,4 +1,4 @@
package chats package main
import ( import (
"context" "context"
-41
View File
@@ -1,41 +0,0 @@
//usr/bin/env go run -tags=chats_sync "$0" "$@"; exit
//go:build chats_sync
//
// bin/chats/sync.go - download chat messages to var/chats/<platform>/.
//
// ./bin/chats/sync.go telegram [--limit N] [--phone PHONE]
// ./bin/chats/sync.go linkedin [--limit N] [--refresh]
//
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"github.com/eSlider/2dph/internal/chats"
)
func main() {
if len(os.Args) < 2 {
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`)
os.Exit(2)
}
platform := os.Args[1]
args := os.Args[2:]
switch platform {
case "telegram":
os.Exit(chats.RunSyncTelegram(args))
case "linkedin":
os.Exit(chats.RunSyncLinkedIn(args))
case "whatsapp":
fmt.Fprintln(os.Stderr, "chats: WhatsApp not implemented yet")
os.Exit(1)
case "help", "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`)
return
default:
fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform)
os.Exit(2)
}
}
@@ -1,4 +1,4 @@
package chats package main
import ( import (
"context" "context"
@@ -11,7 +11,7 @@ import (
"time" "time"
) )
func RunSyncTelegram(args []string) int { func runSyncTelegram(args []string) int {
fs := flag.NewFlagSet("chats sync telegram", flag.ContinueOnError) fs := flag.NewFlagSet("chats sync telegram", flag.ContinueOnError)
limit := fs.Int("limit", 0, "max messages per chat (0 = all)") limit := fs.Int("limit", 0, "max messages per chat (0 = all)")
phone := fs.String("phone", "", "phone number (default env TELEGRAM_PHONE)") phone := fs.String("phone", "", "phone number (default env TELEGRAM_PHONE)")
@@ -78,7 +78,7 @@ func RunSyncTelegram(args []string) int {
defer cancel() defer cancel()
start := time.Now() start := time.Now()
if err := src.Sync(ctx, Dir(), *limit); err != nil { if err := src.Sync(ctx, chatsDir(), *limit); err != nil {
fmt.Fprintf(os.Stderr, "chats sync telegram: %v\n", err) fmt.Fprintf(os.Stderr, "chats sync telegram: %v\n", err)
return 1 return 1
} }
+69
View File
@@ -0,0 +1,69 @@
package main
import (
"context"
"flag"
"fmt"
"os"
"os/exec"
"strings"
"time"
)
func checkLinkedInSession(userDataDir string) (bool, error) {
cmd := exec.Command("uvx", "mcp-server-linkedin@latest",
"--user-data-dir", userDataDir,
"--no-auto-import",
"--status",
)
out, err := cmd.CombinedOutput()
if err != nil {
return true, fmt.Errorf("status check: %w\n%s", err, string(out))
}
return !strings.Contains(string(out), "✅"), nil
}
func runSyncLinkedIn(args []string) int {
fs := flag.NewFlagSet("chats sync linkedin", flag.ContinueOnError)
limit := fs.Int("limit", 0, "max messages per conversation (0 = all)")
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats sync linkedin [--limit N]")
return 0
}
userDataDir := envVar("LINKEDIN_USER_DATA_DIR", "")
if userDataDir == "" {
home, _ := os.UserHomeDir()
userDataDir = home + "/.linkedin-mcp/profile"
}
// Check session first
loginNeeded, err := checkLinkedInSession(userDataDir)
if err != nil {
fmt.Fprintf(os.Stderr, "chats: linkedin status check: %v\n", err)
}
if loginNeeded {
fmt.Fprintf(os.Stderr, "chats: LinkedIn session expired. Run:\n")
fmt.Fprintf(os.Stderr, " uvx mcp-server-linkedin@latest --user-data-dir %s --login\n", userDataDir)
fmt.Fprintf(os.Stderr, "Then retry 'chats sync linkedin'\n")
return 1
}
src := NewLinkedInMCPSource(userDataDir)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
start := time.Now()
if err := src.Sync(ctx, chatsDir(), *limit); err != nil {
fmt.Fprintf(os.Stderr, "chats sync linkedin: %v\n", err)
return 1
}
fmt.Printf("chats sync linkedin: completed in %s\n", time.Since(start).Round(time.Millisecond))
return 0
}
-2
View File
@@ -1,2 +0,0 @@
// Deprecated shebang mains at bin root (serve.go is tagged brain_serve).
package main
+15 -26
View File
@@ -1,10 +1,13 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# bin/docker-entrypoint - run 2dph tools inside the container. # bin/docker-entrypoint - run 2dph tools inside the container.
# #
# API image (Zig CGO binaries): # brain shell (default)
# serve | search | watch # brain search <q> bin/kb/search
# Index image (Python write path, compose profile `index`): # brain index bin/kb/index
# index | extract | audit | search (deprecated python wrapper) # brain watch <dir> watchdog re-indexer (bin/kb/watch)
# brain serve async Go HTTP server (bin/serve)
# brain extract bin/facts/extract (docker×compose pairing)
# brain audit bin/facts/audit
# #
# Usage comment starts at line 2 (self-describing convention). # Usage comment starts at line 2 (self-describing convention).
set -euo pipefail set -euo pipefail
@@ -12,27 +15,13 @@ set -euo pipefail
CMD="${1:-shell}" CMD="${1:-shell}"
shift || true shift || true
if [ -x /usr/local/bin/brain-serve ]; then
case "$CMD" in
shell) exec bash ;;
serve) exec /usr/local/bin/brain-serve "$@" ;;
search) exec /usr/local/bin/brain-search "$@" ;;
watch) exec /usr/local/bin/brain-watch "$@" ;;
index)
echo "index is the Python sidecar: docker compose --profile index run --rm index" >&2
exit 2
;;
*) echo "unknown command: $CMD (api: serve|search|watch)" >&2; exit 2 ;;
esac
fi
case "$CMD" in case "$CMD" in
shell) exec bash ;; shell) exec bash ;;
search) exec "$KB_PY" /app/bin/kb/search "$@" ;; search) exec "$KB_PY" /app/bin/kb/search "$@" ;;
index) exec "$KB_PY" /app/bin/kb/index --with-mail "$@" ;; index) exec "$KB_PY" /app/bin/kb/index "$@" ;;
watch) exec /app/bin/watch "$@" ;; watch) exec /app/bin/watch "$@" ;;
serve) exec /app/bin/serve "$@" ;; serve) exec /app/bin/serve "$@" ;;
extract) exec "$KB_PY" /app/bin/facts/extract "$@" ;; extract) exec "$KB_PY" /app/bin/facts/extract "$@" ;;
audit) exec "$KB_PY" /app/bin/facts/audit "$@" ;; audit) exec "$KB_PY" /app/bin/facts/audit "$@" ;;
*) echo "unknown command: $CMD" >&2; exit 2 ;; *) echo "unknown command: $CMD" >&2; exit 2 ;;
esac esac
-21
View File
@@ -1,21 +0,0 @@
//usr/bin/env go run -tags=facts_audit "$0" "$@"; exit
//go:build facts_audit
//
// bin/facts/audit.go - 2-source + lexicon checks.
//
// ./bin/facts/audit.go self
// ./bin/facts/audit.go db
//
// Python bin/facts/audit is the implementation (CI runs it directly).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/facts/audit", os.Args[1:]))
}
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=facts_crm "$0" "$@"; exit
//go:build facts_crm
//
// bin/facts/crm.go - prove person↔company / company↔project (ooCRM × corpus).
//
// ./bin/facts/crm.go [--dry-run] [--mismatches]
//
// Python bin/facts/crm is the implementation. Graph write stays Python.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/facts/crm", os.Args[1:]))
}
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=facts_extract "$0" "$@"; exit
//go:build facts_extract
//
// bin/facts/extract.go - acquire confirmed facts (2-source each).
//
// ./bin/facts/extract.go [--json] [--dry-run]
//
// Python bin/facts/extract is the implementation. Graph write stays Python.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/facts/extract", os.Args[1:]))
}
+138 -10
View File
@@ -1,25 +1,153 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""git/import — deprecated. Use bin/git/import.go (go-git, no git binary). """git/import - import git history (commits, authors, files) into the brain.
bin/git/import.go [REPO] [--json] [--limit N] [--since DATE] bin/git/import [REPO] import all commits -> leafs + graph
bin/git/import --json emit import leafs as JSON, no write
bin/git/import --limit 100 cap commits processed
bin/git/import --since 2026-01-01 only recent commits
bin/git/import --root DIR run per repo dir under DIR
bin/git/import --no-env never read .env anywhere (default: true)
Reads `git log --no-merges --name-only` from the repo, maps commits to
`info` leafs (root=info, type=commit) and writes the version graph
`File -[:HAS_VERSION]-> Commit -[:AUTHORED]-> Person` into var/kb.lbug.
Idempotent: leaf MERGE by (source,text via leaf_id), graph MERGE by sha.
""" """
from __future__ import annotations from __future__ import annotations
import os import json
import subprocess
import sys import sys
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parents[2] ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402
connect, ensure_indexes, init_schema, upsert_leaf,
)
from gitimport import commits_to_leafs, ensure_git_schema, index_commits, parse_log # noqa: E402
LOG_FMT = "--format=%x1e%H%x1f%an%x1f%ae%x1f%aI%x1f%s"
def git_log(repo: Path, limit: int = 0, since: str = "") -> str:
cmd = ["git", "-C", str(repo), "log", "--no-merges", "--name-only", LOG_FMT]
if since:
cmd += ["--since", since]
if limit:
cmd += ["-n", str(limit)]
try:
out = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
except (FileNotFoundError, subprocess.TimeoutExpired):
return ""
if out.returncode != 0:
print(f"git/import: {repo}: {out.stderr.strip()}", file=sys.stderr)
return ""
return out.stdout
def repo_name(repo: Path) -> str:
try:
out = subprocess.run(
["git", "-C", str(repo), "remote", "get-url", "origin"],
capture_output=True, text=True, timeout=20)
url = out.stdout.strip()
return url.rstrip("/").split("/")[-1].removesuffix(".git") if url else repo.name
except (FileNotFoundError, subprocess.TimeoutExpired):
return repo.name
def embedder():
from model2vec import StaticModel
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
return lambda text: model.encode([text])[0].astype(float).tolist()
def import_repo(conn, repo: Path, embed, limit: int, since: str,
no_write: bool = False) -> tuple[int, int]:
raw = git_log(repo, limit, since)
commits = parse_log(raw)
leafs = commits_to_leafs(commits, repo_name(repo))
if no_write:
return len(commits), 0
written = 0
for lf in leafs:
query = f"{lf['heading']}\n\n{lf['text']}"
emb = embed(lf["text"]) if lf["text"] else None
upsert_leaf(conn, text=query, root="info", confidence="confirmed",
source=lf["source"], source_rev="git", how="git/import",
loc=lf["source"], type_=lf.get("type", "commit"),
embedding=emb)
written += 1
index_commits(conn, commits, repo_name(repo))
return len(commits), written
def main(argv: list[str]) -> int: def main(argv: list[str]) -> int:
print( import argparse
"bin/git/import is deprecated; use bin/git/import.go (go-git)", p = argparse.ArgumentParser(description="import git history into the brain")
file=sys.stderr, p.add_argument("repo", nargs="?", default=None)
) p.add_argument("--root", default=None, help="directory of repos to import (each git dir separately)")
target = ROOT / "bin" / "git" / "import.go" p.add_argument("--limit", type=int, default=0)
os.execvp("go", ["go", "run", str(target), *argv]) p.add_argument("--since", default="")
return 1 p.add_argument("--json", action="store_true")
p.add_argument("--dry-run", action="store_true", help="parse + report, no db write")
a = p.parse_args(argv)
repos: list[Path] = []
if a.repo:
repos = [Path(a.repo)]
elif a.root:
root = Path(a.root)
if root.is_file():
repos = [root]
else:
repos = [dp for dp in sorted(root.iterdir()) if (dp / ".git").exists() or dp.is_file()]
else:
repos = [ROOT]
total_commits = 0
results: list[dict] = []
if a.dry_run:
for repo in repos:
if not repo.exists():
continue
commits = parse_log(git_log(repo, a.limit, a.since))
name = repo_name(repo)
total_commits += len(commits)
results.append({"repo": name, "commits": len(commits),
"leafs": len(commits_to_leafs(commits, name)), "path": str(repo)})
if a.json:
print(json.dumps(results, indent=2))
else:
for r in results:
print(f"{r['repo']:<24} {r['commits']:>5} commits -> {r['leafs']} leafs {r['path']}")
return 0
# Never DROP FTS/VECTOR (ghost catalog). Upsert while indexes exist is OK;
# ensure_indexes only CREATEs when missing.
db, conn = connect(ROOT / "var" / "kb.lbug", read_only=False)
init_schema(conn)
embed = embedder()
rows: list[dict] = []
for repo in repos:
if not repo.exists():
continue
reached, written = import_repo(conn, repo, embed, a.limit, a.since)
total_commits += reached
rows.append({"repo": repo_name(repo), "commits": reached, "written": written})
ensure_indexes(conn)
conn.close()
db.close()
if a.json:
print(json.dumps(rows, indent=2))
else:
for r in rows:
print(f"imported {r['commits']:>5} commits -> {r['written']} leafs {r['repo']}")
print(f"total: {total_commits} commits")
return 0
if __name__ == "__main__": if __name__ == "__main__":
-143
View File
@@ -1,143 +0,0 @@
//usr/bin/env go run "$0" "$@"; exit
//
// bin/git/import.go - read git history with go-git (no git binary).
//
// ./bin/git/import.go [REPO]
// ./bin/git/import.go --json
// ./bin/git/import.go --limit 100 --since 2026-01-01
// ./bin/git/import.go --root DIR
//
// Conversion only: prints commit leafs. Brain write is bin/brain/index.go.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strconv"
"time"
"github.com/eSlider/2dph/internal/cmdbin"
"github.com/eSlider/2dph/internal/gitlog"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
var repo, root, since string
limit := 0
jsonOut := false
i := 0
for i < len(args) {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--limit" && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 0 {
fmt.Fprintf(os.Stderr, "git/import: --limit must be a non-negative integer\n")
return 2
}
limit = n
case a == "--since" && i+1 < len(args):
i++
since = args[i]
case a == "--root" && i+1 < len(args):
i++
root = args[i]
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/git/import.go [REPO] [--json] [--limit N] [--since DATE] [--root DIR]`)
return 0
case len(a) > 0 && a[0] != '-':
repo = a
default:
fmt.Fprintf(os.Stderr, "git/import: unknown flag %s\n", a)
return 2
}
i++
}
var sinceT time.Time
if since != "" {
var err error
sinceT, err = parseSince(since)
if err != nil {
fmt.Fprintf(os.Stderr, "git/import: %v\n", err)
return 2
}
}
repos := []string{}
if repo != "" {
repos = []string{repo}
} else if root != "" {
entries, err := os.ReadDir(root)
if err != nil {
fmt.Fprintf(os.Stderr, "git/import: %v\n", err)
return 1
}
for _, e := range entries {
p := filepath.Join(root, e.Name())
if _, err := os.Stat(filepath.Join(p, ".git")); err == nil {
repos = append(repos, p)
}
}
} else {
repos = []string{cmdbin.Root()}
}
opt := gitlog.Options{Limit: limit, Since: sinceT}
type row struct {
Repo string `json:"repo"`
Path string `json:"path"`
Commits int `json:"commits"`
Leafs []gitlog.Leaf `json:"leafs,omitempty"`
}
var rows []row
for _, p := range repos {
name, err := gitlog.RepoName(p)
if err != nil && name == "" {
fmt.Fprintf(os.Stderr, "git/import: %s: %v\n", p, err)
continue
}
cs, err := gitlog.Log(p, opt)
if err != nil {
fmt.Fprintf(os.Stderr, "git/import: %s: %v\n", p, err)
return 1
}
leafs := make([]gitlog.Leaf, 0, len(cs))
for _, c := range cs {
leafs = append(leafs, gitlog.ToLeaf(c, name))
}
rows = append(rows, row{Repo: name, Path: p, Commits: len(cs), Leafs: leafs})
}
if jsonOut {
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
enc.SetEscapeHTML(false)
if err := enc.Encode(rows); err != nil {
return 1
}
return 0
}
for _, r := range rows {
fmt.Printf("%-24s %5d commits %s\n", r.Repo, r.Commits, r.Path)
}
return 0
}
func parseSince(s string) (time.Time, error) {
for _, layout := range []string{time.RFC3339, "2006-01-02"} {
if t, err := time.Parse(layout, s); err == nil {
return t, nil
}
}
return time.Time{}, fmt.Errorf("cannot parse --since %q", s)
}
+3 -19
View File
@@ -26,7 +26,6 @@ from kblib import ( # noqa: E402
open_readonly, stats, open_readonly, stats,
) )
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402 from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
from mailleafs import from_mail_root # noqa: E402
CORPUS_DEFAULTS = ["README.md", "PLAN.md", "AGENTS.md", "docs", "skills"] CORPUS_DEFAULTS = ["README.md", "PLAN.md", "AGENTS.md", "docs", "skills"]
@@ -100,9 +99,6 @@ def main(argv: list[str]) -> int:
p = argparse.ArgumentParser(description="build the 2dph brain index") p = argparse.ArgumentParser(description="build the 2dph brain index")
p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)") p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)")
p.add_argument("--rebuild", action="store_true", help="fresh db + indexes") p.add_argument("--rebuild", action="store_true", help="fresh db + indexes")
p.add_argument("--with-mail", action="store_true", help="include var/mail message.md leafs")
p.add_argument("--since", default="", help="with --with-mail, only messages dated >= YYYY-MM-DD")
p.add_argument("--dry-run", action="store_true", help="count leafs, write nothing")
p.add_argument( p.add_argument(
"--skip-indexes", "--skip-indexes",
action="store_true", action="store_true",
@@ -113,26 +109,14 @@ def main(argv: list[str]) -> int:
a = p.parse_args(argv) a = p.parse_args(argv)
from kblib import DB_PATH, VAR from kblib import DB_PATH, VAR
VAR.mkdir(exist_ok=True)
if a.rebuild and DB_PATH.exists():
DB_PATH.unlink()
leafs = load_corpus(ROOT) leafs = load_corpus(ROOT)
if a.corpus: if a.corpus:
for source in a.corpus: for source in a.corpus:
leafs.extend(load_corpus_glob(source)) leafs.extend(load_corpus_glob(source))
mail_n = 0
if a.with_mail:
mail = from_mail_root(ROOT / "var" / "mail", since=a.since)
mail_n = len(mail)
leafs.extend(mail)
if a.dry_run:
msg = {"indexed": 0, "corpus_total": len(leafs), "mail_leafs": mail_n, "dry_run": True}
print(json.dumps(msg, indent=2) if a.json else
f"brain/index: {len(leafs)} leafs would be indexed (mail={mail_n})")
return 0
VAR.mkdir(exist_ok=True)
if a.rebuild and DB_PATH.exists():
DB_PATH.unlink()
db, conn = connect(DB_PATH, read_only=False) db, conn = connect(DB_PATH, read_only=False)
init_schema(conn) init_schema(conn)
+20 -21
View File
@@ -1,35 +1,34 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# bin/kb/search — deprecated wrapper. Use bin/brain/search.go. # bin/kb/search - Go deduction search over the brain (model served by daemon).
# CGO via Zig (bin/cgo/zig), not gcc. Builds a binary then execs it. # Builds the kbsearch binary on first run / when source changes, then execs it.
set -euo pipefail set -euo pipefail
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)" KB="$(cd "$(dirname "$0")/../.." && pwd)"
BIN="$ROOT/var/bin/brain-search" BIN="$KB/var/bin/kbsearch"
SRC="$ROOT/internal/brain" SRC="$KB/bin/kbsearch"
CMD="$ROOT/bin/brain"
mkdir -p "$ROOT/var/bin" mkdir -p "$KB/var/bin"
# Rebuild if binary missing or any .go source newer
need_build=0 need_build=0
if [ ! -x "$BIN" ]; then if [ ! -x "$BIN" ]; then
need_build=1 need_build=1
else else
while IFS= read -r -d '' f; do # Check if any .go in kbsearch is newer than binary
if [ "$f" -nt "$BIN" ]; then while IFS= read -r -d '' f; do
need_build=1 if [ "$f" -nt "$BIN" ]; then
break need_build=1
fi break
done < <(find "$SRC" "$CMD" -name '*.go' -print0 2>/dev/null) fi
done < <(find "$SRC" -name '*.go' -print0 2>/dev/null)
fi fi
if [ "$need_build" -eq 1 ]; then if [ "$need_build" -eq 1 ]; then
echo "Building brain/search (zig cc)..." >&2 echo "Building kbsearch..." >&2
( (cd "$SRC" && \
cd "$ROOT" && CGO_CFLAGS="-I$KB/lib-ladybug" \
eval "$("$ROOT/bin/cgo/zig" env)" && CGO_LDFLAGS="-L$KB/lib-ladybug -Wl,-rpath,$KB/lib-ladybug" \
go build -tags system_ladybug -o "$BIN" ./bin/brain go build -tags system_ladybug -o "$BIN" .) || exit 1
) || exit 1
fi fi
echo "bin/kb/search is deprecated; use bin/brain/search.go" >&2
exec "$BIN" "$@" exec "$BIN" "$@"
+1 -1
View File
@@ -1,5 +1,5 @@
//usr/bin/env go run "$0" "$@"; exit //usr/bin/env go run "$0" "$@"; exit
// bin/kb/watch.go — deprecated. Use bin/brain/watch.go. // bin/kb/watch.go - re-index the 2dph brain when corpus files change.
// //
// Usage: // Usage:
// //
+9 -21
View File
@@ -1,12 +1,10 @@
//go:build cgo && system_ladybug // Brain connection management using go-ladybug.
package main
package brain
import ( import (
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
lbug "github.com/LadybugDB/go-ladybug" lbug "github.com/LadybugDB/go-ladybug"
) )
@@ -46,10 +44,10 @@ func dbPath() string {
} }
func openBrain() error { func openBrain() error {
return openWithSandbox(eps()) return openWithOpts(2, eps())
} }
func openWithSandbox(epsv string) error { func openWithOpts(allow int, epsv string) error {
cfg := lbug.DefaultSystemConfig() cfg := lbug.DefaultSystemConfig()
cfg.MaxNumThreads = 8 cfg.MaxNumThreads = 8
cfg.BufferPoolSize = 1 << 30 // 1GB cfg.BufferPoolSize = 1 << 30 // 1GB
@@ -59,30 +57,20 @@ func openWithSandbox(epsv string) error {
if err != nil { if err != nil {
return fmt.Errorf("OpenDatabase: %w", err) return fmt.Errorf("OpenDatabase: %w", err)
} }
if epsv != "" {
if _, err := conn.Query("SET STREAM_SANDBOX = '" + epsv + "'"); err != nil {
return err
}
}
conn, err = lbug.OpenConnection(db) conn, err = lbug.OpenConnection(db)
if err != nil { if err != nil {
closeBrain()
return fmt.Errorf("OpenConnection: %w", err) return fmt.Errorf("OpenConnection: %w", err)
} }
// Session settings need a live connection; running this before
// OpenConnection dereferenced a nil *Connection.
if epsv != "" {
if strings.ContainsAny(epsv, "'\\") {
closeBrain()
return fmt.Errorf("SET STREAM_SANDBOX: invalid value")
}
if _, err := conn.Query("SET STREAM_SANDBOX = '" + epsv + "'"); err != nil {
closeBrain()
return fmt.Errorf("SET STREAM_SANDBOX: %w", err)
}
}
if _, err := conn.Query("LOAD EXTENSION FTS"); err != nil { if _, err := conn.Query("LOAD EXTENSION FTS"); err != nil {
closeBrain()
return fmt.Errorf("LOAD EXTENSION FTS: %w", err) return fmt.Errorf("LOAD EXTENSION FTS: %w", err)
} }
if _, err := conn.Query("LOAD EXTENSION VECTOR"); err != nil { if _, err := conn.Query("LOAD EXTENSION VECTOR"); err != nil {
closeBrain()
return fmt.Errorf("LOAD EXTENSION VECTOR: %w", err) return fmt.Errorf("LOAD EXTENSION VECTOR: %w", err)
} }
return nil return nil
+23
View File
@@ -0,0 +1,23 @@
module github.com/eSlider/2dph/bin/kbsearch
go 1.26.0
require (
github.com/LadybugDB/go-ladybug v0.17.0
github.com/chewxy/math32 v1.11.2
github.com/daulet/tokenizers v1.27.0
)
require (
github.com/apache/arrow-go/v18 v18.6.0 // indirect
github.com/goccy/go-json v0.10.6 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/pierrec/lz4/v4 v4.1.26 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect
golang.org/x/sys v0.43.0 // indirect
)
+44
View File
@@ -0,0 +1,44 @@
github.com/LadybugDB/go-ladybug v0.17.0 h1:RXDbkBjrbRmLdEbhGl4CLOIEzSt09gbP0n9UbKDEfwI=
github.com/LadybugDB/go-ladybug v0.17.0/go.mod h1:GeIXmE8XyF5TFS94NAuTag7vgCC+no/HTBMRA6Rd5Cs=
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/apache/arrow-go/v18 v18.6.0 h1:GX/Jyd3R7mCLiECAwY9FWbbaYblie2WXBSz4Sw8fNpM=
github.com/apache/arrow-go/v18 v18.6.0/go.mod h1:gm3MiPpY82fLYK5VKPB3WoJbsiLVDfT7flD5/vHReKw=
github.com/apache/thrift v0.22.0 h1:r7mTJdj51TMDe6RtcmNdQxgn9XcyfGDOzegMDRg47uc=
github.com/apache/thrift v0.22.0/go.mod h1:1e7J/O1Ae6ZQMTYdy9xa3w9k+XHWPfRvdPyJeynQ+/g=
github.com/chewxy/math32 v1.11.2 h1:IufN08Zwr1NKuWfY+4Tz55BcwKmyKKNdOP7KtumehnM=
github.com/chewxy/math32 v1.11.2/go.mod h1:dOB2rcuFrCn6UHrze36WSLVPKtzPMRAQvBvUwkSsLqs=
github.com/daulet/tokenizers v1.27.0 h1:MmFYAEDFz69s/nNQfHg59DWqHz3v94m99kEZ/JbL+s4=
github.com/daulet/tokenizers v1.27.0/go.mod h1:YjFY1o1HGMyWkQgbXJDghhvke/yFDp2vGdIO2hYs4MQ=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs=
github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU=
golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU=
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+44
View File
@@ -0,0 +1,44 @@
// bin/kbsearch - the Go implementation of bin/kb/search (nested module so the
// root `go test ./...` and CI never compile it against native ladyships).
//
// Usage (built/run by ./bin/kb/search):
//
// kbsearch "query" [--root facts|info] [--repo P] [-n N] [--json]
// kbsearch serve [port] start the embedding daemon
// kbsearch --list-model print the resolved model dir
//
// The potion-multilingual model is loaded only in `serve`; a CLI reuses the
// daemon over localhost HTTP (falling back to in-process embedding).
package main
import (
"fmt"
"log"
"os"
"strconv"
)
func main() {
if len(os.Args) > 1 && os.Args[1] == "serve" {
port := 17830
if len(os.Args) > 2 {
if p, err := strconv.Atoi(os.Args[2]); err == nil {
port = p
}
}
if err := serve(port); err != nil {
log.Fatalf("kbsearch serve: %v", err)
}
return
}
if len(os.Args) > 1 && os.Args[1] == "--list-model" {
dir, err := modelDir()
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(dir)
return
}
os.Exit(runSearch(os.Args[1:]))
}
@@ -1,11 +1,9 @@
//go:build cgo && system_ladybug
// StaticModel wraps the potion-multilingual-128m embedding model. // StaticModel wraps the potion-multilingual-128m embedding model.
// //
// Mirrors model2vec.StaticModel: tokenizer (daulet Unigram) + safetensors matrix. // Mirrors model2vec.StaticModel: tokenizer (daulet Unigram) + safetensors matrix.
// Embed(text) applies the same preprocessing: median_token_length pre-truncation, // Embed(text) applies the same preprocessing: median_token_length pre-truncation,
// add_special_tokens=false, drop unk (id=1), truncate to 512, mean pool, L2 normalize +1e-32. // add_special_tokens=false, drop unk (id=1), truncate to 512, mean pool, L2 normalize +1e-32.
package brain package main
import ( import (
"encoding/json" "encoding/json"
@@ -1,4 +1,5 @@
package brain // modelDir returns the resolved potion-multilingual-128m model directory.
package main
import ( import (
"fmt" "fmt"
@@ -1,6 +1,5 @@
//go:build cgo && system_ladybug // Hybrid FTS + vector search implementation, plus daemon client/server.
package main
package brain
import ( import (
"bytes" "bytes"
@@ -14,12 +13,12 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"sort"
"strconv" "strconv"
"syscall" "strings"
"time" "time"
lbug "github.com/LadybugDB/go-ladybug" lbug "github.com/LadybugDB/go-ladybug"
"github.com/eSlider/2dph/internal/brain/rank"
) )
const defaultPort = 17830 const defaultPort = 17830
@@ -27,15 +26,45 @@ const daemonPath = "/embed"
const healthPath = "/health" const healthPath = "/health"
func runSearch(args []string) int { func runSearch(args []string) int {
opt, err := rank.ParseArgs(args) // Manual flag parsing to allow flags after query (like Python argparse)
if err != nil { root := ""
fmt.Fprintf(os.Stderr, "brain/search: %v\n%s\n", err, rank.Usage) repo := ""
return 2 limit := 20
} jsonOut := false
root, repo, limit, query := opt.Root, opt.Repo, opt.Limit, opt.Query listModel := false
jsonOut := opt.JSONOut
if opt.ListModel { var queryArgs []string
for i := 0; i < len(args); i++ {
switch args[i] {
case "--root":
if i+1 < len(args) {
root = args[i+1]
i++
}
case "--repo":
if i+1 < len(args) {
repo = args[i+1]
i++
}
case "-n":
if i+1 < len(args) {
if n, err := strconv.Atoi(args[i+1]); err == nil {
limit = n
}
i++
}
case "--json":
jsonOut = true
case "--list-model":
listModel = true
default:
if !strings.HasPrefix(args[i], "-") {
queryArgs = append(queryArgs, args[i])
}
}
}
if listModel {
dir, err := modelDir() dir, err := modelDir()
if err != nil { if err != nil {
fmt.Fprintln(os.Stderr, err) fmt.Fprintln(os.Stderr, err)
@@ -45,19 +74,47 @@ func runSearch(args []string) int {
return 0 return 0
} }
query := strings.TrimSpace(strings.Join(queryArgs, " "))
if query == "" {
fmt.Fprintln(os.Stderr, "usage: kbsearch \"query\" [--root facts|info] [--repo REPO] [-n N] [--json]")
return 1
}
if err := openBrain(); err != nil { if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err) fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1 return 1
} }
defer closeBrain() defer closeBrain()
hits, err := searchHits(query, root, repo, limit) emb, err := embedQuery(query)
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "search: %v\n", err) fmt.Fprintf(os.Stderr, "embed: %v\n", err)
return 1 return 1
} }
results := hits fts, err := queryFTS(query, limit*3)
if err != nil {
fmt.Fprintf(os.Stderr, "fts: %v\n", err)
return 1
}
var vec []Hit
if vec, err = queryVector(emb, limit*3); err != nil {
fmt.Fprintf(os.Stderr, "vec: %v\n", err)
}
results := hybrid(fts, vec, limit)
if root != "" {
results = filterRoot(results, root)
}
if repo != "" {
results = filterRepo(results, repo)
}
if len(results) > limit {
results = results[:limit]
}
for i := range results { for i := range results {
if results[i].Text != "" { if results[i].Text != "" {
runes := []rune(results[i].Text) runes := []rune(results[i].Text)
@@ -68,46 +125,23 @@ func runSearch(args []string) int {
} }
} }
webOut := rank.Deduce(results, query, root, opt.NoWeb, func(q string) rank.SecondSource {
return lookupWeb(context.Background(), q)
})
out := Dict{ out := Dict{
{"query", query}, {"query", query},
{"root_filter", root}, {"root_filter", root},
{"count", len(results)}, {"count", len(results)},
{"results", resultsToDicts(results)}, {"results", resultsToDicts(results)},
} }
if webOut != nil {
out = append(out, KV{"web", secondToDict(*webOut)})
}
if jsonOut { if jsonOut {
enc := json.NewEncoder(os.Stdout) enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ") enc.SetIndent("", " ")
enc.SetEscapeHTML(false) enc.SetEscapeHTML(false)
return b2i(enc.Encode(toJSONOut(results, query, root, webOut))) return b2i(enc.Encode(toJSONOut(results, query, root)))
} }
fmt.Print(toYAML(out, 0)) fmt.Print(toYAML(out, 0))
return 0 return 0
} }
func searchHits(query, root, repo string, limit int) ([]Hit, error) {
emb, err := embedQuery(query)
if err != nil {
return nil, fmt.Errorf("embed: %w", err)
}
fts, err := queryFTS(query, limit*3)
if err != nil {
return nil, fmt.Errorf("fts: %w", err)
}
var vec []Hit
if vec, err = queryVector(emb, limit*3); err != nil {
fmt.Fprintf(os.Stderr, "vec: %v\n", err)
}
return rank.RankAndFilter(fts, vec, root, repo, limit), nil
}
func b2i(err error) int { func b2i(err error) int {
if err != nil { if err != nil {
return 1 return 1
@@ -116,7 +150,10 @@ func b2i(err error) int {
} }
func queryFTS(text string, limit int) ([]Hit, error) { func queryFTS(text string, limit int) ([]Hit, error) {
stmt, err := conn.Prepare(rank.FTSStmt) stmt, err := conn.Prepare(
"CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " +
"RETURN node.id, node.text, node.root, node.source, score ORDER BY score LIMIT $n",
)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -133,7 +170,10 @@ func queryVector(emb []float64, limit int) ([]Hit, error) {
for i, v := range emb { for i, v := range emb {
embList[i] = v embList[i] = v
} }
stmt, err := conn.Prepare(rank.VecStmt) stmt, err := conn.Prepare(
"CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n",
)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -175,11 +215,10 @@ func rowsToHits(res *lbug.QueryResult) ([]Hit, error) {
// JSON output types // JSON output types
type jsonOut struct { type jsonOut struct {
Query string `json:"query"` Query string `json:"query"`
RootFilter string `json:"root_filter"` RootFilter string `json:"root_filter"`
Count int `json:"count"` Count int `json:"count"`
Results []jsonHit `json:"results"` Results []jsonHit `json:"results"`
Web *rank.SecondSource `json:"web,omitempty"`
} }
type jsonHit struct { type jsonHit struct {
@@ -190,7 +229,7 @@ type jsonHit struct {
Snippet string `json:"snippet,omitempty"` Snippet string `json:"snippet,omitempty"`
} }
func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *jsonOut { func toJSONOut(hits []Hit, query, rootFilter string) *jsonOut {
out := make([]jsonHit, len(hits)) out := make([]jsonHit, len(hits))
for i, h := range hits { for i, h := range hits {
out[i] = jsonHit{ out[i] = jsonHit{
@@ -206,10 +245,73 @@ func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *js
RootFilter: rootFilter, RootFilter: rootFilter,
Count: len(hits), Count: len(hits),
Results: out, Results: out,
Web: web,
} }
} }
func hybrid(fts, vec []Hit, limit int) []Hit {
byID := make(map[string]Hit)
rrf := make(map[string]float64)
for rank, h := range fts {
byID[h.ID] = h
rrf[h.ID] += 1.0 / (60 + float64(rank+1))
}
for rank, h := range vec {
if _, ok := byID[h.ID]; !ok {
byID[h.ID] = h
} else {
existing := byID[h.ID]
if existing.Score == 0 {
existing.Score = h.Score
byID[h.ID] = existing
}
}
rrf[h.ID] += 1.0 / (60 + float64(rank+1))
}
type scored struct {
id string
rrf float64
}
var scoredList []scored
for id, v := range rrf {
scoredList = append(scoredList, scored{id, v})
}
sort.Slice(scoredList, func(i, j int) bool {
return scoredList[i].rrf > scoredList[j].rrf
})
var out []Hit
for i, s := range scoredList {
if i >= limit {
break
}
h := byID[s.id]
out = append(out, h)
}
return out
}
func filterRoot(hits []Hit, root string) []Hit {
var out []Hit
for _, h := range hits {
if h.Root == root {
out = append(out, h)
}
}
return out
}
func filterRepo(hits []Hit, repo string) []Hit {
var out []Hit
for _, h := range hits {
if strings.Contains(h.Source, repo) {
out = append(out, h)
}
}
return out
}
func resultsToDicts(hits []Hit) []any { func resultsToDicts(hits []Hit) []any {
out := make([]any, len(hits)) out := make([]any, len(hits))
for i, h := range hits { for i, h := range hits {
@@ -261,7 +363,7 @@ func serve(port int) error {
}) })
addr := fmt.Sprintf("127.0.0.1:%d", port) addr := fmt.Sprintf("127.0.0.1:%d", port)
log.Printf("brain search daemon listening on %s", addr) log.Printf("kbsearch daemon listening on %s", addr)
return http.ListenAndServe(addr, mux) return http.ListenAndServe(addr, mux)
} }
@@ -280,16 +382,10 @@ func embedQuery(text string) ([]float64, error) {
port = p port = p
} }
} }
if emb, err := tryDaemon(text, port); err == nil { emb, err := tryDaemon(text, port)
if err == nil {
return emb, nil return emb, nil
} }
if os.Getenv("KBSEARCH_NO_DAEMON") == "" {
if err := ensureDaemon(port); err == nil {
if emb, err := tryDaemon(text, port); err == nil {
return emb, nil
}
}
}
model, err := loadModel() model, err := loadModel()
if err != nil { if err != nil {
@@ -351,13 +447,9 @@ func ensureDaemon(port int) error {
cmd.Dir, _ = filepath.Split(self) cmd.Dir, _ = filepath.Split(self)
cmd.Stdout = nil cmd.Stdout = nil
cmd.Stderr = nil cmd.Stderr = nil
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
if err := cmd.Start(); err != nil { if err := cmd.Start(); err != nil {
return err return err
} }
if cmd.Process != nil {
_ = cmd.Process.Release()
}
for i := 0; i < 40; i++ { for i := 0; i < 40; i++ {
time.Sleep(250 * time.Millisecond) time.Sleep(250 * time.Millisecond)
@@ -371,21 +463,3 @@ func ensureDaemon(port int) error {
} }
return fmt.Errorf("daemon failed to start on port %d", port) return fmt.Errorf("daemon failed to start on port %d", port)
} }
// Main is the bin/brain/search.go entry: search, serve, or --list-model.
func Main(args []string) int {
if len(args) > 0 && args[0] == "serve" {
port := defaultPort
if len(args) > 1 {
if p, err := strconv.Atoi(args[1]); err == nil {
port = p
}
}
if err := serve(port); err != nil {
log.Printf("brain/search serve: %v", err)
return 1
}
return 0
}
return runSearch(args)
}
+16
View File
@@ -0,0 +1,16 @@
// Common types and helpers for kbsearch.
package main
import "os"
func eps() string { return os.Getenv("KBTEST_EPS") }
// Hit is one search result, mirroring the python script's dict shape.
type Hit struct {
ID string `json:"id"`
Text string `json:"text"`
Root string `json:"root"`
Source string `json:"-"` // for repo filtering, not in output
Score float64 `json:"score"`
Snippet string `json:"snippet,omitempty"`
}
@@ -1,5 +1,5 @@
// YAML emitter ported from bin/kb/yamlout.py — preserves insertion order. // YAML emitter ported from bin/kb/yamlout.py — preserves insertion order.
package brain package main
import ( import (
"fmt" "fmt"
+2 -2
View File
@@ -15,8 +15,8 @@ Writes one directory per message: var/mail/{folder}/{message_id}/
attachments/ raw attachment files (zips unpacked to _unpacked/) attachments/ raw attachment files (zips unpacked to _unpacked/)
attachments/*.md converted attachment content attachments/*.md converted attachment content
Indexing is a separate step (`bin/brain/index.go --rebuild`): conversion can Indexing is a separate step (bin/mail/index_mail): conversion can crash in
crash in native docling and must not leave the brain DB mid-transaction. native docling and must not leave the brain DB mid-transaction.
Requires ONLYOFFICE_URL/USER/PASS in .env (or env). Idempotent: a message Requires ONLYOFFICE_URL/USER/PASS in .env (or env). Idempotent: a message
already present (message.md exists) is skipped unless --force. already present (message.md exists) is skipped unless --force.
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=mail_import "$0" "$@"; exit
//go:build mail_import
//
// bin/mail/import.go - message.json → markdown (no brain write).
//
// ./bin/mail/import.go --from-raw var/mail
//
// Indexing is bin/brain/index.go --rebuild, not this command.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/mail/import", os.Args[1:]))
}
+120 -11
View File
@@ -1,26 +1,135 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""mail/index_mail — deprecated. Use bin/brain/index.go --rebuild --with-mail. """mail/index_mail - rebuild the brain with every markdown under var/mail.
Ladybug corrupts its WAL on bulk-insert into an already-indexed DB, so this Ladybug corrupts its WAL when brand-new leafs are bulk-inserted while the
shim always rebuilds (repo corpus + var/mail). Conversion stays in mail/import. FTS/VECTOR indexes already exist, so indexing ALWAYS runs as a fresh rebuild
(repo corpus + var/mail), matching the proven-safe `kb/index --rebuild` path.
Conversion and indexing stay separate: conversion can crash in native docling
and must not leave the brain DB mid-transaction.
bin/mail/index_mail rebuild the index incl. all mail
bin/mail/index_mail --dry-run count without writing
bin/mail/index_mail --limit N cap messages included
bin/mail/index_mail --since D only messages dated >= D (YYYY-MM-DD)
""" """
from __future__ import annotations from __future__ import annotations
import os import argparse
import json
import sys import sys
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parents[2] ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import DB_PATH, VAR, connect, ensure_indexes, init_schema, stats, upsert_leaf # noqa: E402
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
def msg_date(md: Path) -> str:
j = md.parent / "message.json"
try:
d = json.loads(j.read_text(encoding="utf-8"))
return (d.get("receivedDate") or d.get("receivedAt") or "")[:10]
except Exception:
return ""
def mail_leafs(limit: int, since: str, repo: str = "ooMail") -> list[dict]:
root = ROOT / "var" / "mail"
mds = sorted(root.rglob("message.md"))
if since:
mds = [m for m in mds if msg_date(m) >= since]
if limit:
mds = mds[:limit]
leafs: list[dict] = []
for md in mds:
files = [md] + sorted((md.parent / "attachments").glob("*.md"))
for f in files:
if not f.exists():
continue
for lf in to_all(read_markdown(f), f, repo=repo):
lf["source"] = f"ooMail:{md.parent.name}:{f.name}"
lf["how"] = "mail/import"
leafs.append(lf)
return leafs
def main(argv: list[str]) -> int: def main(argv: list[str]) -> int:
print( p = argparse.ArgumentParser(description="rebuild the brain incl. all mail")
"bin/mail/index_mail is deprecated; use bin/brain/index.go --rebuild --with-mail", p.add_argument("--dry-run", action="store_true", help="count only, write nothing")
file=sys.stderr, p.add_argument("--limit", type=int, default=0, help="cap messages included")
) p.add_argument("--since", default="", help="only messages dated >= YYYY-MM-DD")
index = ROOT / "bin" / "kb" / "index" p.add_argument("--json", action="store_true")
os.execv(sys.executable, [sys.executable, str(index), "--rebuild", "--with-mail", *argv]) a = p.parse_args(argv)
return 1
mail = mail_leafs(a.limit, a.since)
if a.dry_run:
print(f"mail/index_mail: {len(mail)} mail leafs would be indexed")
return 0
# Fresh rebuild: delete DB, index repo corpus + mail, create indexes once
# at the end. Never insert into an already-indexed DB (WAL corruption).
VAR.mkdir(exist_ok=True)
if DB_PATH.exists():
DB_PATH.unlink()
corpus = _load_corpus()
leafs = corpus + mail
db, conn = connect(DB_PATH, read_only=False)
init_schema(conn)
embed = _embedder()
done, total = _index_leafs(conn, leafs, embed)
ensure_indexes(conn)
s = stats(conn)
conn.close()
db.close()
result = {"indexed": done, "corpus_total": total, "mail_leafs": len(mail),
**{k: v for k, v in s.items() if k in ("total", "by_root")}}
print(json.dumps(result, indent=2) if a.json else
f"mail/index_mail: indexed {done}/{total} leafs (mail={len(mail)}); db total {s['total']}")
return 0
CORPUS_DEFAULTS = ["README.md", "PLAN.md", "AGENTS.md", "docs", "skills"]
def _load_corpus() -> list[dict]:
files: list[Path] = []
for entry in CORPUS_DEFAULTS:
p = ROOT / entry
if p.is_file():
files.append(p)
elif p.is_dir():
files.extend(walk_markdown(p))
leafs: list[dict] = []
for path in files:
try:
leafs.extend(to_all(read_markdown(path), path, repo="eSlider/2dph"))
except OSError as e:
print(f"mail/index_mail: skip {path}: {e}", file=sys.stderr)
return leafs
def _index_leafs(conn, leafs: list[dict], embed_fn) -> tuple[int, int]:
count = 0
for lf in leafs:
query = f"{lf['heading']}\n\n{lf['text']}"
emb = embed_fn(lf["text"]) if lf["text"] else None
upsert_leaf(conn, text=query, root="info", confidence="confirmed",
source=lf["source"], source_rev="mail" if lf.get("how") == "mail/import" else "working-tree",
how=lf.get("how", "kb/index"), loc=lf["source"], type_=lf.get("type", "reference"),
embedding=emb)
count += 1
return count, len(leafs)
def _embedder():
from model2vec import StaticModel
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
return lambda text: model.encode([text])[0].astype(float).tolist()
if __name__ == "__main__": if __name__ == "__main__":
+1 -1
View File
@@ -6,7 +6,7 @@
// ./bin/mail/sync.go --dry-run // ./bin/mail/sync.go --dry-run
// //
// Writes raw message.json + attachments under var/mail/<folder>/<id>/; run // Writes raw message.json + attachments under var/mail/<folder>/<id>/; run
// bin/mail/import.go --from-raw afterwards to convert everything to markdown. // bin/mail/import --from-raw afterwards to convert everything to markdown.
// //
// Shebang trick: first line is a Go `//` comment; the real code lives in the // Shebang trick: first line is a Go `//` comment; the real code lives in the
// importable package (module path, never a relative import). // importable package (module path, never a relative import).
+1 -3
View File
@@ -32,7 +32,6 @@ func ParseCLI(args []string) (CLIConfig, int, error) {
offset = fs.Int("offset", 0, "skip first N messages per source") offset = fs.Int("offset", 0, "skip first N messages per source")
force = fs.Bool("force", false, "overwrite existing message.json + attachments") force = fs.Bool("force", false, "overwrite existing message.json + attachments")
dryRun = fs.Bool("dry-run", false, "list message counts without writing") dryRun = fs.Bool("dry-run", false, "list message counts without writing")
query = fs.String("query", "in:inbox", "Gmail search query (gmail source only)")
srcs = fs.String("source", "onlyoffice", "comma list: onlyoffice,gmail (default onlyoffice)") srcs = fs.String("source", "onlyoffice", "comma list: onlyoffice,gmail (default onlyoffice)")
help = fs.Bool("help", false, "usage") help = fs.Bool("help", false, "usage")
) )
@@ -61,7 +60,6 @@ func ParseCLI(args []string) (CLIConfig, int, error) {
Offset: *offset, Offset: *offset,
Force: *force, Force: *force,
DryRun: *dryRun, DryRun: *dryRun,
Query: *query,
Policy: RetryPolicy{}, Policy: RetryPolicy{},
} }
cli := CLIConfig{Sync: cfg, Env: *env, Sources: *srcs} cli := CLIConfig{Sync: cfg, Env: *env, Sources: *srcs}
@@ -97,7 +95,7 @@ func Main(args []string) int {
return code return code
} }
if cli.Help { if cli.Help {
fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--query GMAIL_Q] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]") fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]")
return 0 return 0
} }
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour) ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
+4 -17
View File
@@ -112,7 +112,6 @@ type SyncConfig struct {
Offset int // skip first N messages per source Offset int // skip first N messages per source
Force bool // overwrite existing message.json + attachments Force bool // overwrite existing message.json + attachments
DryRun bool // list without writing DryRun bool // list without writing
Query string // Gmail search query; default in:inbox
Policy RetryPolicy Policy RetryPolicy
} }
@@ -137,17 +136,9 @@ type ooSource struct {
c *OOClient c *OOClient
page int page int
} }
// gmailAPI is the Gmail client surface gmailSource needs. *GmailClient implements it.
type gmailAPI interface {
ListIDs(ctx context.Context, q string, maxIDs int, pageToken string) ([]string, string, error)
GetMessage(ctx context.Context, id string) (*Message, error)
DownloadAttachment(ctx context.Context, msgID, attID string) ([]byte, error)
}
type gmailSource struct { type gmailSource struct {
c gmailAPI c *GmailClient
cur string cur string
query string
} }
func (s *ooSource) Folder() string { return "inbox" } func (s *ooSource) Folder() string { return "inbox" }
@@ -180,11 +171,7 @@ func (s *ooSource) DownloadAttachment(ctx context.Context, msg *Message, att Att
} }
func (s *gmailSource) ListIDs(ctx context.Context, limit int, cursor string) ([]string, string, error) { func (s *gmailSource) ListIDs(ctx context.Context, limit int, cursor string) ([]string, string, error) {
q := s.query ids, next, err := s.c.ListIDs(ctx, "in:inbox", limit, cursor)
if q == "" {
q = "in:inbox"
}
ids, next, err := s.c.ListIDs(ctx, q, limit, cursor)
return ids, next, err return ids, next, err
} }
@@ -220,7 +207,7 @@ func Run(ctx context.Context, cfg SyncConfig) (*SyncStats, error) {
if err != nil { if err != nil {
return nil, fmt.Errorf("gmail init: %w", err) return nil, fmt.Errorf("gmail init: %w", err)
} }
sources = append(sources, &gmailSource{c: gm, query: cfg.Query}) sources = append(sources, &gmailSource{c: gm})
} }
if len(sources) == 0 { if len(sources) == 0 {
return nil, errors.New("sync: no source configured (need OO, Gmail, or both)") return nil, errors.New("sync: no source configured (need OO, Gmail, or both)")
-65
View File
@@ -221,71 +221,6 @@ func TestCollectParts(t *testing.T) {
} }
} }
type fakeGmailAPI struct {
lastQ string
lastLimit int
ids []string
}
func (f *fakeGmailAPI) ListIDs(_ context.Context, q string, maxIDs int, _ string) ([]string, string, error) {
f.lastQ = q
f.lastLimit = maxIDs
return f.ids, "", nil
}
func (f *fakeGmailAPI) GetMessage(context.Context, string) (*Message, error) {
return nil, errors.New("unused")
}
func (f *fakeGmailAPI) DownloadAttachment(context.Context, string, string) ([]byte, error) {
return nil, errors.New("unused")
}
func TestGmailSourcePassesQueryToListIDs(t *testing.T) {
fake := &fakeGmailAPI{ids: []string{"m1"}}
src := &gmailSource{c: fake, query: "from:alice@example.com"}
ids, _, err := src.ListIDs(context.Background(), 10, "")
if err != nil {
t.Fatal(err)
}
if fake.lastQ != "from:alice@example.com" {
t.Fatalf("ListIDs q=%q, want from:alice@example.com", fake.lastQ)
}
if fake.lastLimit != 10 {
t.Fatalf("ListIDs limit=%d, want 10", fake.lastLimit)
}
if len(ids) != 1 || ids[0] != "m1" {
t.Fatalf("ids=%v", ids)
}
}
func TestGmailSourceEmptyQueryDefaultsToInbox(t *testing.T) {
fake := &fakeGmailAPI{}
src := &gmailSource{c: fake, query: ""}
if _, _, err := src.ListIDs(context.Background(), 5, ""); err != nil {
t.Fatal(err)
}
if fake.lastQ != "in:inbox" {
t.Fatalf("empty query q=%q, want in:inbox", fake.lastQ)
}
}
func TestParseCLIGmailQuery(t *testing.T) {
cli, code, err := ParseCLI([]string{
"--source", "gmail",
"--query", "from:letrado@example.com",
"--out", t.TempDir(),
"--dry-run",
})
if err != nil || code != 0 {
t.Fatalf("ParseCLI: code=%d err=%v", code, err)
}
if cli.Sync.Query != "from:letrado@example.com" {
t.Fatalf("query=%q", cli.Sync.Query)
}
if cli.Sync.Gmail == nil {
t.Fatal("gmail source not configured")
}
}
func b64(s string) string { func b64(s string) string {
return base64.URLEncoding.EncodeToString([]byte(s)) return base64.URLEncoding.EncodeToString([]byte(s))
} }
-3
View File
@@ -1,3 +0,0 @@
// Commands in this directory are shebang mains (import.go), tagged so
// `go build ./bin/markdown` does not see two mains.
package main
-100
View File
@@ -1,100 +0,0 @@
//usr/bin/env go run "$0" "$@"; exit
//
// bin/markdown/import.go - split markdown into leafs (H2 boundaries).
//
// ./bin/markdown/import.go [dir]
// ./bin/markdown/import.go --files a.md,b.md --json
//
// Conversion only. Brain write is bin/brain/index.go.
// Python bin/md/import remains as a fallback.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"strings"
"github.com/eSlider/2dph/internal/mdleaves"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
jsonOut := false
files := ""
root := "."
for i := 0; i < len(args); i++ {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--files" && i+1 < len(args):
i++
files = args[i]
case strings.HasPrefix(a, "--files="):
files = strings.TrimPrefix(a, "--files=")
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, "bin/markdown/import.go [dir] [--files a.md,b.md] [--json]")
return 0
case strings.HasPrefix(a, "-"):
fmt.Fprintln(os.Stderr, "unknown arg:", a)
return 2
default:
root = a
}
}
var paths []string
if files != "" {
for _, f := range strings.Split(files, ",") {
f = strings.TrimSpace(f)
if f != "" {
paths = append(paths, f)
}
}
} else {
st, err := os.Stat(root)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: no such path %s\n", root)
return 2
}
if !st.IsDir() {
paths = []string{root}
} else {
var err error
paths, err = mdleaves.WalkMarkdown(root)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: %v\n", err)
return 1
}
}
}
if len(paths) == 0 {
fmt.Fprintln(os.Stderr, "md/import: no markdown files")
return 1
}
var all []mdleaves.Leaf
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: %s: %v\n", p, err)
continue
}
all = append(all, mdleaves.ToAll(string(raw), p, "")...)
}
if jsonOut {
s, err := mdleaves.EncodeJSON(all)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Print(s)
return 0
}
fmt.Print(mdleaves.EncodeYAML(all))
return 0
}
-2
View File
@@ -1,2 +0,0 @@
// Commands in this directory are shebang mains (query.go).
package main
-20
View File
@@ -1,20 +0,0 @@
//usr/bin/env go run -tags=postgres_query "$0" "$@"; exit
//go:build postgres_query
//
// bin/postgres/query.go - read-only Postgres as YAML.
//
// ./bin/postgres/query.go --profile onlyoffice -c 'SELECT 1'
//
// Profiles: $HOME/.config/brain/db-profiles.yml (credentials stay out of git).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/db/psql-yq", os.Args[1:]))
}
-91
View File
@@ -1,91 +0,0 @@
//usr/bin/env go run -tags=reasoner_bakeoff "$0" "$@"; exit
//go:build reasoner_bakeoff
//
// bin/reasoner/bakeoff.go - CPU tool-call bake-off against an OpenAI-compatible URL (D18).
//
// REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b ./bin/reasoner/bakeoff.go
// ./bin/reasoner/bakeoff.go --model MichelRosselli/bonsai-27b:Q1_0 --json
//
// Measures OpenAI tool_calls (search/get/audit) and RSS from Ollama /api/ps, not VRAM.
// PicoClaw is not in this repo; the tool names match internal/httpapi MCP ops.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"encoding/json"
"fmt"
"os"
"strings"
"github.com/eSlider/2dph/internal/reasoner"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
base := os.Getenv("REASONER_BASE_URL")
if base == "" {
base = "http://127.0.0.1:11435/v1"
}
model := os.Getenv("REASONER_MODEL")
if model == "" {
model = reasoner.OllamaRAM
}
jsonOut := false
device := "cpu"
for i := 0; i < len(args); i++ {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--model" && i+1 < len(args):
i++
model = args[i]
case strings.HasPrefix(a, "--model="):
model = strings.TrimPrefix(a, "--model=")
case a == "--base-url" && i+1 < len(args):
i++
base = args[i]
case a == "--device" && i+1 < len(args):
i++
device = args[i]
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, "bin/reasoner/bakeoff.go [--model ID] [--base-url URL] [--device cpu] [--json]")
return 0
default:
fmt.Fprintln(os.Stderr, "unknown arg:", a)
return 2
}
}
c := reasoner.Client{BaseURL: base, Model: model, Device: device}
rep := reasoner.Run(c)
raw, err := json.MarshalIndent(rep, "", " ")
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
if jsonOut {
fmt.Println(string(raw))
} else {
fmt.Printf("model: %s\n", rep.Model)
fmt.Printf("hf_id: %s\n", rep.HF)
fmt.Printf("device: %s\n", rep.Device)
fmt.Printf("tool_call: %d/%d\n", rep.ToolCallOK, rep.ToolCallN)
fmt.Printf("xml_leak: %d\n", rep.XMLLeak)
fmt.Printf("rss_mb: %d\n", rep.RSSMB)
fmt.Printf("vram_mb: %d\n", rep.VRAMMB)
for _, p := range rep.Prompts {
status := "fail"
if p.OK {
status = "ok"
}
fmt.Printf(" %s: %s wanted=%s got=%s xml=%v %dms %s\n", p.WantedTool, status, p.WantedTool, p.ToolName, p.XMLLeak, p.LatencyMS, p.Err)
}
}
if rep.ToolCallN == 0 {
return 1
}
return 0
}
-2
View File
@@ -1,2 +0,0 @@
// Commands in this directory are shebang mains (bakeoff.go).
package main
+13 -8
View File
@@ -1,22 +1,27 @@
//usr/bin/env go run -tags=brain_serve "$0" "$@"; exit //usr/bin/env go run "$0" "$@"; exit
//go:build brain_serve // bin/serve.go - async Go HTTP server for the 2dph brain (see bin/server).
// //
// bin/serve.go — deprecated; use bin/brain/serve.go. // KB_ROOT=/path/to/2dph ./bin/serve.go # serve the brain
// KB_SEARCH_CMD=... KB_WORKERS=4 KB_PORT=8630 ./bin/serve.go
//
// Shebang trick: the first line is a Go `//` comment; when executed, env runs
// `go run "$0"` so this file doubles as an executable script. The real code
// lives in the importable package (module path, never a relative import).
// NOTE: never run `gofmt -w` on this file - it rewrites `//usr/bin/env` to
// `// usr/...` and breaks the shebang.
package main package main
import ( import (
"fmt"
"os" "os"
"github.com/eSlider/2dph/internal/httpapi" "github.com/eSlider/2dph/bin/server"
) )
func main() { func main() {
fmt.Fprintln(os.Stderr, "bin/serve.go is deprecated; use bin/brain/serve.go") if env := os.Getenv("KB_ROOT"); env == "" {
if os.Getenv("KB_ROOT") == "" {
if wd, err := os.Getwd(); err == nil { if wd, err := os.Getwd(); err == nil {
os.Setenv("KB_ROOT", wd) os.Setenv("KB_ROOT", wd)
} }
} }
httpapi.Run(nil) server.Run()
} }
+154
View File
@@ -0,0 +1,154 @@
// Package server serves the 2dph brain over HTTP.
//
// Async by design: every request runs on its own goroutine, and CPU-heavy
// searches are serialized through a bounded worker pool (a counting
// semaphore) so N requests can't spawn N Python interpreters at once.
//
// Used by bin/serve.go which is a self-executing shebang script:
//
// ///usr/bin/env go run "$0" "$@"; exit
// package main
// import "github.com/eSlider/2dph/bin/server"
// func main() { server.Run() }
package server
import (
"context"
"encoding/json"
"errors"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
)
type Searcher interface {
Search(ctx context.Context, query string, limit int) ([]byte, error)
}
type Server struct {
searcher Searcher
semaphore chan struct{}
}
const defaultPort = 8630
func NewServer(searcher Searcher, workers int) http.Handler {
return &Server{
searcher: searcher,
semaphore: make(chan struct{}, workers),
}
}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/health":
writeJSON(w, http.StatusOK, map[string]any{"status": "ok"})
case r.URL.Path == "/search":
s.handleSearch(w, r)
default:
writeJSON(w, http.StatusNotFound, map[string]any{"error": "not found"})
}
}
func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
q := strings.TrimSpace(r.URL.Query().Get("q"))
if q == "" {
writeJSON(w, http.StatusBadRequest, map[string]any{"error": "q required"})
return
}
limit := 10
if raw := r.URL.Query().Get("n"); raw != "" {
n, err := strconv.Atoi(raw)
if err != nil || n < 1 || n > 100 {
writeJSON(w, http.StatusBadRequest, map[string]any{"error": "n must be int 1..100"})
return
}
limit = n
}
// Worker pool: block until a slot frees, so burst concurrency still
// bounds memory (no unbounded python processes).
select {
case s.semaphore <- struct{}{}:
defer func() { <-s.semaphore }()
case <-r.Context().Done():
return
}
body, err := s.searcher.Search(r.Context(), q, limit)
if err != nil {
writeJSON(w, http.StatusGatewayTimeout, map[string]any{"error": err.Error()})
return
}
writeRaw(w, http.StatusOK, body)
}
func writeJSON(w http.ResponseWriter, code int, obj any) {
body, _ := json.Marshal(obj)
writeRaw(w, code, body)
}
func writeRaw(w http.ResponseWriter, code int, body []byte) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
w.WriteHeader(code)
w.Write(body)
}
// brainSearcher shells out to bin/kb/search --json. A single python search
// is bounded and short-lived; the worker pool keeps at most N live.
type brainSearcher struct {
cmdPath string
timeout time.Duration
}
func (b *brainSearcher) Search(ctx context.Context, query string, limit int) ([]byte, error) {
ctx, cancel := context.WithTimeout(ctx, b.timeout)
defer cancel()
cmd := exec.CommandContext(ctx, b.cmdPath, "--json", "-n", strconv.Itoa(limit), query)
out, err := cmd.Output()
if err != nil {
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
return nil, errors.New("search backend failed: " + strings.TrimSpace(string(exitErr.Stderr)))
}
return nil, err
}
return out, nil
}
// Run starts the HTTP server. Reads env: KB_SEARCH_CMD (default bin/kb/search,
// relative to the repo root given by KB_ROOT), KB_WORKERS (default 4), KB_PORT
// (default 8630).
func Run() {
root := os.Getenv("KB_ROOT")
searchPath := os.Getenv("KB_SEARCH_CMD")
if searchPath == "" {
searchPath = filepath.Join(root, "bin", "kb", "search")
}
workers := 4
if raw := os.Getenv("KB_WORKERS"); raw != "" {
if n, err := strconv.Atoi(raw); err == nil && n > 0 {
workers = n
}
}
port := defaultPort
if raw := os.Getenv("KB_PORT"); raw != "" {
if n, err := strconv.Atoi(raw); err == nil && n > 0 {
port = n
}
}
searcher := &brainSearcher{cmdPath: searchPath, timeout: 60 * time.Second}
handler := NewServer(searcher, workers)
addr := "127.0.0.1:" + strconv.Itoa(port)
log.Printf("serve: %s (workers=%d)", addr, workers)
if err := http.ListenAndServe(addr, handler); err != nil {
log.Fatal(err)
}
}
@@ -1,12 +1,10 @@
package httpapi package server
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os"
"strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
@@ -48,26 +46,6 @@ func (f *fakeSearcher) Search(ctx context.Context, query string, limit int) ([]b
return []byte(`{"query":"` + query + `","count":0,"results":[]}`), nil return []byte(`{"query":"` + query + `","count":0,"results":[]}`), nil
} }
func (f *fakeSearcher) Get(_ context.Context, id string, body bool) ([]byte, error) {
out := map[string]any{"id": id, "root": "info"}
if body {
out["text"] = "fake body"
}
return json.Marshal(out)
}
func (f *fakeSearcher) Stats(context.Context) ([]byte, error) {
return []byte(`{"total":0,"by_root":{}}`), nil
}
func (f *fakeSearcher) Audit(context.Context) ([]byte, error) {
return []byte(`{"status":"ok"}`), nil
}
func (f *fakeSearcher) Ingest(context.Context) ([]byte, error) {
return []byte(`{"mode":"rebuild","command":"bin/brain/index.go --rebuild"}`), nil
}
func (f *fakeSearcher) count() int { func (f *fakeSearcher) count() int {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
@@ -163,58 +141,6 @@ func TestSearchRejectsBadLimit(t *testing.T) {
} }
} }
func TestGetLeaf(t *testing.T) {
fs := &fakeSearcher{callback: func(q string, limit int) ([]byte, error) {
return []byte(`{}`), nil
}}
h := NewServer(fs, 1)
if code, _ := get(t, h, "/get"); code != http.StatusBadRequest {
t.Fatalf("missing id code = %d, want 400", code)
}
code, body := get(t, h, "/get?id=leaf-1&body=1")
if code != http.StatusOK {
t.Fatalf("get code = %d, want 200 body=%s", code, body)
}
if !strings.Contains(string(body), "leaf-1") {
t.Fatalf("get body %s missing id", body)
}
}
func TestStatsAuditIngest(t *testing.T) {
h := NewServer(&fakeSearcher{}, 1)
for _, path := range []string{"/stats", "/audit", "/ingest"} {
code, body := get(t, h, path)
if code != http.StatusOK {
t.Fatalf("%s code = %d, want 200 (%s)", path, code, body)
}
if !json.Valid(body) {
t.Fatalf("%s body not json: %s", path, body)
}
}
}
func TestHTTPPackageDoesNotExecPython(t *testing.T) {
raw, err := os.ReadFile("server.go")
if err != nil {
t.Fatal(err)
}
lower := strings.ToLower(string(raw))
if strings.Contains(lower, "python3") || strings.Contains(lower, "bin/kb/search") {
t.Fatal("httpapi must not exec Python or bin/kb/search")
}
}
func TestDefaultSearchCmdIsBrainNotPython(t *testing.T) {
t.Setenv("KB_SEARCH_CMD", "")
cmd := defaultSearchCmd("/repo")
if strings.Contains(strings.ToLower(cmd), "python") {
t.Fatalf("search path still python: %s", cmd)
}
if !strings.Contains(cmd, "brain") {
t.Fatalf("search path must be the Go brain binary, got %s", cmd)
}
}
func TestSearchTimeout(t *testing.T) { func TestSearchTimeout(t *testing.T) {
fs := &fakeSearcher{delay: time.Second} fs := &fakeSearcher{delay: time.Second}
h := NewServer(fs, 1) h := NewServer(fs, 1)
+60 -3
View File
@@ -1,12 +1,21 @@
"""gitimport - Ladybug graph writes for Commit/File/Person (no git binary). """gitimport - parse `git log` output and turn commits into brain leafs.
Commit records come from bin/git/import.go (go-git). This module only MERGEs Pure, testable functions. Field grammar (see bin/git/import):
the version graph File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person.
git log --no-merges --name-only \
--format='%x1e%H%x1f%an%x1f%ae%x1f%aI%x1f%s'
0x1e = record separator, 0x1f = field separator.
Files: newline-separated lines following each record's subject.
""" """
from __future__ import annotations from __future__ import annotations
from dataclasses import dataclass, field from dataclasses import dataclass, field
REC_SEP = "\x1e"
FIELD_SEP = "\x1f"
@dataclass @dataclass
class Commit: class Commit:
@@ -17,6 +26,54 @@ class Commit:
subject: str subject: str
files: list[str] = field(default_factory=list) files: list[str] = field(default_factory=list)
def leaf_text(self, repo: str) -> str:
head = f"commit {self.sha[:12]} in {repo}{self.subject}"
body = [head, f"Author: {self.author} <{self.email}>", f"Date: {self.date}"]
if self.files:
body.append("Changing: " + ", ".join(self.files))
return "\n".join(body)
def parse_log(text: str) -> list[Commit]:
"""Parse `git log` output into Commit records.
Records are separated by 0x1e. A record is fields joined by 0x1f,
followed by optional newline-separated file paths inside the next
segment (git emits blank line + files after each record).
"""
commits: list[Commit] = []
# field records and file lists alternate; simpler: split on REC_SEP,
# each chunk = header line, possibly followed by newline + files.
for chunk in text.split(REC_SEP):
chunk = chunk.strip("\n")
if not chunk:
continue
lines = chunk.split("\n", 1)
header = lines[0].split(FIELD_SEP)
if len(header) < 5:
continue
sha, author, email, date, subject = header[:5]
files = [ln.strip() for ln in lines[1].splitlines() if ln.strip()] if len(lines) > 1 else []
commits.append(Commit(sha=sha, author=author, email=email,
date=date, subject=subject, files=files))
return commits
def commits_to_leafs(commits: list[Commit], repo: str) -> list[dict]:
"""Map commits to the leaf shape bin/kb/index expects (source/repo/...)."""
out: list[dict] = []
for c in commits:
out.append({
"source": f"{repo}@{c.sha}",
"repo": repo,
"heading": f"commit {c.sha[:12]}{c.subject}",
"text": c.leaf_text(repo),
"type": "commit",
"status": "current",
"related": ",".join(c.files),
})
return out
GIT_SCHEMA = ( GIT_SCHEMA = (
"CREATE NODE TABLE IF NOT EXISTS Commit (id STRING, repo STRING, subject STRING, " "CREATE NODE TABLE IF NOT EXISTS Commit (id STRING, repo STRING, subject STRING, "
+4 -4
View File
@@ -135,7 +135,7 @@ def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None
`force=True` is accepted for API compatibility but does **not** drop. `force=True` is accepted for API compatibility but does **not** drop.
Fresh indexes require deleting `var/kb.lbug` and rebuilding Fresh indexes require deleting `var/kb.lbug` and rebuilding
(`bin/brain/index.go --rebuild`). (`bin/kb/index --rebuild`).
""" """
del force # API compat; DROP is unsafe — see docstring del force # API compat; DROP is unsafe — see docstring
names = leaf_index_names(conn) names = leaf_index_names(conn)
@@ -145,7 +145,7 @@ def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None
except Exception as e: except Exception as e:
raise RuntimeError( raise RuntimeError(
"CREATE_FTS_INDEX failed (often ghost catalog after DROP INDEX). " "CREATE_FTS_INDEX failed (often ghost catalog after DROP INDEX). "
"Delete var/kb.lbug and run bin/brain/index.go --rebuild. " "Delete var/kb.lbug and run bin/kb/index --rebuild. "
f"Cause: {e}" f"Cause: {e}"
) from e ) from e
if "Leaf_vec" not in names: if "Leaf_vec" not in names:
@@ -158,7 +158,7 @@ def create_fts_and_vector(conn: ladybug.Connection, force: bool = False) -> None
raise RuntimeError( raise RuntimeError(
"CREATE_VECTOR_INDEX failed (often ghost catalog after DROP INDEX " "CREATE_VECTOR_INDEX failed (often ghost catalog after DROP INDEX "
"Leaf.Leaf_vec → `_0_Leaf_vec_UPPER already exists in catalog`). " "Leaf.Leaf_vec → `_0_Leaf_vec_UPPER already exists in catalog`). "
"Delete var/kb.lbug and run bin/brain/index.go --rebuild. " "Delete var/kb.lbug and run bin/kb/index --rebuild. "
f"Cause: {e}" f"Cause: {e}"
) from e ) from e
names = leaf_index_names(conn) names = leaf_index_names(conn)
@@ -237,6 +237,6 @@ def stats(conn: ladybug.Connection) -> dict:
def open_readonly() -> tuple[ladybug.Database, ladybug.Connection]: def open_readonly() -> tuple[ladybug.Database, ladybug.Connection]:
if not DB_PATH.exists(): if not DB_PATH.exists():
raise FileNotFoundError(f"{DB_PATH} missing - run bin/brain/index.go --rebuild first") raise FileNotFoundError(f"{DB_PATH} missing - run bin/kb/index first")
db, conn = connect(read_only=True) db, conn = connect(read_only=True)
return db, conn return db, conn
-37
View File
@@ -1,37 +0,0 @@
"""Mail markdown under var/mail → info leafs. Conversion stays off the brain DB."""
from __future__ import annotations
import json
from pathlib import Path
from mdleaves import read_markdown, to_all
def msg_date(md: Path) -> str:
j = md.parent / "message.json"
try:
d = json.loads(j.read_text(encoding="utf-8"))
return (d.get("receivedDate") or d.get("receivedAt") or "")[:10]
except (OSError, json.JSONDecodeError, TypeError):
return ""
def from_mail_root(root: Path, limit: int = 0, since: str = "", repo: str = "ooMail") -> list[dict]:
if not root.is_dir():
return []
mds = sorted(root.rglob("message.md"))
if since:
mds = [m for m in mds if msg_date(m) >= since]
if limit:
mds = mds[:limit]
leafs: list[dict] = []
for md in mds:
files = [md] + sorted((md.parent / "attachments").glob("*.md"))
for f in files:
if not f.exists():
continue
for lf in to_all(read_markdown(f), f, repo=repo):
lf["source"] = f"ooMail:{md.parent.name}:{f.name}"
lf["how"] = "mail/import"
leafs.append(lf)
return leafs
-198
View File
@@ -1,198 +0,0 @@
"""D14 layout: bin/{subject}/{method}.go, libs in internal/, one go.mod."""
from __future__ import annotations
import os
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class BinLayoutTest(unittest.TestCase):
def test_brain_search_shebang_exists(self) -> None:
p = ROOT / "bin" / "brain" / "search.go"
self.assertTrue(p.is_file(), "missing bin/brain/search.go")
first = p.read_text().splitlines()[0]
self.assertTrue(
first.startswith("//usr/bin/env go run"),
f"shebang first line, got {first!r}",
)
def test_no_nested_go_mod_under_bin(self) -> None:
nested = list((ROOT / "bin").rglob("go.mod"))
self.assertEqual(nested, [], f"nested go.mod files: {nested}")
def test_rank_lives_in_internal_brain(self) -> None:
self.assertTrue(
(ROOT / "internal" / "brain" / "rank" / "rank.go").is_file(),
"ranking must live in internal/brain/rank (cgo-free)",
)
self.assertFalse(
(ROOT / "bin" / "kbsearch").exists(),
"bin/kbsearch nested module must be gone",
)
def test_no_main_go_under_bin_brain(self) -> None:
main = ROOT / "bin" / "brain" / "main.go"
self.assertFalse(main.exists(), "bin/brain/main.go is not a method")
def test_chats_methods_are_shebangs_not_main(self) -> None:
chats = ROOT / "bin" / "chats"
self.assertFalse(
(chats / "main.go").exists(),
"bin/chats/main.go is a dispatcher, not a method",
)
self.assertFalse(
(chats / "index_cmd.go").exists(),
"chats index is a brain write hiding under the wrong subject",
)
for method in ("sync.go", "import.go", "facts.go", "apply.go"):
p = chats / method
self.assertTrue(p.is_file(), f"missing bin/chats/{method}")
first = p.read_text().splitlines()[0]
self.assertTrue(
first.startswith("//usr/bin/env go run"),
f"{method} shebang, got {first!r}",
)
def test_chats_lib_lives_in_internal(self) -> None:
self.assertTrue(
(ROOT / "internal" / "chats" / "linkedin.go").is_file(),
"LinkedIn parser must live in internal/chats",
)
self.assertFalse(
(ROOT / "bin" / "chats" / "linkedin.go").exists(),
"parser must not stay under bin/chats as a second main",
)
def _assert_shebang(self, rel: str) -> None:
p = ROOT / rel
self.assertTrue(p.is_file(), f"missing {rel}")
first = p.read_text().splitlines()[0]
self.assertTrue(
first.startswith("//usr/bin/env go run"),
f"{rel} shebang, got {first!r}",
)
def test_brain_methods_are_shebangs(self) -> None:
for method in ("index.go", "get.go", "stats.go", "eval.go", "watch.go"):
self._assert_shebang(f"bin/brain/{method}")
def test_brain_get_stats_eval_are_not_python_exec(self) -> None:
for method in ("get.go", "stats.go", "eval.go"):
text = (ROOT / "bin" / "brain" / method).read_text()
self.assertNotIn(
"ExecFile",
text,
f"bin/brain/{method} must call internal/brain, not ExecFile Python",
)
self.assertNotIn(
"cmdbin",
text,
f"bin/brain/{method} must not import internal/cmdbin",
)
self.assertIn(
"system_ladybug",
text.splitlines()[0],
f"bin/brain/{method} shebang must pass -tags=system_ladybug",
)
self.assertIn(
"github.com/eSlider/2dph/internal/brain",
text,
)
def test_eval_control_questions_live_in_rank(self) -> None:
rank = (ROOT / "internal" / "brain" / "rank" / "evalq.go").read_text()
py = (ROOT / "bin" / "kb" / "eval").read_text()
for frag in ("BM25", "DevOps", "LadybugDB"):
self.assertIn(frag, rank)
self.assertIn(frag, py)
self.assertIn("0.95", rank)
def test_facts_methods_are_shebangs(self) -> None:
for method in ("audit.go", "extract.go", "crm.go"):
self._assert_shebang(f"bin/facts/{method}")
text = (ROOT / "bin" / "facts" / method).read_text()
self.assertIn("cmdbin.ExecFile", text)
self.assertIn(f"bin/facts/{method.removesuffix('.go')}", text)
def test_mail_import_is_shebang_not_brain_write(self) -> None:
self._assert_shebang("bin/mail/import.go")
index_mail = (ROOT / "bin" / "mail" / "index_mail").read_text()
self.assertIn(
"bin/brain/index.go",
index_mail,
"index_mail must point at bin/brain/index.go",
)
def test_markdown_import_is_go_not_python_exec(self) -> None:
self._assert_shebang("bin/markdown/import.go")
text = (ROOT / "bin" / "markdown" / "import.go").read_text()
self.assertNotIn("ExecFile", text)
self.assertNotIn("cmdbin", text)
self.assertIn("internal/mdleaves", text)
self.assertNotIn("kb.lbug", text)
def test_import_adapters_do_not_write_ladybug(self) -> None:
for rel in (
"bin/mail/import.go",
"bin/mail/import",
"bin/markdown/import.go",
"bin/chats/import.go",
"bin/git/import.go",
):
text = (ROOT / rel).read_text()
self.assertNotIn("upsert_leaf", text, rel)
self.assertNotIn("kb.lbug", text, rel)
self.assertNotIn("var/brain.lbug", text, rel)
index = (ROOT / "bin" / "brain" / "index.go").read_text()
self.assertIn("bin/kb/index", index)
def test_postgres_query_is_shebang(self) -> None:
self._assert_shebang("bin/postgres/query.go")
def test_git_import_is_gogit_shebang(self) -> None:
self._assert_shebang("bin/git/import.go")
py = (ROOT / "bin" / "git" / "import").read_text()
self.assertNotIn(
'["git"',
py,
"Python git/import must not subprocess the git binary",
)
self.assertIn("bin/git/import.go", py)
def test_web_search_is_shebang(self) -> None:
self._assert_shebang("bin/web/search.go")
py = (ROOT / "bin" / "web" / "search").read_text()
self.assertIn("bin/web/search.go", py)
def test_gitimport_py_has_no_git_binary(self) -> None:
py = (ROOT / "bin" / "tools" / "gitimport.py").read_text()
self.assertNotIn("subprocess", py)
self.assertNotIn("git log", py)
def test_gogit_is_direct_go_mod_require(self) -> None:
text = (ROOT / "go.mod").read_text()
first = text.split("require (")[1].split(")")[0]
self.assertRegex(first, r"github.com/go-git/go-git/v5\s+v")
for line in first.splitlines():
if "go-git/go-git" in line:
self.assertNotIn("indirect", line)
def test_cgo_uses_zig_not_gcc(self) -> None:
for rel in ("bin/cgo/zig", "bin/cgo/zcc", "bin/cgo/zc++"):
p = ROOT / rel
self.assertTrue(p.is_file(), f"missing {rel}")
self.assertTrue(
os.access(p, os.X_OK),
f"{rel} must be executable",
)
zig = (ROOT / "bin" / "cgo" / "zig").read_text()
self.assertIn("zig cc", zig)
self.assertIn("0.14.1", zig)
zcc = (ROOT / "bin" / "cgo" / "zcc").read_text()
self.assertIn('exec "$ZIG" cc', zcc)
self.assertNotIn("command -v gcc", zcc)
search = (ROOT / "bin" / "kb" / "search").read_text()
self.assertIn("bin/cgo/zig", search)
self.assertNotIn("command -v gcc", search)
+10 -13
View File
@@ -9,6 +9,12 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
import kblib # noqa: E402 import kblib # noqa: E402
import gitimport # noqa: E402 import gitimport # noqa: E402
SAMPLE = (
"\x1e" + "a1b2c3d" + "\x1f" + "Ada Lovelace" + "\x1f" + "ada@example.com"
+ "\x1f" + "2026-08-10T12:00:00+01:00" + "\x1f" + "feat: first commit"
+ "\n\nREADME.md\nsrc/main.c\n"
)
COMMIT_PERSON_SCHEMA = ( COMMIT_PERSON_SCHEMA = (
"CREATE NODE TABLE IF NOT EXISTS Commit (id STRING, repo STRING, subject STRING, " "CREATE NODE TABLE IF NOT EXISTS Commit (id STRING, repo STRING, subject STRING, "
"author STRING, email STRING, date STRING, PRIMARY KEY(id))" "author STRING, email STRING, date STRING, PRIMARY KEY(id))"
@@ -20,17 +26,6 @@ HAS_VERSION_SCHEMA = "CREATE REL TABLE IF NOT EXISTS HAS_VERSION (FROM File TO C
AUTHORED_SCHEMA = "CREATE REL TABLE IF NOT EXISTS AUTHORED (FROM Commit TO Person)" AUTHORED_SCHEMA = "CREATE REL TABLE IF NOT EXISTS AUTHORED (FROM Commit TO Person)"
def sample_commit() -> gitimport.Commit:
return gitimport.Commit(
sha="a1b2c3d",
author="Ada Lovelace",
email="ada@example.com",
date="2026-08-10T12:00:00+01:00",
subject="feat: first commit",
files=["README.md", "src/main.c"],
)
class GitGraphTest(unittest.TestCase): class GitGraphTest(unittest.TestCase):
def setUp(self): def setUp(self):
self.dir = tempfile.mkdtemp() self.dir = tempfile.mkdtemp()
@@ -47,12 +42,14 @@ class GitGraphTest(unittest.TestCase):
self.db.close() self.db.close()
def test_index_commits_creates_nodes_and_edges(self): def test_index_commits_creates_nodes_and_edges(self):
gitimport.index_commits(self.conn, [sample_commit()], "sample-repo") cs = gitimport.parse_log(SAMPLE)
gitimport.index_commits(self.conn, cs, "sample-repo")
rp = self.conn.execute("MATCH (p:Person) RETURN p.name, p.email").get_all() rp = self.conn.execute("MATCH (p:Person) RETURN p.name, p.email").get_all()
self.assertEqual([tuple(r) for r in rp], [("Ada Lovelace", "ada@example.com")]) self.assertEqual([tuple(r) for r in rp], [("Ada Lovelace", "ada@example.com")])
rc = self.conn.execute("MATCH (c:Commit) RETURN c.id, c.repo").get_all() rc = self.conn.execute("MATCH (c:Commit) RETURN c.id, c.repo").get_all()
self.assertEqual(len(rc), 1) self.assertEqual(len(rc), 1)
self.assertEqual(rc[0][1], "sample-repo") self.assertEqual(rc[0][1], "sample-repo")
# File -[:HAS_VERSION]-> Commit -[:AUTHORED]-> Person
rf = self.conn.execute( rf = self.conn.execute(
"MATCH (f:File)-[:HAS_VERSION]->(c:Commit)-[:AUTHORED]->(p:Person) " "MATCH (f:File)-[:HAS_VERSION]->(c:Commit)-[:AUTHORED]->(p:Person) "
"RETURN f.path, c.id, p.email").get_all() "RETURN f.path, c.id, p.email").get_all()
@@ -61,7 +58,7 @@ class GitGraphTest(unittest.TestCase):
self.assertTrue(all(r[2] == "ada@example.com" for r in rf)) self.assertTrue(all(r[2] == "ada@example.com" for r in rf))
def test_index_commits_idempotent(self): def test_index_commits_idempotent(self):
cs = [sample_commit()] cs = gitimport.parse_log(SAMPLE)
gitimport.index_commits(self.conn, cs, "sample-repo") gitimport.index_commits(self.conn, cs, "sample-repo")
gitimport.index_commits(self.conn, cs, "sample-repo") gitimport.index_commits(self.conn, cs, "sample-repo")
n = self.conn.execute("MATCH (c:Commit) RETURN count(*)").get_all()[0][0] n = self.conn.execute("MATCH (c:Commit) RETURN count(*)").get_all()[0][0]
+57
View File
@@ -0,0 +1,57 @@
import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import gitimport # noqa: E402
SAMPLE = (
"\x1e" + "a1b2c3d" + "\x1f" + "Ada Lovelace" + "\x1f" + "ada@example.com"
+ "\x1f" + "2026-08-10T12:00:00+01:00" + "\x1f" + "feat: first commit"
+ "\n\nREADME.md\nsrc/main.c\n"
+ "\x1e" + "e4f5a6b" + "\x1f" + "Bob Babbage" + "\x1f" + "bob@example.com"
+ "\x1f" + "2026-08-11T09:30:00+01:00" + "\x1f" + "fix: typo"
+ "\n\ndocs/notes.md"
)
class GitparseTest(unittest.TestCase):
def test_parses_records(self):
cs = gitimport.parse_log(SAMPLE)
self.assertEqual(len(cs), 2)
def test_parses_commit_fields(self):
cs = gitimport.parse_log(SAMPLE)
c = cs[0]
self.assertEqual(c.sha, "a1b2c3d")
self.assertEqual(c.author, "Ada Lovelace")
self.assertEqual(c.email, "ada@example.com")
self.assertEqual(c.date, "2026-08-10T12:00:00+01:00")
self.assertEqual(c.subject, "feat: first commit")
def test_parses_changed_files(self):
cs = gitimport.parse_log(SAMPLE)
self.assertEqual(cs[0].files, ["README.md", "src/main.c"])
self.assertEqual(cs[1].files, ["docs/notes.md"])
def test_ignores_empty(self):
self.assertEqual(gitimport.parse_log(""), [])
def test_skip_malformed_record(self):
self.assertEqual(gitimport.parse_log("\x1eweird\x1e"), [])
def test_commit_leaf_shape(self):
leafs = gitimport.commits_to_leafs(gitimport.parse_log(SAMPLE), "sample-repo")
self.assertEqual(len(leafs), 2)
lf = leafs[0]
self.assertEqual(lf["type"], "commit")
self.assertEqual(lf["repo"], "sample-repo")
self.assertEqual(lf["source"], "sample-repo@a1b2c3d")
self.assertIn("Ada Lovelace", lf["text"])
self.assertIn("README.md", lf["related"])
self.assertIn("feat: first commit", lf["heading"])
if __name__ == "__main__":
unittest.main()
-41
View File
@@ -1,41 +0,0 @@
"""Import adapters write files only. Index rebuild is brain/index (D14 / Gitea #7)."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class IndexAdapterTest(unittest.TestCase):
def test_dry_run_fixture_corpus_does_not_write_lbug(self) -> None:
tmp = Path(tempfile.mkdtemp())
(tmp / "note.md").write_text("# Fixture\n\n## Leaf\n\nhello corpus\n", encoding="utf-8")
lbug = tmp / "kb.lbug"
try:
import ladybug # noqa: F401
except ImportError:
venv_py = ROOT / ".venv" / "bin" / "python"
if not venv_py.is_file():
self.skipTest("ladybug missing")
py = str(venv_py)
else:
py = sys.executable
proc = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "index"), "--dry-run", "--json", "--corpus", str(tmp)],
cwd=ROOT,
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
msg = json.loads(proc.stdout)
self.assertTrue(msg.get("dry_run"))
self.assertGreaterEqual(msg.get("corpus_total", 0), 1)
self.assertFalse(lbug.exists(), "dry-run must not create a Ladybug file")
-46
View File
@@ -1,46 +0,0 @@
"""Mail markdown → leafs (no Ladybug). Brain index --with-mail uses this."""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import mailleafs # noqa: E402
class MailLeafsTest(unittest.TestCase):
def test_message_md_becomes_info_leaf(self) -> None:
root = Path(tempfile.mkdtemp())
msg = root / "inbox" / "alice-1"
msg.mkdir(parents=True)
(msg / "message.json").write_text(
json.dumps({"receivedDate": "2026-01-15T10:00:00Z", "subject": "Hello"}),
encoding="utf-8",
)
(msg / "message.md").write_text(
"---\nroot: info\n---\n\n# Hello\n\nFrom Alice to Bob.\n",
encoding="utf-8",
)
leafs = mailleafs.from_mail_root(root)
self.assertEqual(len(leafs), 1)
self.assertIn("Alice", leafs[0]["text"])
self.assertTrue(leafs[0]["source"].startswith("ooMail:"))
self.assertEqual(leafs[0]["how"], "mail/import")
def test_since_filters_by_message_json_date(self) -> None:
root = Path(tempfile.mkdtemp())
for name, day in (("old", "2025-01-01"), ("new", "2026-06-01")):
d = root / "inbox" / name
d.mkdir(parents=True)
(d / "message.json").write_text(
json.dumps({"receivedDate": f"{day}T00:00:00Z"}),
encoding="utf-8",
)
(d / "message.md").write_text(f"# {name}\n\nbody\n", encoding="utf-8")
leafs = mailleafs.from_mail_root(root, since="2026-01-01")
self.assertEqual(len(leafs), 1)
self.assertIn("new", leafs[0]["text"])
-157
View File
@@ -1,157 +0,0 @@
"""Published docs must match live commands (Gitea SoT, brain/search, no fake --hop)."""
from __future__ import annotations
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class PublishedDocsTest(unittest.TestCase):
def test_readme_points_issues_at_gitea(self) -> None:
text = (ROOT / "README.md").read_text()
self.assertIn(
"https://git.produktor.io/eSlider/2dph/issues",
text,
"README must point issues at Gitea",
)
def test_plan_d15_names_gitea_origin(self) -> None:
text = (ROOT / "PLAN.md").read_text()
self.assertIn("D15", text)
self.assertIn("git.produktor.io/eSlider/2dph", text)
def test_readme_primary_search_is_brain(self) -> None:
text = (ROOT / "README.md").read_text()
self.assertIn(
"bin/brain/search.go",
text,
"README deduction search must name bin/brain/search.go",
)
def test_readme_index_is_brain_not_index_mail(self) -> None:
text = (ROOT / "README.md").read_text()
self.assertIn("bin/brain/index.go", text)
self.assertNotIn(
"bin/mail/index_mail",
text,
"mail index is a brain write; README must name bin/brain/index.go",
)
def test_readme_git_import_is_gogit(self) -> None:
text = (ROOT / "README.md").read_text()
self.assertIn("bin/git/import.go", text)
self.assertIn("go-git", text)
self.assertIn("D19", (ROOT / "PLAN.md").read_text())
def test_web_search_is_go_not_ops_host(self) -> None:
readme = (ROOT / "README.md").read_text()
self.assertIn("bin/web/search.go", readme)
skill = (ROOT / "skills" / "web-search" / "SKILL.md").read_text()
self.assertIn("bin/web/search.go", skill)
self.assertNotIn("search.ops.io", skill)
self.assertNotIn("search.ops.io", readme)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn("searxng", compose)
self.assertNotIn("search.ops.io", compose)
settings = (ROOT / "deploy" / "searxng" / "settings.yml").read_text()
self.assertNotIn("password", settings.lower())
self.assertIn("json", settings)
def test_picoclaw_compose_profile_has_mcp_example(self) -> None:
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('profiles: ["picoclaw"]', compose)
self.assertIn("127.0.0.1:8630", compose)
example = (ROOT / "deploy" / "picoclaw" / "mcp.json.example").read_text()
self.assertIn("127.0.0.1:8630/mcp", example)
self.assertNotIn("password", example.lower())
self.assertNotIn("token", example.lower())
docs = (ROOT / "docs" / "picoclaw.md").read_text()
self.assertIn("search", docs)
self.assertIn("throttled", docs)
def test_readme_read_path_is_go(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("get.go", plan)
self.assertIn("CI fallback", plan)
design = (ROOT / "docs" / "design.md").read_text()
self.assertIn("internal/brain/rank", design)
self.assertIn("They do not exec Python", design)
def test_openapi_mcp_from_same_handlers(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D20", plan)
self.assertIn("/openapi.json", (ROOT / "README.md").read_text())
self.assertIn("/mcp", (ROOT / "README.md").read_text())
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
self.assertIn("/mcp", skill)
self.assertFalse((ROOT / "skills" / "db-yaml").exists())
self.assertTrue((ROOT / "skills" / "postgres" / "SKILL.md").is_file())
def test_cgo_zig_and_index_profile(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D21", plan)
self.assertIn("zig cc", plan)
dockerfile = (ROOT / "Dockerfile").read_text()
self.assertIn("bin/cgo/zcc", dockerfile)
self.assertIn("FROM debian:bookworm-slim AS api", dockerfile)
self.assertIn("FROM python:3.12-slim AS index", dockerfile)
api = dockerfile[dockerfile.index("FROM debian:bookworm-slim AS api") :]
self.assertNotIn("pip install", api)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('profiles: ["index"]', compose)
self.assertIn("target: api", compose)
def test_reasoner_docs_name_real_hf_ids_cpu_sidecar(self) -> None:
docs = (ROOT / "docs" / "reasoner.md").read_text()
for hf in (
"Qwen/Qwen3.5-9B",
"Qwen/Qwen3.6-27B",
"prism-ml/Bonsai-27B-gguf",
):
self.assertIn(hf, docs)
self.assertIn("no official qwen3.6-9b", docs.lower())
self.assertIn("OLLAMA_NUM_GPU", docs)
self.assertIn("rss_mb", docs)
self.assertIn("vram_mb", docs)
self.assertIn("3/3", docs)
self.assertIn("Do not claim 9B is better at tools", docs)
self.assertNotIn("Qwen/Qwen3.6-9B", docs)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D18", plan)
self.assertIn("Qwen/Qwen3.5-9B", plan)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('profiles: ["reasoner"]', compose)
self.assertIn("OLLAMA_NUM_GPU", compose)
self.assertIn("127.0.0.1:11435", compose)
dockerfile = (ROOT / "Dockerfile").read_text()
self.assertNotIn(".gguf", dockerfile.lower())
self.assertNotIn(".safetensors", dockerfile.lower())
api = dockerfile[dockerfile.index("FROM debian:bookworm-slim AS api") :]
self.assertNotIn("COPY models", api)
self.assertNotIn("qwen", api.lower())
def test_readme_search_escalates_web(self) -> None:
text = (ROOT / "README.md").read_text()
self.assertIn("--no-web", text)
self.assertIn("D17", (ROOT / "PLAN.md").read_text())
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
self.assertIn("`web` block", skill)
def test_docs_do_not_claim_hop_walks(self) -> None:
paths = [
ROOT / "README.md",
ROOT / "docs" / "design.md",
ROOT / "skills" / "brain" / "SKILL.md",
ROOT / "skills" / "diataxis-docs" / "SKILL.md",
]
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
# Honest "not implemented" notes must not match.
lie = re.compile(r"--hop (?:N|1).*(?:follow|walk)", re.I | re.S)
for path in paths:
text = path.read_text()
self.assertIsNone(
lie.search(text),
f"{path.relative_to(ROOT)} still claims --hop walks the graph",
)
-49
View File
@@ -1,49 +0,0 @@
"""Skills must name live commands; every bin/ path in SKILL.md must exist."""
from __future__ import annotations
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
BIN_PATH = re.compile(r"(bin/[A-Za-z0-9_./-]+)")
class SkillsTest(unittest.TestCase):
def test_db_yaml_renamed_to_postgres(self) -> None:
self.assertFalse(
(ROOT / "skills" / "db-yaml").exists(),
"skills/db-yaml must be skills/postgres",
)
self.assertTrue((ROOT / "skills" / "postgres" / "SKILL.md").is_file())
text = (ROOT / "skills" / "postgres" / "SKILL.md").read_text()
self.assertIn("bin/postgres/query.go", text)
self.assertNotIn("search.ops.io", text)
def test_every_bin_path_in_skills_exists(self) -> None:
missing: list[str] = []
for path in (ROOT / "skills").rglob("SKILL.md"):
text = path.read_text()
for m in BIN_PATH.finditer(text):
rel = m.group(1).rstrip(")`.,;")
candidate = ROOT / rel
if not candidate.exists():
missing.append(f"{path.relative_to(ROOT)}: {rel}")
self.assertEqual(missing, [], "skill bin paths must exist")
def test_brain_skill_lists_generated_tools(self) -> None:
tools = (ROOT / "skills" / "brain" / "tools.md").read_text()
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
self.assertIn("tools.md", skill)
for name in ("search", "get", "stats", "audit"):
self.assertIn(f"`{name}`", tools)
def test_picoclaw_lists_tool_order(self) -> None:
skill = (ROOT / "skills" / "picoclaw" / "SKILL.md").read_text()
agents = (ROOT / "AGENTS.md").read_text()
self.assertIn("**`search`**", skill)
self.assertIn("**`get`**", skill)
self.assertIn("**`audit`**", skill)
self.assertIn("throttled", skill.lower())
self.assertIn("not a negative finding", agents)
self.assertIn("Fact-check every", agents)
-33
View File
@@ -1,33 +0,0 @@
"""Every bin/ path named in skills/ must exist on disk."""
from __future__ import annotations
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
BIN_PATH = re.compile(r"\b(bin/[A-Za-z0-9_./-]+)")
class SkillsBinPathsTest(unittest.TestCase):
def test_agent_cost_skill_is_gone(self) -> None:
self.assertFalse(
(ROOT / "skills" / "agent-cost").exists(),
"skills/agent-cost documents bin/agents/cost which does not exist",
)
def test_brain_skill_replaces_kb_search(self) -> None:
self.assertTrue((ROOT / "skills" / "brain" / "SKILL.md").is_file())
self.assertFalse((ROOT / "skills" / "kb-search").exists())
def test_skill_bin_paths_exist(self) -> None:
missing: list[str] = []
for skill in sorted((ROOT / "skills").rglob("SKILL.md")):
text = skill.read_text()
for match in BIN_PATH.findall(text):
rel = match.rstrip("`'.,")
if rel.endswith(".go") or Path(rel).suffix == "" or Path(rel).suffix in {".go", ".py"}:
p = ROOT / rel
if not p.exists():
missing.append(f"{skill.relative_to(ROOT)}: {rel}")
self.assertEqual(missing, [], "SKILL.md names bin/ paths that do not exist")
+4 -4
View File
@@ -1,4 +1,4 @@
// Package watch polls corpus directories for changes and re-runs brain/index. // Package watch polls corpus directories for changes and re-runs bin/kb/index.
// //
// Port of the former bin/kb-watch bash script to an importable, testable Go // Port of the former bin/kb-watch bash script to an importable, testable Go
// package. Polls file mtimes (no inotify deps); cheap and reliable. // package. Polls file mtimes (no inotify deps); cheap and reliable.
@@ -18,8 +18,8 @@ import (
type Options struct { type Options struct {
Dirs []string Dirs []string
Interval time.Duration Interval time.Duration
// IndexCmd is the index command template. %s is replaced by the repo // IndexCmd is the kb/index command template. %s is replaced by the repo
// root (from KB_ROOT). Defaults to `python3 <root>/bin/kb/index --with-mail`. // root (from KB_ROOT). Defaults to `python3 <root>/bin/kb/index`.
IndexCmd string IndexCmd string
} }
@@ -67,7 +67,7 @@ func fromEnv(args []string) Options {
if pys == "" { if pys == "" {
pys = "python3" pys = "python3"
} }
opts.IndexCmd = pys + " <root>/bin/kb/index --with-mail" opts.IndexCmd = pys + " <root>/bin/kb/index"
return opts return opts
} }
+2 -6
View File
@@ -3,7 +3,6 @@ package watch
import ( import (
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"time" "time"
) )
@@ -44,10 +43,7 @@ func TestFromEnvDefaults(t *testing.T) {
if opts.Interval != 30*time.Second { if opts.Interval != 30*time.Second {
t.Fatalf("default interval = %s, want 30s", opts.Interval) t.Fatalf("default interval = %s, want 30s", opts.Interval)
} }
if !strings.Contains(opts.IndexCmd, "kb/index") { if opts.IndexCmd == "" {
t.Fatalf("default index cmd = %q, want kb/index", opts.IndexCmd) t.Fatal("default index cmd is empty")
}
if !strings.Contains(opts.IndexCmd, "--with-mail") {
t.Fatalf("default index cmd must include --with-mail, got %q", opts.IndexCmd)
} }
} }
+136 -12
View File
@@ -1,26 +1,150 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""web/search — deprecated. Use bin/web/search.go (SearXNG, no Python client). """web/search - web search through the self-hosted SearXNG at search.ops.io.
bin/web/search.go QUERY [--json] [-n N] [--site HOST] bin/web/search "LadybugDB vector search"
bin/web/search "model2vec multilingual" --site github.com
bin/web/search "uclancy" --category it -n 3 --json | jq -r '.results[].url'
bin/web/search "sqlite-vec" --refresh # ignore the cached answer
This complements bin/kb/search: the knowledge base holds our own facts, this
reaches the public web. Use it as the second, independent source that the
detective method asks for.
Exit codes: 0 results, 2 refused as possible PII, 3 throttled (not "nothing
found" - the instance answers 200 with an empty list when it throttles).
""" """
from __future__ import annotations from __future__ import annotations
import argparse
import fcntl
import json
import os import os
import sys import sys
import time
import urllib.parse
import urllib.request
from pathlib import Path from pathlib import Path
ROOT = Path(__file__).resolve().parents[2] TOOLS = Path(__file__).resolve().parents[1] / "tools"
sys.path.insert(0, str(TOOLS))
sys.path.insert(0, str(TOOLS / "web-search"))
import websearch as ws # noqa: E402
from yamlout import to_yaml # noqa: E402
CONFIG = Path(os.environ.get("BRAIN_SEARCH_ENV", Path.home() / ".config/brain/search.env"))
CACHE = Path(os.environ.get("BRAIN_SEARCH_CACHE", Path.home() / ".cache/brain/web-search.sqlite"))
LOCK = CACHE.with_suffix(".lock")
def main(argv: list[str]) -> int: def load_config() -> dict:
print( if not CONFIG.exists():
"bin/web/search is deprecated; use bin/web/search.go", sys.exit(f"no credentials at {CONFIG} (mode 600, BRAIN_SEARCH_URL/USER/PASS)")
file=sys.stderr, conf = {}
) for line in CONFIG.read_text().splitlines():
target = ROOT / "bin" / "web" / "search.go" line = line.strip()
os.execvp("go", ["go", "run", str(target), *argv]) if not line or line.startswith("#") or "=" not in line:
return 1 continue
key, _, value = line.partition("=")
conf[key.strip()] = value.strip().strip("\"'")
missing = {"BRAIN_SEARCH_URL", "BRAIN_SEARCH_USER", "BRAIN_SEARCH_PASS"} - conf.keys()
if missing:
sys.exit(f"{CONFIG} is missing {', '.join(sorted(missing))}")
return conf
def fetch(conf: dict, query: str, params: dict, timeout: int) -> dict:
args = {"q": query, "format": "json", **params}
url = f"{conf['BRAIN_SEARCH_URL'].rstrip('/')}/search?{urllib.parse.urlencode(args)}"
request = urllib.request.Request(url)
token = f"{conf['BRAIN_SEARCH_USER']}:{conf['BRAIN_SEARCH_PASS']}".encode()
import base64
request.add_header("Authorization", "Basic " + base64.b64encode(token).decode())
with urllib.request.urlopen(request, timeout=timeout) as response:
return json.loads(response.read().decode())
def main() -> int:
parser = argparse.ArgumentParser(description="web search via SearXNG")
parser.add_argument("query")
parser.add_argument("-n", "--limit", type=int, default=ws.DEFAULT_LIMIT)
parser.add_argument("--site", help="restrict to one domain")
parser.add_argument("--lang", help="language code, e.g. de")
parser.add_argument("--fresh", choices=["day", "week", "month", "year"],
help="time range")
parser.add_argument("--category", help="SearXNG category, e.g. it, science, news")
parser.add_argument("--engines", help="comma separated engine list")
parser.add_argument("--json", action="store_true")
parser.add_argument("--refresh", action="store_true", help="bypass the cache")
parser.add_argument("--ttl", type=float, default=ws.CACHE_TTL)
parser.add_argument("--timeout", type=int, default=25)
parser.add_argument("--force", action="store_true",
help="send even if the query looks like PII")
args = parser.parse_args()
query = f"site:{args.site} {args.query}" if args.site else args.query
reason = ws.phi_reason(query)
if reason and not args.force:
print(f"refused: {reason}. This query would leave the host.", file=sys.stderr)
print("Rephrase without identifiers, or pass --force if it is genuinely public.",
file=sys.stderr)
return 2
params = {}
if args.lang:
params["language"] = args.lang
if args.fresh:
params["time_range"] = args.fresh
if args.category:
params["categories"] = args.category
if args.engines:
params["engines"] = args.engines
key = ws.cache_key(query, params)
conn = ws.open_cache(CACHE)
if not args.refresh:
cached = ws.cache_get(conn, key, ttl=args.ttl)
if cached is not None:
out = ws.project(cached, limit=args.limit)
out["cached"] = True
sys.stdout.write(json.dumps(out, indent=2, ensure_ascii=False) + "\n"
if args.json else to_yaml(out))
return 0
conf = load_config()
LOCK.parent.mkdir(parents=True, exist_ok=True)
# One request at a time across every agent on this host: the instance
# suspends engines for minutes when several of us ask at once.
with open(LOCK, "w") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
payload = None
for attempt in range(1 + len(ws.RETRY_BACKOFF)):
delay = ws.wait_for(ws.last_call(conn), time.time())
if delay:
time.sleep(delay)
ws.mark_call(conn)
try:
payload = fetch(conf, query, params, args.timeout)
except Exception as error: # noqa: BLE001 - report, do not crash
print(f"request failed: {error}", file=sys.stderr)
return 3
if ws.classify(payload) == "ok":
break
if attempt < len(ws.RETRY_BACKOFF):
time.sleep(ws.RETRY_BACKOFF[attempt])
if ws.classify(payload) == "ok":
ws.cache_put(conn, key, payload)
out = ws.project(payload, limit=args.limit)
sys.stdout.write(json.dumps(out, indent=2, ensure_ascii=False) + "\n"
if args.json else to_yaml(out))
return 0 if out["status"] == "ok" else 3
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(main(sys.argv[1:])) sys.exit(main())
-232
View File
@@ -1,232 +0,0 @@
//usr/bin/env go run "$0" "$@"; exit
//
// bin/web/search.go - SearXNG as the second independent source (D3).
//
// ./bin/web/search.go "LadybugDB vector search"
// ./bin/web/search.go "model2vec" --category it --json
// ./bin/web/search.go "postgres" --site github.com --fresh year
//
// Empty results mean throttled, not "nothing exists". Exit 2 = PII refuse, 3 = throttled.
// Config: $BRAIN_SEARCH_ENV (default $HOME/.config/brain/search.env).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"encoding/json"
"fmt"
"net/http"
"os"
"strconv"
"time"
"github.com/eSlider/2dph/internal/websearch"
"golang.org/x/sys/unix"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
var (
query, site, lang, fresh, category, engines string
limit = websearch.DefaultLimit
jsonOut, refresh, force bool
ttl = float64(websearch.CacheTTL)
timeout = 25
)
i := 0
for i < len(args) {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--refresh":
refresh = true
case a == "--force":
force = true
case (a == "-n" || a == "--limit") && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 0 {
fmt.Fprintln(os.Stderr, "web/search: --limit must be a non-negative integer")
return 2
}
limit = n
case a == "--site" && i+1 < len(args):
i++
site = args[i]
case a == "--lang" && i+1 < len(args):
i++
lang = args[i]
case a == "--fresh" && i+1 < len(args):
i++
fresh = args[i]
case a == "--category" && i+1 < len(args):
i++
category = args[i]
case a == "--engines" && i+1 < len(args):
i++
engines = args[i]
case a == "--ttl" && i+1 < len(args):
i++
v, err := strconv.ParseFloat(args[i], 64)
if err != nil {
fmt.Fprintln(os.Stderr, "web/search: --ttl must be a number")
return 2
}
ttl = v
case a == "--timeout" && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n <= 0 {
fmt.Fprintln(os.Stderr, "web/search: --timeout must be a positive integer")
return 2
}
timeout = n
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/web/search.go QUERY [--json] [-n N] [--site HOST] [--lang LANG] [--fresh day|week|month|year] [--category CAT] [--engines LIST] [--refresh] [--force]`)
return 0
case len(a) > 0 && a[0] != '-' && query == "":
query = a
default:
fmt.Fprintf(os.Stderr, "web/search: unknown flag %s\n", a)
return 2
}
i++
}
if query == "" {
fmt.Fprintln(os.Stderr, "web/search: query required")
return 2
}
if site != "" {
query = "site:" + site + " " + query
}
if reason := websearch.PHIReason(query); reason != "" && !force {
fmt.Fprintf(os.Stderr, "refused: %s. This query would leave the host.\n", reason)
fmt.Fprintln(os.Stderr, "Rephrase without identifiers, or pass --force if it is genuinely public.")
return 2
}
params := map[string]string{}
if lang != "" {
params["language"] = lang
}
if fresh != "" {
params["time_range"] = fresh
}
if category != "" {
params["categories"] = category
}
if engines != "" {
params["engines"] = engines
}
cachePath := os.Getenv("BRAIN_SEARCH_CACHE")
if cachePath == "" {
cachePath = os.Getenv("HOME") + "/.cache/brain/web-search.sqlite"
}
cache, err := websearch.OpenCache(cachePath)
if err != nil {
fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err)
return 1
}
defer cache.Close()
key := websearch.CacheKey(query, params)
now := float64(time.Now().Unix())
if !refresh {
if cached, err := cache.Get(key, ttl, now); err != nil {
fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err)
return 1
} else if cached != nil {
out := websearch.Project(*cached, limit, websearch.DefaultSnippetChars)
out.Cached = true
return writeOut(out, jsonOut)
}
}
envPath := os.Getenv("BRAIN_SEARCH_ENV")
if envPath == "" {
envPath = os.Getenv("HOME") + "/.config/brain/search.env"
}
conf, err := websearch.LoadConfig(envPath)
if err != nil {
fmt.Fprintf(os.Stderr, "web/search: %v\n", err)
return 1
}
lockPath := cachePath + ".lock"
lock, err := os.OpenFile(lockPath, os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
fmt.Fprintf(os.Stderr, "web/search: lock: %v\n", err)
return 1
}
defer lock.Close()
if err := unix.Flock(int(lock.Fd()), unix.LOCK_EX); err != nil {
fmt.Fprintf(os.Stderr, "web/search: lock: %v\n", err)
return 1
}
defer unix.Flock(int(lock.Fd()), unix.LOCK_UN)
var payload websearch.Payload
attempts := 1 + len(websearch.RetryBackoff)
client := &http.Client{}
for attempt := 0; attempt < attempts; attempt++ {
last, err := cache.LastCall()
if err != nil {
fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err)
return 1
}
if delay := websearch.WaitFor(last, float64(time.Now().Unix()), websearch.MinInterval); delay > 0 {
time.Sleep(time.Duration(delay * float64(time.Second)))
}
if err := cache.MarkCall(float64(time.Now().Unix())); err != nil {
fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err)
return 1
}
payload, err = websearch.Fetch(client, conf, query, params, time.Duration(timeout)*time.Second)
if err != nil {
fmt.Fprintf(os.Stderr, "request failed: %v\n", err)
return 3
}
if websearch.Classify(payload) == websearch.StatusOK {
break
}
if attempt < len(websearch.RetryBackoff) {
time.Sleep(time.Duration(websearch.RetryBackoff[attempt] * float64(time.Second)))
}
}
if websearch.Classify(payload) == websearch.StatusOK {
if err := cache.Put(key, payload, float64(time.Now().Unix())); err != nil {
fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err)
}
}
out := websearch.Project(payload, limit, websearch.DefaultSnippetChars)
code := writeOut(out, jsonOut)
if out.Status != websearch.StatusOK && code == 0 {
return 3
}
return code
}
func writeOut(out websearch.Output, jsonOut bool) int {
if jsonOut {
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
enc.SetEscapeHTML(false)
if err := enc.Encode(out); err != nil {
return 1
}
if out.Status != websearch.StatusOK {
return 3
}
return 0
}
fmt.Print(out.YAML())
if out.Status != websearch.StatusOK {
return 3
}
return 0
}
+17 -88
View File
@@ -1,137 +1,66 @@
# 2dph — docker composition # 2dph — docker composition
# #
# docker compose up -d brain # API (Zig CGO serve) # docker compose run --rm brain index # rebuild graph
# docker compose --profile index run --rm index # Python rebuild # docker compose run --rm brain search "Matrix fed" # one-shot query
# docker compose --profile picoclaw up brain-mcp # docker compose run --rm brain serve # async Go server
# docker compose --profile reasoner up -d reasoner # CPU Ollama :11435 # docker compose up brain-watch # auto re-index
# docker compose --profile searxng up -d
# #
# Secrets never baked in: search.env + db-profiles.yml from ~/.config/brain. # Caching: the 128M model (HF_HOME) and kb.lbug (VAR_DIR) live in named
# volumes, so rebuilds never redownload the model or re-derive the graph.
# Secrets are never baked into the image: search.env + db-profiles.yml mount
# read-only from ~/.config/brain.
name: 2dph name: 2dph
services: services:
brain: brain:
image: ghcr.io/eslider/2dph:api image: ghcr.io/eslider/2dph:latest
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
target: api
cache_from: cache_from:
- ghcr.io/eslider/2dph:cache - ghcr.io/eslider/2dph:cache
command: ["serve"] command: ["brain", "search", "help"]
environment: &env environment: &env
HF_HOME: /data/hf HF_HOME: /data/hf
BRAIN_SEARCH_CACHE: /data/cache/web-search.sqlite BRAIN_SEARCH_CACHE: /data/cache/web-search.sqlite
BRAIN_DB_PROFILES: /secret/db-profiles.yml BRAIN_DB_PROFILES: /secret/db-profiles.yml
BRAIN_SEARCH_ENV: /secret/search.env BRAIN_SEARCH_ENV: /secret/search.env
KB_ROOT: /data KB_SEARCH_CMD: /app/bin/kb/search
KB_WORKERS: "4" KB_WORKERS: "4"
KB_PORT: "8630"
volumes: volumes:
- kb-model:/data/hf - kb-model:/data/hf
- kb-var:/data - kb-var:/data
# corpus is read-only on the host, never written from the container
- ..:/corpus:ro
- ~/.config/brain:/secret:ro - ~/.config/brain:/secret:ro
ports:
- "127.0.0.1:8630:8630"
read_only: true read_only: true
tmpfs: tmpfs:
- /tmp - /tmp
healthcheck: healthcheck:
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8630/health"] test: ["CMD", "python3", "-c", "import ladybug, model2vec, mistune; print('ok')"]
interval: 30s interval: 30s
timeout: 5s timeout: 5s
retries: 3 retries: 3
restart: unless-stopped restart: unless-stopped
stop_grace_period: 20s stop_grace_period: 20s
# watcher: re-index on corpus file change (watchdog script)
brain-watch: brain-watch:
image: ghcr.io/eslider/2dph:api image: ghcr.io/eslider/2dph:latest
environment: *env environment: *env
volumes: volumes:
- kb-model:/data/hf - kb-model:/data/hf
- kb-var:/data - kb-var:/data
- ..:/corpus:ro - ..:/corpus:ro
- ~/.config/brain:/secret:ro - ~/.config/brain:/secret:ro
command: ["watch", "/corpus"] command: ["brain", "watch", "/corpus"]
read_only: true read_only: true
tmpfs: tmpfs:
- /tmp - /tmp
restart: unless-stopped restart: unless-stopped
stop_grace_period: 20s stop_grace_period: 20s
# Python write path (Ladybug rebuild). Not in the API image.
# docker compose --profile index run --rm index
index:
profiles: ["index"]
image: ghcr.io/eslider/2dph:index
build:
context: .
dockerfile: Dockerfile
target: index
environment:
HF_HOME: /data/hf
KB_PY: python3
volumes:
- kb-model:/data/hf
- kb-var:/app/var
- ..:/corpus:ro
- ~/.config/brain:/secret:ro
command: ["index"]
read_only: true
tmpfs:
- /tmp
# Optional local SearXNG (D3). Skip if BRAIN_SEARCH_URL already points at a
# live instance — do not run a second copy on that host.
# SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d
searxng:
profiles: ["searxng"]
image: docker.io/searxng/searxng:2026.8.10-0a118066d
ports:
- "127.0.0.1:8888:8080"
environment:
SEARXNG_SECRET: ${SEARXNG_SECRET:-}
volumes:
- ./deploy/searxng/settings.yml:/etc/searxng/settings.yml:ro
- ./deploy/searxng/limiter.toml:/etc/searxng/limiter.toml:ro
restart: unless-stopped
# MCP endpoint for an external agent (PicoClaw is not shipped here).
# docker compose --profile picoclaw up brain-mcp
brain-mcp:
profiles: ["picoclaw"]
image: ghcr.io/eslider/2dph:api
environment: *env
volumes:
- kb-model:/data/hf
- kb-var:/data
- ~/.config/brain:/secret:ro
command: ["serve"]
ports:
- "127.0.0.1:8630:8630"
read_only: true
tmpfs:
- /tmp
restart: unless-stopped
# CPU OpenAI-compatible sidecar (D18). Weights are pulled at runtime, not
# baked into the 2dph image. Does not touch host Ollama on :11434.
# docker compose --profile reasoner up -d reasoner
# docker compose --profile reasoner exec reasoner ollama pull qwen3.5:9b
reasoner:
profiles: ["reasoner"]
image: docker.io/ollama/ollama:latest
environment:
OLLAMA_NUM_GPU: "0"
OLLAMA_HOST: "0.0.0.0:11434"
ports:
- "127.0.0.1:11435:11434"
volumes:
- reasoner-ollama:/root/.ollama
restart: unless-stopped
volumes: volumes:
kb-model: kb-model:
kb-var: kb-var:
reasoner-ollama:
-8
View File
@@ -1,8 +0,0 @@
{
"mcpServers": {
"2dph": {
"url": "http://127.0.0.1:8630/mcp",
"description": "2dph fact gate. Tool order: search → get → audit. throttled is not absence."
}
}
}
-7
View File
@@ -1,7 +0,0 @@
[botdetection.ip_lists]
# RFC1918 only. Do not copy a live instance egress IP into git.
pass_ip = [
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16",
]
-28
View File
@@ -1,28 +0,0 @@
use_default_settings: true
general:
instance_name: "2dph"
search:
formats:
- html
- json
suspended_times:
SearxEngineCaptcha: 300
SearxEngineTooManyRequests: 120
SearxEngineAccessDenied: 300
server:
limiter: true
image_proxy: false
# secret_key comes from SEARXNG_SECRET (never commit a real secret)
engines:
- name: bing
disabled: false
- name: google
disabled: false
- name: duckduckgo
disabled: false
- name: wikipedia
disabled: false
-6
View File
@@ -6,11 +6,5 @@ Brain/ops/eSlider stack. Facts need proof or they are
- [PLAN.md](../PLAN.md) — decisions, execution order, open questions (v2) - [PLAN.md](../PLAN.md) — decisions, execution order, open questions (v2)
- [design](design.md) — schema, deduction model, sources - [design](design.md) — schema, deduction model, sources
- [reasoner](reasoner.md) — D18 CPU bake-off (Qwen3.5-9B vs Bonsai / Qwen3.6-27B)
- [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) — work board (origin)
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is
not a walk; the flag errors until File/FROM_FILE edges exist.
Published docs live here and mirror the project state. Published docs live here and mirror the project state.
+5 -6
View File
@@ -15,10 +15,9 @@ OO_CLI (default: $HOME/go/bin/oo)
## Quick reference ## Quick reference
``` ```
./bin/chats/sync.go telegram --limit 100 ./bin/chat sync telegram --limit 100
./bin/chats/import.go ./bin/chat import
./bin/chats/facts.go ./bin/chat index
./bin/chats/apply.go --dry-run ./bin/chat facts
./bin/chat apply --dry-run
``` ```
JSONL → markdown only. Brain ingest is `bin/brain/index.go` (not a `chats index`).
-33
View File
@@ -1,33 +0,0 @@
# CRM association proof (oo CLI ↔ corpus)
Proven with `oo` (eslider/go-onlyoffice) against the OnlyOffice portal
(`office.produktor.io`). Portal CRM is the SSOT for company ↔ person ↔
project associations; the corpus SoT (`eslider/cv/projects/knowledge-mesh-seed.yaml`)
is the second, independent source. Facts that can be backed by both are
written to the brain under `root=facts` by `bin/facts/crm`.
## What was verified
- Logical counts (portal MySQL): 1300 contacts = 897 persons + 404 companies,
198 projects, 998 deals, 939 project↔contact links.
- Every client company linked to a project has ≥1 person underneath.
- Every person `company_id` resolves to an existing company.
- Corpus org list (9) maps 1:1 onto CRM companies:
ProProdukt SL / produktor.io, Dyvenia, Immowelt AG, WhereGroup,
Keynote SIGOS, D2S/SYSTEMS, GRID, Pack und Cup, Markets Platform.
- 78 person↔company association facts written to the brain
(`how=crm-crosscheck`, `type=association`). Recall@5 in `bin/kb/eval` = 1.0.
## Mistakes found
| # | Mistake | Fix |
|---|---------|-----|
| 1 | Duplicate legal entity `GoldenRatio.Exchange` (contact 759) vs `Golden Ratio Exchange` (763); 3 deals (211, 287, 559) were linked to 759 | `oo contacts merge 759 763` — 763 kept, 759 removed, deal links re-pointed to 763 |
| 2 | `env/`-wide: OnlyOffice creds file used wrong UX (user `eslider`, password with `$2` suffix) making `oo` auth fail | `.env` fixed to `eslider@gmail.com` + clean password; `.env` stays gitignored |
## Gates after fix
- `uv run python -m unittest discover -s bin/tools -t .` → 26 tests OK
- `bin/facts/audit self` + `bin/facts/audit db` → ok
- `bin/kb/eval` → recall@5 = 1.0
- `go test ./...` (bin/server + bin/watch) → ok
+4 -32
View File
@@ -17,16 +17,14 @@ as one consistent state.
## Deduction search ## Deduction search
``` ```
bin/brain/search.go "question" bin/kb/search "question"
1. facts root — confirmed answers only → return with evidence links 1. facts root — confirmed answers only → return with evidence links
2. info root — supporting narrative → snippets, marked (not confirmed) 2. info root — supporting narrative → snippets, marked (not confirmed)
3. web-search — second independent source → upgrade hypothesis to confirmed 3. web-search — second independent source → upgrade hypothesis to confirmed
(`web` block from `bin/web/search.go` when no facts hit; status `throttled`
is not evidence of absence; `--no-web` / `--root` skip it)
``` ```
`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an `--hop N` follows graph edges (sibling leaves under a heading, owning file,
error; it is not a graph walk. `related:` files, vector-neighbour leaves) — the deduction walk.
## Who / What / How / Where / When + evidence ## Who / What / How / Where / When + evidence
@@ -48,8 +46,7 @@ Every assertion edge carries:
Content leafs: `sha256`, `observed_at`, `source_rev`, `confidence`. Stale = a Content leafs: `sha256`, `observed_at`, `source_rev`, `confidence`. Stale = a
file changed on disk (git HEAD/mtime) after its last observed `source_rev`. file changed on disk (git HEAD/mtime) after its last observed `source_rev`.
`File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person` records the history of every `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person` records the history of every
content leaf. Commit records come from `bin/git/import.go` (go-git, no git content leaf.
binary); conversion prints leafs, brain write is `bin/brain/index.go`.
`bin/facts/audit stale` flags leafs whose observed revision is behind the `bin/facts/audit stale` flags leafs whose observed revision is behind the
corpus HEAD. corpus HEAD.
@@ -62,28 +59,3 @@ corpus HEAD.
Confirmed = A×B or B×C agreement. Single source = hypothesis + `(not confirmed)`. Confirmed = A×B or B×C agreement. Single source = hypothesis + `(not confirmed)`.
Conflicting pairings (≥2 yes vs ≥2 no) = hypothesis (OQ1 → v2 resolution). Conflicting pairings (≥2 yes vs ≥2 no) = hypothesis (OQ1 → v2 resolution).
## Read path
`bin/brain/get.go`, `stats.go`, and `eval.go` call `internal/brain` with cgo
(`system_ladybug`), compiled by **Zig** (`bin/cgo/zcc`, D21), not gcc.
They do not exec Python. Control questions for recall@5 live in
`internal/brain/rank` so CI can test the table without libladybug.
Python `bin/kb/{get,stats,eval}` remain for GitHub Actions until the runner
fetches Zig + libs (`bin/cgo/zig`). Index/write is still `bin/kb/index`
(`docker compose --profile index`).
## Agent API (D20)
`bin/brain/serve.go` exposes the same `internal/httpapi.Ops` table as OpenAPI
(`GET /openapi.json`) and MCP (`POST /mcp` JSON-RPC `tools/list` +
`tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`.
Agents should use these endpoints instead of shebang CLIs.
## Reasoner (D18)
Pluggable OpenAI-compatible URL. RAM: `Qwen/Qwen3.5-9B`. Quality:
`prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B.
CPU sidecar: compose profile `reasoner` (`OLLAMA_NUM_GPU=0`,
`127.0.0.1:11435`). Bake-off: `bin/reasoner/bakeoff.go`. Weights stay out
of the 2dph image. See [docs/reasoner.md](reasoner.md).
-18
View File
@@ -1,18 +0,0 @@
# PicoClaw profile (reference agent)
2dph is the memory/fact gate. PicoClaw (or any MCP client) is the agent loop
and is **not** shipped in this repo.
```bash
docker compose --profile picoclaw up brain-mcp
```
The API listens on `127.0.0.1:8630`. Point the agent at
`http://127.0.0.1:8630/mcp` using [deploy/picoclaw/mcp.json.example](../deploy/picoclaw/mcp.json.example).
OpenAPI: `GET http://127.0.0.1:8630/openapi.json`.
Before a factual reply: `search``get``audit`. `throttled` is not a
negative finding. See `skills/picoclaw/SKILL.md`.
No Cursor required. A live PicoClaw binary/image is an operator choice.
-69
View File
@@ -1,69 +0,0 @@
# Reasoner bake-off (D18)
Pluggable OpenAI-compatible URL. 2dph does not ship weights. PicoClaw is
not in this repo; the bake-off hits the same tool names PicoClaw would
(`search``get``audit` from `internal/httpapi.Ops`).
```bash
docker compose --profile reasoner up -d reasoner
docker compose --profile reasoner exec reasoner ollama pull qwen3.5:9b
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \
./bin/reasoner/bakeoff.go --json
```
Host Ollama on `:11434` is left alone. This sidecar binds `127.0.0.1:11435`
with `OLLAMA_NUM_GPU=0` (CPU). Measure RSS (`/api/ps` `size`), not VRAM.
If Compose cannot allocate a project network (Docker IPAM pool exhausted),
the same sidecar is:
```bash
docker run -d --name 2dph-reasoner \
-e OLLAMA_NUM_GPU=0 \
-p 127.0.0.1:11435:11434 \
-v 2dph-reasoner-ollama:/root/.ollama \
ollama/ollama:latest
```
## Real Hugging Face ids
| Role | HF id | Ollama tag (this bake-off) |
|------|-------|----------------------------|
| RAM / 9B | `Qwen/Qwen3.5-9B` | `qwen3.5:9b` |
| Quality 27B (CPU) | `prism-ml/Bonsai-27B-gguf` (derived from Qwen3.6-27B) | `MichelRosselli/bonsai-27b:Q1_0` |
| Quality 27B (full) | `Qwen/Qwen3.6-27B` | not pulled on this CPU box |
There is **no official Qwen3.6-9B**. Do not invent that id.
Qwen3.5-9B has documented upstream tool-call XML bugs. A 9B win on tools
is only claimed if this bake-off records OpenAI `tool_calls` (not
`<tool_call>` XML in `content`).
The 2dph API image does not `COPY` GGUF/safetensors. Pull at runtime into
the `reasoner-ollama` volume.
## Live CPU run
Host sidecar: Ollama **0.32.9**, `OLLAMA_NUM_GPU=0`, `127.0.0.1:11435`,
`device: cpu`, `vram_mb: 0`. Date: 2026-08-13. Same three prompts
(`search` / `get` / `audit`). PicoClaw binary was not used; the OpenAI
tools payload is the surface it would send.
| Model | HF id | tool_call | xml_leak | rss_mb | latency_ms (search/get/audit) |
|-------|-------|-----------|----------|--------|-------------------------------|
| `qwen3.5:9b` | `Qwen/Qwen3.5-9B` | 3/3 | 0 | 5790 | 50059 / 51678 / 31980 |
| `MichelRosselli/bonsai-27b:Q1_0` | `prism-ml/Bonsai-27B-gguf` | 3/3 | 0 | 21951 | 327702 / 166830 / 119808 |
| `Qwen/Qwen3.6-27B` | `Qwen/Qwen3.6-27B` | not loaded | — | — | too heavy for this CPU box |
Both loaded models emitted OpenAI `tool_calls` (not `<tool_call>` XML) on
this runtime. **Do not claim 9B is better at tools** — the score is tied
at 3/3. 9B is smaller and faster. Bonsai RSS includes weights + KV
(`size` from `/api/ps`); first Bonsai prompt includes cold load.
Re-run:
```bash
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \
./bin/reasoner/bakeoff.go --json
REASONER_MODEL=MichelRosselli/bonsai-27b:Q1_0 ./bin/reasoner/bakeoff.go --json
```
+1 -45
View File
@@ -1,52 +1,8 @@
module github.com/eSlider/2dph module github.com/eSlider/2dph
go 1.26 go 1.25.0
require ( require (
github.com/LadybugDB/go-ladybug v0.17.0
github.com/arran4/golang-ical v0.3.5 github.com/arran4/golang-ical v0.3.5
github.com/chewxy/math32 v1.11.2
github.com/daulet/tokenizers v1.27.0
github.com/go-git/go-git/v5 v5.19.2
golang.org/x/sys v0.47.0
golang.org/x/text v0.40.0 golang.org/x/text v0.40.0
modernc.org/sqlite v1.56.0
)
require (
dario.cat/mergo v1.0.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/ProtonMail/go-crypto v1.1.6 // indirect
github.com/apache/arrow-go/v18 v18.6.0 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/goccy/go-json v0.10.6 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/pierrec/lz4/v4 v4.1.26 // indirect
github.com/pjbgf/sha1cd v0.6.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
golang.org/x/net v0.56.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
modernc.org/libc v1.74.4 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
) )
+6 -176
View File
@@ -1,184 +1,14 @@
dario.cat/mergo v1.0.0 h1:AGCNq9Evsj31mOgNPcLyXc+4PNABt905YmuqPYYpBWk=
dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
github.com/LadybugDB/go-ladybug v0.17.0 h1:RXDbkBjrbRmLdEbhGl4CLOIEzSt09gbP0n9UbKDEfwI=
github.com/LadybugDB/go-ladybug v0.17.0/go.mod h1:GeIXmE8XyF5TFS94NAuTag7vgCC+no/HTBMRA6Rd5Cs=
github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/ProtonMail/go-crypto v1.1.6 h1:ZcV+Ropw6Qn0AX9brlQLAUXfqLBc7Bl+f/DmNxpLfdw=
github.com/ProtonMail/go-crypto v1.1.6/go.mod h1:rA3QumHc/FZ8pAHreoekgiAbzpNsfQAosU5td4SnOrE=
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4=
github.com/apache/arrow-go/v18 v18.6.0 h1:GX/Jyd3R7mCLiECAwY9FWbbaYblie2WXBSz4Sw8fNpM=
github.com/apache/arrow-go/v18 v18.6.0/go.mod h1:gm3MiPpY82fLYK5VKPB3WoJbsiLVDfT7flD5/vHReKw=
github.com/apache/thrift v0.22.0 h1:r7mTJdj51TMDe6RtcmNdQxgn9XcyfGDOzegMDRg47uc=
github.com/apache/thrift v0.22.0/go.mod h1:1e7J/O1Ae6ZQMTYdy9xa3w9k+XHWPfRvdPyJeynQ+/g=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
github.com/arran4/golang-ical v0.3.5 h1:bbz6ld4dC+MmCKiFfOd6SkmIGnhNMBACZ485ULh7p9A= github.com/arran4/golang-ical v0.3.5 h1:bbz6ld4dC+MmCKiFfOd6SkmIGnhNMBACZ485ULh7p9A=
github.com/arran4/golang-ical v0.3.5/go.mod h1:OnguFgjN0Hmx8jzpmWcC+AkHio94ujmLHKoaef7xQh8= github.com/arran4/golang-ical v0.3.5/go.mod h1:OnguFgjN0Hmx8jzpmWcC+AkHio94ujmLHKoaef7xQh8=
github.com/chewxy/math32 v1.11.2 h1:IufN08Zwr1NKuWfY+4Tz55BcwKmyKKNdOP7KtumehnM= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/chewxy/math32 v1.11.2/go.mod h1:dOB2rcuFrCn6UHrze36WSLVPKtzPMRAQvBvUwkSsLqs=
github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8=
github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4=
github.com/cyphar/filepath-securejoin v0.6.1 h1:5CeZ1jPXEiYt3+Z6zqprSAgSWiggmpVyciv8syjIpVE=
github.com/cyphar/filepath-securejoin v0.6.1/go.mod h1:A8hd4EnAeyujCJRrICiOWqjS1AX0a9kM5XL+NwKoYSc=
github.com/daulet/tokenizers v1.27.0 h1:MmFYAEDFz69s/nNQfHg59DWqHz3v94m99kEZ/JbL+s4=
github.com/daulet/tokenizers v1.27.0/go.mod h1:YjFY1o1HGMyWkQgbXJDghhvke/yFDp2vGdIO2hYs4MQ=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o=
github.com/elazarl/goproxy v1.7.2/go.mod h1:82vkLNir0ALaW14Rc399OTTjyNREgmdL2cVoIbS6XaE=
github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FMNAnJvWQ=
github.com/gliderlabs/ssh v0.3.8 h1:a4YXD1V7xMF9g5nTkdfnja3Sxy1PVDCj1Zg4Wb8vY6c=
github.com/gliderlabs/ssh v0.3.8/go.mod h1:xYoytBv1sV0aL3CavoDuJIQNURXkkfPA/wxQ1pL1fAU=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic=
github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA=
github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399 h1:eMje31YglSBqCdIqdhKBW8lokaMrL3uTkpGYlE2OOT4=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII=
github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY=
github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk=
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8/go.mod h1:wcDNUvekVysuuOpQKo3191zZyTpiI6se1N1ULghS0sw=
github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs=
github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4=
github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/onsi/gomega v1.34.1 h1:EUMJIKUjM8sKjYbtxQI9A4z2o+rruxnzNvpknOXie6k=
github.com/onsi/gomega v1.34.1/go.mod h1:kU1QgUvBDLXBJq618Xvm2LUX6rSAfRaFRTcdOeDLwwY=
github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pjbgf/sha1cd v0.6.0 h1:3WJ8Wz8gvDz29quX1OcEmkAlUg9diU4GxJHqs0/XiwU=
github.com/pjbgf/sha1cd v0.6.0/go.mod h1:lhpGlyHLpQZoxMv8HcgXvZEhcGs0PG/vsZnEJ7H0iCM=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 h1:n661drycOFuPLCN3Uc8sB6B/s6Z4t2xvBgU1htSHuq8=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/skeema/knownhosts v1.3.1 h1:X2osQ+RAjK76shCbvhHHHVl3ZlgDm8apHEHFqRjnBY8=
github.com/skeema/knownhosts v1.3.1/go.mod h1:r7KTdC8l4uxWRyK2TpQZ/1o5HaSzh06ePQNxPwTcfiY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI=
modernc.org/cc/v4 v4.29.1/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=
modernc.org/ccgo/v4 v4.34.6/go.mod h1:SZ8YcN9NG7XVsQYdm6jYBvi8PQP1qi+kqB6OhjqI3Fk=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.4 h1:2g65LGVSmFQrXeITAw97x7hCRvZFcyE1uDP+7Vng7JI=
modernc.org/gc/v3 v3.1.4/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.74.4 h1:fX1Omw4o2/1C2iRkkIsrQTasJQldLhRmuPreXLoWs9k=
modernc.org/libc v1.74.4/go.mod h1:eeQAS9W3sZeKYMFubydxJpII9ybHWshk+7or7bLG9co=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.56.0 h1:/D8e2RfFqoy/Zc6PuC76U28zFwmI/sYx1Kjm4yEn9e0=
modernc.org/sqlite v1.56.0/go.mod h1:yCJ2cmAaIkHQ25oXWrF8H4O1lIfPYPR26yCEDj2P3pQ=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=

Some files were not shown because too many files have changed in this diff Show More