Compare commits

..
Author SHA1 Message Date
eSlider e04b037b33 feat: parse all Go CLIs with flaggy; dump bash complete.
Tests / Test (push) Skipped
Tests / OCR (tesseract fixture) (push) Skipped
Tests / Release (semver) (push) Skipped
Tests / Test (pull_request) Failing after 6s
Tests / OCR (tesseract fixture) (pull_request) Failing after 4s
Tests / Release (semver) (pull_request) Skipped
stdlib flag dropped --hop after the query. One wrapper in internal/cli,
source <(./bin/cli/complete.go bash). Gitea #34.
2026-08-14 12:08:52 +01:00
eSliderandGitHub 0065655e03 feat: D16 adjudication — 2v2 stays hypothesis until a rule fires. (#35)
Tests / Test (push) Skipped
Tests / OCR (tesseract fixture) (push) Skipped
Tests / Release (semver) (push) Skipped
temporal_freshness then authority_pairing; unresolved keeps (not confirmed).
bin/facts/audit contradict. Gitea #29.
2026-08-14 11:43:00 +01:00
eSliderandGitHub a331042488 feat: DuckDB quantiles in-process (gcc CGO), not Ladybug. (#34)
Tests / Test (push) Failing after 6s
Tests / OCR (tesseract fixture) (push) Failing after 4s
Tests / Release (semver) (push) Skipped
OQ3: internal/duckstats + bin/qa/stats.go. Zig stays Ladybug-only.
mikefarah/yq for small structured slices. Gitea #30.
2026-08-14 11:32:23 +01:00
eSliderandGitHub f99dfea104 feat: OCR scans with tesseract, drop docling from the default path. (#33)
Tests / Test (push) Failing after 5s
Tests / OCR (tesseract fixture) (push) Failing after 4s
Tests / Release (semver) (push) Skipped
pdftotext still wins on born-digital PDFs. Empty text layers go through
pdftoppm + tesseract eng+deu. Optional OCR_ENGINE=paddle. Gitea #6.
2026-08-14 11:26:04 +01:00
eSliderandGitHub bae1494258 ci: recall@5 SoT is Zig bin/brain/eval.go (#19) (#32)
Tests / Test (push) Failing after 6s
Tests / Release (semver) (push) Skipped
* ci: recall@5 SoT is Zig bin/brain/eval.go.

GitHub Actions rebuilds the repo corpus and runs the Go eval gate
instead of Python bin/kb/eval. Audit self is a hard fail (Gitea #19).

* ci: point eval at the checkout and keep DevOps in the corpus.

/tmp/brain-eval cannot walk up to var/kb.lbug; KB_ROOT is required.
Recall fragments must exist in README/docs so a repo-only rebuild gates.
2026-08-14 11:03:39 +01:00
eSliderandGitHub c5be3f19be feat: index facts extract and chats markdown on rebuild. (#31)
Tests / Test (push) Failing after 5s
Tests / Release (semver) (push) Skipped
--with-facts / --facts-json write root=facts leafs; --with-chats
picks up var/chats/md as info. WhatsApp sync stays out of v1 (Gitea #18).
2026-08-14 10:57:40 +01:00
eSliderandGitHub a88dbb490c feat: search --hop walks FROM_FILE to Person. (#30)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
Parser no longer errors; hop 1 returns File, hop 3 reaches Person.
Rebuild writes Leaf-[:FROM_FILE]->File so the walk is not empty
on a fresh index (Gitea #17).
2026-08-14 10:51:35 +01:00
eSliderandGitHub 9d1a3f3c70 feat: write leafs incrementally without rebuilding the graph. (#29)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
Ladybug 0.19 stays FTS/HNSW queryable on MERGE of new ids; DROP INDEX
was the fatal path. bin/brain/add.go and POST /ingest land facts+info
in one transaction so watch/mail/git can become leafs now (Gitea #14).
2026-08-14 10:46:59 +01:00
eSliderandGitHub 8b5be9b659 docs: record v1 detective epic and remaining gaps. (#28)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
Read path and MCP are in; write, hops, corpus, and CI eval are still open. Board is Gitea epic #16.
2026-08-14 10:32:19 +01:00
eSliderandGitHub 1edb158f35 docs: portable runbook and Diataxis index (#27)
Tests / Test (push) Failing after 5s
Tests / Release (semver) (push) Skipped
Public face is a product, not a laptop tool. Run steps live in
docs/runbook.md; decisions D3/D6/D14/D15/D17/D18 stay in the docs index.
2026-08-13 23:31:41 +01:00
eSliderandGitHub 85caff90b9 feat: markdown import splits H2 leafs in Go (#26)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Drop the Python ExecFile wrapper. Conversion still does not write
Ladybug; add a dry-run fixture check for the index adapter split.
2026-08-13 23:27:42 +01:00
eSliderandGitHub b317968a4c feat: CPU reasoner bake-off for Qwen3.5-9B vs Bonsai (#25)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Measure OpenAI tool_calls (search/get/audit) on a CPU Ollama sidecar
instead of gating D18 on GPU or PicoClaw. Weights stay out of the image.
2026-08-13 23:19:51 +01:00
eSliderandGitHub 1f9bdb0bf6 feat: compile ladybug CGO with Zig, not gcc (#24)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
API image is Go-only (compose target api). Python rebuild is profile
index. bin/cgo/zig pins Zig 0.14.1, liblbug, and tokenizers.
2026-08-13 21:54:31 +01:00
106 changed files with 5366 additions and 2334 deletions
+1
View File
@@ -3,6 +3,7 @@
var
.git
.github
lib-ladybug
__pycache__
*.pyc
*.lbug
+38 -8
View File
@@ -37,31 +37,61 @@ jobs:
bash -n bin/db/ssh-tunnel
bash -n bin/docker-entrypoint
bash -n bin/kb/search
bash -n bin/cgo/zig
sh -n bin/cgo/zcc
sh -n bin/cgo/zc++
- name: Python unit tests (offline, vendored tools)
run: |
uv run python -m unittest discover -s bin/tools -t .
- name: Go tests (root module, no ladybug cgo)
- name: Go tests (root module; duckdb-go CGO via gcc, no ladybug)
run: |
go vet ./...
go test ./... -count=1
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go vet ./...
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go test ./... -count=1
- name: brain ranking tests (no cgo / no ladybug)
run: go test ./internal/brain/rank -count=1
- name: facts/audit self (lexicon consistency, no network)
run: |
./bin/facts/audit self 2>/dev/null || echo "audit: not yet implemented; gate skipped"
run: ./bin/facts/audit self
- name: kb/eval recall gate
- name: CGO via Zig (compile brain/search + eval)
run: |
./bin/kb/eval 2>/dev/null || echo "eval: not yet implemented; gate skipped"
chmod +x bin/cgo/zig bin/cgo/zcc bin/cgo/zc++
bin/cgo/zig go build -tags system_ladybug -o /tmp/brain-search ./bin/brain/search.go
bin/cgo/zig go build -tags 'system_ladybug,brain_eval' -o /tmp/brain-eval ./bin/brain/eval.go
- uses: actions/cache@v4
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-hf-potion-multilingual-128M
- name: recall@5 SoT (Zig bin/brain/eval.go)
run: |
uv run python bin/kb/index --rebuild --json
KB_ROOT="$PWD" /tmp/brain-eval --json
ocr:
name: OCR (tesseract fixture)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Install tesseract + poppler
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu poppler-utils
- name: Go OCR tests (synthetic HELLO PNG)
run: go test ./internal/ocr -count=1
release:
name: Release (semver)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: test
needs: [test, ocr]
runs-on: ubuntu-latest
permissions:
contents: write
+3
View File
@@ -11,3 +11,6 @@ __pycache__/
.secrets/
lib-ladybug/
go.work.local
models/
# Purged from git history. Do not re-add.
docs/crm-associations-proof.md
+31 -17
View File
@@ -3,7 +3,8 @@
Evidence-first brain over the ops/eSlider stack. Facts need proof or they are
`(not confirmed)`.
Read first: [PLAN](PLAN.md) → [docs](docs/).
Read first: [PLAN](PLAN.md) → [docs](docs/) → [roadmap](docs/roadmap.md)
(epic [#16](https://git.produktor.io/eSlider/2dph/issues/16)).
## Method (detective, no fork)
@@ -39,19 +40,22 @@ PLAN.md decisions + execution + open questions
docs/ published docs
skills/ in-project agent skills (vendored, no external links)
bin/ self-describing tools bin/{subject}/{method}.go (shebang)
bin/brain/ search.go serve.go index.go get.go stats.go eval.go watch.go
bin/brain/ search.go serve.go index.go add.go get.go stats.go eval.go watch.go
bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats
bin/mail/ sync.go import.go (index_mail → brain/index.go)
bin/markdown/ import.go (mistune leafs)
bin/mail/ sync.go import.go ocr.go (index_mail → brain/index.go)
bin/markdown/ import.go (H2 leaf split; Python bin/md/import fallback)
bin/postgres/ query.go (read-only YAML)
bin/git/ import.go (go-git history; Python shim execs it)
bin/web/ search.go (SearXNG; Python shim execs it)
internal/ shared Go (brain/rank is cgo-free; chats parsers; gitlog; websearch)
bin/reasoner/ bakeoff.go (D18 CPU OpenAI tool-call bake-off)
internal/ shared Go (brain/rank is cgo-free; facts D16; cli flaggy D23; chats; gitlog; websearch; reasoner; duckstats)
bin/qa/ stats.go (DuckDB quantiles / JSONL count; gcc CGO, not Zig)
bin/watch/ corpus watcher (used by bin/brain/watch.go)
bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch)
bin/docker-entrypoint container entrypoint (brain index|search|serve|watch)
bin/cgo/ zig zcc zc++ (CGO via zig cc, not gcc)
bin/docker-entrypoint container entrypoint (api: serve|search|watch; index: python)
compose.yaml docker composition (root level, not docker/)
Dockerfile multi-stage: python deps + static Go binaries
Dockerfile api (Zig CGO, no Python) + index (Python write)
var/ kb.lbug, var/mail/*, caches (gitignored)
.venv/ ladybug + model2vec + mistune
```
@@ -62,43 +66,53 @@ var/ kb.lbug, var/mail/*, caches (gitignored)
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments
bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox)
bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only)
bin/brain/index.go --rebuild # rebuild brain incl. all mail (fresh DB)
bin/brain/index.go --rebuild --with-facts --with-chats
```
- `sync` (Go) downloads messages + attachments; Gmail uses paginated list +
`body.attachmentId` (not partId) for attachments.
- `import` converts body + attachments to markdown. PDFs use poppler
`pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to
docling (isolated subprocess — its native onnx can segfault the parent).
Conversion never touches the brain DB (crash safety).
- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Ladybug
corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector
indexes exist; a fresh DB with indexes created last is the only safe path.
`pdftotext -layout` fast path (~15ms); textless/scanned PDFs use
`pdftoppm` + tesseract `eng+deu` (`bin/mail/ocr.go`). Optional
`OCR_ENGINE=paddle`. Conversion never touches the brain DB (crash safety).
- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Bulk
rebuild still deletes `var/kb.lbug` and creates FTS/HNSW last. Single-leaf
write is `bin/brain/add.go` (safe while indexes exist; do not DROP INDEX).
Keep conversion + indexing separate so a conversion crash can't leave the
DB mid-transaction.
## Tools
```bash
bin/facts/audit.go ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate
bin/facts/audit.go ["self"|"db"|"contradict"] # 2-source + D16 adjudication
bin/facts/crm.go [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT)
bin/kb/search "query" [--repo X] # deprecated wrapper → bin/brain/search.go
bin/brain/search.go "query" [--root facts|info] # deduction search → YAML
bin/brain/search.go "query" --no-web # local graph only
source <(./bin/cli/complete.go bash) # flaggy completions (D23)
eval "$(bin/cgo/zig env)" # Zig cc + liblbug (not gcc)
bin/brain/index.go --rebuild [--with-mail] [--with-facts] [--with-chats]
bin/brain/add.go --text T --root facts --source "a.md x b.md" # incremental write
bin/brain/add.go --json # stdin leaf or {leafs:[...]}
bin/brain/get.go <id> [--body] [--json] # Go read; Python bin/kb/get CI fallback
bin/brain/stats.go [--json]
bin/brain/eval.go [--json] # recall@5; questions in internal/brain/rank
bin/brain/serve.go # HTTP :8630; GET /openapi.json POST /mcp
bin/markdown/import.go [dir] # mistune leaves → YAML
bin/markdown/import.go [dir] # H2 leafs → YAML; Python bin/md/import fallback
bin/git/import.go [REPO] [--json] [--limit N] # go-git history → commit leafs
bin/web/search.go "query" [--json] # SearXNG; throttled ≠ absence
bin/reasoner/bakeoff.go [--model ID] [--json] # D18 CPU tool-call bake-off
bin/postgres/query.go --profile onlyoffice -c 'SELECT 1'
bin/qa/stats.go # D22 DuckDB quantiles / JSONL (gcc CGO)
bin/mail/ocr.go <image|pdf> # tesseract eng+deu (scans)
bin/md/tables # what the graph holds → YAML
bin/brain/deduce "question" # thinking wrapper
```
Never start a shell command with `cd` — use the tool working-directory
parameter. Search before reading whole files.
parameter. Search before reading whole files. For YAML/JSON/XML/CSV/TOML/HCL
prefer mikefarah/yq (`skills/yq/SKILL.md`). For bulk rows and quantiles use
duckdb-go (`internal/duckstats`, `skills/duckdb/SKILL.md`), not Ladybug.
## GitHub safety rules (ABSOLUTE — never violate)
+60 -16
View File
@@ -1,5 +1,13 @@
# syntax=docker/dockerfile:1
FROM python:3.12-slim AS base
#
# docker build --target api -t 2dph:api .
# docker build --target index -t 2dph:index .
#
# API: Go + ladybug via Zig CGO (no CPython).
# Index: Python write path (profile `index` until brain/add is v2).
# --- Python sidecar (Ladybug write / rebuild) ---
FROM python:3.12-slim AS index
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
@@ -8,26 +16,16 @@ ENV PYTHONUNBUFFERED=1 \
WORKDIR /app
RUN id -u 2dph 2>/dev/null || useradd --create-home --uid 1001 2dph
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
poppler-utils tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu \
&& rm -rf /var/lib/apt/lists/*
# deps layer-first: rebuild only on dependency change
COPY requirements.lock.txt /tmp/requirements.lock.txt
RUN python -m pip install --no-cache-dir -r /tmp/requirements.lock.txt \
&& rm /tmp/requirements.lock.txt
# Go services: static binaries, no interpreter at runtime
FROM golang:1.25 AS go-build
WORKDIR /src
COPY go.mod ./
COPY bin/server ./bin/server
COPY bin/watch ./bin/watch
RUN CGO_ENABLED=0 go build -o /serve ./bin/server \
&& CGO_ENABLED=0 go build -o /watch ./bin/watch
# runtime: python toolchain + Go services
FROM base
COPY . .
COPY --from=go-build /serve /app/bin/serve
COPY --from=go-build /watch /app/bin/watch
RUN chmod +x /app/bin/docker-entrypoint \
&& chown -R 2dph:2dph /app
USER 2dph
@@ -37,5 +35,51 @@ ENV PATH="/app/bin:${PATH}" \
KB_ROOT=/app
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import model2vec, ladybug, mistune; print('ok')" || exit 1
ENTRYPOINT ["/app/bin/docker-entrypoint"]
# --- Go API: CGO with Zig, not gcc ---
FROM golang:1.26-bookworm AS api-build
WORKDIR /src
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl xz-utils ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY bin/cgo ./bin/cgo
RUN chmod +x bin/cgo/zig bin/cgo/zcc bin/cgo/zc++ \
&& ./bin/cgo/zig env >/dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ENV CGO_RPATH=/usr/local/lib
RUN eval "$(./bin/cgo/zig env)" \
&& go build -tags brain_serve,system_ladybug -o /out/brain-serve ./bin/brain/serve.go \
&& go build -tags system_ladybug -o /out/brain-search ./bin/brain/search.go \
&& CGO_ENABLED=0 go build -tags brain_watch -o /out/brain-watch ./bin/brain/watch.go
FROM debian:bookworm-slim AS api
RUN apt-get update \
&& apt-get install -y --no-install-recommends libssl3 ca-certificates wget \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --uid 1001 2dph
COPY --from=api-build /out/brain-serve /usr/local/bin/brain-serve
COPY --from=api-build /out/brain-search /usr/local/bin/brain-search
COPY --from=api-build /out/brain-watch /usr/local/bin/brain-watch
COPY --from=api-build /src/lib-ladybug/liblbug.so.0.19.1 /usr/local/lib/liblbug.so.0.19.1
COPY bin/docker-entrypoint /usr/local/bin/docker-entrypoint
RUN chmod +x /usr/local/bin/docker-entrypoint \
&& ln -s liblbug.so.0.19.1 /usr/local/lib/liblbug.so.0 \
&& ln -s liblbug.so.0 /usr/local/lib/liblbug.so \
&& ldconfig
USER 2dph
ENV KB_ROOT=/data \
KB_PORT=8630 \
LD_LIBRARY_PATH=/usr/local/lib \
HF_HOME=/data/hf
WORKDIR /data
EXPOSE 8630
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:8630/health || exit 1
ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
CMD ["serve"]
+61 -23
View File
@@ -4,7 +4,13 @@ A brain that loves facts and deduction. Evidence-first knowledge graph + hybrid
RAG over the operational Brain/ops/eSlider stack. Built like Sherlock
Holmes: nothing is asserted unless it has proof.
Status: **in progress** — this file is the plan and the record of decisions.
Status: **v1 in** (epic [#16](https://git.produktor.io/eSlider/2dph/issues/16) closed).
v2 board: milestone [v2](https://git.produktor.io/eSlider/2dph/milestone/13) —
OCR [#6](https://git.produktor.io/eSlider/2dph/issues/6) in,
[#29](https://git.produktor.io/eSlider/2dph/issues/29) OQ1 in,
[#30](https://git.produktor.io/eSlider/2dph/issues/30) OQ3 in,
[#34](https://git.produktor.io/eSlider/2dph/issues/34) D23 in.
Gap: [docs/roadmap.md](docs/roadmap.md).
## What
@@ -29,9 +35,9 @@ detective method: **a fact needs ≥2 independent sources or it is
| D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. |
| D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. |
| D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). |
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path (`get.go` / `stats.go` / `eval.go`) is Go + cgo. Python `bin/kb/{get,stats,eval}` is the CI fallback (GitHub runners have no ladybug cgo). Index/write stays Python until the Go write path is safe. |
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Incremental write is Python `bin/kb/add` (`bin/brain/add.go`). Bulk rebuild stays `compose --profile index` until the Go write path is safe. |
| D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). |
| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. |
| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. 2-source auto-pair docker ps × compose × ssh-config × docs. |
| D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. |
| D10 | versioning | everything is a leaf with `sha256 + observed_at + source_rev`; `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`. Stale = `source_rev` < git HEAD. |
| D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. |
@@ -39,11 +45,14 @@ detective method: **a fact needs ≥2 independent sources or it is
| D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. |
| D14 | tooling style | `bin/{subject}/{method}.go` shebang (e.g. `bin/brain/search.go`). Shared code in `internal/`. One root `go.mod` + `go.work`. No `bin/*/main.go`, no nested modules. |
| D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. |
| D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. |
| D16 | contradictions | ≥2 yes vs ≥2 no → hypothesis → `(not confirmed)` until a rule fires. Order: **temporal_freshness** (fresh ≥2 vs stale minority), then **authority_pairing** (runtime/config A×B beats narrative C). Store as `a x b vs c x d` on hypothesis leafs. `bin/facts/audit contradict`. [#29](https://git.produktor.io/eSlider/2dph/issues/29). |
| D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. |
| D18 | reasoner | Pluggable OpenAI-compatible URL. RAM: Qwen3.5-9B. Quality: Bonsai-27B or Qwen3.6-27B. No official Qwen3.6-9B. |
| D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is compose profile `picoclaw`; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. Agent lever/loop: [#15](https://git.produktor.io/eSlider/2dph/issues/15). |
| D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. |
| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). |
| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`/`ingest`). |
| D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc``zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. |
| D22 | analytics | **duckdb-go** in-process (`internal/duckstats`, `bin/qa/stats.go`) for quantiles/JSONL. Links with **gcc/g++**, not Zig. Ladybug stays the graph; web-search cache stays modernc sqlite. Slice small structured docs with **mikefarah/yq**, not kislyuk/jq. [#30](https://git.produktor.io/eSlider/2dph/issues/30). |
| D23 | CLI | **flaggy** (`github.com/integrii/flaggy`, 0 deps). Flags at any position. Wrapper `internal/cli`. Bash complete: `source <(./bin/cli/complete.go bash)`. No cobra, no stdlib `flag` in Go tools. Search does not intercept the word `completion`. [#34](https://git.produktor.io/eSlider/2dph/issues/34). |
## Architecture
@@ -54,19 +63,25 @@ detective method: **a fact needs ≥2 independent sources or it is
skills/ in-project skills (web-search, postgres, brain, picoclaw, diataxis-docs)
bin/
facts/extract.go audit.go crm.go # D14 shebang; Python implementation
kb/index Python write path (called by bin/brain/index.go)
kb/index Python bulk write (called by bin/brain/index.go)
kb/add Python incremental write (called by bin/brain/add.go)
brain/index.go rebuild FTS + HNSW (incl. --with-mail)
brain/add.go incremental leaf write (no rebuild)
brain/get.go stats.go eval.go # Go read (cgo); Python bin/kb/* CI fallback
brain/watch.go
brain/search.go deduction: facts → info → web-search
brain/serve.go HTTP API in-process + OpenAPI/MCP (D20); compose profile picoclaw
brain/search.go deduction: facts → info → web
cli/complete.go flaggy bash/zsh/fish complete (D23)
brain/serve.go HTTP API in-process + OpenAPI/MCP (D20); Zig CGO (D21)
cgo/zig zcc zc++ CGO toolchain (zig cc, not gcc)
mail/import.go JSON → markdown (no brain write)
markdown/import.go mistune leaves
markdown/import.go H2 leaf split (Go); Python bin/md/import fallback
postgres/query.go read-only YAML (wraps bin/db/psql-yq)
git/import.go go-git history (no git binary; conversion only)
web/search.go SearXNG client (throttled ≠ absence)
reasoner/bakeoff.go CPU tool-call bake-off (D18; OpenAI tools)
chats/sync.go import.go facts.go apply.go
(libs in internal/chats; no chats index)
mail/ocr.go tesseract eng+deu (pdftoppm scans)
md/import (deprecated; bin/markdown/import.go)
brain/extract brain/audit brain/deduce (thinking wrapper)
web/search (deprecated shim → web/search.go)
@@ -80,7 +95,10 @@ detective method: **a fact needs ≥2 independent sources or it is
Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`.
`Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`.
Edges: `RUNS / USES / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
Edges: `RUNS / USES / FROM_FILE / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
`FROM_FILE` / `HAS_VERSION` / `AUTHORED`: `bin/brain/search.go --hop N` walks
them from each hit (1=File, 2=Commit, 3=Person). Rebuild writes
`Leaf-[:FROM_FILE]->File`; git import writes the rest.
Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
`where`, `when`, `source_rev`.
@@ -98,17 +116,20 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
- `bin/{subject}/{method}` — line 2 is a usage comment (mirrors `psql-yq`).
- bash + python primary; golang via Go shebang when a compiled helper is right.
- YAML default output, `--json` for machines. Slice with `yq`.
- YAML default output, `--json` for machines. Slice with mikefarah/yq.
- Everything that touches the network / DB is read-only, throttled, cached.
- Tests (TDD) gate every commit; `gh` + CI/CD on every push.
## Open questions (v2)
- OQ1: mutually-contradicting evidence — how to resolve (authority weighting,
temporal freshness, audit adjudication).
- OQ2: OCR pipeline for pdfs/images/docs — mostly solved: poppler pdftotext
fast-path for born-digital PDFs, docling fallback for the ~5% textless ones.
- OQ3: optional duckdb-md layer for `SELECT … FORMAT MARKDOWN` export/write-back.
- OQ1: **in** — D16 adjudication: `temporal_freshness` then `authority_pairing`.
Unresolved 2v2 stays hypothesis. [#29](https://git.produktor.io/eSlider/2dph/issues/29).
- OQ2: OCR **in**. `pdftotext -layout` first; scans `pdftoppm` + tesseract
`eng+deu` (`bin/mail/ocr.go`, `internal/ocr`). No gocv, no gosseract CGO
(D21 Zig owns Ladybug CGO). Optional `OCR_ENGINE=paddle` / compose profile
`ocr-paddle`. Docling left the default path. [#6](https://git.produktor.io/eSlider/2dph/issues/6).
- OQ3: **in** — duckdb-go (`internal/duckstats`, `bin/qa/stats.go`) for
quantiles / JSONL count. Not a second graph. [#30](https://git.produktor.io/eSlider/2dph/issues/30).
- OQ4: YAML-first storage for leafs — deferred: JSON is ~10x faster to
serialize and unambiguous; YAML only where humans edit files.
@@ -117,7 +138,8 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download.
Gmail attachments key off `body.attachmentId`, not MIME `partId`.
2. `bin/mail/import.go --from-raw` — message.json → message.md; PDFs via
`pdftotext -layout` (~15ms) with docling subprocess fallback; ICS sidecars
`pdftotext -layout` (~15ms); textless/scanned PDFs `pdftoppm` + tesseract
`eng+deu`. ICS sidecars
Latin-1→UTF-8 normalized.
3. `bin/brain/index.go --rebuild` — fresh rebuild (repo corpus + mail) because ladybug
corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and
@@ -134,10 +156,9 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser)
3. python -m unittest discover -s bin/tools (includes published-docs SoT)
4. `bin/facts/audit self` (lexicon internal consistency; `bin/facts/audit.go` is the D14 wrapper)
5. `bin/kb/eval` (recall@5 ≥ 0.95). Local SoT is `bin/brain/eval.go`; CI uses
the Python twin until the runner has ladybug cgo. Questions live in
`internal/brain/rank`.
6. md-docs build/lint if docs tooling arrives.
5. `bin/brain/eval.go` via Zig (recall@5 ≥ 0.95). Python `bin/kb/eval` is an
explicit fallback, not the CI SoT.
6. `bin/cgo/zig go build -tags system_ladybug` (compile search with zig cc; fetches pinned zig+libs).
Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as
`db/tech-poc`: contract first where there is an OpenAPI/message shape.
@@ -150,5 +171,22 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
4. .venv: ladybug + model2vec + mistune
5. schema + tools with TDD (kb + md + facts + brain)
6. ~/.config/brain config
7. corpus extraction (facts/info)
7. corpus extraction (facts/info)**in**: [#18](https://git.produktor.io/eSlider/2dph/issues/18)
8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit
## Gap to v1 (epic #16)
Remaining: none for epic #16 (v1). Board:
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16),
milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
Narrative: [docs/roadmap.md](docs/roadmap.md).
| Order | Issue | Gap |
|-------|-------|-----|
| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in**`bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. |
| 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | **in**`--hop N` walks `FROM_FILE``HAS_VERSION``AUTHORED` (max 3). |
| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | **in**`--with-facts` / `--facts-json` land `root=facts`; `--with-chats` indexes `var/chats/md`. WhatsApp sync is out of v1. |
| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search``get``audit`). |
| 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | **in** — CI recall SoT is `bin/brain/eval.go` via Zig. Python `bin/kb/eval` stays as an explicit fallback. |
Does **not** block epic close: OQ4. OCR [#6](https://git.produktor.io/eSlider/2dph/issues/6), OQ1 [#29](https://git.produktor.io/eSlider/2dph/issues/29), OQ3 [#30](https://git.produktor.io/eSlider/2dph/issues/30) are **in**.
+40 -30
View File
@@ -7,14 +7,16 @@
[![Latest Release](https://img.shields.io/github/v/tag/eSlider/2dph?sort=semver&label=release)](https://github.com/eSlider/2dph/releases)
[![GitHub Stars](https://img.shields.io/github/stars/eSlider/2dph?style=social)](https://github.com/eSlider/2dph/stargazers)
An evidence-first brain over the operational eSlider stack. **Facts need two
independent sources, or they are `(not confirmed)`.**
An evidence-first brain. **Facts need two independent sources, or they are
`(not confirmed)`.** Cursor is not the runtime.
`2dph` is a single embedded knowledge graph (LadybugDB = Kuzu successor) with
native **HNSW vector** + **BM25 full-text** indexes, built from markdown,
compose files, ssh config, docker state, and git history. Search is
*deduction*: confirmed facts first, supporting info second, `web-search` as
the independent second source when the local graph cannot confirm.
`2dph` is a single embedded knowledge graph (LadybugDB) with native **HNSW
vector** + **BM25 full-text** indexes. Search is *deduction*: confirmed facts
first, supporting info second, `web-search` as the independent second source
when the local graph cannot confirm.
Run it: [docs/runbook.md](docs/runbook.md). Design: [docs/design.md](docs/design.md).
Docs index: [docs/README.md](docs/README.md).
## Architecture
@@ -31,7 +33,7 @@ graph TB
EX["bin/facts/extract.go<br/>2-source pairing"]
AU["bin/facts/audit.go<br/>confidence + staleness"]
IDX["bin/brain/index.go<br/>chunk + embed"]
MD["bin/markdown/import.go<br/>mistune leaves"]
MD["bin/markdown/import.go<br/>H2 leaf split"]
SR["bin/brain/search.go<br/>deduction"]
end
@@ -74,7 +76,7 @@ graph TB
## The method
Every assertion is `Who / What / How / Where / When + evidence + confidence`,
mirroring the detective detective skill: **≥2 independent sources confirm a
mirroring the detective method: **≥2 independent sources confirm a
fact; conflicting sources or a single source → `hypothesis``(not confirmed)`.**
| root | meaning | used for answers |
@@ -94,7 +96,7 @@ bin/brain/stats.go # index health
bin/brain/eval.go # recall@5 gate
```
`--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
`--hop N` walks File/Commit/Person from each hit (max 3). `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary):
@@ -118,23 +120,24 @@ Mail is a first-class corpus (retrievable through the same search):
```bash
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go)
bin/mail/import.go --from-raw var/mail # JSON → markdown
bin/brain/add.go --text T --root facts --source "a.md x b.md"
bin/brain/index.go --rebuild --with-facts --with-chats # facts extract + chats md
bin/brain/index.go --rebuild # rebuild brain (incl. mail)
bin/brain/search.go "invoice from last week" # same search over mail leafs
```
## Storage
- **LadybugDB** — single `var/kb.lbug`, Cypher property graph, HNSW + BM25
in one engine, embedded (no server), ACID, read-only-safe for concurrent
readers. Read tools (`get` / `stats` / `eval`) are Go + cgo; Python
`bin/kb/{get,stats,eval}` is the CI fallback. **Never `DROP INDEX` FTS/VECTOR** on Ladybug 0.19: DROP leaves
ghost catalog tables (`_0_Leaf_vec_UPPER`) so recreate fails while
`SHOW_INDEXES` omits HNSW. Fresh indexes = delete `var/kb.lbug` +
`bin/brain/index.go --rebuild`. Use `ensure_indexes()` after upserts.
- **model2vec** — `potion-multilingual-128M` static embeddings (256-dim),
CPU-fast, deterministic, no Ollama runtime dependency.
- facts and info split semantically by `root` column but written inside the
same transaction.
- **LadybugDB** — single `var/kb.lbug`, Cypher + HNSW + BM25, embedded.
Read tools (`get` / `stats` / `eval`) are Go + Zig CGO (`bin/cgo/zcc`).
Python fallbacks stay for CI until the runner fetches Zig. Incremental
write is `bin/brain/add.go` (Python `kblib.add_leafs`). Bulk rebuild is
Compose profile `index` (`bin/brain/index.go --rebuild`).
- **model2vec** — `potion-multilingual-128M` (256-dim), CPU, no Ollama
runtime dependency.
- facts and info split by `root` but written in the same transaction.
Ladybug 0.19 DROP INDEX warning: [docs/runbook.md](docs/runbook.md).
## Tooling conventions
@@ -145,32 +148,39 @@ machines. Tests gate every commit. HTTP: `bin/brain/serve.go` calls
## Development
See the portable runbook: [docs/runbook.md](docs/runbook.md).
```bash
uv venv .venv # Python 3.12, uv-managed
uv pip install -r requirements.lock.txt # pinned toolchain
bin/facts/audit.go self # lexicon consistency gate
go test ./... && python -m unittest discover -s bin/tools -t .
uv venv .venv
uv pip install -r requirements.lock.txt
bin/facts/audit.go self
go test ./... && uv run python -m unittest discover -s bin/tools -t .
```
Docker (optional, cached model + var volumes):
```bash
docker compose run --rm brain index # (re)index corpus
docker compose run --rm brain search "query" # one-shot query
docker compose run --rm brain serve # bin/brain/serve.go
docker compose up -d brain # API (Zig CGO serve :8630)
docker compose --profile index run --rm index # Python Ladybug rebuild
docker compose --profile picoclaw up brain-mcp # MCP on 127.0.0.1:8630
docker compose --profile reasoner up -d reasoner # CPU Ollama 127.0.0.1:11435
docker compose up brain-watch # auto re-index on change
```
## Related
eSlider DevOps engineer practice: ops, OnlyOffice, and mail feed the facts
root through `bin/facts/extract` (two-source pairing).
- [go-second-brain](https://github.com/eSlider/go-second-brain) — the earlier
Neo4j + Qdrant + Matrix RAG brain
- [agent-skills](https://github.com/eSlider/agent-skills) — upstream
skills (`web-search`, `postgres`, …) that 2dph integrates
- detective method — the two-source method
Work board (issues): [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
Work board (issues): [epic #16](https://git.produktor.io/eSlider/2dph/issues/16)
on [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).
See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions.
See [PLAN.md](PLAN.md) for decisions, [docs/roadmap.md](docs/roadmap.md) for
the gap to v1, and v2 open questions.
+21
View File
@@ -0,0 +1,21 @@
//usr/bin/env go run -tags=brain_add "$0" "$@"; exit
//go:build brain_add
//
// bin/brain/add.go - incremental leaf write (Python kblib, no rebuild).
//
// ./bin/brain/add.go --text T --root facts --source "a.md x b.md"
// ./bin/brain/add.go --json
//
// D6: write stays Python. Does not delete var/kb.lbug.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/kb/add", os.Args[1:]))
}
+1
View File
@@ -8,6 +8,7 @@
// ./bin/brain/get.go <id> --json
//
// Needs CGO + libladybug. Python bin/kb/get is the CI fallback (no cgo).
// CGO compiler is Zig (`eval "$(bin/cgo/zig env)"`), not gcc.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
+3 -3
View File
@@ -3,12 +3,12 @@
//
// bin/brain/index.go - rebuild the Ladybug graph (Python write path).
//
// ./bin/brain/index.go --rebuild
// ./bin/brain/index.go --rebuild --with-facts --with-chats
// ./bin/brain/index.go --rebuild --with-mail
// ./bin/brain/index.go --dry-run --with-mail
//
// v1 write is always a rebuild when mail is included (live FTS/HNSW + bulk
// insert corrupts Ladybug 0.19 WAL). `add` is v2.
// v1 write: bin/brain/add.go for one/few leafs (indexes may already exist).
// Bulk mail/corpus still --rebuild (fresh file, indexes last).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
+2 -2
View File
@@ -3,11 +3,11 @@
//
// bin/brain/search.go - deduction search over the 2dph brain.
//
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--json] [--no-web]
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--hop N] [--json] [--no-web]
// ./bin/brain/search.go serve [port]
// ./bin/brain/search.go --list-model
//
// Needs CGO + libladybug (CGO_CFLAGS/CGO_LDFLAGS). Prefer the wrapper
// Needs CGO + libladybug via Zig (`eval "$(bin/cgo/zig env)"`), not gcc.
// bin/kb/search which sets those and builds a binary for the embed daemon.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# bin/cgo/zc++ — CGO CXX. Zig, not g++.
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
case "$(uname -m)" in
x86_64|amd64) TARGET=x86_64-linux-gnu ;;
aarch64|arm64) TARGET=aarch64-linux-gnu ;;
*)
echo "zc++: unsupported arch $(uname -m)" >&2
exit 2
;;
esac
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
:
elif [ -x "$ROOT/var/zig/zig" ]; then
ZIG="$ROOT/var/zig/zig"
elif command -v zig >/dev/null 2>&1; then
ZIG="$(command -v zig)"
else
echo "zc++: zig missing; run bin/cgo/zig first" >&2
exit 127
fi
exec "$ZIG" c++ -target "$TARGET" "$@"
Executable
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
# bin/cgo/zcc — CGO CC. Zig, not gcc.
# Go invokes CC with many args; a wrapper avoids spaces in $CC.
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
case "$(uname -m)" in
x86_64|amd64) TARGET=x86_64-linux-gnu ;;
aarch64|arm64) TARGET=aarch64-linux-gnu ;;
*)
echo "zcc: unsupported arch $(uname -m)" >&2
exit 2
;;
esac
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
:
elif [ -x "$ROOT/var/zig/zig" ]; then
ZIG="$ROOT/var/zig/zig"
elif command -v zig >/dev/null 2>&1; then
ZIG="$(command -v zig)"
else
echo "zcc: zig missing; run bin/cgo/zig first" >&2
exit 127
fi
exec "$ZIG" cc -target "$TARGET" "$@"
Executable
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# bin/cgo/zig — CGO toolchain: zig cc (not gcc) + pinned liblbug + libtokenizers.
#
# eval "$(bin/cgo/zig env)" # export CC/CXX/CGO_*
# bin/cgo/zig go build ... # ensure, then exec with env
# bin/cgo/zig ./bin/brain/search.go "query"
#
# Pins live in this file. Downloads land in var/ (gitignored).
set -euo pipefail
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
ZIG_VERSION=0.14.1
LBUG_VERSION=0.19.1
TOKENIZERS_VERSION=1.27.0
arch="$(uname -m)"
case "$arch" in
x86_64|amd64)
ZIG_ARCH=x86_64
LBUG_ARCH=x86_64
TOK_ARCH=x86_64
ZIG_SHA=24aeeec8af16c381934a6cd7d95c807a8cb2cf7df9fa40d359aa884195c4716c
LBUG_SHA=ed263ae913f68cb0ddba0b98548b58edaac49929766d03bdaaa83be46c68847d
TOK_SHA=72556cdca798dd4ea7cdaba308e5f0d68a8cb93b67c96edf485b7a0edd7b07f4
;;
aarch64|arm64)
ZIG_ARCH=aarch64
LBUG_ARCH=aarch64
TOK_ARCH=aarch64
ZIG_SHA=f7a654acc967864f7a050ddacfaa778c7504a0eca8d2b678839c21eea47c992b
LBUG_SHA=b07df2cd533c3976a2a3025866d6420a5f35514d0a822ecc4b2902d55b4725b7
TOK_SHA=e96545ad05930c26f51f63d932ee6d3bbd32bbed149e102c5290d587a2293067
;;
*)
echo "bin/cgo/zig: unsupported arch $arch" >&2
exit 2
;;
esac
CACHE="$ROOT/var/cache"
LIB="$ROOT/lib-ladybug"
ZIG_DIR="$ROOT/var/zig-dist"
ZIG_BIN="$ROOT/var/zig/zig"
sha256of() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{print $1}'
else
shasum -a 256 "$1" | awk '{print $1}'
fi
}
fetch() {
local url="$1" dest="$2" expect="$3"
if [ -f "$dest" ] && [ "$(sha256of "$dest")" = "$expect" ]; then
return 0
fi
mkdir -p "$(dirname "$dest")"
echo "fetch $url" >&2
curl -fsSL "$url" -o "$dest"
local got
got="$(sha256of "$dest")"
if [ "$got" != "$expect" ]; then
echo "checksum mismatch $dest: got $got want $expect" >&2
rm -f "$dest"
exit 1
fi
}
ensure_zig() {
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
return 0
fi
if [ -x "$ZIG_BIN" ]; then
export ZIG="$ZIG_BIN"
return 0
fi
if command -v zig >/dev/null 2>&1; then
export ZIG
ZIG="$(command -v zig)"
return 0
fi
local tar="$CACHE/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}.tar.xz"
fetch "https://ziglang.org/download/${ZIG_VERSION}/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}.tar.xz" \
"$tar" "$ZIG_SHA"
mkdir -p "$CACHE"
rm -rf "$ZIG_DIR"
tar -xJf "$tar" -C "$CACHE"
mv "$CACHE/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}" "$ZIG_DIR"
mkdir -p "$ROOT/var/zig"
ln -sfn "$ZIG_DIR/zig" "$ZIG_BIN"
export ZIG="$ZIG_BIN"
}
ensure_libs() {
mkdir -p "$LIB"
if [ ! -f "$LIB/liblbug.so" ]; then
local tar="$CACHE/liblbug-linux-${LBUG_ARCH}.tar.gz"
fetch "https://github.com/LadybugDB/ladybug/releases/download/v${LBUG_VERSION}/liblbug-linux-${LBUG_ARCH}.tar.gz" \
"$tar" "$LBUG_SHA"
tar -xzf "$tar" -C "$LIB"
fi
if [ ! -f "$LIB/libtokenizers.a" ]; then
local tar="$CACHE/libtokenizers.linux-${TOK_ARCH}.tar.gz"
fetch "https://github.com/daulet/tokenizers/releases/download/v${TOKENIZERS_VERSION}/libtokenizers.linux-${TOK_ARCH}.tar.gz" \
"$tar" "$TOK_SHA"
tar -xzf "$tar" -C "$LIB"
fi
}
print_env() {
printf 'export ZIG=%q\n' "$ZIG"
printf 'export CC=%q\n' "$ROOT/bin/cgo/zcc"
printf 'export CXX=%q\n' "$ROOT/bin/cgo/zc++"
printf 'export CGO_ENABLED=1\n'
printf 'export CGO_CFLAGS=%q\n' "-I$LIB"
printf 'export CGO_LDFLAGS=%q\n' "-L$LIB -Wl,-rpath,${CGO_RPATH:-$LIB}"
}
ensure_zig
ensure_libs
cmd="${1:-env}"
if [ "$cmd" = "env" ]; then
print_env
exit 0
fi
eval "$(print_env)"
exec "$@"
+3 -2
View File
@@ -29,10 +29,11 @@ func main() {
case "linkedin":
os.Exit(chats.RunSyncLinkedIn(args))
case "whatsapp":
fmt.Fprintln(os.Stderr, "chats: WhatsApp not implemented yet")
fmt.Fprintln(os.Stderr, "chats: WhatsApp sync is out of v1")
os.Exit(1)
case "help", "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`)
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]
WhatsApp sync is out of v1.`)
return
default:
fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform)
+91
View File
@@ -0,0 +1,91 @@
//usr/bin/env go run "$0" "$@"; exit
//
// bin/cli/complete.go - dump flaggy shell completions for all Go shebang tools (D23).
//
// source <(./bin/cli/complete.go bash)
// ./bin/cli/complete.go zsh|fish|powershell|nushell
//
// Search does not steal the word "completion"; this binary dumps scripts.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"strings"
mailsync "github.com/eSlider/2dph/bin/mail/sync"
"github.com/eSlider/2dph/internal/brain/rank"
"github.com/eSlider/2dph/internal/chats"
"github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/gitlog"
"github.com/eSlider/2dph/internal/mdleaves"
"github.com/eSlider/2dph/internal/ocr"
"github.com/eSlider/2dph/internal/reasoner"
"github.com/eSlider/2dph/internal/websearch"
"github.com/integrii/flaggy"
)
func tools() []cli.Tool {
return []cli.Tool{
{Path: "bin/brain/search.go", Name: "brain-search", New: rank.Parser},
{Path: "bin/brain/get.go", Name: "brain-get", New: func() *flaggy.Parser {
o := rank.GetOptions{}
return rank.GetParser(&o)
}},
{Path: "bin/brain/stats.go", Name: "brain-stats", New: rank.StatsParser},
{Path: "bin/brain/eval.go", Name: "brain-eval", New: rank.EvalParser},
{Path: "bin/web/search.go", Name: "web-search", New: websearch.Parser},
{Path: "bin/git/import.go", Name: "git-import", New: gitlog.Parser},
{Path: "bin/markdown/import.go", Name: "markdown-import", New: mdleaves.Parser},
{Path: "bin/qa/stats.go", Name: "qa-stats", New: cli.QAParser},
{Path: "bin/reasoner/bakeoff.go", Name: "reasoner-bakeoff", New: reasoner.Parser},
{Path: "bin/mail/ocr.go", Name: "mail-ocr", New: ocr.Parser},
{Path: "bin/mail/sync.go", Name: "mail-sync", New: mailsync.Parser},
{Path: "bin/chats/sync.go", Name: "chats-sync", New: chats.SyncParser},
{Path: "bin/chats/import.go", Name: "chats-import", New: chats.ImportParser},
{Path: "bin/chats/facts.go", Name: "chats-facts", New: chats.FactsParser},
{Path: "bin/chats/apply.go", Name: "chats-apply", New: chats.ApplyParser},
}
}
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
shell := "bash"
if len(args) > 0 {
switch args[0] {
case "bash", "zsh", "fish", "powershell", "nushell":
shell = args[0]
case "-h", "--help", "help":
fmt.Fprintln(os.Stderr, "usage: bin/cli/complete.go [bash|zsh|fish|powershell|nushell]")
return 0
default:
fmt.Fprintf(os.Stderr, "cli/complete: unknown shell %q\n", args[0])
return 2
}
}
if shell == "bash" {
fmt.Print(cli.BashScript(tools()))
return 0
}
var b strings.Builder
for _, t := range tools() {
p := t.New()
p.Name = t.Name
switch shell {
case "zsh":
b.WriteString(flaggy.GenerateZshCompletion(p))
case "fish":
b.WriteString(flaggy.GenerateFishCompletion(p))
case "powershell":
b.WriteString(flaggy.GeneratePowerShellCompletion(p))
case "nushell":
b.WriteString(flaggy.GenerateNushellCompletion(p))
}
}
fmt.Print(b.String())
return 0
}
+18 -7
View File
@@ -1,13 +1,10 @@
#!/usr/bin/env bash
# bin/docker-entrypoint - run 2dph tools inside the container.
#
# brain shell (default)
# brain search <q> bin/brain/search.go
# brain index bin/kb/index --with-mail
# brain watch <dir> compiled /app/bin/watch (bin/brain/watch.go)
# brain serve compiled /app/bin/serve (bin/brain/serve.go)
# brain extract bin/facts/extract (docker×compose pairing)
# brain audit bin/facts/audit
# API image (Zig CGO binaries):
# serve | search | watch
# Index image (Python write path, compose profile `index`):
# index | extract | audit | search (deprecated python wrapper)
#
# Usage comment starts at line 2 (self-describing convention).
set -euo pipefail
@@ -15,6 +12,20 @@ set -euo pipefail
CMD="${1:-shell}"
shift || true
if [ -x /usr/local/bin/brain-serve ]; then
case "$CMD" in
shell) exec bash ;;
serve) exec /usr/local/bin/brain-serve "$@" ;;
search) exec /usr/local/bin/brain-search "$@" ;;
watch) exec /usr/local/bin/brain-watch "$@" ;;
index)
echo "index is the Python sidecar: docker compose --profile index run --rm index" >&2
exit 2
;;
*) echo "unknown command: $CMD (api: serve|search|watch)" >&2; exit 2 ;;
esac
fi
case "$CMD" in
shell) exec bash ;;
search) exec "$KB_PY" /app/bin/kb/search "$@" ;;
+45 -18
View File
@@ -1,14 +1,15 @@
#!/usr/bin/env python3
"""facts/audit - evidence & lexicon checks for the 2dph brain.
bin/facts/audit self # lexicon: every fact in db has >=2 sources
bin/facts/audit db # evidence gate: run against var/kb.lbug
bin/facts/audit self # lexicon: docs + two-source rule
bin/facts/audit db # evidence gate against var/kb.lbug
bin/facts/audit contradict # D16 adjudication (JSON claim(s) on stdin)
`self` mode checks the repo itself (no network, no runtime deps). It greps
for known-good two-source pairings and confirms the docs are consistent.
`db` mode loads every Leaf with root=facts and asserts each has source_rev
and a non-empty `loc` (the "where did you see it" evidence pointer) and that
'confirmed' facts carry a two-source `source` field.
`self` mode checks the repo itself (no network, no runtime deps).
`db` mode loads every Leaf with root=facts. Confirmed facts need ` x `;
hypothesis contradictions need `a x b vs c x d` (both sides ≥2).
`contradict` applies temporal_freshness then authority_pairing; ≥2 vs ≥2
with no rule stays hypothesis / `(not confirmed)`.
Exit 0 = all checks pass, 1 = audit failures, 2 = could not evaluate.
"""
@@ -22,6 +23,8 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from contradict import adjudicate, check_fact_row # noqa: E402
def audit_db() -> list[str]:
from kblib import connect
@@ -33,14 +36,8 @@ def audit_db() -> list[str]:
r = conn.execute("MATCH (l:Leaf {root:'facts'}) RETURN l.id, l.source, l.loc, l.how, l.confidence")
problems: list[str] = []
for lid, source, loc, how, conf in r.get_all():
if conf != "confirmed":
problems.append(f"{lid}: facts require confidence='confirmed', got '{conf}'")
if not source or " x " not in source:
problems.append(f"{lid}: needs 2-source evidence in source, got '{source}'")
if not loc:
problems.append(f"{lid}: missing loc (evidence pointer)")
if not how:
problems.append(f"{lid}: missing how")
problems.extend(check_fact_row(str(lid), str(source or ""), str(loc or ""),
str(how or ""), str(conf or "")))
conn.close()
db.close()
return problems
@@ -54,20 +51,50 @@ def audit_self() -> list[str]:
problems.append("PLAN.md missing recall@5 gate")
if re.search(r"(?i)facts must have.*2 sources|2.source", plan) is None:
problems.append("PLAN.md missing the two-source evidence rule for facts")
if "temporal_freshness" not in plan or "authority_pairing" not in plan:
problems.append("PLAN.md missing D16 adjudication rules")
if re.search(r"(?i)HNSW|BM25|deduction", (ROOT / "README.md").read_text()) is None:
problems.append("README.md missing search/retrieval description")
return problems
def audit_contradict(raw: str) -> tuple[list[str], list[dict]]:
raw = raw.strip()
if not raw:
return ["contradict: empty stdin (JSON claim or {claims:[...]})"], []
try:
payload = json.loads(raw)
except json.JSONDecodeError as e:
return [f"contradict: invalid JSON: {e}"], []
if isinstance(payload, dict) and "claims" in payload:
claims = list(payload.get("claims") or [])
elif isinstance(payload, dict):
claims = [payload]
elif isinstance(payload, list):
claims = payload
else:
return ["contradict: expected object or list"], []
details = [adjudicate(c) for c in claims]
return [], details
def main(argv: list[str]) -> int:
import argparse
p = argparse.ArgumentParser(description="evidence & lexicon audit")
p.add_argument("mode", choices=("self", "db"))
p.add_argument("mode", choices=("self", "db", "contradict"))
p.add_argument("--json", action="store_true")
a = p.parse_args(argv)
problems = audit_self() if a.mode == "self" else audit_db()
out = {"mode": a.mode, "ok": not problems, "problems": problems}
details: list[dict] = []
if a.mode == "self":
problems = audit_self()
elif a.mode == "db":
problems = audit_db()
else:
problems, details = audit_contradict(sys.stdin.read())
out: dict = {"mode": a.mode, "ok": not problems, "problems": problems}
if details:
out["contradictions"] = details
if a.json:
print(json.dumps(out, indent=2))
else:
+1
View File
@@ -5,6 +5,7 @@
//
// ./bin/facts/audit.go self
// ./bin/facts/audit.go db
// ./bin/facts/audit.go contradict --json < claim.json
//
// Python bin/facts/audit is the implementation (CI runs it directly).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
+6 -49
View File
@@ -16,9 +16,8 @@ import (
"fmt"
"os"
"path/filepath"
"strconv"
"time"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/cmdbin"
"github.com/eSlider/2dph/internal/gitlog"
)
@@ -28,50 +27,17 @@ func main() {
}
func run(args []string) int {
var repo, root, since string
limit := 0
jsonOut := false
i := 0
for i < len(args) {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--limit" && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 0 {
fmt.Fprintf(os.Stderr, "git/import: --limit must be a non-negative integer\n")
return 2
}
limit = n
case a == "--since" && i+1 < len(args):
i++
since = args[i]
case a == "--root" && i+1 < len(args):
i++
root = args[i]
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/git/import.go [REPO] [--json] [--limit N] [--since DATE] [--root DIR]`)
return 0
case len(a) > 0 && a[0] != '-':
repo = a
default:
fmt.Fprintf(os.Stderr, "git/import: unknown flag %s\n", a)
return 2
}
i++
c, err := gitlog.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
repo, root, since, limit, jsonOut := c.Repo, c.Root, c.Since, c.Limit, c.JSONOut
var sinceT time.Time
if since != "" {
var err error
sinceT, err = parseSince(since)
sinceT, err := gitlog.ParseSince(since)
if err != nil {
fmt.Fprintf(os.Stderr, "git/import: %v\n", err)
return 2
}
}
repos := []string{}
if repo != "" {
@@ -132,12 +98,3 @@ func run(args []string) int {
}
return 0
}
func parseSince(s string) (time.Time, error) {
for _, layout := range []string{time.RFC3339, "2006-01-02"} {
if t, err := time.Parse(layout, s); err == nil {
return t, nil
}
}
return time.Time{}, fmt.Errorf("cannot parse --since %q", s)
}
Executable
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""kb/add - incremental leaf write (no rebuild).
bin/kb/add --text T --root facts|info --source S
bin/kb/add --json # stdin: one object or {"leafs":[...]}
bin/kb/add --db PATH --json
Writes facts+info in one Ladybug transaction. Does not delete kb.lbug.
Embedding is used when provided; otherwise model2vec encodes the text.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402
EMBED_DIM,
add_leafs,
connect,
ensure_indexes,
init_schema,
)
def _as_leafs(payload: object) -> list[dict]:
if isinstance(payload, list):
return [dict(x) for x in payload]
if isinstance(payload, dict):
if "leafs" in payload:
return [dict(x) for x in payload["leafs"]]
return [dict(payload)]
raise ValueError("json must be an object, a list, or {leafs:[...]}")
def _embed_missing(leafs: list[dict]) -> None:
missing = [lf for lf in leafs if not lf.get("embedding")]
if not missing:
return
from model2vec import StaticModel
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
for lf in missing:
text = str(lf.get("text") or "")
vec = model.encode([text])[0].astype(float).tolist()
if len(vec) != EMBED_DIM:
vec = (vec + [0.0] * EMBED_DIM)[:EMBED_DIM]
lf["embedding"] = vec
def main(argv: list[str]) -> int:
import argparse
p = argparse.ArgumentParser(description="add leafs without rebuilding the brain")
p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)")
p.add_argument("--json", action="store_true", help="read leaf JSON from stdin")
p.add_argument("--text", default="", help="leaf text")
p.add_argument("--root", default="info", choices=("facts", "info"))
p.add_argument("--source", default="")
p.add_argument("--confidence", default="confirmed")
p.add_argument("--source-rev", default="working-tree")
p.add_argument("--how", default="brain/add")
p.add_argument("--loc", default="")
p.add_argument("--type", default="reference", dest="type_")
args = p.parse_args(argv)
if args.json:
raw = sys.stdin.read()
if not raw.strip():
print("kb/add: empty stdin", file=sys.stderr)
return 2
leafs = _as_leafs(json.loads(raw))
else:
if not args.text or not args.source:
print("kb/add: --text and --source are required (or --json)", file=sys.stderr)
return 2
leafs = [{
"text": args.text,
"root": args.root,
"source": args.source,
"confidence": args.confidence,
"source_rev": args.source_rev,
"how": args.how,
"loc": args.loc or args.source,
"type": args.type_,
}]
for lf in leafs:
if not lf.get("text") or not lf.get("source"):
print("kb/add: each leaf needs text and source", file=sys.stderr)
return 2
_embed_missing(leafs)
from kblib import DB_PATH, VAR
dbpath = Path(args.db) if args.db else DB_PATH
dbpath.parent.mkdir(parents=True, exist_ok=True)
VAR.mkdir(exist_ok=True)
db, conn = connect(dbpath, read_only=False)
init_schema(conn)
ids = add_leafs(conn, leafs)
ensure_indexes(conn)
conn.close()
db.close()
print(json.dumps({"mode": "add", "ids": ids, "db": str(dbpath)}))
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+98 -14
View File
@@ -2,12 +2,15 @@
"""kb/index - build the 2dph brain from markdown + factual leafs.
bin/kb/index [--corpus DIR] [--rebuild] [--limit N]
bin/kb/index --rebuild --with-facts --with-chats
bin/kb/index --json # emit stats as JSON
Reads every .md under the corpus (default: repo root docs, skills, READMEs)
as `info` leafs, embeds them with model2vec (potion-multilingual-128M), and
writes them into var/kb.lbug with FTS + HNSW indexes. `facts` leafs come
from bin/facts/extract (docker x compose x ssh-config pairing).
from bin/facts/extract (docker × compose × ssh-config pairing) when
`--with-facts` is set. `--with-chats` indexes markdown under var/chats/md
(or a given dir) as info. WhatsApp sync stays out of v1.
--rebuild drops the database file and indexes from scratch. Without it a run
is idempotent (MERGE by (source,text) id).
@@ -22,7 +25,7 @@ ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402
connect, ensure_indexes, init_schema, upsert_leaf,
add_leafs, connect, ensure_indexes, init_schema, upsert_leaf, link_from_file,
open_readonly, stats,
)
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
@@ -81,10 +84,11 @@ def index_leafs(conn, leafs: list[dict], embed_fn, limit: int) -> tuple[int, int
for lf in leafs[:limit] if limit else leafs:
query = f"{lf['heading']}\n\n{lf['text']}"
emb = embed_fn(lf["text"]) if lf["text"] else None
upsert_leaf(conn, text=query, root="info", confidence="confirmed",
lid = upsert_leaf(conn, text=query, root="info", confidence="confirmed",
source=lf["source"], source_rev="working-tree",
how="kb/index", loc=lf["source"], type_=lf.get("type", "reference"),
embedding=emb)
link_from_file(conn, lid, lf["source"], repo=str(lf.get("repo") or ""))
count += 1
return count, len(leafs)
@@ -95,12 +99,65 @@ def embedder():
return lambda text: model.encode([text])[0].astype(float).tolist()
def index_fact_dicts(conn, facts: list[dict], embed_fn) -> int:
"""Write extract-shaped dicts as root=facts leafs (2-source source field)."""
leafs = []
for f in facts:
text = str(f.get("text") or "")
source = str(f.get("source") or "")
if not text or not source:
continue
leafs.append({
"text": text,
"root": "facts",
"confidence": "confirmed",
"source": source,
"source_rev": f.get("source_rev") or "working-tree",
"how": f.get("how") or "facts/extract",
"loc": f.get("loc") or source,
"type": "fact",
"embedding": embed_fn(text) if text else None,
})
return len(add_leafs(conn, leafs))
def facts_from_extract() -> list[dict]:
import subprocess
proc = subprocess.run(
[sys.executable, str(ROOT / "bin" / "facts" / "extract"), "--json", "--dry-run"],
cwd=ROOT,
capture_output=True,
text=True,
check=False,
)
if proc.returncode != 0:
print(f"kb/index: facts/extract failed: {proc.stderr}", file=sys.stderr)
return []
try:
payload = json.loads(proc.stdout)
except json.JSONDecodeError:
print("kb/index: facts/extract produced non-JSON", file=sys.stderr)
return []
return list(payload.get("facts") or [])
def main(argv: list[str]) -> int:
import argparse
p = argparse.ArgumentParser(description="build the 2dph brain index")
p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)")
p.add_argument("--rebuild", action="store_true", help="fresh db + indexes")
p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)")
p.add_argument("--no-defaults", action="store_true", help="do not index repo README/docs/skills")
p.add_argument("--with-mail", action="store_true", help="include var/mail message.md leafs")
p.add_argument("--with-facts", action="store_true", help="run facts/extract into root=facts")
p.add_argument("--facts-json", default="", help="JSON list (or {facts:[...]}) of fact dicts")
p.add_argument(
"--with-chats",
nargs="?",
const=str(ROOT / "var" / "chats" / "md"),
default="",
help="index chat markdown as info (default var/chats/md)",
)
p.add_argument("--since", default="", help="with --with-mail, only messages dated >= YYYY-MM-DD")
p.add_argument("--dry-run", action="store_true", help="count leafs, write nothing")
p.add_argument(
@@ -114,44 +171,71 @@ def main(argv: list[str]) -> int:
from kblib import DB_PATH, VAR
leafs = load_corpus(ROOT)
dbpath = Path(a.db) if a.db else DB_PATH
leafs: list[dict] = [] if a.no_defaults else load_corpus(ROOT)
if a.corpus:
for source in a.corpus:
leafs.extend(load_corpus_glob(source))
chat_n = 0
if a.with_chats:
chats = load_corpus_glob(a.with_chats)
chat_n = len(chats)
leafs.extend(chats)
mail_n = 0
if a.with_mail:
mail = from_mail_root(ROOT / "var" / "mail", since=a.since)
mail_n = len(mail)
leafs.extend(mail)
facts: list[dict] = []
if a.facts_json:
raw = Path(a.facts_json).read_text(encoding="utf-8")
payload = json.loads(raw)
facts = list(payload.get("facts") if isinstance(payload, dict) else payload)
if a.with_facts:
facts.extend(facts_from_extract())
if a.dry_run:
msg = {"indexed": 0, "corpus_total": len(leafs), "mail_leafs": mail_n, "dry_run": True}
msg = {
"indexed": 0,
"corpus_total": len(leafs),
"mail_leafs": mail_n,
"chat_leafs": chat_n,
"facts_leafs": len(facts),
"dry_run": True,
}
print(json.dumps(msg, indent=2) if a.json else
f"brain/index: {len(leafs)} leafs would be indexed (mail={mail_n})")
f"brain/index: {len(leafs)} info + {len(facts)} facts would be indexed")
return 0
VAR.mkdir(exist_ok=True)
if a.rebuild and DB_PATH.exists():
DB_PATH.unlink()
dbpath.parent.mkdir(parents=True, exist_ok=True)
if a.rebuild and dbpath.exists():
dbpath.unlink()
db, conn = connect(DB_PATH, read_only=False)
db, conn = connect(dbpath, read_only=False)
init_schema(conn)
# Never DROP FTS/VECTOR (ghost catalog). Write leafs, then ensure indexes
# unless --skip-indexes (seed facts first — MERGE under live FTS corrupts it).
# --rebuild already deleted kb.lbug above, so CREATE runs on a clean DB.
embed = embedder()
done, total = index_leafs(conn, leafs, embed, a.limit)
fact_n = index_fact_dicts(conn, facts, embed) if facts else 0
if not a.skip_indexes:
ensure_indexes(conn)
s = stats(conn)
conn.close()
db.close()
result = {"indexed": done, "corpus_total": total, **{k: v for k, v in s.items() if k in ("total", "by_root")}}
result = {
"indexed": done,
"corpus_total": total,
"facts_leafs": fact_n,
"chat_leafs": chat_n,
**{k: v for k, v in s.items() if k in ("total", "by_root")},
}
if a.skip_indexes:
result["indexes"] = "skipped"
print(json.dumps(result, indent=2) if a.json else f"indexed {done}/{total} leafs; db total {s['total']}")
print(json.dumps(result, indent=2) if a.json else
f"indexed {done}/{total} info + {fact_n} facts; db total {s['total']}")
return 0
+4 -6
View File
@@ -1,10 +1,9 @@
#!/usr/bin/env bash
# bin/kb/search — deprecated wrapper. Use bin/brain/search.go.
# Sets CGO for ladybug, builds a binary (embed daemon needs a real executable),
# then execs it. Prints one deprecation line.
# CGO via Zig (bin/cgo/zig), not gcc. Builds a binary then execs it.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
BIN="$ROOT/var/bin/brain-search"
SRC="$ROOT/internal/brain"
CMD="$ROOT/bin/brain"
@@ -24,11 +23,10 @@ else
fi
if [ "$need_build" -eq 1 ]; then
echo "Building brain/search..." >&2
echo "Building brain/search (zig cc)..." >&2
(
cd "$ROOT" &&
CGO_CFLAGS="-I$ROOT/lib-ladybug" \
CGO_LDFLAGS="-L$ROOT/lib-ladybug -Wl,-rpath,$ROOT/lib-ladybug" \
eval "$("$ROOT/bin/cgo/zig" env)" &&
go build -tags system_ladybug -o "$BIN" ./bin/brain
) || exit 1
fi
+11 -77
View File
@@ -7,7 +7,7 @@
bin/mail/import --since 2026-01-01 only messages after a date
bin/mail/import --limit 50 cap messages per run
bin/mail/import --no-attachments body only, skip attachment conversion
bin/mail/import --ocr OCR scanned PDFs/images via docling
bin/mail/import --ocr OCR images (PDFs OCR when textless)
bin/mail/import --dry-run list messages without writing anything
Writes one directory per message: var/mail/{folder}/{message_id}/
@@ -16,10 +16,11 @@ Writes one directory per message: var/mail/{folder}/{message_id}/
attachments/*.md converted attachment content
Indexing is a separate step (`bin/brain/index.go --rebuild`): conversion can
crash in native docling and must not leave the brain DB mid-transaction.
crash and must not leave the brain DB mid-transaction.
Requires ONLYOFFICE_URL/USER/PASS in .env (or env). Idempotent: a message
already present (message.md exists) is skipped unless --force.
Requires ONLYOFFICE_URL/USER/PASS in .env (or env) except `--from-raw`.
Idempotent: a message already present (message.md exists) is skipped unless
--force.
"""
from __future__ import annotations
@@ -41,10 +42,10 @@ from mailconv import ( # noqa: E402
IMAGE_SUFFIXES,
LEGACY_OFFICE_SUFFIXES,
TEXT_SUFFIXES,
convert_pdf,
html_to_markdown,
is_convertible,
normalize_markdown,
subject_to_filename,
ocr_image,
zip_extract_safe,
)
@@ -147,9 +148,9 @@ def convert_file_to_md(path: Path, ocr: bool) -> str | None:
except Exception as e:
return f"\n<!-- conversion failed: {e} -->\n"
if suffix == ".pdf":
return _convert_pdf(path, ocr)
return convert_pdf(path, ocr)
if suffix in IMAGE_SUFFIXES and ocr:
return _convert_pdf(path, ocr)
return ocr_image(path) or "\n<!-- ocr unavailable -->\n"
if suffix in LEGACY_OFFICE_SUFFIXES:
return _convert_legacy(path)
if suffix in ARCHIVE_SUFFIXES:
@@ -157,67 +158,6 @@ def convert_file_to_md(path: Path, ocr: bool) -> str | None:
return None
def _convert_pdf(path: Path, ocr: bool) -> str:
"""Convert one PDF to markdown.
Fast path: poppler's pdftotext (-layout) extracts exact text from
born-digital PDFs in ~15ms vs docling's 1-3s. Only textless PDFs (scanned
pages, layout-heavy) fall back to docling, which runs isolated in a
subprocess because its native onnx/RT-DETR has segfaulted the main process.
"""
text = _pdf_fast_text(path)
if ocr or text is None or not text.strip():
return _convert_pdf_docling(path, ocr)
return normalize_markdown(text)
def _pdf_fast_text(path: Path) -> str | None:
"""pdftotext -layout; None when poppler is unavailable (or the PDF has no text layer)."""
try:
proc = subprocess.run(
["pdftotext", "-layout", str(path), "-"],
capture_output=True, timeout=60)
except (OSError, subprocess.TimeoutExpired):
return None
if proc.returncode != 0:
return None
return proc.stdout.decode("utf-8", errors="replace")
def _convert_pdf_docling(path: Path, ocr: bool) -> str:
try:
proc = subprocess.run(
[sys.executable, os.path.abspath(__file__), "--pdf-worker", str(path),
"--ocr" if ocr else "--no-ocr"],
capture_output=True, text=True, timeout=600)
except subprocess.TimeoutExpired:
return "\n<!-- pdf conversion timed out -->\n"
if proc.returncode != 0:
tail = proc.stderr.strip().splitlines()[-3:]
return f"\n<!-- pdf conversion failed: {proc.returncode}: {' | '.join(tail)} -->\n"
return proc.stdout
def _pdf_worker(path: Path, ocr: bool) -> None:
"""docling worker entry: prints converted markdown on stdout, exits non-zero on error."""
try:
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.pipeline_options import PdfPipelineOptions
opts = PdfPipelineOptions()
opts.do_ocr = bool(ocr)
opts.do_table_structure = True
conv = DocumentConverter(format_options={"pdf": PdfFormatOption(pipeline_options=opts)})
res = conv.convert(str(path))
sys.stdout.write(normalize_markdown(res.document.export_to_markdown()))
sys.exit(0)
except Exception as e:
# errors/stacktraces to stderr; the caller only reports a one-liner
print(f"pdf-worker: {e}", file=sys.stderr)
import traceback
traceback.print_exc(file=sys.stderr)
sys.exit(1)
def _convert_legacy(path: Path) -> str:
"""Legacy .doc/.xls/.ppt -> md via pandoc (installed) or a stub."""
try:
@@ -356,19 +296,12 @@ def main(argv: list[str]) -> int:
p.add_argument("--from-raw", default="",
help="convert Go-synced dirs (var/mail/<folder>/<id>/message.json) to markdown")
p.add_argument("--no-attachments", action="store_true", help="skip attachment download+convert")
p.add_argument("--ocr", action="store_true", help="OCR scanned PDFs/images via docling")
p.add_argument("--ocr", action="store_true", help="OCR images (PDFs OCR when textless)")
p.add_argument("--force", action="store_true", help="re-import even if message.md exists")
p.add_argument("--dry-run", action="store_true", help="list messages, write nothing")
p.add_argument("--json", action="store_true")
p.add_argument("--pdf-worker", default="", help=argparse.SUPPRESS)
p.add_argument("--no-ocr", action="store_true", help=argparse.SUPPRESS)
a = p.parse_args(argv)
if a.pdf_worker:
_pdf_worker(Path(a.pdf_worker), ocr=not a.no_ocr)
return 0
conf = load_env()
fid = folder_id(a.folder)
out_root = ROOT / "var" / "mail"
summary: list[dict] = []
@@ -394,6 +327,7 @@ def main(argv: list[str]) -> int:
target_dir=msg_dir.parent))
summary.append(entry)
else:
conf = load_env()
OOCLIENT = OOClient(conf)
if a.id:
messages = [{"id": i} for i in a.id]
+46
View File
@@ -0,0 +1,46 @@
//usr/bin/env go run -tags=mail_ocr "$0" "$@"; exit
//go:build mail_ocr
//
// bin/mail/ocr.go - OCR an image or scanned PDF (tesseract eng+deu).
//
// ./bin/mail/ocr.go scan.png
// ./bin/mail/ocr.go scan.pdf
// OCR_ENGINE=paddle ./bin/mail/ocr.go scan.png
//
// PDFs try pdftotext -layout first; empty text layer uses pdftoppm + tesseract.
// No gocv. Tesseract CGO bindings are not used (D21 Zig owns Ladybug CGO).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"strings"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/ocr"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := ocr.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
path := c.Path
var text string
if strings.HasSuffix(strings.ToLower(path), ".pdf") {
text, err = ocr.PDFFile(path)
} else {
text, err = ocr.ImageFile(path)
}
if err != nil {
fmt.Fprintf(os.Stderr, "mail/ocr: %v\n", err)
return 1
}
fmt.Println(text)
return 0
}
+68 -40
View File
@@ -5,12 +5,15 @@ package sync
import (
"context"
"flag"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
// CLIConfig is a superset of SyncConfig plus flag parsing results.
@@ -21,58 +24,84 @@ type CLIConfig struct {
Help bool
}
// ParseCLI reads os.Args into a CLIConfig. Exit codes: 0 ok, 2 usage.
type flagVals struct {
env, out, srcs, query string
workers, limit, offset int
force, dryRun bool
}
func Parser() *flaggy.Parser {
v := flagVals{workers: 4, query: "in:inbox", srcs: "onlyoffice"}
return bind(&v)
}
func bind(v *flagVals) *flaggy.Parser {
if v.workers == 0 {
v.workers = 4
}
if v.query == "" {
v.query = "in:inbox"
}
if v.srcs == "" {
v.srcs = "onlyoffice"
}
p := cliparse.New("mail-sync")
p.Description = "download mail to var/mail"
p.String(&v.env, "", "env", ".env file")
p.String(&v.out, "", "out", "var/mail root")
p.Int(&v.workers, "", "workers", "concurrent downloads")
p.Int(&v.limit, "", "limit", "max messages per source (0 = all)")
p.Int(&v.offset, "", "offset", "skip first N messages per source")
p.Bool(&v.force, "", "force", "overwrite existing message.json")
p.Bool(&v.dryRun, "", "dry-run", "list counts without writing")
p.String(&v.query, "", "query", "Gmail search query")
p.String(&v.srcs, "", "source", "comma list: onlyoffice,gmail")
return p
}
// ParseCLI reads args into a CLIConfig. Exit codes: 0 ok, 2 usage.
func ParseCLI(args []string) (CLIConfig, int, error) {
fs := flag.NewFlagSet("mail/sync", flag.ContinueOnError)
var (
env = fs.String("env", "", ".env file (default: <cwd>/.env)")
out = fs.String("out", "", "var/mail root (default: <cwd>/var/mail)")
workers = fs.Int("workers", 4, "concurrent downloads")
limit = fs.Int("limit", 0, "max messages per source (0 = all)")
offset = fs.Int("offset", 0, "skip first N messages per source")
force = fs.Bool("force", false, "overwrite existing message.json + attachments")
dryRun = fs.Bool("dry-run", false, "list message counts without writing")
query = fs.String("query", "in:inbox", "Gmail search query (gmail source only)")
srcs = fs.String("source", "onlyoffice", "comma list: onlyoffice,gmail (default onlyoffice)")
help = fs.Bool("help", false, "usage")
)
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
v := flagVals{workers: 4, query: "in:inbox", srcs: "onlyoffice"}
p := bind(&v)
if err := cliparse.Parse(p, args); err != nil {
if errors.Is(err, cliparse.ErrHelp) {
return CLIConfig{Help: true}, 0, nil
}
return CLIConfig{}, 2, err
}
if *help || fs.NArg() > 0 {
if len(p.TrailingArguments) > 0 {
return CLIConfig{Help: true}, 0, nil
}
wd, err := os.Getwd()
if err != nil {
return CLIConfig{}, 2, err
}
if *env == "" {
*env = filepath.Join(wd, ".env")
if v.env == "" {
v.env = filepath.Join(wd, ".env")
}
if *out == "" {
*out = filepath.Join(wd, "var", "mail")
if v.out == "" {
v.out = filepath.Join(wd, "var", "mail")
}
envVars := readEnv(*env)
envVars := readEnv(v.env)
cfg := SyncConfig{
Out: *out,
Workers: *workers,
Limit: *limit,
Offset: *offset,
Force: *force,
DryRun: *dryRun,
Query: *query,
Out: v.out,
Workers: v.workers,
Limit: v.limit,
Offset: v.offset,
Force: v.force,
DryRun: v.dryRun,
Query: v.query,
Policy: RetryPolicy{},
}
cli := CLIConfig{Sync: cfg, Env: *env, Sources: *srcs}
for _, s := range strings.Split(*srcs, ",") {
out := CLIConfig{Sync: cfg, Env: v.env, Sources: v.srcs}
for _, s := range strings.Split(v.srcs, ",") {
switch strings.TrimSpace(s) {
case "onlyoffice":
u := pick(envVars["ONLYOFFICE_URL"], envVars["OO_URL"])
user := pick(envVars["ONLYOFFICE_USER"], envVars["OO_USER"])
pass := pick(envVars["ONLYOFFICE_PASS"], envVars["OO_PASSWORD"])
if u == "" || user == "" || pass == "" {
return CLIConfig{}, 2, fmt.Errorf("onlyoffice source needs ONLYOFFICE_URL/USER/PASS in %s", *env)
return CLIConfig{}, 2, fmt.Errorf("onlyoffice source needs ONLYOFFICE_URL/USER/PASS in %s", v.env)
}
cfg.OO = &OOConfig{URL: u, User: user, Password: pass}
case "gmail":
@@ -85,30 +114,30 @@ func ParseCLI(args []string) (CLIConfig, int, error) {
return CLIConfig{}, 2, fmt.Errorf("unknown source %q", s)
}
}
cli.Sync = cfg
return cli, 0, nil
out.Sync = cfg
return out, 0, nil
}
// Main is the CLI entry: returns process exit code.
func Main(args []string) int {
cli, code, err := ParseCLI(args)
cfg, code, err := ParseCLI(args)
if err != nil {
fmt.Fprintln(os.Stderr, "mail/sync:", err)
return code
}
if cli.Help {
if cfg.Help {
fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--query GMAIL_Q] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]")
return 0
}
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
defer cancel()
start := time.Now()
stats, err := Run(ctx, cli.Sync)
stats, err := Run(ctx, cfg.Sync)
if err != nil {
fmt.Fprintln(os.Stderr, "mail/sync:", err)
return 1
}
if cli.Sync.DryRun {
if cfg.Sync.DryRun {
fmt.Printf("mail/sync: dry-run checked=%d (no writes)\n", stats.Checked)
return 0
}
@@ -135,7 +164,6 @@ func readEnv(path string) map[string]string {
k, v, _ := strings.Cut(line, "=")
out[strings.TrimSpace(k)] = strings.Trim(strings.TrimSpace(v), "\"'")
}
// env overrides file
for _, kv := range os.Environ() {
k, v, ok := strings.Cut(kv, "=")
if !ok {
+70 -5
View File
@@ -1,20 +1,85 @@
//usr/bin/env go run -tags=markdown_import "$0" "$@"; exit
//go:build markdown_import
//usr/bin/env go run "$0" "$@"; exit
//
// bin/markdown/import.go - split markdown into leafs (mistune).
// bin/markdown/import.go - split markdown into leafs (H2 boundaries).
//
// ./bin/markdown/import.go [dir]
// ./bin/markdown/import.go --files a.md,b.md --json
//
// Conversion only. Brain write is bin/brain/index.go.
// Python bin/md/import remains as a fallback.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"strings"
"github.com/eSlider/2dph/internal/cmdbin"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/mdleaves"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/md/import", os.Args[1:]))
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := mdleaves.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
jsonOut := c.JSONOut
files := c.Files
root := c.Root
var paths []string
if files != "" {
for _, f := range strings.Split(files, ",") {
f = strings.TrimSpace(f)
if f != "" {
paths = append(paths, f)
}
}
} else {
st, err := os.Stat(root)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: no such path %s\n", root)
return 2
}
if !st.IsDir() {
paths = []string{root}
} else {
var err error
paths, err = mdleaves.WalkMarkdown(root)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: %v\n", err)
return 1
}
}
}
if len(paths) == 0 {
fmt.Fprintln(os.Stderr, "md/import: no markdown files")
return 1
}
var all []mdleaves.Leaf
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: %s: %v\n", p, err)
continue
}
all = append(all, mdleaves.ToAll(string(raw), p, "")...)
}
if jsonOut {
s, err := mdleaves.EncodeJSON(all)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Print(s)
return 0
}
fmt.Print(mdleaves.EncodeYAML(all))
return 0
}
+61
View File
@@ -0,0 +1,61 @@
//usr/bin/env go run -tags=qa_stats "$0" "$@"; exit
//go:build qa_stats
//
// bin/qa/stats.go - DuckDB quantiles over a JSON number array or JSONL count.
//
// ./bin/qa/stats.go <<< '[1,2,3,4,5]'
// ./bin/qa/stats.go --jsonl rows.jsonl
//
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
// DuckDB CGO needs gcc/g++ (not Zig). After eval "$(bin/cgo/zig env)":
// CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= ./bin/qa/stats.go
package main
import (
"encoding/json"
"fmt"
"io"
"os"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/duckstats"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := cliparse.ParseQAStats(args)
if err != nil {
return cliparse.Fail(err)
}
jsonl := c.JSONL
if jsonl != "" {
n, err := duckstats.CountJSONL(jsonl)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Printf("n: %d\n", n)
return 0
}
raw, err := io.ReadAll(os.Stdin)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
var samples []float64
if err := json.Unmarshal(raw, &samples); err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
s, err := duckstats.Quantiles(samples)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Printf("n: %d\nmin: %g\np50: %g\np95: %g\nmax: %g\navg: %g\n",
s.N, s.Min, s.P50, s.P95, s.Max, s.Avg)
return 0
}
+73
View File
@@ -0,0 +1,73 @@
//usr/bin/env go run -tags=reasoner_bakeoff "$0" "$@"; exit
//go:build reasoner_bakeoff
//
// bin/reasoner/bakeoff.go - CPU tool-call bake-off against an OpenAI-compatible URL (D18).
//
// REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b ./bin/reasoner/bakeoff.go
// ./bin/reasoner/bakeoff.go --model MichelRosselli/bonsai-27b:Q1_0 --json
//
// Measures OpenAI tool_calls (search/get/audit) and RSS from Ollama /api/ps, not VRAM.
// PicoClaw is compose profile picoclaw; tool names match internal/httpapi MCP ops.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"encoding/json"
"fmt"
"os"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/duckstats"
"github.com/eSlider/2dph/internal/reasoner"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := reasoner.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
base, model, jsonOut, device := c.Base, c.Model, c.JSONOut, c.Device
client := reasoner.Client{BaseURL: base, Model: model, Device: device}
rep := reasoner.Run(client)
lat := make([]float64, 0, len(rep.Prompts))
for _, p := range rep.Prompts {
lat = append(lat, float64(p.LatencyMS))
}
if st, err := duckstats.Quantiles(lat); err == nil {
rep.LatencyP50MS = st.P50
rep.LatencyP95MS = st.P95
}
raw, err := json.MarshalIndent(rep, "", " ")
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
if jsonOut {
fmt.Println(string(raw))
} else {
fmt.Printf("model: %s\n", rep.Model)
fmt.Printf("hf_id: %s\n", rep.HF)
fmt.Printf("device: %s\n", rep.Device)
fmt.Printf("tool_call: %d/%d\n", rep.ToolCallOK, rep.ToolCallN)
fmt.Printf("xml_leak: %d\n", rep.XMLLeak)
fmt.Printf("rss_mb: %d\n", rep.RSSMB)
fmt.Printf("vram_mb: %d\n", rep.VRAMMB)
fmt.Printf("latency_p50_ms: %g\n", rep.LatencyP50MS)
fmt.Printf("latency_p95_ms: %g\n", rep.LatencyP95MS)
for _, p := range rep.Prompts {
status := "fail"
if p.OK {
status = "ok"
}
fmt.Printf(" %s: %s wanted=%s got=%s xml=%v %dms %s\n", p.WantedTool, status, p.WantedTool, p.ToolName, p.XMLLeak, p.LatencyMS, p.Err)
}
}
if rep.ToolCallN == 0 {
return 1
}
return 0
}
+2
View File
@@ -0,0 +1,2 @@
// Commands in this directory are shebang mains (bakeoff.go).
package main
+103
View File
@@ -0,0 +1,103 @@
"""D16 contradiction adjudication (same rules as internal/facts)."""
from __future__ import annotations
from typing import Any
CONF_CONFIRMED = "confirmed"
CONF_HYPOTHESIS = "hypothesis"
RULE_UNRESOLVED = "unresolved"
RULE_TEMPORAL = "temporal_freshness"
RULE_AUTHORITY = "authority_pairing"
RULE_TWO_SOURCE = "two_source"
RULE_SINGLE = "single_source"
KIND_RUNTIME = "runtime"
KIND_CONFIG = "config"
KIND_NARRATIVE = "narrative"
def _independent(sources: list[dict]) -> int:
seen: set[str] = set()
for i, s in enumerate(sources):
sid = str(s.get("id") or "") or f"{s.get('kind', '')}#{i}"
seen.add(sid)
return len(seen)
def _fresh_n(sources: list[dict]) -> int:
return sum(1 for s in sources if not s.get("stale"))
def _strong_n(sources: list[dict]) -> int:
return sum(1 for s in sources if s.get("kind") in (KIND_RUNTIME, KIND_CONFIG))
def adjudicate(claim: dict[str, Any]) -> dict[str, Any]:
yes = list(claim.get("yes") or [])
no = list(claim.get("no") or [])
yes_n, no_n = _independent(yes), _independent(no)
text = str(claim.get("text") or "")
def out(conf: str, rule: str, winner: str = "") -> dict[str, Any]:
return {
"text": text,
"confidence": conf,
"confirmed": conf == CONF_CONFIRMED,
"rule": rule,
"winner": winner,
"yes": yes_n,
"no": no_n,
}
if yes_n < 2 or no_n < 2:
if yes_n >= 2:
return out(CONF_CONFIRMED, RULE_TWO_SOURCE, "yes")
if no_n >= 2:
return out(CONF_CONFIRMED, RULE_TWO_SOURCE, "no")
return out(CONF_HYPOTHESIS, RULE_SINGLE)
yf, nf = _fresh_n(yes), _fresh_n(no)
if yf >= 2 and nf < 2:
return out(CONF_CONFIRMED, RULE_TEMPORAL, "yes")
if nf >= 2 and yf < 2:
return out(CONF_CONFIRMED, RULE_TEMPORAL, "no")
ys, ns = _strong_n(yes), _strong_n(no)
if ys >= 2 and ns < 2:
return out(CONF_CONFIRMED, RULE_AUTHORITY, "yes")
if ns >= 2 and ys < 2:
return out(CONF_CONFIRMED, RULE_AUTHORITY, "no")
return out(CONF_HYPOTHESIS, RULE_UNRESOLVED)
def parse_source_field(source: str) -> tuple[str, str]:
"""Split `a x b vs c x d` into (yes, no). Empty no if no ` vs `."""
if " vs " not in source:
return source, ""
yes, _, no = source.partition(" vs ")
return yes.strip(), no.strip()
def check_fact_row(lid: str, source: str, loc: str, how: str, conf: str) -> list[str]:
"""Lexicon checks for one facts leaf (no Ladybug)."""
problems: list[str] = []
src = source or ""
if conf == CONF_CONFIRMED:
if " vs " in src:
problems.append(f"{lid}: confirmed fact cannot keep a vs-contradiction")
if " x " not in src:
problems.append(f"{lid}: needs 2-source evidence in source, got '{source}'")
elif conf == CONF_HYPOTHESIS:
yes, no = parse_source_field(src)
if not no or " x " not in yes or " x " not in no:
problems.append(
f"{lid}: hypothesis contradiction needs 'a x b vs c x d', got '{source}'"
)
elif conf == "partial":
pass
else:
problems.append(f"{lid}: unknown confidence '{conf}'")
if not loc:
problems.append(f"{lid}: missing loc (evidence pointer)")
if not how:
problems.append(f"{lid}: missing how")
return problems
+92 -1
View File
@@ -4,7 +4,7 @@ Single embedded graph `var/kb.lbug`. Two roots: facts (assertions backed by
>=2 independent sources) and info (narrative leafs). Hybrid retrieval: BM25
(FTS extension) + HNSW cosine (VECTOR extension) + Cypher graph hops.
All access is read-only unless `--rebuild` is passed to kb/index.
All access is read-only unless `--rebuild` (kb/index) or `kb/add`.
"""
from __future__ import annotations
@@ -118,6 +118,97 @@ def upsert_leaf(conn: ladybug.Connection, *, text: str, root: str, confidence: s
return lid
def add_leafs(conn: ladybug.Connection, leafs: list[dict]) -> list[str]:
"""Write facts+info leafs in one transaction. Safe while FTS/HNSW exist.
Each leaf dict: text, source, optional root/confidence/source_rev/how/loc/type/embedding.
Does not delete the database file. Measured on Ladybug 0.19: MERGE of new
ids (and updates) stays FTS+HNSW queryable; DROP INDEX is the fatal path.
"""
if not leafs:
return []
started = False
try:
conn.execute("BEGIN TRANSACTION")
started = True
except Exception:
started = False
ids: list[str] = []
try:
for lf in leafs:
ids.append(
upsert_leaf(
conn,
text=str(lf["text"]),
root=str(lf.get("root") or ROOT_INFO),
confidence=str(lf.get("confidence") or CONF_CONFIRMED),
source=str(lf["source"]),
source_rev=str(lf.get("source_rev") or "working-tree"),
how=str(lf.get("how") or "brain/add"),
loc=str(lf.get("loc") or lf.get("source") or ""),
type_=str(lf.get("type") or lf.get("type_") or "reference"),
embedding=lf.get("embedding"),
)
)
if started:
conn.execute("COMMIT")
except Exception:
if started:
try:
conn.execute("ROLLBACK")
except Exception:
pass
raise
return ids
def file_id(repo: str, path: str) -> str:
"""Stable File.id matching gitimport (`repo:path`)."""
return f"{repo}:{path}" if repo else path
def link_from_file(conn: ladybug.Connection, leaf_id: str, path: str,
repo: str = "", mtime: str = "") -> str:
"""MERGE File and Leaf-[:FROM_FILE]->File so --hop 1 can walk."""
fid = file_id(repo, path)
conn.execute(
"MERGE (f:File {id:$id}) SET f.path=$path, f.repo=$repo, f.mtime=$mtime",
parameters={"id": fid, "path": path, "repo": repo, "mtime": mtime},
)
conn.execute(
"MATCH (l:Leaf {id:$lid}), (f:File {id:$fid}) "
"MERGE (l)-[:FROM_FILE]->(f)",
parameters={"lid": leaf_id, "fid": fid},
)
return fid
HOP_STMTS = {
1: "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File) RETURN f.id, f.path, 1",
2: ("MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit) "
"RETURN c.id, c.subject, 2"),
3: ("MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit)"
"-[:AUTHORED]->(p:Person) RETURN p.id, p.name, 3"),
}
HOP_LABELS = {1: "File", 2: "Commit", 3: "Person"}
def hop_walk(conn: ladybug.Connection, leaf_id: str, n: int) -> list[dict]:
"""Walk Leaf → File → Commit → Person up to n hops (max 3)."""
depth = min(max(int(n), 0), 3)
out: list[dict] = []
for d in range(1, depth + 1):
rows = conn.execute(HOP_STMTS[d], parameters={"id": leaf_id}).get_all()
for row in rows:
out.append({
"id": row[0],
"label": HOP_LABELS[d],
"name": row[1],
"depth": int(row[2]),
})
return out
def leaf_index_names(conn: ladybug.Connection) -> set[str]:
"""Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'})."""
rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()
+84 -1
View File
@@ -7,7 +7,10 @@ offline against fixtures.
from __future__ import annotations
import html
import os
import re
import subprocess
import tempfile
import zipfile
from pathlib import Path
@@ -18,9 +21,10 @@ OFFICE_SUFFIXES = {".docx", ".pptx", ".xlsx", ".html", ".htm", ".epub", ".eml",
PDF_SUFFIXES = {".pdf"}
IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".tif", ".webp"}
ARCHIVE_SUFFIXES = {".zip"}
# Legacy binary Office (doc/xls/ppt) — markitdown/docling skip them; we try
# Legacy binary Office (doc/xls/ppt) — markitdown skip them; we try
# pandoc first, else leave a stub.
LEGACY_OFFICE_SUFFIXES = {".doc", ".xls", ".ppt"}
TESS_LANG = "eng+deu"
CONVERTIBLE_SUFFIXES = (
TEXT_SUFFIXES | OFFICE_SUFFIXES | PDF_SUFFIXES | IMAGE_SUFFIXES | ARCHIVE_SUFFIXES | LEGACY_OFFICE_SUFFIXES
@@ -146,3 +150,82 @@ def zip_extract_safe(zip_path: Path, dest: Path) -> list[Path]:
def is_convertible(suffix: str) -> bool:
return suffix.lower() in CONVERTIBLE_SUFFIXES
def convert_pdf(path: Path, ocr: bool = False) -> str:
"""pdftotext -layout first; empty text layer → pdftoppm + tesseract.
`ocr` is unused for born-digital PDFs (text layer wins). Scans OCR
automatically. This path never execs an ONNX document converter.
"""
del ocr # scans OCR when the text layer is empty; flag is for images
text = pdf_fast_text(path)
if text and text.strip():
return normalize_markdown(text)
scanned = ocr_pdf(path)
if scanned and scanned.strip():
return normalize_markdown(scanned)
if text:
return normalize_markdown(text)
return "\n<!-- pdf has no text layer (ocr unavailable) -->\n"
def pdf_fast_text(path: Path) -> str | None:
"""pdftotext -layout; None when poppler is missing or the command fails."""
try:
proc = subprocess.run(
["pdftotext", "-layout", str(path), "-"],
capture_output=True, timeout=60)
except (OSError, subprocess.TimeoutExpired):
return None
if proc.returncode != 0:
return None
return proc.stdout.decode("utf-8", errors="replace")
def ocr_pdf(path: Path) -> str:
"""Rasterize with pdftoppm and OCR each page (tesseract or paddle)."""
try:
with tempfile.TemporaryDirectory(prefix="2dph-ocr-") as tmp:
prefix = str(Path(tmp) / "page")
proc = subprocess.run(
["pdftoppm", "-png", "-r", "200", str(path), prefix],
capture_output=True, timeout=120)
if proc.returncode != 0:
return ""
pages = sorted(Path(tmp).glob("page*.png"))
parts = [ocr_image(p) for p in pages]
return "\n\n".join(p for p in parts if p and p.strip())
except (OSError, subprocess.TimeoutExpired):
return ""
def ocr_image(path: Path) -> str:
engine = os.environ.get("OCR_ENGINE", "tesseract")
if engine == "paddle":
return _ocr_paddle(path)
return _ocr_tesseract(path)
def _ocr_tesseract(path: Path) -> str:
try:
proc = subprocess.run(
["tesseract", str(path), "stdout", "-l", TESS_LANG, "--psm", "6"],
capture_output=True, timeout=120)
except (OSError, subprocess.TimeoutExpired):
return ""
if proc.returncode != 0:
return ""
return proc.stdout.decode("utf-8", errors="replace").strip()
def _ocr_paddle(path: Path) -> str:
try:
proc = subprocess.run(
["paddleocr", "ocr", "-i", str(path)],
capture_output=True, timeout=180)
except (OSError, subprocess.TimeoutExpired):
return ""
if proc.returncode != 0:
return ""
return proc.stdout.decode("utf-8", errors="replace").strip()
+159 -2
View File
@@ -1,6 +1,7 @@
"""D14 layout: bin/{subject}/{method}.go, libs in internal/, one go.mod."""
from __future__ import annotations
import os
import unittest
from pathlib import Path
@@ -74,9 +75,20 @@ class BinLayoutTest(unittest.TestCase):
)
def test_brain_methods_are_shebangs(self) -> None:
for method in ("index.go", "get.go", "stats.go", "eval.go", "watch.go"):
for method in ("index.go", "add.go", "get.go", "stats.go", "eval.go", "watch.go"):
self._assert_shebang(f"bin/brain/{method}")
def test_brain_add_is_python_write_not_rebuild(self) -> None:
self._assert_shebang("bin/brain/add.go")
text = (ROOT / "bin" / "brain" / "add.go").read_text()
self.assertIn("cmdbin.ExecFile", text)
self.assertIn("bin/kb/add", text)
self.assertNotIn("--rebuild", text)
py = (ROOT / "bin" / "kb" / "add").read_text()
self.assertIn("add_leafs", py)
self.assertIn("--json", py)
self.assertNotIn("unlink", py.lower())
def test_brain_get_stats_eval_are_not_python_exec(self) -> None:
for method in ("get.go", "stats.go", "eval.go"):
text = (ROOT / "bin" / "brain" / method).read_text()
@@ -115,6 +127,38 @@ class BinLayoutTest(unittest.TestCase):
self.assertIn("cmdbin.ExecFile", text)
self.assertIn(f"bin/facts/{method.removesuffix('.go')}", text)
def test_d16_adjudication_is_cgo_free(self) -> None:
self.assertTrue((ROOT / "internal" / "facts" / "contradict.go").is_file())
go = (ROOT / "internal" / "facts" / "contradict.go").read_text()
py = (ROOT / "bin" / "tools" / "contradict.py").read_text()
audit = (ROOT / "bin" / "facts" / "audit").read_text()
for token in ("temporal_freshness", "authority_pairing", "unresolved"):
self.assertIn(token, go)
self.assertIn(token, py)
self.assertIn("contradict", audit)
self.assertIn(" vs ", py)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("temporal_freshness", plan)
self.assertIn("authority_pairing", plan)
shebang = (ROOT / "bin" / "facts" / "audit.go").read_text()
self.assertIn("contradict", shebang)
def test_d23_flaggy_cli(self) -> None:
self.assertTrue((ROOT / "internal" / "cli" / "cli.go").is_file())
self.assertIn("github.com/integrii/flaggy", (ROOT / "go.mod").read_text())
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D23", plan)
self.assertIn("flaggy", plan)
complete = (ROOT / "bin" / "cli" / "complete.go").read_text()
first = complete.splitlines()[0]
self.assertTrue(first.startswith("//usr/bin/env go run"), first)
self.assertIn("complete.go bash", complete)
self.assertIn("brain-search", complete)
chats_import = (ROOT / "internal" / "chats" / "import.go").read_text()
self.assertNotIn("flag.NewFlagSet", chats_import)
args = (ROOT / "internal" / "brain" / "rank" / "args.go").read_text()
self.assertIn("internal/cli", args)
def test_mail_import_is_shebang_not_brain_write(self) -> None:
self._assert_shebang("bin/mail/import.go")
index_mail = (ROOT / "bin" / "mail" / "index_mail").read_text()
@@ -124,8 +168,55 @@ class BinLayoutTest(unittest.TestCase):
"index_mail must point at bin/brain/index.go",
)
def test_markdown_import_is_shebang(self) -> None:
def test_mail_ocr_is_tesseract_not_docling(self) -> None:
self._assert_shebang("bin/mail/ocr.go")
ocr = (ROOT / "bin" / "mail" / "ocr.go").read_text()
self.assertIn("internal/ocr", ocr)
self.assertIn("mail_ocr", ocr)
self.assertNotIn("github.com/otiai10/gosseract", ocr)
py = (ROOT / "bin" / "mail" / "import").read_text()
self.assertNotIn("from docling", py)
self.assertNotIn("import docling", py)
self.assertIn("convert_pdf", py)
conv = (ROOT / "bin" / "tools" / "mailconv.py").read_text()
self.assertIn("pdftotext", conv)
self.assertIn("pdftoppm", conv)
self.assertIn("tesseract", conv)
self.assertIn("eng+deu", conv)
self.assertNotIn("from docling", conv)
self.assertNotIn("import docling", conv)
self.assertNotIn("gocv", conv.lower())
proj = (ROOT / "pyproject.toml").read_text()
self.assertNotIn("docling", proj)
ci = (ROOT / ".github" / "workflows" / "ci.yml").read_text()
self.assertIn("tesseract-ocr", ci)
self.assertIn("./internal/ocr", ci)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn("ocr-paddle", compose)
self.assertIn("OCR_ENGINE", compose)
def test_markdown_import_is_go_not_python_exec(self) -> None:
self._assert_shebang("bin/markdown/import.go")
text = (ROOT / "bin" / "markdown" / "import.go").read_text()
self.assertNotIn("ExecFile", text)
self.assertNotIn("cmdbin", text)
self.assertIn("internal/mdleaves", text)
self.assertNotIn("kb.lbug", text)
def test_import_adapters_do_not_write_ladybug(self) -> None:
for rel in (
"bin/mail/import.go",
"bin/mail/import",
"bin/markdown/import.go",
"bin/chats/import.go",
"bin/git/import.go",
):
text = (ROOT / rel).read_text()
self.assertNotIn("upsert_leaf", text, rel)
self.assertNotIn("kb.lbug", text, rel)
self.assertNotIn("var/brain.lbug", text, rel)
index = (ROOT / "bin" / "brain" / "index.go").read_text()
self.assertIn("bin/kb/index", index)
def test_postgres_query_is_shebang(self) -> None:
self._assert_shebang("bin/postgres/query.go")
@@ -157,3 +248,69 @@ class BinLayoutTest(unittest.TestCase):
for line in first.splitlines():
if "go-git/go-git" in line:
self.assertNotIn("indirect", line)
def test_duckdb_go_is_direct_require(self) -> None:
text = (ROOT / "go.mod").read_text()
first = text.split("require (")[1].split(")")[0]
self.assertRegex(first, r"github.com/duckdb/duckdb-go/v2\s+v")
for line in first.splitlines():
if "duckdb/duckdb-go" in line:
self.assertNotIn("indirect", line)
skill = (ROOT / "skills" / "duckdb" / "SKILL.md").read_text()
self.assertIn("github.com/duckdb/duckdb-go", skill)
self.assertIn("Ladybug", skill)
self.assertIn("sqlite", skill.lower())
self.assertIn("gcc", skill.lower())
self.assertIn("Zig", skill)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D22", plan)
self.assertIn("duckdb-go", plan)
self._assert_shebang("bin/qa/stats.go")
reasoner = (ROOT / "internal" / "reasoner" / "client.go").read_text()
self.assertNotIn("duckdb", reasoner)
self.assertNotIn("duckstats", reasoner)
bakeoff = (ROOT / "bin" / "reasoner" / "bakeoff.go").read_text()
self.assertIn("internal/duckstats", bakeoff)
webcache = (ROOT / "internal" / "websearch" / "cache.go").read_text()
self.assertNotIn("duckdb", webcache)
self.assertIn("modernc.org/sqlite", webcache)
def test_cgo_uses_zig_not_gcc(self) -> None:
for rel in ("bin/cgo/zig", "bin/cgo/zcc", "bin/cgo/zc++"):
p = ROOT / rel
self.assertTrue(p.is_file(), f"missing {rel}")
self.assertTrue(
os.access(p, os.X_OK),
f"{rel} must be executable",
)
zig = (ROOT / "bin" / "cgo" / "zig").read_text()
self.assertIn("zig cc", zig)
self.assertIn("0.14.1", zig)
zcc = (ROOT / "bin" / "cgo" / "zcc").read_text()
self.assertIn('exec "$ZIG" cc', zcc)
self.assertNotIn("command -v gcc", zcc)
search = (ROOT / "bin" / "kb" / "search").read_text()
self.assertIn("bin/cgo/zig", search)
self.assertNotIn("command -v gcc", search)
def test_ci_recall_sot_is_zig_brain_eval(self) -> None:
ci = (ROOT / ".github" / "workflows" / "ci.yml").read_text()
self.assertIn("bin/brain/eval.go", ci)
self.assertIn("system_ladybug,brain_eval", ci)
self.assertIn("/tmp/brain-eval", ci)
self.assertIn("KB_ROOT", ci)
self.assertNotIn("bin/kb/eval", ci)
self.assertNotIn("gate skipped", ci)
self.assertIn("./bin/facts/audit self", ci)
def test_eval_fragments_live_in_default_corpus(self) -> None:
"""CI --rebuild indexes README/PLAN/docs/skills; fragments must be there."""
corpus = []
for rel in ("README.md", "PLAN.md", "AGENTS.md"):
corpus.append((ROOT / rel).read_text())
for d in ("docs", "skills"):
for p in (ROOT / d).rglob("*.md"):
corpus.append(p.read_text())
blob = "\n".join(corpus)
for frag in ("BM25", "DevOps", "LadybugDB"):
self.assertIn(frag, blob, f"{frag} must appear in default index corpus")
+104
View File
@@ -0,0 +1,104 @@
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(__file__))
from contradict import ( # noqa: E402
RULE_AUTHORITY,
RULE_SINGLE,
RULE_TEMPORAL,
RULE_TWO_SOURCE,
RULE_UNRESOLVED,
adjudicate,
check_fact_row,
parse_source_field,
)
def src(i, kind, stale=False):
return {"id": i, "kind": kind, "stale": stale}
class TestContradict(unittest.TestCase):
def test_two_vs_two_stays_hypothesis(self):
r = adjudicate({
"text": "svc listens on 443",
"yes": [src("docker-ps", "runtime"), src("compose", "config")],
"no": [src("docker-old", "runtime"), src("compose-old", "config")],
})
self.assertFalse(r["confirmed"])
self.assertEqual(r["rule"], RULE_UNRESOLVED)
self.assertEqual(r["winner"], "")
def test_temporal_freshness(self):
r = adjudicate({
"text": "svc listens on 443",
"yes": [src("docker-ps", "runtime"), src("compose", "config")],
"no": [src("old-readme", "narrative", True), src("old-wiki", "narrative", True)],
})
self.assertTrue(r["confirmed"])
self.assertEqual(r["rule"], RULE_TEMPORAL)
self.assertEqual(r["winner"], "yes")
def test_authority_pairing(self):
r = adjudicate({
"text": "svc listens on 443",
"yes": [src("docker-ps", "runtime"), src("compose", "config")],
"no": [src("readme", "narrative"), src("wiki", "narrative")],
})
self.assertTrue(r["confirmed"])
self.assertEqual(r["rule"], RULE_AUTHORITY)
self.assertEqual(r["winner"], "yes")
def test_two_source_and_single(self):
two = adjudicate({
"text": "arc-1 runs Matrix",
"yes": [src("compose", "config"), src("docker-ps", "runtime")],
})
self.assertTrue(two["confirmed"])
self.assertEqual(two["rule"], RULE_TWO_SOURCE)
one = adjudicate({"text": "maybe", "yes": [src("readme", "narrative")]})
self.assertFalse(one["confirmed"])
self.assertEqual(one["rule"], RULE_SINGLE)
def test_parse_source_field(self):
yes, no = parse_source_field("docker ps x compose.yml vs old.md x wiki.md")
self.assertIn(" x ", yes)
self.assertIn(" x ", no)
def test_check_fact_row_allows_hypothesis_vs(self):
p = check_fact_row(
"L1", "a.md x b.md vs c.md x d.md", "var/", "audit", "hypothesis",
)
self.assertEqual(p, [])
p = check_fact_row("L2", "a.md x b.md", "var/", "audit", "confirmed")
self.assertEqual(p, [])
p = check_fact_row("L3", "a.md x b.md vs c.md x d.md", "var/", "audit", "confirmed")
self.assertTrue(any("vs-contradiction" in x for x in p))
p = check_fact_row("L4", "only-one.md", "var/", "audit", "hypothesis")
self.assertTrue(any("a x b vs" in x for x in p))
def test_audit_contradict_cli_unresolved(self):
import json
import subprocess
from pathlib import Path
root = Path(__file__).resolve().parents[2]
payload = json.dumps({
"text": "svc 443",
"yes": [src("a", "runtime"), src("b", "config")],
"no": [src("c", "runtime"), src("d", "config")],
})
proc = subprocess.run(
[sys.executable, str(root / "bin" / "facts" / "audit"), "contradict", "--json"],
input=payload, capture_output=True, text=True, check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
out = json.loads(proc.stdout)
self.assertTrue(out["ok"])
self.assertEqual(out["contradictions"][0]["rule"], RULE_UNRESOLVED)
self.assertFalse(out["contradictions"][0]["confirmed"])
if __name__ == "__main__":
unittest.main()
+97
View File
@@ -0,0 +1,97 @@
"""Import adapters write files only. Index rebuild is brain/index (D14 / Gitea #7)."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class IndexAdapterTest(unittest.TestCase):
def test_dry_run_fixture_corpus_does_not_write_lbug(self) -> None:
tmp = Path(tempfile.mkdtemp())
(tmp / "note.md").write_text("# Fixture\n\n## Leaf\n\nhello corpus\n", encoding="utf-8")
lbug = tmp / "kb.lbug"
try:
import ladybug # noqa: F401
except ImportError:
venv_py = ROOT / ".venv" / "bin" / "python"
if not venv_py.is_file():
self.skipTest("ladybug missing")
py = str(venv_py)
else:
py = sys.executable
proc = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "index"), "--dry-run", "--json", "--corpus", str(tmp)],
cwd=ROOT,
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
msg = json.loads(proc.stdout)
self.assertTrue(msg.get("dry_run"))
self.assertGreaterEqual(msg.get("corpus_total", 0), 1)
self.assertFalse(lbug.exists(), "dry-run must not create a Ladybug file")
def test_facts_json_and_chats_land_on_rebuild(self) -> None:
"""Gitea #18: facts (2-source) + chats markdown become leafs on rebuild."""
tmp = Path(tempfile.mkdtemp())
dbpath = tmp / "kb.lbug"
chats = tmp / "chats"
chats.mkdir()
(chats / "alice.md").write_text(
"# Chat\n\n## Alice and Bob\n\nhello from chats fixture unique-chat-token\n",
encoding="utf-8",
)
facts_path = tmp / "facts.json"
facts_path.write_text(json.dumps([{
"text": "container 'brain' unique-fact-token is running and declared in compose.yaml",
"source": "docker ps x compose.yaml",
"loc": "compose.yaml:brain",
"how": "facts/extract",
}]), encoding="utf-8")
venv_py = ROOT / ".venv" / "bin" / "python"
py = str(venv_py) if venv_py.is_file() else sys.executable
proc = subprocess.run(
[
py, str(ROOT / "bin" / "kb" / "index"),
"--rebuild", "--db", str(dbpath), "--no-defaults",
"--with-chats", str(chats),
"--facts-json", str(facts_path),
"--json",
],
cwd=ROOT,
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
msg = json.loads(proc.stdout)
self.assertGreaterEqual(msg.get("facts_leafs", 0), 1)
self.assertGreaterEqual(msg.get("chat_leafs", 0), 1)
self.assertTrue(dbpath.exists())
sys.path.insert(0, str(ROOT / "bin" / "tools"))
import kblib
db, conn = kblib.connect(dbpath, read_only=True)
try:
stats = kblib.stats(conn)
self.assertGreaterEqual(stats["by_root"].get("facts", 0), 1)
fts = kblib.query_fts(conn, "unique-chat-token", 5)
self.assertTrue(fts, "chats markdown must be FTS-searchable")
fact_hits = kblib.query_fts(conn, "unique-fact-token", 5)
self.assertTrue(any(h.get("root") == "facts" for h in fact_hits))
src = conn.execute(
"MATCH (l:Leaf {root:'facts'}) RETURN l.source"
).get_all()
self.assertTrue(any(" x " in str(r[0]) for r in src))
finally:
conn.close()
db.close()
+69
View File
@@ -0,0 +1,69 @@
"""Incremental add writes leafs without deleting kb.lbug."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class KbAddCLITest(unittest.TestCase):
def test_json_add_does_not_delete_db(self) -> None:
tmp = Path(tempfile.mkdtemp())
dbpath = tmp / "kb.lbug"
py = sys.executable
venv_py = ROOT / ".venv" / "bin" / "python"
if venv_py.is_file():
py = str(venv_py)
payload = {
"text": "cli zebra leaf",
"root": "info",
"source": "cli-test",
"confidence": "confirmed",
"how": "test",
"loc": str(tmp),
"type": "reference",
"embedding": [0.0] * 256,
}
payload["embedding"][0] = 0.3
proc = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "add"), "--db", str(dbpath), "--json"],
cwd=ROOT,
input=json.dumps(payload),
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
self.assertTrue(dbpath.exists(), "add must create the db, not skip write")
out = json.loads(proc.stdout)
self.assertEqual(out.get("mode"), "add")
self.assertEqual(len(out.get("ids") or []), 1)
again = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "add"), "--db", str(dbpath), "--json"],
cwd=ROOT,
input=json.dumps({
**payload,
"text": "second moose leaf",
"source": "cli-test-2",
}),
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(again.returncode, 0, again.stderr)
self.assertTrue(dbpath.exists())
second = json.loads(again.stdout)
self.assertEqual(len(second.get("ids") or []), 1)
self.assertNotEqual(out["ids"][0], second["ids"][0])
if __name__ == "__main__":
unittest.main()
+96
View File
@@ -71,6 +71,69 @@ class KblibTest(unittest.TestCase):
self.assertTrue(hits)
self.assertIn("Leaf_vec", kblib.leaf_index_names(self.conn))
def test_add_after_indexes_keeps_fts_queryable(self):
"""Incremental add after FTS+HNSW must find the new leaf on both indexes."""
kblib.upsert_leaf(self.conn, text="seed fox leaf", root="info",
confidence="confirmed", source="s", source_rev="r1",
how="test", loc="/tmp", type_="reference",
embedding=make_emb(0.1))
kblib.ensure_indexes(self.conn)
ids = kblib.add_leafs(self.conn, [{
"text": "added zebra after index",
"root": "facts",
"confidence": "confirmed",
"source": "a.md x b.md",
"source_rev": "r1",
"how": "test",
"loc": "/tmp",
"type": "fact",
"embedding": make_emb(0.9),
}])
self.assertEqual(len(ids), 1)
fts = kblib.query_fts(self.conn, "zebra", 5)
self.assertTrue(fts)
self.assertIn("zebra", fts[0]["text"])
self.assertEqual(fts[0]["root"], "facts")
vec = kblib.query_vector(self.conn, make_emb(0.9), 5)
self.assertTrue(any("zebra" in h["text"] for h in vec))
fox = kblib.query_fts(self.conn, "fox", 5)
self.assertTrue(fox)
self.assertIn("fox", fox[0]["text"])
def test_add_facts_and_info_one_transaction(self):
"""D12: facts and info land in the same transaction."""
kblib.ensure_indexes(self.conn)
ids = kblib.add_leafs(self.conn, [
{
"text": "tx fact leaf two-source",
"root": "facts",
"confidence": "confirmed",
"source": "compose.yml x docker ps",
"source_rev": "r1",
"how": "test",
"loc": "/tmp",
"type": "fact",
"embedding": make_emb(0.4),
},
{
"text": "tx info narrative",
"root": "info",
"confidence": "confirmed",
"source": "note.md",
"source_rev": "r1",
"how": "test",
"loc": "/tmp",
"type": "reference",
"embedding": make_emb(0.5),
},
])
self.assertEqual(len(ids), 2)
stats = kblib.stats(self.conn)
self.assertEqual(stats["by_root"].get("facts"), 1)
self.assertEqual(stats["by_root"].get("info"), 1)
self.assertTrue(kblib.query_fts(self.conn, "two-source", 5))
self.assertTrue(kblib.query_fts(self.conn, "narrative", 5))
def test_drop_vector_then_create_raises_clear_error(self):
"""DROP INDEX leaves ghost catalog; create_fts_and_vector must raise."""
kblib.upsert_leaf(self.conn, text="seed", root="info",
@@ -98,6 +161,39 @@ class KblibTest(unittest.TestCase):
self.assertEqual(stats["total"], 2)
self.assertEqual(stats["by_root"], {"facts": 1, "info": 1})
def test_hop_1_returns_file_hop_3_reaches_person(self):
"""--hop walks FROM_FILE / HAS_VERSION / AUTHORED (Gitea #17)."""
import gitimport
lid = kblib.upsert_leaf(
self.conn, text="readme hop fixture", root="info",
confidence="confirmed", source="README.md", source_rev="r1",
how="test", loc="README.md", type_="reference",
embedding=make_emb(0.3),
)
kblib.link_from_file(self.conn, lid, "README.md", repo="sample-repo")
gitimport.index_commits(self.conn, [gitimport.Commit(
sha="a1b2c3d",
author="Ada Lovelace",
email="ada@example.com",
date="2026-08-10T12:00:00Z",
subject="feat: first commit",
files=["README.md"],
)], "sample-repo")
hop1 = kblib.hop_walk(self.conn, lid, 1)
self.assertEqual(len(hop1), 1)
self.assertEqual(hop1[0]["label"], "File")
self.assertEqual(hop1[0]["name"], "README.md")
self.assertEqual(hop1[0]["depth"], 1)
hop3 = kblib.hop_walk(self.conn, lid, 3)
labels = {n["label"] for n in hop3}
self.assertIn("File", labels)
self.assertIn("Commit", labels)
self.assertIn("Person", labels)
person = [n for n in hop3 if n["label"] == "Person"][0]
self.assertEqual(person["name"], "Ada Lovelace")
self.assertEqual(person["depth"], 3)
if __name__ == "__main__":
unittest.main()
+89
View File
@@ -8,10 +8,13 @@ from pathlib import Path
sys.path.insert(0, os.path.dirname(__file__))
from mailconv import ( # noqa: E402
TESS_LANG,
clean_email_address,
convert_pdf,
html_to_markdown,
is_convertible,
normalize_markdown,
ocr_image,
split_zip_members,
subject_to_filename,
zip_extract_safe,
@@ -100,6 +103,92 @@ class TestMailConv(unittest.TestCase):
self.assertFalse(is_convertible(".exe"))
self.assertFalse(is_convertible(".unknown"))
def test_convert_pdf_prefers_pdftotext(self):
import mailconv as mc
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(list(cmd))
class P:
returncode = 0
stdout = b"Invoice BM25 layout"
stderr = b""
return P()
self._patch_run(mc, fake_run)
out = convert_pdf(Path(self._tmp("born.pdf")))
self.assertIn("BM25", out)
self.assertEqual(calls[0][:2], ["pdftotext", "-layout"])
self.assertFalse(any(c[0] == "tesseract" for c in calls))
self.assertFalse(any(c[0] == "pdftoppm" for c in calls))
def test_convert_pdf_empty_layer_uses_pdftoppm_tesseract(self):
import mailconv as mc
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(list(cmd))
class P:
returncode = 0
stdout = b""
stderr = b""
if cmd[0] == "pdftotext":
P.stdout = b" \n"
return P()
if cmd[0] == "pdftoppm":
prefix = Path(cmd[-1])
(prefix.parent / "page-1.png").write_bytes(b"fake")
return P()
if cmd[0] == "tesseract":
P.stdout = b"scanned HELLO"
return P()
return P()
self._patch_run(mc, fake_run)
out = convert_pdf(Path(self._tmp("scan.pdf")))
self.assertIn("HELLO", out)
bins = [c[0] for c in calls]
self.assertIn("pdftotext", bins)
self.assertIn("pdftoppm", bins)
self.assertIn("tesseract", bins)
tess = next(c for c in calls if c[0] == "tesseract")
self.assertIn(TESS_LANG, tess)
self.assertNotIn("docling", " ".join(bins))
def test_ocr_image_paddle_engine(self):
import mailconv as mc
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(list(cmd))
class P:
returncode = 0
stdout = b"paddle text"
stderr = b""
return P()
self._patch_run(mc, fake_run)
os.environ["OCR_ENGINE"] = "paddle"
try:
out = ocr_image(Path(self._tmp("x.png")))
finally:
os.environ.pop("OCR_ENGINE", None)
self.assertEqual(out, "paddle text")
self.assertEqual(calls[0][:2], ["paddleocr", "ocr"])
def _patch_run(self, mod, fn) -> None:
self.addCleanup(setattr, mod.subprocess, "run", mod.subprocess.run)
mod.subprocess.run = fn
def _mk_zip(self, members):
zpath = Path(self._tmp("arc.zip"))
with zipfile.ZipFile(zpath, "w") as zf:
+84 -10
View File
@@ -1,7 +1,6 @@
"""Published docs must match live commands (Gitea SoT, brain/search, no fake --hop)."""
"""Published docs must match live commands (Gitea SoT, brain/search)."""
from __future__ import annotations
import re
import unittest
from pathlib import Path
@@ -89,6 +88,49 @@ class PublishedDocsTest(unittest.TestCase):
self.assertFalse((ROOT / "skills" / "db-yaml").exists())
self.assertTrue((ROOT / "skills" / "postgres" / "SKILL.md").is_file())
def test_cgo_zig_and_index_profile(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D21", plan)
self.assertIn("zig cc", plan)
dockerfile = (ROOT / "Dockerfile").read_text()
self.assertIn("bin/cgo/zcc", dockerfile)
self.assertIn("FROM debian:bookworm-slim AS api", dockerfile)
self.assertIn("FROM python:3.12-slim AS index", dockerfile)
api = dockerfile[dockerfile.index("FROM debian:bookworm-slim AS api") :]
self.assertNotIn("pip install", api)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('profiles: ["index"]', compose)
self.assertIn("target: api", compose)
def test_reasoner_docs_name_real_hf_ids_cpu_sidecar(self) -> None:
docs = (ROOT / "docs" / "reasoner.md").read_text()
for hf in (
"Qwen/Qwen3.5-9B",
"Qwen/Qwen3.6-27B",
"prism-ml/Bonsai-27B-gguf",
):
self.assertIn(hf, docs)
self.assertIn("no official qwen3.6-9b", docs.lower())
self.assertIn("OLLAMA_NUM_GPU", docs)
self.assertIn("rss_mb", docs)
self.assertIn("vram_mb", docs)
self.assertIn("3/3", docs)
self.assertIn("Do not claim 9B is better at tools", docs)
self.assertNotIn("Qwen/Qwen3.6-9B", docs)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D18", plan)
self.assertIn("Qwen/Qwen3.5-9B", plan)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('"reasoner"', compose)
self.assertIn("OLLAMA_NUM_GPU", compose)
self.assertIn("127.0.0.1:11435", compose)
dockerfile = (ROOT / "Dockerfile").read_text()
self.assertNotIn(".gguf", dockerfile.lower())
self.assertNotIn(".safetensors", dockerfile.lower())
api = dockerfile[dockerfile.index("FROM debian:bookworm-slim AS api") :]
self.assertNotIn("COPY models", api)
self.assertNotIn("qwen", api.lower())
def test_readme_search_escalates_web(self) -> None:
text = (ROOT / "README.md").read_text()
self.assertIn("--no-web", text)
@@ -96,19 +138,51 @@ class PublishedDocsTest(unittest.TestCase):
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
self.assertIn("`web` block", skill)
def test_docs_do_not_claim_hop_walks(self) -> None:
def test_docs_say_hop_walks_from_file(self) -> None:
paths = [
ROOT / "README.md",
ROOT / "docs" / "design.md",
ROOT / "skills" / "brain" / "SKILL.md",
ROOT / "skills" / "diataxis-docs" / "SKILL.md",
ROOT / "docs" / "runbook.md",
ROOT / "docs" / "README.md",
]
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
# Honest "not implemented" notes must not match.
lie = re.compile(r"--hop (?:N|1).*(?:follow|walk)", re.I | re.S)
for path in paths:
text = path.read_text()
self.assertIsNone(
lie.search(text),
f"{path.relative_to(ROOT)} still claims --hop walks the graph",
self.assertIn("--hop", text, f"{path.relative_to(ROOT)} must document --hop")
self.assertNotIn(
"not implemented",
text.lower(),
f"{path.relative_to(ROOT)} still says hop is not implemented",
)
def test_docs_are_portable_diataxis(self) -> None:
index = (ROOT / "docs" / "README.md").read_text()
self.assertIn("type: reference", index)
for d in ("D3", "D6", "D14", "D15", "D17", "D18"):
self.assertIn(d, index)
runbook = (ROOT / "docs" / "runbook.md").read_text()
self.assertIn("type: howto", runbook)
self.assertIn("bin/brain/search.go", runbook)
self.assertIn("bin/brain/index.go", runbook)
self.assertNotIn("search.ops.io", runbook)
self.assertNotIn("/mnt/", runbook)
self.assertNotIn("/home/", runbook)
readme = (ROOT / "README.md").read_text()
self.assertIn("docs/runbook.md", readme)
self.assertNotIn("search.ops.io", readme)
def test_v1_epic_is_named_in_docs(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("Gap to v1", plan)
self.assertIn("eSlider/2dph/issues/16", plan)
self.assertIn("eSlider/2dph/issues/17", plan)
self.assertIn("eSlider/2dph/milestone/12", plan)
road = (ROOT / "docs" / "roadmap.md").read_text()
self.assertIn("type: explanation", road)
self.assertIn("issues/16", road)
self.assertIn("issues/14", road)
index = (ROOT / "docs" / "README.md").read_text()
self.assertIn("roadmap.md", index)
self.assertIn("epic #16", index)
agents = (ROOT / "AGENTS.md").read_text()
self.assertIn("roadmap.md", agents)
+14
View File
@@ -47,3 +47,17 @@ class SkillsTest(unittest.TestCase):
self.assertIn("throttled", skill.lower())
self.assertIn("not a negative finding", agents)
self.assertIn("Fact-check every", agents)
def test_yq_is_mikefarah_for_structured_data(self) -> None:
skill = (ROOT / "skills" / "yq" / "SKILL.md").read_text()
self.assertIn("https://github.com/mikefarah/yq", skill)
for fmt in ("YAML", "JSON", "XML", "CSV", "TOML", "HCL"):
self.assertIn(fmt, skill)
self.assertIn("not kislyuk", skill.lower())
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("mikefarah/yq", plan)
agents = (ROOT / "AGENTS.md").read_text()
self.assertIn("mikefarah/yq", agents)
web = (ROOT / "skills" / "web-search" / "SKILL.md").read_text()
self.assertIn("| yq ", web)
self.assertNotIn("| jq ", web)
+36
View File
@@ -0,0 +1,36 @@
"""qa/system_perf.py is an offline-gated system test (no live brain in CI)."""
from __future__ import annotations
import ast
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class SystemPerfScriptTest(unittest.TestCase):
def test_script_compiles_and_is_read_only(self) -> None:
path = ROOT / "qa" / "system_perf.py"
src = path.read_text()
compile(src, str(path), "exec")
self.assertIn("--json", src)
self.assertIn("qwen3.5:9b", src)
self.assertIn("--picoclaw", src)
self.assertIn("BRAIN_URL", src)
self.assertIn("tools/list", src)
self.assertIn("tools/call", src)
self.assertIn("GATE_HEALTH_MS", src)
self.assertIn("GATE_GET_P50_MS", src)
self.assertNotIn("kb.lbug", src)
self.assertNotIn("password", src.lower())
self.assertNotIn("token", src.lower())
def test_script_does_not_write_ladybug(self) -> None:
tree = ast.parse((ROOT / "qa" / "system_perf.py").read_text())
writes = [
n.func.attr
for n in ast.walk(tree)
if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)
and n.func.attr in {"write_text", "write_bytes", "dump"}
]
self.assertEqual(writes, [], f"system_perf must not write files: {writes}")
+8 -70
View File
@@ -16,9 +16,9 @@ import (
"fmt"
"net/http"
"os"
"strconv"
"time"
"github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/websearch"
"golang.org/x/sys/unix"
)
@@ -28,77 +28,15 @@ func main() {
}
func run(args []string) int {
var (
query, site, lang, fresh, category, engines string
limit = websearch.DefaultLimit
jsonOut, refresh, force bool
ttl = float64(websearch.CacheTTL)
timeout = 25
)
i := 0
for i < len(args) {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--refresh":
refresh = true
case a == "--force":
force = true
case (a == "-n" || a == "--limit") && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 0 {
fmt.Fprintln(os.Stderr, "web/search: --limit must be a non-negative integer")
return 2
}
limit = n
case a == "--site" && i+1 < len(args):
i++
site = args[i]
case a == "--lang" && i+1 < len(args):
i++
lang = args[i]
case a == "--fresh" && i+1 < len(args):
i++
fresh = args[i]
case a == "--category" && i+1 < len(args):
i++
category = args[i]
case a == "--engines" && i+1 < len(args):
i++
engines = args[i]
case a == "--ttl" && i+1 < len(args):
i++
v, err := strconv.ParseFloat(args[i], 64)
c, err := websearch.ParseArgs(args)
if err != nil {
fmt.Fprintln(os.Stderr, "web/search: --ttl must be a number")
return 2
}
ttl = v
case a == "--timeout" && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n <= 0 {
fmt.Fprintln(os.Stderr, "web/search: --timeout must be a positive integer")
return 2
}
timeout = n
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/web/search.go QUERY [--json] [-n N] [--site HOST] [--lang LANG] [--fresh day|week|month|year] [--category CAT] [--engines LIST] [--refresh] [--force]`)
return 0
case len(a) > 0 && a[0] != '-' && query == "":
query = a
default:
fmt.Fprintf(os.Stderr, "web/search: unknown flag %s\n", a)
return 2
}
i++
}
if query == "" {
fmt.Fprintln(os.Stderr, "web/search: query required")
return 2
return cli.Fail(err)
}
query, site, lang, fresh, category, engines := c.Query, c.Site, c.Lang, c.Fresh, c.Category, c.Engines
limit := c.Limit
jsonOut, refresh, force := c.JSONOut, c.Refresh, c.Force
ttl := c.TTL
timeout := c.Timeout
if site != "" {
query = "site:" + site + " " + query
}
+98 -22
View File
@@ -1,66 +1,96 @@
# 2dph — docker composition
#
# docker compose run --rm brain index # rebuild graph
# docker compose run --rm brain search "Matrix fed" # one-shot query
# docker compose run --rm brain serve # async Go server
# docker compose up brain-watch # auto re-index
# docker compose up -d brain # API (Zig CGO serve)
# docker compose --profile index run --rm index # Python rebuild
# docker compose --profile picoclaw up -d # brain-mcp + CPU reasoner + PicoClaw gateway
# docker compose --profile reasoner up -d reasoner # CPU Ollama :11435
# docker compose --profile searxng up -d
# OCR_ENGINE=paddle docker compose --profile ocr-paddle run --rm ocr-paddle
#
# Caching: the 128M model (HF_HOME) and kb.lbug (VAR_DIR) live in named
# volumes, so rebuilds never redownload the model or re-derive the graph.
# Secrets are never baked into the image: search.env + db-profiles.yml mount
# read-only from ~/.config/brain.
# Secrets never baked in: search.env + db-profiles.yml from ~/.config/brain.
name: 2dph
networks:
default:
name: 2dph_sys
driver: bridge
ipam:
config:
- subnet: 10.23.42.0/24
services:
brain:
image: ghcr.io/eslider/2dph:latest
image: ghcr.io/eslider/2dph:api
build:
context: .
dockerfile: Dockerfile
target: api
cache_from:
- ghcr.io/eslider/2dph:cache
command: ["brain", "search", "help"]
command: ["serve"]
environment: &env
HF_HOME: /data/hf
BRAIN_SEARCH_CACHE: /data/cache/web-search.sqlite
BRAIN_DB_PROFILES: /secret/db-profiles.yml
BRAIN_SEARCH_ENV: /secret/search.env
KB_SEARCH_CMD: /app/bin/kb/search
KB_ROOT: /data
KB_WORKERS: "4"
KB_PORT: "8630"
volumes:
- kb-model:/data/hf
- kb-var:/data
# corpus is read-only on the host, never written from the container
- ..:/corpus:ro
- ~/.config/brain:/secret:ro
ports:
- "127.0.0.1:8630:8630"
read_only: true
tmpfs:
- /tmp
healthcheck:
test: ["CMD", "python3", "-c", "import ladybug, model2vec, mistune; print('ok')"]
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8630/health"]
interval: 30s
timeout: 5s
retries: 3
restart: unless-stopped
stop_grace_period: 20s
# watcher: re-index on corpus file change (watchdog script)
brain-watch:
image: ghcr.io/eslider/2dph:latest
image: ghcr.io/eslider/2dph:api
environment: *env
volumes:
- kb-model:/data/hf
- kb-var:/data
- ..:/corpus:ro
- ~/.config/brain:/secret:ro
command: ["brain", "watch", "/corpus"]
command: ["watch", "/corpus"]
read_only: true
tmpfs:
- /tmp
restart: unless-stopped
stop_grace_period: 20s
# Python write path (Ladybug rebuild). Not in the API image.
# docker compose --profile index run --rm index
index:
profiles: ["index"]
image: ghcr.io/eslider/2dph:index
build:
context: .
dockerfile: Dockerfile
target: index
environment:
HF_HOME: /data/hf
KB_PY: python3
volumes:
- kb-model:/data/hf
- kb-var:/app/var
- ..:/corpus:ro
- ~/.config/brain:/secret:ro
command: ["index"]
read_only: true
tmpfs:
- /tmp
# Optional local SearXNG (D3). Skip if BRAIN_SEARCH_URL already points at a
# live instance — do not run a second copy on that host.
# SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d
@@ -76,18 +106,17 @@ services:
- ./deploy/searxng/limiter.toml:/etc/searxng/limiter.toml:ro
restart: unless-stopped
# MCP endpoint for an external agent (PicoClaw is not shipped here).
# docker compose --profile picoclaw up brain-mcp
# Point the agent at http://127.0.0.1:8630/mcp (see deploy/picoclaw/).
# MCP endpoint for PicoClaw (and any MCP client).
# docker compose --profile picoclaw up -d
brain-mcp:
profiles: ["picoclaw"]
image: ghcr.io/eslider/2dph:latest
image: ghcr.io/eslider/2dph:api
environment: *env
volumes:
- kb-model:/data/hf
- kb-var:/data
- ~/.config/brain:/secret:ro
command: ["brain", "serve"]
command: ["serve"]
ports:
- "127.0.0.1:8630:8630"
read_only: true
@@ -95,6 +124,53 @@ services:
- /tmp
restart: unless-stopped
# CPU OpenAI-compatible sidecar (D18). Weights are pulled at runtime, not
# baked into the 2dph image. Does not touch host Ollama on :11434.
# docker compose --profile reasoner up -d reasoner
# docker compose --profile reasoner exec reasoner ollama pull qwen3.5:9b
reasoner:
profiles: ["reasoner", "picoclaw"]
image: docker.io/ollama/ollama:latest
environment:
OLLAMA_NUM_GPU: "0"
OLLAMA_HOST: "0.0.0.0:11434"
ports:
- "127.0.0.1:11435:11434"
volumes:
- reasoner-ollama:/root/.ollama
restart: unless-stopped
# Official PicoClaw gateway. Config has no secrets (Ollama + HTTP MCP).
# Host network: brain/reasoner bind 127.0.0.1 only, so host.docker.internal
# (docker0) cannot reach them. Gateway 127.0.0.1:18790 (not the 18800 launcher).
# If :8630/:11435 are already bound, do not start brain-mcp/reasoner:
# docker compose --profile picoclaw up -d --no-deps picoclaw
picoclaw:
profiles: ["picoclaw"]
image: docker.io/sipeed/picoclaw:v0.3.1
network_mode: host
depends_on:
- brain-mcp
- reasoner
environment:
PICOCLAW_GATEWAY_HOST: "127.0.0.1"
entrypoint: ["picoclaw", "gateway"]
volumes:
- picoclaw-home:/root/.picoclaw
- ./deploy/picoclaw/config.json:/root/.picoclaw/config.json:ro
restart: unless-stopped
# Optional PP-OCRv5 (not default). Default OCR is tesseract eng+deu.
# OCR_ENGINE=paddle docker compose --profile ocr-paddle run --rm ocr-paddle
ocr-paddle:
profiles: ["ocr-paddle"]
image: python:3.12-slim
environment:
OCR_ENGINE: paddle
command: ["python", "-c", "print('OCR_ENGINE=paddle; install paddleocr on PATH')"]
volumes:
kb-model:
kb-var:
reasoner-ollama:
picoclaw-home:
+33
View File
@@ -0,0 +1,33 @@
{
"agents": {
"defaults": {
"model_name": "qwen3.5-9b",
"max_tool_iterations": 8,
"max_tokens": 512,
"context_window": 8192
}
},
"model_list": [
{
"model_name": "qwen3.5-9b",
"model": "ollama/qwen3.5:9b",
"api_base": "http://127.0.0.1:11435/v1",
"request_timeout": 600
}
],
"tools": {
"web": {
"enabled": false
},
"mcp": {
"enabled": true,
"servers": {
"2dph": {
"enabled": true,
"type": "http",
"url": "http://127.0.0.1:8630/mcp"
}
}
}
}
}
+32 -9
View File
@@ -1,15 +1,38 @@
# 2dph (deductionphile)
---
type: reference
status: current
related:
- docs/runbook.md
- docs/design.md
- PLAN.md
- docs/roadmap.md
---
Evidence-first knowledge graph + hybrid RAG over the operational
Brain/ops/eSlider stack. Facts need proof or they are
# 2dph docs (Diataxis)
Evidence-first knowledge graph. Facts need proof or they are
`(not confirmed)`.
- [PLAN.md](../PLAN.md) — decisions, execution order, open questions (v2)
- [design](design.md) — schema, deduction model, sources
- [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) — work board (origin)
| Type | Doc |
|------|-----|
| tutorial / howto | [runbook](runbook.md) — run anywhere (uv, Go, Docker) |
| explanation | [design](design.md) — two roots, deduction, D17/D20/D18 |
| explanation | [roadmap](roadmap.md) — gap to v1 (epic #16) |
| howto | [picoclaw](picoclaw.md) — MCP agent profile |
| howto | [reasoner](reasoner.md) — CPU bake-off (D18) |
| reference | [PLAN.md](../PLAN.md) — decisions D1D22 |
Decisions the public face must name: **D3** SearXNG compose, **D6** Go service /
Python write sidecar, **D14** `bin/{subject}/{method}.go`, **D15** Gitea origin,
**D17** assertion gate (facts → info → web), **D18** pluggable reasoner.
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is
not a walk; the flag errors until File/FROM_FILE edges exist.
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop N` walks
`FROM_FILE` → Commit → Person from each hit (max 3). Rebuild writes
File edges ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
Published docs live here and mirror the project state.
Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues)
([epic #16](https://git.produktor.io/eSlider/2dph/issues/16)).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).
Published docs live here and match live commands.
+2 -1
View File
@@ -21,4 +21,5 @@ OO_CLI (default: $HOME/go/bin/oo)
./bin/chats/apply.go --dry-run
```
JSONL → markdown only. Brain ingest is `bin/brain/index.go` (not a `chats index`).
JSONL → markdown only. Brain ingest is `bin/brain/index.go --with-chats`
(default `var/chats/md`). WhatsApp sync is out of v1.
+38 -7
View File
@@ -1,3 +1,12 @@
---
type: explanation
status: current
related:
- docs/README.md
- docs/runbook.md
- docs/roadmap.md
---
# Design — facts, info, deduction
## Two roots, one transaction
@@ -25,8 +34,13 @@ bin/brain/search.go "question"
is not evidence of absence; `--no-web` / `--root` skip it)
```
`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an
error; it is not a graph walk.
`--hop N` walks `Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`
from each hit (1=File, 2=Commit, 3=Person). Rebuild writes FROM_FILE;
git import writes HAS_VERSION/AUTHORED ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
Go CLIs parse with **flaggy** via `internal/cli` (D23). Flags may appear
after positionals (`search q --hop 1`). Completions:
`source <(./bin/cli/complete.go bash)`.
## Who / What / How / Where / When + evidence
@@ -61,19 +75,36 @@ corpus HEAD.
- C: narrative — READMEs, AGENTS.md, docs
Confirmed = A×B or B×C agreement. Single source = hypothesis + `(not confirmed)`.
Conflicting pairings (≥2 yes vs ≥2 no) = hypothesis (OQ1 → v2 resolution).
Conflicting pairings (≥2 yes vs ≥2 no) stay hypothesis until
`temporal_freshness` or `authority_pairing` fires (`bin/facts/audit contradict`,
[#29](https://git.produktor.io/eSlider/2dph/issues/29)).
## Read path
`bin/brain/get.go`, `stats.go`, and `eval.go` call `internal/brain` with cgo
(`system_ladybug`). They do not exec Python. Control questions for recall@5
live in `internal/brain/rank` so CI can test the table without libladybug.
(`system_ladybug`), compiled by **Zig** (`bin/cgo/zcc`, D21), not gcc.
They do not exec Python. Control questions for recall@5 live in
`internal/brain/rank` so CI can test the table without libladybug.
Python `bin/kb/{get,stats,eval}` remain for GitHub Actions until the runner
has ladybug cgo. Index/write is still `bin/kb/index`.
fetches Zig + libs (`bin/cgo/zig`). Incremental write is `bin/kb/add`
(`bin/brain/add.go`). Bulk index/write is still `bin/kb/index`
(`docker compose --profile index`).
## Agent API (D20)
`bin/brain/serve.go` exposes the same `internal/httpapi.Ops` table as OpenAPI
(`GET /openapi.json`) and MCP (`POST /mcp` JSON-RPC `tools/list` +
`tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`.
`tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`,
`ingest` (add a leaf; omit body for the CLI hint).
Agents should use these endpoints instead of shebang CLIs.
## Reasoner (D18)
Pluggable OpenAI-compatible URL. RAM: `Qwen/Qwen3.5-9B`. Quality:
`prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B.
CPU sidecar: compose profile `reasoner` (`OLLAMA_NUM_GPU=0`,
`127.0.0.1:11435`). Bake-off: `bin/reasoner/bakeoff.go`. Weights stay out
of the 2dph image. See [docs/reasoner.md](reasoner.md).
Gap to v1 (hops, corpus, CI eval): [roadmap](roadmap.md),
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
+22 -6
View File
@@ -1,18 +1,34 @@
# PicoClaw profile (reference agent)
2dph is the memory/fact gate. PicoClaw (or any MCP client) is the agent loop
and is **not** shipped in this repo.
2dph is the memory/fact gate. Compose profile `picoclaw` runs the official
PicoClaw gateway (`docker.io/sipeed/picoclaw:v0.3.1`) plus `brain-mcp` and the
CPU reasoner. Default agent model is `qwen3.5:9b` (RAM path, D18). Weights stay
in the reasoner volume, not in the 2dph image.
No secrets in git: Ollama needs no key; MCP is local HTTP.
```bash
docker compose --profile picoclaw up brain-mcp
docker compose --profile picoclaw up -d
# already serving :8630 / :11435:
docker compose --profile picoclaw up -d --no-deps picoclaw
```
The API listens on `127.0.0.1:8630`. Point the agent at
`http://127.0.0.1:8630/mcp` using [deploy/picoclaw/mcp.json.example](../deploy/picoclaw/mcp.json.example).
Gateway: `127.0.0.1:18790`. Brain MCP: `http://127.0.0.1:8630/mcp`.
Cursor-style clients can use [deploy/picoclaw/mcp.json.example](../deploy/picoclaw/mcp.json.example).
PicoClaw itself uses [deploy/picoclaw/config.json](../deploy/picoclaw/config.json)
(`127.0.0.1` + host network — loopback publishes are not reachable via docker0).
OpenAPI: `GET http://127.0.0.1:8630/openapi.json`.
Before a factual reply: `search``get``audit`. `throttled` is not a
negative finding. See `skills/picoclaw/SKILL.md`.
No Cursor required. A live PicoClaw binary/image is an operator choice.
System performance (MCP gates + qwen3.5:9b tool_call + PicoClaw gateway):
```bash
./qa/system_perf.py --json | yq '.gates'
REASONER_MODEL=qwen3.5:9b ./qa/system_perf.py --reasoner --picoclaw --json | yq '.reasoner'
```
The default agent model is `qwen3.5:9b`. PicoClaw `context_window` is 8192
(heuristic `max_tokens*4` at 512 is 2048, too small for MCP tool schemas).
`request_timeout` is 600s for a CPU turn (tool_call + MCP search + answer).
+71
View File
@@ -0,0 +1,71 @@
# Reasoner bake-off (D18)
Pluggable OpenAI-compatible URL. 2dph does not ship weights. PicoClaw is
compose profile `picoclaw` (`sipeed/picoclaw`); the bake-off hits the same
tool names (`search``get``audit` from `internal/httpapi.Ops`).
```bash
docker compose --profile reasoner up -d reasoner
docker compose --profile reasoner exec reasoner ollama pull qwen3.5:9b
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \
./bin/reasoner/bakeoff.go --json
```
JSON includes `latency_p50_ms` / `latency_p95_ms` from DuckDB (`internal/duckstats`, D22).
Host Ollama on `:11434` is left alone. This sidecar binds `127.0.0.1:11435`
with `OLLAMA_NUM_GPU=0` (CPU). Measure RSS (`/api/ps` `size`), not VRAM.
If Compose cannot allocate a project network (Docker IPAM pool exhausted),
the same sidecar is:
```bash
docker run -d --name 2dph-reasoner \
-e OLLAMA_NUM_GPU=0 \
-p 127.0.0.1:11435:11434 \
-v 2dph-reasoner-ollama:/root/.ollama \
ollama/ollama:latest
```
## Real Hugging Face ids
| Role | HF id | Ollama tag (this bake-off) |
|------|-------|----------------------------|
| RAM / 9B | `Qwen/Qwen3.5-9B` | `qwen3.5:9b` |
| Quality 27B (CPU) | `prism-ml/Bonsai-27B-gguf` (derived from Qwen3.6-27B) | `MichelRosselli/bonsai-27b:Q1_0` |
| Quality 27B (full) | `Qwen/Qwen3.6-27B` | not pulled on this CPU box |
There is **no official Qwen3.6-9B**. Do not invent that id.
Qwen3.5-9B has documented upstream tool-call XML bugs. A 9B win on tools
is only claimed if this bake-off records OpenAI `tool_calls` (not
`<tool_call>` XML in `content`).
The 2dph API image does not `COPY` GGUF/safetensors. Pull at runtime into
the `reasoner-ollama` volume.
## Live CPU run
Host sidecar: Ollama **0.32.9**, `OLLAMA_NUM_GPU=0`, `127.0.0.1:11435`,
`device: cpu`, `vram_mb: 0`. Date: 2026-08-13. Same three prompts
(`search` / `get` / `audit`). PicoClaw binary was not used; the OpenAI
tools payload is the surface it would send.
| Model | HF id | tool_call | xml_leak | rss_mb | latency_ms (search/get/audit) |
|-------|-------|-----------|----------|--------|-------------------------------|
| `qwen3.5:9b` | `Qwen/Qwen3.5-9B` | 3/3 | 0 | 5790 | 50059 / 51678 / 31980 |
| `MichelRosselli/bonsai-27b:Q1_0` | `prism-ml/Bonsai-27B-gguf` | 3/3 | 0 | 21951 | 327702 / 166830 / 119808 |
| `Qwen/Qwen3.6-27B` | `Qwen/Qwen3.6-27B` | not loaded | — | — | too heavy for this CPU box |
Both loaded models emitted OpenAI `tool_calls` (not `<tool_call>` XML) on
this runtime. **Do not claim 9B is better at tools** — the score is tied
at 3/3. 9B is smaller and faster. Bonsai RSS includes weights + KV
(`size` from `/api/ps`); first Bonsai prompt includes cold load.
Re-run:
```bash
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \
./bin/reasoner/bakeoff.go --json
REASONER_MODEL=MichelRosselli/bonsai-27b:Q1_0 ./bin/reasoner/bakeoff.go --json
```
+67
View File
@@ -0,0 +1,67 @@
---
type: explanation
status: current
related:
- PLAN.md
- docs/design.md
- docs/runbook.md
---
# Gap to v1 — detective brain
Goal: a brain that does not assert without proof. Search is deduction
(`facts` ≥2 sources → `info``web`). `confirmed` only from the facts root.
**v1 is a living graph the agent can write and walk**, not “more RAG”.
Epic: [Gitea #16](https://git.produktor.io/eSlider/2dph/issues/16).
Milestone: [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
Decisions: [PLAN.md](../PLAN.md).
## In (do not reopen)
Read path Go + Zig CGO (D21). HTTP + OpenAPI + MCP (D20). PicoClaw compose
profile + CPU reasoner (D18). Mail sync → import → rebuild. D14 shebangs.
Compose `api` (no CPython) / `index` (Python write). Issues #1#5, #7#13.
[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop.
[#14](https://git.produktor.io/eSlider/2dph/issues/14) `bin/brain/add.go` /
`POST /ingest` (Python `kblib.add_leafs`; no Go upsert port).
[#17](https://git.produktor.io/eSlider/2dph/issues/17) `--hop N` walks
FROM_FILE / HAS_VERSION / AUTHORED.
[#18](https://git.produktor.io/eSlider/2dph/issues/18) `--with-facts` /
`--with-chats` on rebuild (WhatsApp out of v1).
[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI recall SoT =
`bin/brain/eval.go` via Zig.
Epic [#16](https://git.produktor.io/eSlider/2dph/issues/16) closed.
## v2
[#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR — **in**.
[#30](https://git.produktor.io/eSlider/2dph/issues/30) OQ3 duckdb-go — **in**.
[#29](https://git.produktor.io/eSlider/2dph/issues/29) OQ1 contradiction
resolution — **in** (`temporal_freshness`, `authority_pairing`).
[#34](https://git.produktor.io/eSlider/2dph/issues/34) D23 flaggy CLI — **in**.
## Blockers
None for epic #16 (closed). Remaining v2: OQ4.
```
question
├─ FTS + HNSW ← in
├─ facts / info roots ← in
├─ web (D17) ← in
├─ brain/add ACID ← in
├─ Cypher hop ← in
└─ facts+chats corpus ← in
```
## Not v1
OQ4 YAML-first leafs.
OCR (OQ2), duckdb-go (OQ3/D22), and D16 adjudication (OQ1) are in.
## Close epic #16 when
Children #14, #15, #17, #18, #19 are closed. MCP tool order stays gated by tests.
+86
View File
@@ -0,0 +1,86 @@
---
type: howto
status: current
related:
- docs/README.md
- PLAN.md
---
# Run 2dph (portable)
No laptop-absolute paths. Config lives in env files under `$HOME/.config/brain/`
(mode 0600), not in git.
## Toolchain
- Go (see `go.mod`)
- Python 3.12 + [uv](https://docs.astral.sh/uv)
- Optional: Docker, Zig CGO via `bin/cgo/zig` (not gcc)
- Optional: poppler (`pdftotext`/`pdftoppm`) + tesseract `eng+deu` for mail OCR
```bash
uv venv .venv
uv pip install -r requirements.lock.txt
eval "$(bin/cgo/zig env)" # when compiling Ladybug read tools
go test ./...
uv run python -m unittest discover -s bin/tools -t .
```
## Config
| File / env | Purpose |
|------------|---------|
| `$BRAIN_SEARCH_ENV` (default `$HOME/.config/brain/search.env`) | `BRAIN_SEARCH_URL` (SearXNG). Optional Basic Auth. |
| `$HOME/.config/brain/db-profiles.yml` | read-only Postgres profiles (OnlyOffice via tunnel) |
If the host already runs SearXNG, point `BRAIN_SEARCH_URL` at it. Do not start
a second copy (D3). Optional Compose instance:
```bash
SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d
```
That binds `127.0.0.1:8888`. JSON format must stay enabled.
## Index then search
Write path is `bin/brain/add.go` for a leaf (or `POST /ingest`). Bulk
corpus rebuild remains `bin/brain/index.go --rebuild` (Compose profile
`index`). Do not DROP INDEX on Ladybug 0.19.
```bash
bin/brain/add.go --text "arc-1 runs Matrix" --root facts --source "compose.yml x docker ps"
bin/brain/index.go --rebuild --with-facts --with-chats
bin/brain/search.go "LadybugDB vector index" # facts → info → web (D17)
bin/brain/search.go "upstream flag" --no-web
source <(./bin/cli/complete.go bash) # D23 flaggy complete
bin/brain/get.go <id> --body
bin/brain/stats.go
```
`--hop N` walks File → Commit → Person from each hit. Empty web results are `throttled`, not absence.
Gap to v1: [roadmap](roadmap.md) / [epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
Ladybug 0.19: never `DROP INDEX` FTS/VECTOR (ghost catalog). Fresh indexes =
delete `var/kb.lbug` then `--rebuild`.
## HTTP / MCP
```bash
docker compose up -d brain # :8630 Zig CGO serve
docker compose --profile index run --rm index # rebuild
docker compose --profile picoclaw up brain-mcp # MCP 127.0.0.1:8630
```
`GET /openapi.json`, `POST /mcp`. Agent tool order: `search``get``audit`.
## Reasoner (optional, D18)
CPU sidecar on `127.0.0.1:11435`. Weights are not in the 2dph image.
```bash
docker compose --profile reasoner up -d reasoner
REASONER_BASE_URL=http://127.0.0.1:11435/v1 ./bin/reasoner/bakeoff.go --json
```
See [reasoner.md](reasoner.md).
+9
View File
@@ -7,7 +7,9 @@ require (
github.com/arran4/golang-ical v0.3.5
github.com/chewxy/math32 v1.11.2
github.com/daulet/tokenizers v1.27.0
github.com/duckdb/duckdb-go/v2 v2.10505.0
github.com/go-git/go-git/v5 v5.19.2
github.com/integrii/flaggy v1.8.0
golang.org/x/sys v0.47.0
golang.org/x/text v0.40.0
modernc.org/sqlite v1.56.0
@@ -20,10 +22,17 @@ require (
github.com/apache/arrow-go/v18 v18.6.0 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
github.com/duckdb/duckdb-go-bindings v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/goccy/go-json v0.10.6 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
+18
View File
@@ -31,6 +31,20 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/duckdb/duckdb-go-bindings v0.10505.0 h1:/0pPsTLrcCsTGxT0VrHgJWnOcPe1tQL1vrki1v3jbAI=
github.com/duckdb/duckdb-go-bindings v0.10505.0/go.mod h1:HoD5xePkDj3VZbBnVVfxVVYIljZ9khCprWA7FgwIiC4=
github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0 h1:FrMqquFBQlMsi34h2KZgCku54rqA8xEbXZ0NLVDKwYs=
github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0/go.mod h1:EnAvZh1kNJHp5yF+M1ZHNEvapnmt6anq1xXHVrAGqMo=
github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0 h1:lbRbpQwT1MmUhh/VTwukV9K8bxKByV3UghAP3MvsbBo=
github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0/go.mod h1:IGLSeEcFhNeZF16aVjQCULD7TsFZKG5G7SyKJAXKp5c=
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0 h1:nrsaVYj3XYCRbS2FpdOMD/KHE7egRMr+/NR1IHmjT84=
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0/go.mod h1:KAIynZ0GHCS7X5fRyuFnQMg/SZBPK/bS9OCOVojClxw=
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0 h1:qM6oGDgwXBILJGbTY4fCy6QOczLpucUA6yn6g3ORjh4=
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0/go.mod h1:81SGOYoEUs8qaAfSk1wRfM5oobrIJ5KI7AzYhK6/bvQ=
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0 h1:DjqZl9rYreHkSOqnqLmkrqH5T8UdQNcxZLJVZzGmXXA=
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0/go.mod h1:K25pJL26ARblGDeuAkrdblFvUen92+CwksLtPEHRqqQ=
github.com/duckdb/duckdb-go/v2 v2.10505.0 h1:SWwvLn2Qx/RQSnQNupwgIF8VbnJ5A6OQU9lYb/mDETI=
github.com/duckdb/duckdb-go/v2 v2.10505.0/go.mod h1:m0PW4J4FG9hlFlVdXi6Ds9owpyIDaBdE2jyce00fGcE=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o=
@@ -47,6 +61,8 @@ github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399 h1:eMj
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII=
github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY=
github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
@@ -61,6 +77,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/integrii/flaggy v1.8.0 h1:tC1qWwg4fhF2Qdaj+MpPK04cxlOSq0+HoMZqAW6Arao=
github.com/integrii/flaggy v1.8.0/go.mod h1:QS4c80m87SXG0pmVUT/Lx2RY5EbkLvLp7IKBD2jwcFA=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4=
+16 -4
View File
@@ -7,6 +7,8 @@ import (
"context"
"encoding/json"
"fmt"
"os/exec"
"path/filepath"
"github.com/eSlider/2dph/internal/brain/rank"
)
@@ -137,13 +139,23 @@ func (HTTP) Audit(context.Context) ([]byte, error) {
return json.Marshal(map[string]any{"status": "ok", "by_confidence": rows})
}
func (HTTP) Ingest(context.Context) ([]byte, error) {
func (HTTP) Ingest(ctx context.Context, body []byte) ([]byte, error) {
if len(bytes.TrimSpace(body)) == 0 {
return json.Marshal(map[string]any{
"mode": "rebuild",
"command": "bin/brain/index.go --rebuild",
"add": "v2",
"mode": "add",
"command": "bin/brain/add.go",
"rebuild": "bin/brain/index.go --rebuild",
})
}
cmd := exec.CommandContext(ctx, filepath.Join(repoRoot(), "bin", "kb", "add"), "--json")
cmd.Stdin = bytes.NewReader(body)
cmd.Dir = repoRoot()
out, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("add: %w", err)
}
return out, nil
}
func asInt(v any) int64 {
switch n := v.(type) {
+46 -41
View File
@@ -3,73 +3,78 @@ package rank
import (
"fmt"
"strconv"
"strings"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--json] [--no-web]
const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--hop N] [--json] [--no-web]
bin/brain/search.go serve [port]
bin/brain/search.go --list-model`
bin/brain/search.go --list-model
source <(./bin/cli/complete.go bash)`
type Options struct {
Query string
Root string
Repo string
Limit int
Hop int
JSONOut bool
ListModel bool
NoWeb bool
}
// NewParser is the flaggy schema for search (also used by bin/cli/complete.go).
func NewParser(opt *Options) *flaggy.Parser {
if opt.Limit == 0 {
opt.Limit = 20
}
p := cli.New("brain-search")
p.Description = "deduction search: facts → info → web"
p.String(&opt.Root, "", "root", "facts or info")
p.String(&opt.Repo, "", "repo", "filter by repo")
p.Int(&opt.Limit, "n", "n", "max hits")
p.Int(&opt.Hop, "", "hop", "walk FROM_FILE depth 1-3")
p.Bool(&opt.JSONOut, "", "json", "JSON output")
p.Bool(&opt.NoWeb, "", "no-web", "stay local")
p.Bool(&opt.ListModel, "", "list-model", "print embedding model")
return p
}
// ParseArgs reads flags. Unknown flags are an error: silently dropping them
// meant `--hop 1` vanished and its argument `1` was appended to the query.
// --hop is recognised so it cannot be swallowed; it is not implemented until
// File/FROM_FILE edges exist.
func ParseArgs(args []string) (Options, error) {
opt := Options{Limit: 20}
var queryArgs []string
for i := 0; i < len(args); i++ {
arg := args[i]
wantsValue := arg == "--root" || arg == "--repo" || arg == "-n" || arg == "--hop"
if wantsValue && i+1 >= len(args) {
return opt, fmt.Errorf("%s needs a value", arg)
p := NewParser(&opt)
var q string
p.AddPositionalValue(&q, "query", 1, false, "search query")
if err := cli.Parse(p, args); err != nil {
return opt, err
}
switch arg {
case "--root":
i++
opt.Root = args[i]
if opt.Root != "facts" && opt.Root != "info" {
opt.Query = cli.Query(q, p.TrailingArguments)
if opt.Root != "" && opt.Root != "facts" && opt.Root != "info" {
return opt, fmt.Errorf("--root must be facts or info, got %q", opt.Root)
}
case "--repo":
i++
opt.Repo = args[i]
case "-n":
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 1 {
return opt, fmt.Errorf("-n must be a positive integer, got %q", args[i])
if opt.Limit < 1 {
return opt, fmt.Errorf("-n must be a positive integer, got %q", strconv.Itoa(opt.Limit))
}
opt.Limit = n
case "--hop":
return opt, fmt.Errorf("--hop is not implemented yet (needs File/FROM_FILE edges)")
case "--json":
opt.JSONOut = true
case "--no-web":
opt.NoWeb = true
case "--list-model":
opt.ListModel = true
default:
if strings.HasPrefix(arg, "-") {
return opt, fmt.Errorf("unknown flag %q", arg)
if opt.Hop < 0 {
return opt, fmt.Errorf("--hop must be a positive integer, got %q", strconv.Itoa(opt.Hop))
}
queryArgs = append(queryArgs, arg)
if opt.Hop > 3 {
return opt, fmt.Errorf("--hop max is 3 (File → Commit → Person)")
}
}
opt.Query = strings.TrimSpace(strings.Join(queryArgs, " "))
if opt.Query == "" && !opt.ListModel {
return opt, fmt.Errorf("no query given")
}
return opt, nil
}
// Parser is the search schema for bin/cli/complete.go.
func Parser() *flaggy.Parser {
opt := Options{Limit: 20}
p := NewParser(&opt)
var q string
p.AddPositionalValue(&q, "query", 1, false, "search query")
return p
}
+17 -2
View File
@@ -19,20 +19,35 @@ type SecondSourceHit struct {
type WebFn func(query string) SecondSource
// ShouldEscalate is true when the default deduction path has no facts hit.
// ShouldEscalate is true when the default deduction path has no confirmed
// facts hit. Hypothesis/partial facts are `(not confirmed)` (D16).
// `--root facts|info` is a single-root ask: do not mix in the web.
func ShouldEscalate(hits []Hit, rootFilter string) bool {
if rootFilter != "" {
return false
}
for _, h := range hits {
if h.Root == "facts" {
if ConfirmedFact(h) {
return false
}
}
return true
}
// ConfirmedFact is a facts-root hit that is not hypothesis/partial.
// Empty confidence is treated as confirmed (legacy leafs).
func ConfirmedFact(h Hit) bool {
if h.Root != "facts" {
return false
}
switch h.Confidence {
case "hypothesis", "partial":
return false
default:
return true
}
}
// Deduce returns the second-source block, or nil when web must not run.
func Deduce(hits []Hit, query, rootFilter string, noWeb bool, web WebFn) *SecondSource {
if noWeb || web == nil || !ShouldEscalate(hits, rootFilter) {
+10
View File
@@ -14,6 +14,16 @@ func TestShouldEscalateWhenNoFacts(t *testing.T) {
}
}
func TestShouldEscalateWhenHypothesisFacts(t *testing.T) {
hyp := Hit{ID: "c", Root: "facts", Confidence: "hypothesis", Source: "a x b vs c x d"}
if !ShouldEscalate([]Hit{hyp}, "") {
t.Fatal("hypothesis facts are (not confirmed); escalate")
}
if ConfirmedFact(hyp) {
t.Fatal("hypothesis is not confirmed")
}
}
func TestShouldNotEscalateWhenFactsConfirm(t *testing.T) {
hits := []Hit{h("f", "facts", "docker ps x compose"), h("i", "info", "docs/a.md")}
if ShouldEscalate(hits, "") {
+29 -2
View File
@@ -3,7 +3,34 @@ package rank
// BM25 ranks best-first, so the top hits are the *highest* scores; cosine
// distance ranks best-first ascending. Both mirror kblib.py.
const FTSStmt = "CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " +
"RETURN node.id, node.text, node.root, node.source, score ORDER BY score DESC LIMIT $n"
"RETURN node.id, node.text, node.root, node.source, score, node.confidence ORDER BY score DESC LIMIT $n"
const VecStmt = "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n"
"RETURN node.id, node.text, node.root, node.source, distance, node.confidence ORDER BY distance LIMIT $n"
// HopStmt is the Cypher walk from a search hit. Depth 1 = File, 2 = Commit, 3 = Person.
func HopStmt(depth int) string {
switch depth {
case 1:
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File) RETURN f.id, f.path, 1"
case 2:
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit) RETURN c.id, c.subject, 2"
case 3:
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit)-[:AUTHORED]->(p:Person) RETURN p.id, p.name, 3"
default:
return ""
}
}
func HopLabel(depth int) string {
switch depth {
case 1:
return "File"
case 2:
return "Commit"
case 3:
return "Person"
default:
return ""
}
}
+9
View File
@@ -7,14 +7,23 @@ import (
"strings"
)
type HopNode struct {
ID string `json:"id"`
Label string `json:"label"`
Name string `json:"name"`
Depth int `json:"depth"`
}
// Hit is one search result, mirroring the python script's dict shape.
type Hit struct {
ID string `json:"id"`
Text string `json:"text"`
Root string `json:"root"`
Confidence string `json:"confidence,omitempty"`
Source string `json:"-"`
Score float64 `json:"score"`
Snippet string `json:"snippet,omitempty"`
Hops []HopNode `json:"hops,omitempty"`
}
// rrfK dampens the contribution of low ranks; same constant as kblib.py.
+36 -7
View File
@@ -91,15 +91,41 @@ func TestHybridKeepsVectorScoreForSharedHit(t *testing.T) {
}
// The old parser dropped unknown flags and appended their arguments to the
// query, so `search "q" --hop 1` searched for "q 1". --hop is not implemented
// here (needs File edges); it must still fail closed instead of changing q.
// query, so `search "q" --hop 1` searched for "q 1". --hop must stay a flag.
func TestParseHopIsNotSwallowedIntoTheQuery(t *testing.T) {
_, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"})
if err == nil {
t.Fatal("expected --hop to error (not implemented), not be swallowed")
opt, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !strings.Contains(err.Error(), "--hop") {
t.Fatalf("error should name --hop, got %v", err)
if opt.Query != "what runs on arc-2" {
t.Fatalf("query swallowed hop arg: %q", opt.Query)
}
if opt.Hop != 1 {
t.Fatalf("hop = %d, want 1", opt.Hop)
}
}
func TestParseHopMaxIsThree(t *testing.T) {
if _, err := ParseArgs([]string{"q", "--hop", "4"}); err == nil {
t.Fatal("expected --hop 4 to error")
}
opt, err := ParseArgs([]string{"q", "--hop", "3"})
if err != nil || opt.Hop != 3 {
t.Fatalf("hop 3: %+v err=%v", opt, err)
}
}
func TestHopStmtWalksFromFile(t *testing.T) {
s := HopStmt(1)
if !strings.Contains(s, "FROM_FILE") || !strings.Contains(s, "File") {
t.Fatalf("hop 1 must walk FROM_FILE, got %q", s)
}
s3 := HopStmt(3)
if !strings.Contains(s3, "HAS_VERSION") || !strings.Contains(s3, "AUTHORED") || !strings.Contains(s3, "Person") {
t.Fatalf("hop 3 must reach Person, got %q", s3)
}
if HopLabel(1) != "File" || HopLabel(3) != "Person" {
t.Fatal("hop labels")
}
}
@@ -146,4 +172,7 @@ func TestFTSQueryOrdersByScoreDescending(t *testing.T) {
if !strings.Contains(FTSStmt, "ORDER BY score DESC") {
t.Fatalf("FTS query must order by score DESC, got:\n%s", FTSStmt)
}
if !strings.Contains(FTSStmt, "node.confidence") {
t.Fatal("FTS must return confidence for D16")
}
}
+59
View File
@@ -0,0 +1,59 @@
package rank
import (
"fmt"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type GetOptions struct {
ID string
Body bool
JSONOut bool
}
func GetParser(opt *GetOptions) *flaggy.Parser {
p := cli.New("brain-get")
p.Description = "read one leaf"
p.Bool(&opt.Body, "", "body", "full text instead of snippet")
p.Bool(&opt.JSONOut, "", "json", "JSON output")
p.AddPositionalValue(&opt.ID, "id", 1, false, "leaf id")
return p
}
func ParseGet(args []string) (GetOptions, error) {
var opt GetOptions
if err := cli.Parse(GetParser(&opt), args); err != nil {
return opt, err
}
if opt.ID == "" {
return opt, fmt.Errorf("id required")
}
return opt, nil
}
type JSONFlag struct {
JSONOut bool
}
func bindJSON(name string, opt *JSONFlag) *flaggy.Parser {
p := cli.New(name)
p.Bool(&opt.JSONOut, "", "json", "JSON output")
return p
}
func StatsParser() *flaggy.Parser {
opt := JSONFlag{}
return bindJSON("brain-stats", &opt)
}
func EvalParser() *flaggy.Parser {
opt := JSONFlag{}
return bindJSON("brain-eval", &opt)
}
func ParseJSONFlag(name string, args []string) (JSONFlag, error) {
var opt JSONFlag
return opt, cli.Parse(bindJSON(name, &opt), args)
}
+13 -48
View File
@@ -11,30 +11,15 @@ import (
"unicode/utf8"
"github.com/eSlider/2dph/internal/brain/rank"
"github.com/eSlider/2dph/internal/cli"
)
func MainGet(args []string) int {
id, body, jsonOut := "", false, false
for _, a := range args {
switch {
case a == "--body":
body = true
case a == "--json":
jsonOut = true
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/brain/get.go <id> [--body] [--json]`)
return 0
case strings.HasPrefix(a, "-"):
fmt.Fprintf(os.Stderr, "brain/get: unknown flag %s\n", a)
return 2
default:
id = a
}
}
if id == "" {
fmt.Fprintln(os.Stderr, "brain/get: id required")
return 2
opt, err := rank.ParseGet(args)
if err != nil {
return cli.Fail(err)
}
id, body, jsonOut := opt.ID, opt.Body, opt.JSONOut
if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1
@@ -72,21 +57,11 @@ func MainGet(args []string) int {
}
func MainStats(args []string) int {
jsonOut := false
for _, a := range args {
switch a {
case "--json":
jsonOut = true
case "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/brain/stats.go [--json]`)
return 0
default:
if strings.HasPrefix(a, "-") {
fmt.Fprintf(os.Stderr, "brain/stats: unknown flag %s\n", a)
return 2
}
}
opt, err := rank.ParseJSONFlag("brain-stats", args)
if err != nil {
return cli.Fail(err)
}
jsonOut := opt.JSONOut
if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1
@@ -124,21 +99,11 @@ func MainStats(args []string) int {
}
func MainEval(args []string) int {
jsonOut := false
for _, a := range args {
switch a {
case "--json":
jsonOut = true
case "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/brain/eval.go [--json]`)
return 0
default:
if strings.HasPrefix(a, "-") {
fmt.Fprintf(os.Stderr, "brain/eval: unknown flag %s\n", a)
return 2
}
}
opt, err := rank.ParseJSONFlag("brain-eval", args)
if err != nil {
return cli.Fail(err)
}
jsonOut := opt.JSONOut
if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1
+72 -1
View File
@@ -20,6 +20,7 @@ import (
lbug "github.com/LadybugDB/go-ladybug"
"github.com/eSlider/2dph/internal/brain/rank"
"github.com/eSlider/2dph/internal/cli"
)
const defaultPort = 17830
@@ -29,6 +30,9 @@ const healthPath = "/health"
func runSearch(args []string) int {
opt, err := rank.ParseArgs(args)
if err != nil {
if errors.Is(err, cli.ErrHelp) {
return 0
}
fmt.Fprintf(os.Stderr, "brain/search: %v\n%s\n", err, rank.Usage)
return 2
}
@@ -56,6 +60,12 @@ func runSearch(args []string) int {
fmt.Fprintf(os.Stderr, "search: %v\n", err)
return 1
}
if opt.Hop > 0 {
if err := attachHops(hits, opt.Hop); err != nil {
fmt.Fprintf(os.Stderr, "hop: %v\n", err)
return 1
}
}
results := hits
for i := range results {
@@ -108,6 +118,44 @@ func searchHits(query, root, repo string, limit int) ([]Hit, error) {
return rank.RankAndFilter(fts, vec, root, repo, limit), nil
}
func attachHops(hits []Hit, n int) error {
if conn == nil {
return fmt.Errorf("brain not open")
}
for i := range hits {
var hops []rank.HopNode
for d := 1; d <= n; d++ {
stmt, err := conn.Prepare(rank.HopStmt(d))
if err != nil {
return err
}
res, err := conn.Execute(stmt, map[string]any{"id": hits[i].ID})
stmt.Close()
if err != nil {
return err
}
for res.HasNext() {
row, err := res.Next()
if err != nil {
return err
}
vals, err := row.GetAsSlice()
if err != nil || len(vals) < 3 {
continue
}
hops = append(hops, rank.HopNode{
ID: fmt.Sprint(vals[0]),
Label: rank.HopLabel(d),
Name: fmt.Sprint(vals[1]),
Depth: int(asInt(vals[2])),
})
}
}
hits[i].Hops = hops
}
return nil
}
func b2i(err error) int {
if err != nil {
return 1
@@ -168,7 +216,11 @@ func rowsToHits(res *lbug.QueryResult) ([]Hit, error) {
root := fmt.Sprint(vals[2])
source := fmt.Sprint(vals[3])
score := float64(vals[4].(float64))
hits = append(hits, Hit{ID: id, Text: text, Root: root, Source: source, Score: score})
conf := ""
if len(vals) >= 6 {
conf = fmt.Sprint(vals[5])
}
hits = append(hits, Hit{ID: id, Text: text, Root: root, Source: source, Score: score, Confidence: conf})
}
return hits, nil
}
@@ -186,8 +238,10 @@ type jsonHit struct {
ID string `json:"id"`
Text string `json:"text"`
Root string `json:"root"`
Confidence string `json:"confidence,omitempty"`
Score float64 `json:"score"`
Snippet string `json:"snippet,omitempty"`
Hops []rank.HopNode `json:"hops,omitempty"`
}
func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *jsonOut {
@@ -197,8 +251,10 @@ func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *js
ID: h.ID,
Text: h.Text,
Root: h.Root,
Confidence: h.Confidence,
Score: h.Score,
Snippet: h.Snippet,
Hops: h.Hops,
}
}
return &jsonOut{
@@ -219,9 +275,24 @@ func resultsToDicts(hits []Hit) []any {
{"root", h.Root},
{"score", h.Score},
}
if h.Confidence != "" {
d = append(d, KV{"confidence", h.Confidence})
}
if h.Snippet != "" {
d = append(d, KV{"snippet", h.Snippet})
}
if len(h.Hops) > 0 {
nodes := make([]any, len(h.Hops))
for j, n := range h.Hops {
nodes[j] = Dict{
{"id", n.ID},
{"label", n.Label},
{"name", n.Name},
{"depth", n.Depth},
}
}
d = append(d, KV{"hops", nodes})
}
out[i] = d
}
return out
+6 -12
View File
@@ -3,12 +3,13 @@ package chats
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
cliparse "github.com/eSlider/2dph/internal/cli"
)
type ooContact struct {
@@ -25,16 +26,9 @@ type ooContact struct {
}
func RunApply(args []string) int {
fs := flag.NewFlagSet("chats apply", flag.ContinueOnError)
dryRun := fs.Bool("dry-run", false, "show what would be done without writing")
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats apply [--dry-run]")
return 0
dryRun, err := parseApplyFlags(args)
if err != nil {
return cliparse.Fail(err)
}
ooCLI := findOO()
@@ -128,7 +122,7 @@ func RunApply(args []string) int {
fmt.Printf("\nchats apply: %d actions to apply\n", len(resolved))
if *dryRun {
if dryRun {
for _, r := range resolved {
switch r.Action {
case "info-add":
+74
View File
@@ -0,0 +1,74 @@
package chats
import (
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type syncTelegramFlags struct {
Limit int
Phone string
}
type syncLinkedInFlags struct {
Limit int
Refresh bool
}
func SyncParser() *flaggy.Parser {
p := cliparse.New("chats-sync")
p.Description = "download chats to var/chats"
tg := flaggy.NewSubcommand("telegram")
li := flaggy.NewSubcommand("linkedin")
var limit int
var phone string
var refresh bool
tg.Int(&limit, "", "limit", "max messages per chat")
tg.String(&phone, "", "phone", "phone (default TELEGRAM_PHONE)")
li.Int(&limit, "", "limit", "max messages per conversation")
li.Bool(&refresh, "", "refresh", "refresh webtop session")
p.AttachSubcommand(tg, 1)
p.AttachSubcommand(li, 1)
return p
}
func ImportParser() *flaggy.Parser {
return cliparse.New("chats-import")
}
func FactsParser() *flaggy.Parser {
return cliparse.New("chats-facts")
}
func ApplyParser() *flaggy.Parser {
p := cliparse.New("chats-apply")
dry := false
p.Bool(&dry, "", "dry-run", "show without writing")
return p
}
func parseTelegramFlags(args []string) (syncTelegramFlags, error) {
var f syncTelegramFlags
p := cliparse.New("chats-sync-telegram")
p.Int(&f.Limit, "", "limit", "max messages per chat")
p.String(&f.Phone, "", "phone", "phone (default TELEGRAM_PHONE)")
return f, cliparse.Parse(p, args)
}
func parseLinkedInFlags(args []string) (syncLinkedInFlags, error) {
var f syncLinkedInFlags
p := cliparse.New("chats-sync-linkedin")
p.Int(&f.Limit, "", "limit", "max messages per conversation")
p.Bool(&f.Refresh, "", "refresh", "refresh webtop session")
return f, cliparse.Parse(p, args)
}
func parseApplyFlags(args []string) (dryRun bool, err error) {
p := cliparse.New("chats-apply")
p.Bool(&dryRun, "", "dry-run", "show without writing")
return dryRun, cliparse.Parse(p, args)
}
func parseNoFlags(name string, args []string) error {
return cliparse.Parse(cliparse.New(name), args)
}
+4 -10
View File
@@ -3,12 +3,13 @@ package chats
import (
"bufio"
"encoding/json"
"flag"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
cliparse "github.com/eSlider/2dph/internal/cli"
)
var (
@@ -76,15 +77,8 @@ type ExtractedFact struct {
}
func RunFacts(args []string) int {
fs := flag.NewFlagSet("chats facts", flag.ContinueOnError)
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats facts")
return 0
if err := parseNoFlags("chats-facts", args); err != nil {
return cliparse.Fail(err)
}
root := Dir()
+4 -10
View File
@@ -4,25 +4,19 @@ import (
"bufio"
"bytes"
"encoding/json"
"flag"
"fmt"
"html"
"os"
"path/filepath"
"sort"
"strings"
cliparse "github.com/eSlider/2dph/internal/cli"
)
func RunImport(args []string) int {
fs := flag.NewFlagSet("chats import", flag.ContinueOnError)
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats import")
return 0
if err := parseNoFlags("chats-import", args); err != nil {
return cliparse.Fail(err)
}
root := Dir()
+9 -14
View File
@@ -2,12 +2,13 @@ package chats
import (
"context"
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"time"
cliparse "github.com/eSlider/2dph/internal/cli"
)
func checkLinkedInSession(userDataDir string) (bool, error) {
@@ -29,18 +30,12 @@ func checkLinkedInSession(userDataDir string) (bool, error) {
}
func RunSyncLinkedIn(args []string) int {
fs := flag.NewFlagSet("chats sync linkedin", flag.ContinueOnError)
limit := fs.Int("limit", 0, "max messages per conversation (0 = all)")
refresh := fs.Bool("refresh", false, "refresh session from live webtop browser before sync")
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats sync linkedin [--limit N] [--refresh]")
return 0
f, err := parseLinkedInFlags(args)
if err != nil {
return cliparse.Fail(err)
}
limit := f.Limit
refresh := f.Refresh
userDataDir := envVar("LINKEDIN_USER_DATA_DIR", "")
if userDataDir == "" {
@@ -48,7 +43,7 @@ func RunSyncLinkedIn(args []string) int {
userDataDir = home + "/.linkedin-mcp/profile"
}
if *refresh {
if refresh {
if code := refreshLinkedInSession(userDataDir); code != 0 {
return code
}
@@ -72,7 +67,7 @@ func RunSyncLinkedIn(args []string) int {
defer cancel()
start := time.Now()
if err := src.Sync(ctx, Dir(), *limit); err != nil {
if err := src.Sync(ctx, Dir(), limit); err != nil {
fmt.Fprintf(os.Stderr, "chats sync linkedin: %v\n", err)
return 1
}
+9 -14
View File
@@ -2,33 +2,28 @@ package chats
import (
"context"
"flag"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"time"
cliparse "github.com/eSlider/2dph/internal/cli"
)
func RunSyncTelegram(args []string) int {
fs := flag.NewFlagSet("chats sync telegram", flag.ContinueOnError)
limit := fs.Int("limit", 0, "max messages per chat (0 = all)")
phone := fs.String("phone", "", "phone number (default env TELEGRAM_PHONE)")
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats sync telegram [--limit N] [--phone PHONE]")
return 0
f, err := parseTelegramFlags(args)
if err != nil {
return cliparse.Fail(err)
}
limit := f.Limit
phone := f.Phone
apiIDStr := envVar("TELEGRAM_API_ID", "")
apiHash := envVar("TELEGRAM_API_HASH", "")
sessionStr := envVar("TELEGRAM_SESSION_STRING", "")
phoneNum := *phone
phoneNum := phone
if phoneNum == "" {
phoneNum = envVar("TELEGRAM_PHONE", "")
}
@@ -78,7 +73,7 @@ func RunSyncTelegram(args []string) int {
defer cancel()
start := time.Now()
if err := src.Sync(ctx, Dir(), *limit); err != nil {
if err := src.Sync(ctx, Dir(), limit); err != nil {
fmt.Fprintf(os.Stderr, "chats sync telegram: %v\n", err)
return 1
}
+190
View File
@@ -0,0 +1,190 @@
// Package cli is the shared flaggy wrapper (D23).
//
// flaggy: zero deps, flags at any position, shell completion scripts.
// Individual tools keep ShowCompletion off so a query like "completion" is
// not stolen; dump scripts with bin/cli/complete.go.
package cli
import (
"errors"
"fmt"
"os"
"strings"
"sync"
"github.com/integrii/flaggy"
)
// ErrHelp means -h/--help was requested (exit 0).
var ErrHelp = errors.New("help")
var parseMu sync.Mutex
// New returns a per-call parser. Never reuse: flaggy parses once.
func New(name string) *flaggy.Parser {
p := flaggy.NewParser(name)
p.ShowVersionWithVersionFlag = false
p.ShowCompletion = false
// Extra positionals become TrailingArguments (search "two words --json").
// Unknown dash tokens are rejected in Parse after flaggy returns.
p.ShowHelpOnUnexpected = false
p.ShowHelpWithHFlag = true
return p
}
// Parse runs p.ParseArgs and turns flaggy's os.Exit into an error.
// Not safe to call in parallel (flaggy.PanicInsteadOfExit is process-global).
func Parse(p *flaggy.Parser, args []string) error {
parseMu.Lock()
defer parseMu.Unlock()
prev := flaggy.PanicInsteadOfExit
flaggy.PanicInsteadOfExit = true
defer func() { flaggy.PanicInsteadOfExit = prev }()
var exitMsg string
err := func() error {
defer func() {
if r := recover(); r != nil {
exitMsg = fmt.Sprint(r)
}
}()
return p.ParseArgs(args)
}()
if err != nil {
return err
}
if exitMsg != "" {
if strings.Contains(exitMsg, "code: 0") {
return ErrHelp
}
return errors.New(exitMsg)
}
if u := unknownFlags(p, args); len(u) > 0 {
return fmt.Errorf("unknown flag %q", u[0])
}
return nil
}
func unknownFlags(p *flaggy.Parser, args []string) []string {
flags := collectFlags(&p.Subcommand)
var out []string
skipNext := false
for _, a := range args {
if skipNext {
skipNext = false
continue
}
if a == "--" {
break
}
name, inline := flagName(a)
if name == "" {
continue
}
if name == "h" || name == "help" {
continue
}
f := findFlag(flags, name)
if f == nil {
out = append(out, a)
continue
}
if !inline && !isBoolFlag(f) {
skipNext = true
}
}
return out
}
func flagName(a string) (name string, inline bool) {
if a == "-" || !strings.HasPrefix(a, "-") {
return "", false
}
rest := strings.TrimLeft(a, "-")
name, _, inline = strings.Cut(rest, "=")
return name, inline
}
func collectFlags(sc *flaggy.Subcommand) []*flaggy.Flag {
out := append([]*flaggy.Flag{}, sc.Flags...)
for _, sub := range sc.Subcommands {
out = append(out, collectFlags(sub)...)
}
return out
}
func findFlag(flags []*flaggy.Flag, name string) *flaggy.Flag {
for _, f := range flags {
if f.HasName(name) {
return f
}
}
return nil
}
func isBoolFlag(f *flaggy.Flag) bool {
_, ok := f.AssignmentVar.(*bool)
return ok
}
// Query joins the first positional with leftover trailing words.
func Query(first string, trailing []string) string {
parts := make([]string, 0, 1+len(trailing))
if s := strings.TrimSpace(first); s != "" {
parts = append(parts, s)
}
for _, t := range trailing {
if s := strings.TrimSpace(t); s != "" {
parts = append(parts, s)
}
}
return strings.Join(parts, " ")
}
// Code maps parse errors to process exit codes (0 help, 2 usage).
func Code(err error) int {
if err == nil || errors.Is(err, ErrHelp) {
return 0
}
return 2
}
// Fail prints err unless it is help or a flaggy exit that already wrote stderr.
func Fail(err error) int {
if err == nil || errors.Is(err, ErrHelp) {
return 0
}
if strings.HasPrefix(err.Error(), "Panic instead of exit") {
return 2
}
fmt.Fprintln(os.Stderr, err)
return 2
}
// Tool is one shebang CLI for completion dump.
type Tool struct {
Path string
Name string
New func() *flaggy.Parser
}
// BashScript concatenates flaggy bash complete scripts and binds each
// function to the shebang path (./bin/subject/method.go).
func BashScript(tools []Tool) string {
var b strings.Builder
b.WriteString("# 2dph flaggy completions (D23). source <(./bin/cli/complete.go bash)\n")
for _, t := range tools {
p := t.New()
p.Name = t.Name
script := flaggy.GenerateBashCompletion(p)
b.WriteString(script)
fn := "_" + strings.ReplaceAll(t.Name, "-", "_") + "_complete"
if t.Path != "" && t.Path != t.Name {
fmt.Fprintf(&b, "complete -F %s %s\n", fn, t.Path)
if !strings.HasPrefix(t.Path, "./") {
fmt.Fprintf(&b, "complete -F %s ./%s\n", fn, t.Path)
}
}
}
return b.String()
}
+77
View File
@@ -0,0 +1,77 @@
package cli
import (
"errors"
"strings"
"testing"
"github.com/integrii/flaggy"
)
func TestParseBoolAndIntAnyPosition(t *testing.T) {
p := New("t")
jsonOut := false
n := 20
q := ""
p.Bool(&jsonOut, "", "json", "JSON")
p.Int(&n, "n", "n", "limit")
p.AddPositionalValue(&q, "query", 1, false, "q")
if err := Parse(p, []string{"two", "words", "--json", "-n", "5"}); err != nil {
t.Fatal(err)
}
got := Query(q, p.TrailingArguments)
if got != "two words" || !jsonOut || n != 5 {
t.Fatalf("q=%q json=%v n=%d", got, jsonOut, n)
}
}
func TestParseUnknownFlagIsError(t *testing.T) {
p := New("t")
jsonOut := false
p.Bool(&jsonOut, "", "json", "JSON")
if err := Parse(p, []string{"--nope"}); err == nil {
t.Fatal("unknown flag accepted")
}
}
func TestParseHelpIsErrHelp(t *testing.T) {
p := New("t")
jsonOut := false
p.Bool(&jsonOut, "", "json", "JSON")
err := Parse(p, []string{"--help"})
if !errors.Is(err, ErrHelp) {
t.Fatalf("got %v", err)
}
}
func TestParseMissingFlagValueIsError(t *testing.T) {
p := New("t")
n := 0
p.Int(&n, "", "hop", "hop")
if err := Parse(p, []string{"--hop"}); err == nil {
t.Fatal("expected missing value error")
}
}
func TestBashScriptNamesShebangPath(t *testing.T) {
out := BashScript([]Tool{{
Path: "bin/brain/search.go",
Name: "brain-search",
New: newSearchLike,
}})
if !strings.Contains(out, "--json") || !strings.Contains(out, "--hop") {
t.Fatalf("flags missing:\n%s", out)
}
if !strings.Contains(out, "complete -F") || !strings.Contains(out, "bin/brain/search.go") {
t.Fatalf("shebang complete missing:\n%s", out)
}
}
func newSearchLike() *flaggy.Parser {
p := New("brain-search")
jsonOut := false
hop := 0
p.Bool(&jsonOut, "", "json", "JSON")
p.Int(&hop, "", "hop", "graph hop")
return p
}
+24
View File
@@ -0,0 +1,24 @@
package cli
import "github.com/integrii/flaggy"
type QAStats struct {
JSONL string
}
func QAParser() *flaggy.Parser {
c := QAStats{}
return BindQA(&c)
}
func BindQA(c *QAStats) *flaggy.Parser {
p := New("qa-stats")
p.Description = "DuckDB quantiles / JSONL count"
p.String(&c.JSONL, "", "jsonl", "JSONL file (else stdin JSON [float,…])")
return p
}
func ParseQAStats(args []string) (QAStats, error) {
var c QAStats
return c, Parse(BindQA(&c), args)
}
+47
View File
@@ -0,0 +1,47 @@
// Package duckstats runs in-process DuckDB for columnar aggregates.
// Graph facts stay in Ladybug. Web-search KV cache stays modernc sqlite.
package duckstats
import (
"database/sql"
"fmt"
_ "github.com/duckdb/duckdb-go/v2"
)
type Stats struct {
N int `json:"n"`
Min float64 `json:"min"`
P50 float64 `json:"p50"`
P95 float64 `json:"p95"`
Max float64 `json:"max"`
Avg float64 `json:"avg"`
}
func Quantiles(samples []float64) (Stats, error) {
if len(samples) == 0 {
return Stats{}, fmt.Errorf("duckstats: empty samples")
}
db, err := sql.Open("duckdb", "")
if err != nil {
return Stats{}, err
}
defer db.Close()
var s Stats
err = db.QueryRow(`
SELECT count(v), min(v), quantile_cont(v, 0.5), quantile_cont(v, 0.95), max(v), avg(v)
FROM (SELECT unnest(?) AS v)`, samples).Scan(
&s.N, &s.Min, &s.P50, &s.P95, &s.Max, &s.Avg)
return s, err
}
func CountJSONL(path string) (int64, error) {
db, err := sql.Open("duckdb", "")
if err != nil {
return 0, err
}
defer db.Close()
var n int64
err = db.QueryRow(`SELECT count(*) FROM read_json_auto(?)`, path).Scan(&n)
return n, err
}
+51
View File
@@ -0,0 +1,51 @@
package duckstats
import (
"os"
"testing"
)
func TestQuantilesEmpty(t *testing.T) {
_, err := Quantiles(nil)
if err == nil {
t.Fatal("empty slice must error")
}
}
func TestQuantilesOdd(t *testing.T) {
s, err := Quantiles([]float64{1, 2, 3, 4, 5})
if err != nil {
t.Fatal(err)
}
if s.N != 5 {
t.Fatalf("n=%d", s.N)
}
if s.Min != 1 || s.Max != 5 {
t.Fatalf("min=%v max=%v", s.Min, s.Max)
}
if s.P50 != 3 {
t.Fatalf("p50=%v want 3", s.P50)
}
if s.Avg != 3 {
t.Fatalf("avg=%v want 3", s.Avg)
}
if s.P95 < 4.5 || s.P95 > 5 {
t.Fatalf("p95=%v want in [4.5,5]", s.P95)
}
}
func TestCountJSONL(t *testing.T) {
dir := t.TempDir()
p := dir + "/rows.jsonl"
body := "{\"ms\":1}\n{\"ms\":2}\n{\"ms\":3}\n"
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
n, err := CountJSONL(p)
if err != nil {
t.Fatal(err)
}
if n != 3 {
t.Fatalf("count=%d want 3", n)
}
}
+120
View File
@@ -0,0 +1,120 @@
// Package facts is cgo-free evidence rules (D16 contradictions).
package facts
import "strconv"
const (
ConfConfirmed = "confirmed"
ConfHypothesis = "hypothesis"
RuleUnresolved = "unresolved"
RuleTemporalFreshness = "temporal_freshness"
RuleAuthorityPairing = "authority_pairing"
RuleTwoSource = "two_source"
RuleSingleSource = "single_source"
KindRuntime = "runtime"
KindConfig = "config"
KindNarrative = "narrative"
)
// Source is one independent pointer on a yes or no side.
type Source struct {
ID string `json:"id"`
Kind string `json:"kind"`
When string `json:"when,omitempty"`
Stale bool `json:"stale,omitempty"`
}
// Claim is one assertion with yes/no evidence lists.
type Claim struct {
Text string `json:"text"`
Yes []Source `json:"yes"`
No []Source `json:"no"`
}
// Result is audit output. Confirmed=false means `(not confirmed)`.
type Result struct {
Text string `json:"text"`
Confidence string `json:"confidence"`
Confirmed bool `json:"confirmed"`
Rule string `json:"rule"`
Winner string `json:"winner,omitempty"`
YesN int `json:"yes"`
NoN int `json:"no"`
}
func independent(ss []Source) int {
seen := map[string]struct{}{}
for i, s := range ss {
id := s.ID
if id == "" {
id = s.Kind + "#" + strconv.Itoa(i)
}
seen[id] = struct{}{}
}
return len(seen)
}
func freshN(ss []Source) int {
n := 0
for _, s := range ss {
if !s.Stale {
n++
}
}
return n
}
func strongN(ss []Source) int {
n := 0
for _, s := range ss {
if s.Kind == KindRuntime || s.Kind == KindConfig {
n++
}
}
return n
}
func out(c Claim, conf, rule, winner string) Result {
return Result{
Text: c.Text,
Confidence: conf,
Confirmed: conf == ConfConfirmed,
Rule: rule,
Winner: winner,
YesN: independent(c.Yes),
NoN: independent(c.No),
}
}
// Adjudicate applies D16: ≥2 yes vs ≥2 no stays hypothesis until a rule fires.
// Order: temporal_freshness, then authority_pairing (A/B beats narrative C).
func Adjudicate(c Claim) Result {
yesN := independent(c.Yes)
noN := independent(c.No)
if yesN < 2 || noN < 2 {
if yesN >= 2 {
return out(c, ConfConfirmed, RuleTwoSource, "yes")
}
if noN >= 2 {
return out(c, ConfConfirmed, RuleTwoSource, "no")
}
return out(c, ConfHypothesis, RuleSingleSource, "")
}
yf, nf := freshN(c.Yes), freshN(c.No)
if yf >= 2 && nf < 2 {
return out(c, ConfConfirmed, RuleTemporalFreshness, "yes")
}
if nf >= 2 && yf < 2 {
return out(c, ConfConfirmed, RuleTemporalFreshness, "no")
}
ys, ns := strongN(c.Yes), strongN(c.No)
if ys >= 2 && ns < 2 {
return out(c, ConfConfirmed, RuleAuthorityPairing, "yes")
}
if ns >= 2 && ys < 2 {
return out(c, ConfConfirmed, RuleAuthorityPairing, "no")
}
return out(c, ConfHypothesis, RuleUnresolved, "")
}
+86
View File
@@ -0,0 +1,86 @@
package facts
import "testing"
func src(id, kind string, stale bool) Source {
return Source{ID: id, Kind: kind, Stale: stale}
}
func TestTwoVsTwoStaysHypothesis(t *testing.T) {
c := Claim{
Text: "svc listens on 443",
Yes: []Source{
src("docker-ps", KindRuntime, false),
src("compose", KindConfig, false),
},
No: []Source{
src("docker-ps-old", KindRuntime, false),
src("compose-old", KindConfig, false),
},
}
r := Adjudicate(c)
if r.Confirmed || r.Confidence != ConfHypothesis || r.Rule != RuleUnresolved {
t.Fatalf("2v2 must stay (not confirmed): %+v", r)
}
if r.Winner != "" {
t.Fatalf("unresolved must not name a winner: %+v", r)
}
}
func TestTemporalFreshnessResolvesStaleSide(t *testing.T) {
c := Claim{
Text: "svc listens on 443",
Yes: []Source{
src("docker-ps", KindRuntime, false),
src("compose", KindConfig, false),
},
No: []Source{
src("old-readme", KindNarrative, true),
src("old-wiki", KindNarrative, true),
},
}
r := Adjudicate(c)
if !r.Confirmed || r.Rule != RuleTemporalFreshness || r.Winner != "yes" {
t.Fatalf("fresh yes vs stale no: %+v", r)
}
}
func TestAuthorityPairingBeatsNarrative(t *testing.T) {
c := Claim{
Text: "svc listens on 443",
Yes: []Source{
src("docker-ps", KindRuntime, false),
src("compose", KindConfig, false),
},
No: []Source{
src("readme", KindNarrative, false),
src("wiki", KindNarrative, false),
},
}
r := Adjudicate(c)
if !r.Confirmed || r.Rule != RuleAuthorityPairing || r.Winner != "yes" {
t.Fatalf("A×B vs C×C: %+v", r)
}
}
func TestTwoSourceYesIsConfirmed(t *testing.T) {
c := Claim{
Text: "arc-1 runs Matrix",
Yes: []Source{
src("compose", KindConfig, false),
src("docker-ps", KindRuntime, false),
},
}
r := Adjudicate(c)
if !r.Confirmed || r.Rule != RuleTwoSource || r.Winner != "yes" {
t.Fatalf("%+v", r)
}
}
func TestSingleSourceIsHypothesis(t *testing.T) {
c := Claim{Text: "maybe", Yes: []Source{src("readme", KindNarrative, false)}}
r := Adjudicate(c)
if r.Confirmed || r.Rule != RuleSingleSource {
t.Fatalf("%+v", r)
}
}
+51
View File
@@ -0,0 +1,51 @@
package gitlog
import (
"fmt"
"time"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Repo, Root, Since string
Limit int
JSONOut bool
}
func Parser() *flaggy.Parser {
c := CLI{}
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
p := cli.New("git-import")
p.Description = "go-git history → commit leafs"
p.Bool(&c.JSONOut, "", "json", "JSON output")
p.Int(&c.Limit, "", "limit", "max commits (0 = all)")
p.String(&c.Since, "", "since", "RFC3339 or YYYY-MM-DD")
p.String(&c.Root, "", "root", "scan dir for git repos")
p.AddPositionalValue(&c.Repo, "repo", 1, false, "git repo path")
return p
}
func ParseArgs(args []string) (CLI, error) {
var c CLI
if err := cli.Parse(Bind(&c), args); err != nil {
return c, err
}
return c, nil
}
func ParseSince(s string) (time.Time, error) {
if s == "" {
return time.Time{}, nil
}
for _, layout := range []string{time.RFC3339, "2006-01-02"} {
if t, err := time.Parse(layout, s); err == nil {
return t, nil
}
}
return time.Time{}, fmt.Errorf("cannot parse --since %q", s)
}
+9 -1
View File
@@ -144,7 +144,15 @@ func (s *Server) mcpCall(r *http.Request, params json.RawMessage) (any, error) {
return nil, fmt.Errorf("cancelled")
}
defer s.release()
body, err = s.api.Ingest(r.Context())
var payload []byte
text := strings.TrimSpace(fmt.Sprint(p.Arguments["text"]))
if text != "" && text != "<nil>" {
payload, err = json.Marshal(p.Arguments)
if err != nil {
return nil, err
}
}
body, err = s.api.Ingest(r.Context(), payload)
default:
return nil, fmt.Errorf("unknown tool %s", p.Name)
}
+43 -5
View File
@@ -11,6 +11,7 @@ import (
"context"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"os"
@@ -27,7 +28,7 @@ type API interface {
Get(ctx context.Context, id string, body bool) ([]byte, error)
Stats(ctx context.Context) ([]byte, error)
Audit(ctx context.Context) ([]byte, error)
Ingest(ctx context.Context) ([]byte, error)
Ingest(ctx context.Context, body []byte) ([]byte, error)
}
type Server struct {
@@ -59,7 +60,7 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
case PathAudit:
s.handleJSON(w, r, s.api.Audit)
case PathIngest:
s.handleJSON(w, r, s.api.Ingest)
s.handleIngest(w, r)
case PathOpenAPI:
s.handleOpenAPI(w, r)
case PathMCP:
@@ -116,6 +117,24 @@ func (s *Server) handleJSON(w http.ResponseWriter, r *http.Request, fn func(cont
writeAPI(w, body, err)
}
func (s *Server) handleIngest(w http.ResponseWriter, r *http.Request) {
var raw []byte
if r.Method == http.MethodPost {
b, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err != nil {
writeJSON(w, http.StatusBadRequest, map[string]any{"error": "read body"})
return
}
raw = b
}
if !s.acquire(w, r) {
return
}
defer s.release()
body, err := s.api.Ingest(r.Context(), raw)
writeAPI(w, body, err)
}
func (s *Server) tryAcquire(r *http.Request) bool {
return s.acquire(nopWriter{}, r)
}
@@ -191,12 +210,31 @@ func (ExecSearcher) Get(context.Context, string, bool) ([]byte, error) {
}
func (ExecSearcher) Stats(context.Context) ([]byte, error) { return nil, errUnimplemented }
func (ExecSearcher) Audit(context.Context) ([]byte, error) { return nil, errUnimplemented }
func (ExecSearcher) Ingest(context.Context) ([]byte, error) {
func (b ExecSearcher) Ingest(ctx context.Context, body []byte) ([]byte, error) {
if len(strings.TrimSpace(string(body))) == 0 {
return json.Marshal(map[string]any{
"mode": "rebuild",
"command": "bin/brain/index.go --rebuild",
"mode": "add",
"command": "bin/brain/add.go",
"rebuild": "bin/brain/index.go --rebuild",
})
}
root := os.Getenv("KB_ROOT")
if root == "" {
root = "."
}
cmd := exec.CommandContext(ctx, filepath.Join(root, "bin/kb/add"), "--json")
cmd.Stdin = strings.NewReader(string(body))
cmd.Dir = root
out, err := cmd.Output()
if err != nil {
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
return nil, errors.New("add failed: " + strings.TrimSpace(string(exitErr.Stderr)))
}
return nil, err
}
return out, nil
}
func defaultSearchCmd(root string) string {
if env := os.Getenv("KB_SEARCH_CMD"); env != "" {
+27 -2
View File
@@ -1,6 +1,7 @@
package httpapi
import (
"bytes"
"context"
"encoding/json"
"net/http"
@@ -64,8 +65,11 @@ func (f *fakeSearcher) Audit(context.Context) ([]byte, error) {
return []byte(`{"status":"ok"}`), nil
}
func (f *fakeSearcher) Ingest(context.Context) ([]byte, error) {
return []byte(`{"mode":"rebuild","command":"bin/brain/index.go --rebuild"}`), nil
func (f *fakeSearcher) Ingest(_ context.Context, body []byte) ([]byte, error) {
if len(bytes.TrimSpace(body)) == 0 {
return []byte(`{"mode":"add","command":"bin/brain/add.go"}`), nil
}
return []byte(`{"mode":"add","ids":["fake-leaf"]}`), nil
}
func (f *fakeSearcher) count() int {
@@ -193,6 +197,27 @@ func TestStatsAuditIngest(t *testing.T) {
}
}
func TestIngestIsAddNotRebuildHint(t *testing.T) {
h := NewServer(&fakeSearcher{}, 1)
code, body := get(t, h, "/ingest")
if code != http.StatusOK {
t.Fatalf("GET /ingest code = %d body=%s", code, body)
}
if strings.Contains(string(body), `"add":"v2"`) || strings.Contains(string(body), "write is v2") {
t.Fatalf("GET /ingest still a v2 hint: %s", body)
}
if !strings.Contains(string(body), "bin/brain/add.go") {
t.Fatalf("GET /ingest should name add.go: %s", body)
}
code, body = postJSON(t, h, "/ingest", `{"text":"hello","root":"info","source":"t"}`)
if code != http.StatusOK {
t.Fatalf("POST /ingest code = %d body=%s", code, body)
}
if !strings.Contains(string(body), "fake-leaf") {
t.Fatalf("POST /ingest should add: %s", body)
}
}
func TestHTTPPackageDoesNotExecPython(t *testing.T) {
raw, err := os.ReadFile("server.go")
if err != nil {
+9 -1
View File
@@ -47,7 +47,15 @@ var Ops = []Op{
},
{Path: PathStats, Method: "get", ID: "stats", Summary: "index health", MCP: true},
{Path: PathAudit, Method: "get", ID: "audit", Summary: "facts confidence histogram", MCP: true},
{Path: PathIngest, Method: "get", ID: "ingest", Summary: "rebuild hint (write is v2)", MCP: true},
{
Path: PathIngest, Method: "post", ID: "ingest", Summary: "add a leaf without rebuild",
MCP: true,
Params: []Param{
{Name: "text", In: "query", Type: "string", Description: "leaf text (omit for CLI hint)"},
{Name: "root", In: "query", Type: "string", Description: "facts or info (default info)"},
{Name: "source", In: "query", Type: "string", Description: "evidence pointer; facts need two sources"},
},
},
{Path: PathOpenAPI, Method: "get", ID: "openapi", Summary: "OpenAPI 3 document for this server"},
}
+14 -1
View File
@@ -38,11 +38,24 @@ func TestMCPToolsMatchOpenAPIPaths(t *testing.T) {
t.Fatalf("MCP tool %s has no OpenAPI path %s", tool.Name, path)
}
}
for _, need := range []string{"search", "get", "stats", "audit"} {
for _, need := range []string{"search", "get", "stats", "audit", "ingest"} {
if !names[need] {
t.Fatalf("MCP tools missing %s: %v", need, names)
}
}
var ingest MCPTool
for _, tool := range tools {
if tool.Name == "ingest" {
ingest = tool
break
}
}
if strings.Contains(ingest.Description, "v2") {
t.Fatalf("ingest still a v2 hint: %s", ingest.Description)
}
if !strings.Contains(ingest.Description, "add") {
t.Fatalf("ingest should describe add: %s", ingest.Description)
}
}
func TestOpenAPIHTTP(t *testing.T) {
+41
View File
@@ -0,0 +1,41 @@
package mdleaves
import (
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Root string
Files string
JSONOut bool
}
func Parser() *flaggy.Parser {
c := CLI{Root: "."}
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
if c.Root == "" {
c.Root = "."
}
p := cli.New("markdown-import")
p.Description = "split markdown H2 leafs"
p.Bool(&c.JSONOut, "", "json", "JSON output")
p.String(&c.Files, "", "files", "comma-separated paths")
p.AddPositionalValue(&c.Root, "dir", 1, false, "markdown root")
return p
}
func ParseArgs(args []string) (CLI, error) {
c := CLI{Root: "."}
p := Bind(&c)
if err := cli.Parse(p, args); err != nil {
return c, err
}
if extra := cli.Query("", p.TrailingArguments); extra != "" && c.Root == "." {
c.Root = extra
}
return c, nil
}
+146
View File
@@ -0,0 +1,146 @@
package mdleaves
import (
"encoding/json"
"os"
"path/filepath"
"regexp"
"strings"
)
type Leaf struct {
Source string `json:"source"`
Repo string `json:"repo"`
Heading string `json:"heading"`
Text string `json:"text"`
Type string `json:"type"`
Status string `json:"status"`
Related string `json:"related"`
}
type chunk struct {
Heading string
Text string
}
var (
h1 = regexp.MustCompile(`^# \S`)
h2 = regexp.MustCompile(`^## \S`)
)
func ExtractFrontmatter(text string) (map[string]string, string) {
if !strings.HasPrefix(text, "---") {
return map[string]string{}, text
}
end := strings.Index(text[3:], "\n---")
if end == -1 {
return map[string]string{}, text
}
fm := strings.TrimSpace(text[3 : 3+end])
body := text[3+end+4:]
meta := map[string]string{}
for _, line := range strings.Split(fm, "\n") {
key, value, ok := strings.Cut(line, ":")
if !ok {
continue
}
meta[strings.TrimSpace(key)] = strings.Trim(strings.TrimSpace(value), `"'`)
}
return meta, body
}
func SplitLeafs(meta map[string]string, body string) []chunk {
lines := strings.Split(body, "\n")
title := ""
type hdr struct {
heading string
start int
}
var headers []hdr
for i, line := range lines {
switch {
case h1.MatchString(line):
title = strings.TrimSpace(strings.TrimLeft(line, "#"))
case h2.MatchString(line):
headers = append(headers, hdr{strings.TrimSpace(strings.TrimLeft(line, "#")), i})
}
}
if len(headers) == 0 {
var kept []string
for _, l := range lines {
if strings.TrimSpace(l) != "" {
kept = append(kept, l)
}
}
return []chunk{{Heading: title, Text: strings.TrimSpace(strings.Join(kept, "\n"))}}
}
out := make([]chunk, 0, len(headers))
for idx, h := range headers {
end := len(lines)
if idx+1 < len(headers) {
end = headers[idx+1].start
}
var kept []string
for _, l := range lines[h.start:end] {
if strings.TrimSpace(l) != "" {
kept = append(kept, l)
}
}
text := strings.Join(kept, "\n")
if idx == 0 && title != "" {
text = title + "\n\n" + text
}
out = append(out, chunk{Heading: h.heading, Text: strings.TrimSpace(text)})
}
return out
}
func ToAll(text, path, repo string) []Leaf {
meta, body := ExtractFrontmatter(text)
if meta["type"] == "" {
meta["type"] = "reference"
}
if meta["status"] == "" {
meta["status"] = "current"
}
chunks := SplitLeafs(meta, body)
out := make([]Leaf, 0, len(chunks))
for _, c := range chunks {
out = append(out, Leaf{
Source: path,
Repo: repo,
Heading: c.Heading,
Text: c.Text,
Type: meta["type"],
Status: meta["status"],
Related: meta["related"],
})
}
return out
}
func WalkMarkdown(root string) ([]string, error) {
var out []string
err := filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
ext := strings.ToLower(filepath.Ext(p))
if ext == ".md" || ext == ".markdown" {
out = append(out, p)
}
return nil
})
return out, err
}
func EncodeJSON(leafs []Leaf) (string, error) {
raw, err := json.MarshalIndent(leafs, "", " ")
if err != nil {
return "", err
}
return string(raw) + "\n", nil
}
+73
View File
@@ -0,0 +1,73 @@
package mdleaves
import (
"strings"
"testing"
)
func TestSplitLeafsOnH2(t *testing.T) {
body := "# Title\n\n## One\n\nalpha\n\n## Two\n\nbeta\n"
leafs := SplitLeafs(map[string]string{}, body)
if len(leafs) != 2 {
t.Fatalf("n=%d", len(leafs))
}
if leafs[0].Heading != "One" || !strings.Contains(leafs[0].Text, "Title") {
t.Fatalf("first=%+v", leafs[0])
}
if leafs[1].Heading != "Two" || strings.Contains(leafs[1].Text, "Title") {
t.Fatalf("second=%+v", leafs[1])
}
}
func TestSplitLeafsNoH2IsWholeDoc(t *testing.T) {
body := "# Title\n\njust a paragraph\n"
leafs := SplitLeafs(nil, body)
if len(leafs) != 1 {
t.Fatalf("n=%d", len(leafs))
}
if leafs[0].Heading != "Title" {
t.Fatalf("heading=%q", leafs[0].Heading)
}
if !strings.Contains(leafs[0].Text, "just a paragraph") {
t.Fatalf("text=%q", leafs[0].Text)
}
}
func TestFrontmatterAndToAll(t *testing.T) {
raw := "---\ntype: howto\nstatus: current\nrelated: docs/design.md\n---\n# Doc\n\n## Step\n\ndo it\n"
got := ToAll(raw, "docs/x.md", "eSlider/2dph")
if len(got) != 1 {
t.Fatalf("n=%d", len(got))
}
if got[0].Type != "howto" || got[0].Status != "current" {
t.Fatalf("%+v", got[0])
}
if got[0].Source != "docs/x.md" || got[0].Repo != "eSlider/2dph" {
t.Fatalf("%+v", got[0])
}
if got[0].Related != "docs/design.md" {
t.Fatalf("related=%q", got[0].Related)
}
}
func TestEncodeJSONAndYAML(t *testing.T) {
leafs := ToAll("# Hi\n\nbody\n", "a.md", "")
js, err := EncodeJSON(leafs)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(js, `"heading": "Hi"`) {
t.Fatalf("json=%s", js)
}
y := EncodeYAML(leafs)
if !strings.Contains(y, "heading: Hi") {
t.Fatalf("yaml=%s", y)
}
}
func TestDefaultsTypeAndStatus(t *testing.T) {
got := ToAll("# Hi\n\nbody\n", "a.md", "")
if got[0].Type != "reference" || got[0].Status != "current" {
t.Fatalf("%+v", got[0])
}
}
+36
View File
@@ -0,0 +1,36 @@
package mdleaves
import (
"fmt"
"strconv"
"strings"
)
func EncodeYAML(leafs []Leaf) string {
if len(leafs) == 0 {
return "[]\n"
}
var b strings.Builder
for _, lf := range leafs {
b.WriteString("-\n")
writeKV(&b, "source", lf.Source)
writeKV(&b, "repo", lf.Repo)
writeKV(&b, "heading", lf.Heading)
writeKV(&b, "text", lf.Text)
writeKV(&b, "type", lf.Type)
writeKV(&b, "status", lf.Status)
writeKV(&b, "related", lf.Related)
}
return b.String()
}
func writeKV(b *strings.Builder, k, v string) {
fmt.Fprintf(b, " %s: %s\n", k, yamlScalar(v))
}
func yamlScalar(s string) string {
if strings.Contains(s, "\n") || s == "" || strings.ContainsAny(s, ":#'\"[]{}&*!|>%@`") || s != strings.TrimSpace(s) {
return strconv.Quote(s)
}
return s
}
+35
View File
@@ -0,0 +1,35 @@
package ocr
import (
"fmt"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Path string
}
func Parser() *flaggy.Parser {
c := CLI{}
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
p := cli.New("mail-ocr")
p.Description = "tesseract eng+deu on image or scanned PDF"
p.AddPositionalValue(&c.Path, "file", 1, false, "image or pdf")
return p
}
func ParseArgs(args []string) (CLI, error) {
var c CLI
if err := cli.Parse(Bind(&c), args); err != nil {
return c, err
}
if c.Path == "" {
return c, fmt.Errorf("usage: bin/mail/ocr.go <image|pdf>")
}
return c, nil
}
+162
View File
@@ -0,0 +1,162 @@
// Package ocr runs Tesseract (eng+deu) on images and scanned PDFs.
//
// Default engine is the tesseract CLI, not gosseract CGO: Ladybug CGO stays
// Zig-only (D21). Same engine, no gocv. OCR_ENGINE=paddle selects paddleocr
// when that binary is on PATH (compose profile ocr-paddle).
package ocr
import (
"fmt"
"image"
"image/color"
"image/png"
"os"
"os/exec"
"path/filepath"
"strings"
)
const TessLang = "eng+deu"
func ImageFile(path string) (string, error) {
engine := os.Getenv("OCR_ENGINE")
if engine == "paddle" {
return runPaddle(path)
}
return runTesseract(path)
}
func PDFFile(path string) (string, error) {
text, err := pdfToText(path)
if err == nil && strings.TrimSpace(text) != "" {
return strings.TrimSpace(text), nil
}
ocr, oerr := pdfPages(path)
if oerr != nil {
if err != nil {
return "", err
}
return "", oerr
}
if strings.TrimSpace(ocr) != "" {
return strings.TrimSpace(ocr), nil
}
if text != "" {
return strings.TrimSpace(text), nil
}
return "", fmt.Errorf("pdf has no text layer (ocr unavailable)")
}
func pdfToText(path string) (string, error) {
cmd := exec.Command("pdftotext", "-layout", path, "-")
out, err := cmd.Output()
if err != nil {
return "", err
}
return string(out), nil
}
func pdfPages(path string) (string, error) {
dir, err := os.MkdirTemp("", "2dph-ocr-")
if err != nil {
return "", err
}
defer os.RemoveAll(dir)
prefix := filepath.Join(dir, "page")
cmd := exec.Command("pdftoppm", "-png", "-r", "200", path, prefix)
if err := cmd.Run(); err != nil {
return "", err
}
matches, err := filepath.Glob(prefix + "*.png")
if err != nil {
return "", err
}
var parts []string
for _, img := range matches {
t, err := ImageFile(img)
if err != nil {
continue
}
if s := strings.TrimSpace(t); s != "" {
parts = append(parts, s)
}
}
return strings.Join(parts, "\n\n"), nil
}
func runTesseract(path string) (string, error) {
pre, err := preprocessFile(path)
if err != nil {
pre = path
} else {
defer os.Remove(pre)
}
cmd := exec.Command("tesseract", pre, "stdout", "-l", TessLang, "--psm", "6")
out, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(out)), nil
}
func runPaddle(path string) (string, error) {
cmd := exec.Command("paddleocr", "ocr", "-i", path)
out, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(out)), nil
}
func preprocessFile(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
img, err := png.Decode(f)
if err != nil {
return "", err
}
out := filepath.Join(os.TempDir(), filepath.Base(path)+".gray.png")
w, err := os.Create(out)
if err != nil {
return "", err
}
defer w.Close()
if err := png.Encode(w, GrayContrast(img)); err != nil {
os.Remove(out)
return "", err
}
return out, nil
}
// GrayContrast is a stdlib preprocess (no gocv): grayscale + stretch.
func GrayContrast(src image.Image) image.Image {
b := src.Bounds()
dst := image.NewGray(b)
var minL, maxL uint8 = 255, 0
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
g := color.GrayModel.Convert(src.At(x, y)).(color.Gray)
if g.Y < minL {
minL = g.Y
}
if g.Y > maxL {
maxL = g.Y
}
}
}
span := int(maxL) - int(minL)
if span < 1 {
span = 1
}
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
g := color.GrayModel.Convert(src.At(x, y)).(color.Gray)
v := uint8((int(g.Y) - int(minL)) * 255 / span)
dst.SetGray(x, y, color.Gray{Y: v})
}
}
return dst
}
+54
View File
@@ -0,0 +1,54 @@
package ocr
import (
"image"
"image/color"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestGrayContrastStretches(t *testing.T) {
img := image.NewGray(image.Rect(0, 0, 2, 2))
img.SetGray(0, 0, color.Gray{Y: 64})
img.SetGray(0, 1, color.Gray{Y: 64})
img.SetGray(1, 0, color.Gray{Y: 64})
img.SetGray(1, 1, color.Gray{Y: 192})
out := GrayContrast(img).(*image.Gray)
if out.GrayAt(0, 0).Y != 0 {
t.Fatalf("min should map to 0, got %d", out.GrayAt(0, 0).Y)
}
if out.GrayAt(1, 1).Y != 255 {
t.Fatalf("max should map to 255, got %d", out.GrayAt(1, 1).Y)
}
}
func TestHelloPNGFixtureOCR(t *testing.T) {
if _, err := exec.LookPath("tesseract"); err != nil {
t.Skip("tesseract not installed")
}
path := filepath.Join("testdata", "hello.png")
got, err := ImageFile(path)
if err != nil {
t.Fatal(err)
}
up := strings.ToUpper(got)
if !strings.Contains(up, "HELLO") {
t.Fatalf("ocr %q missing HELLO", got)
}
}
func TestPaddleEngineUsesPaddleocrBinary(t *testing.T) {
t.Setenv("OCR_ENGINE", "paddle")
_, err := ImageFile(filepath.Join("testdata", "hello.png"))
if _, look := exec.LookPath("paddleocr"); look != nil {
if err == nil {
t.Fatal("expected error when paddleocr is missing")
}
return
}
if err != nil {
t.Fatal(err)
}
}
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 KiB

+47
View File
@@ -0,0 +1,47 @@
package reasoner
import (
"os"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Base string
Model string
Device string
JSONOut bool
}
func Parser() *flaggy.Parser {
c := NewCLI()
return Bind(&c)
}
func NewCLI() CLI {
base := os.Getenv("REASONER_BASE_URL")
if base == "" {
base = "http://127.0.0.1:11435/v1"
}
model := os.Getenv("REASONER_MODEL")
if model == "" {
model = OllamaRAM
}
return CLI{Base: base, Model: model, Device: "cpu"}
}
func Bind(c *CLI) *flaggy.Parser {
p := cli.New("reasoner-bakeoff")
p.Description = "CPU tool-call bake-off"
p.Bool(&c.JSONOut, "", "json", "JSON output")
p.String(&c.Model, "", "model", "Ollama/HF model id")
p.String(&c.Base, "", "base-url", "OpenAI-compatible URL")
p.String(&c.Device, "", "device", "cpu")
return p
}
func ParseArgs(args []string) (CLI, error) {
c := NewCLI()
return c, cli.Parse(Bind(&c), args)
}
+327
View File
@@ -0,0 +1,327 @@
package reasoner
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// HF IDs named in docs. No Qwen3.6-9B exists.
const (
HFQwen35_9B = "Qwen/Qwen3.5-9B"
HFQwen36_27B = "Qwen/Qwen3.6-27B"
HFBonsai27B = "prism-ml/Bonsai-27B-gguf"
OllamaRAM = "qwen3.5:9b"
OllamaQuality = "MichelRosselli/bonsai-27b:Q1_0"
)
type ToolCall struct {
Name string
Arguments string
}
type Result struct {
Model string `json:"model"`
OK bool `json:"ok"`
ToolName string `json:"tool_name,omitempty"`
XMLLeak bool `json:"xml_leak"`
Err string `json:"error,omitempty"`
LatencyMS int64 `json:"latency_ms"`
RSSMB int `json:"rss_mb,omitempty"`
Device string `json:"device"`
WantedTool string `json:"wanted_tool"`
}
type Prompt struct {
Name string
Want string
User string
}
var BakePrompts = []Prompt{
{
Name: "search-before-claim",
Want: "search",
User: "Use tools. Search the 2dph brain for LadybugDB before you answer. Call search.",
},
{
Name: "get-leaf",
Want: "get",
User: "Use tools. Fetch leaf id leaf-demo with get. Do not invent the body.",
},
{
Name: "audit-index",
Want: "audit",
User: "Use tools. Call audit on the brain index health.",
},
}
func MCPTools() []map[string]any {
return []map[string]any{
openaiTool("search", "deduction search (facts → info → web)", map[string]any{
"type": "object",
"properties": map[string]any{
"q": map[string]any{"type": "string", "description": "search query"},
"n": map[string]any{"type": "integer"},
},
"required": []string{"q"},
}),
openaiTool("get", "read one leaf by id", map[string]any{
"type": "object",
"properties": map[string]any{
"id": map[string]any{"type": "string"},
"body": map[string]any{"type": "boolean"},
},
"required": []string{"id"},
}),
openaiTool("audit", "facts confidence histogram", map[string]any{
"type": "object",
"properties": map[string]any{},
}),
}
}
func openaiTool(name, desc string, schema map[string]any) map[string]any {
return map[string]any{
"type": "function",
"function": map[string]any{
"name": name,
"description": desc,
"parameters": schema,
},
}
}
type Client struct {
BaseURL string
Model string
HTTP *http.Client
Device string
ToolChoice string
}
type Report struct {
Model string `json:"model"`
HF string `json:"hf_id"`
Device string `json:"device"`
ToolCallOK int `json:"tool_call_ok"`
ToolCallN int `json:"tool_call_n"`
XMLLeak int `json:"xml_leak"`
RSSMB int `json:"rss_mb"`
VRAMMB int `json:"vram_mb"`
LatencyP50MS float64 `json:"latency_p50_ms,omitempty"`
LatencyP95MS float64 `json:"latency_p95_ms,omitempty"`
Prompts []Result `json:"prompts"`
}
func HFFor(model string) string {
switch model {
case OllamaRAM:
return HFQwen35_9B
case OllamaQuality:
return HFBonsai27B
default:
if strings.Contains(model, "qwen3.6") || strings.Contains(model, "Qwen3.6") {
return HFQwen36_27B
}
return ""
}
}
func (c *Client) httpc() *http.Client {
if c.HTTP == nil {
c.HTTP = &http.Client{Timeout: 10 * time.Minute}
}
return c.HTTP
}
func Origin(base string) string {
s := strings.TrimRight(base, "/")
return strings.TrimSuffix(s, "/v1")
}
func (c Client) ChatTools(user string) (ToolCall, string, error) {
choice := c.ToolChoice
if choice == "" {
choice = "required"
}
body, _ := json.Marshal(map[string]any{
"model": c.Model,
"messages": []map[string]string{
{"role": "system", "content": "You are PicoClaw talking to 2dph MCP. Always call a tool before a factual claim. search then get then audit."},
{"role": "user", "content": user},
},
"tools": MCPTools(),
"tool_choice": choice,
})
base := strings.TrimRight(c.BaseURL, "/")
if !strings.HasSuffix(base, "/v1") {
base += "/v1"
}
url := base + "/chat/completions"
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return ToolCall{}, "", err
}
req.Header.Set("Content-Type", "application/json")
res, err := c.httpc().Do(req)
if err != nil {
return ToolCall{}, "", err
}
defer res.Body.Close()
raw, _ := io.ReadAll(res.Body)
if res.StatusCode >= 300 {
return ToolCall{}, string(raw), fmt.Errorf("http %d", res.StatusCode)
}
return ParseToolResponse(raw)
}
func ParseToolResponse(raw []byte) (ToolCall, string, error) {
var wrap struct {
Choices []struct {
Message struct {
Content string `json:"content"`
ToolCalls []struct {
Function struct {
Name string `json:"name"`
Arguments json.RawMessage `json:"arguments"`
} `json:"function"`
} `json:"tool_calls"`
} `json:"message"`
} `json:"choices"`
}
if err := json.Unmarshal(raw, &wrap); err != nil {
return ToolCall{}, "", err
}
content := ""
if len(wrap.Choices) > 0 {
content = wrap.Choices[0].Message.Content
if n := len(wrap.Choices[0].Message.ToolCalls); n > 0 {
fn := wrap.Choices[0].Message.ToolCalls[0].Function
return ToolCall{Name: fn.Name, Arguments: rawArgs(fn.Arguments)}, content, nil
}
}
return ToolCall{}, content, fmt.Errorf("no tool_calls")
}
func rawArgs(raw json.RawMessage) string {
if len(raw) == 0 {
return ""
}
var s string
if err := json.Unmarshal(raw, &s); err == nil {
return s
}
return string(raw)
}
func XMLLeak(content string) bool {
s := strings.ToLower(content)
return strings.Contains(s, "<tool_call>") ||
strings.Contains(s, "<function=") ||
strings.Contains(s, "<parameter")
}
func RunPrompt(c Client, p Prompt) Result {
start := time.Now()
tc, content, err := c.ChatTools(p.User)
out := Result{
Model: c.Model,
WantedTool: p.Want,
LatencyMS: time.Since(start).Milliseconds(),
Device: c.Device,
XMLLeak: XMLLeak(content),
}
if err != nil {
out.Err = err.Error()
if content != "" && out.XMLLeak {
out.Err = "xml tool call instead of openai tool_calls"
}
return out
}
out.ToolName = tc.Name
out.OK = tc.Name == p.Want
if !out.OK {
out.Err = "wanted " + p.Want + " got " + tc.Name
}
return out
}
type ProcMem struct {
Name string
SizeMB int
VRAMMB int
}
func ParsePS(raw []byte) []ProcMem {
var wrap struct {
Models []struct {
Name string `json:"name"`
Size int64 `json:"size"`
SizeVRAM int64 `json:"size_vram"`
} `json:"models"`
}
if err := json.Unmarshal(raw, &wrap); err != nil {
return nil
}
out := make([]ProcMem, 0, len(wrap.Models))
for _, m := range wrap.Models {
out = append(out, ProcMem{
Name: m.Name,
SizeMB: int(m.Size / (1024 * 1024)),
VRAMMB: int(m.SizeVRAM / (1024 * 1024)),
})
}
return out
}
func (c Client) FetchPS() []ProcMem {
url := Origin(c.BaseURL) + "/api/ps"
res, err := c.httpc().Get(url)
if err != nil {
return nil
}
defer res.Body.Close()
raw, _ := io.ReadAll(res.Body)
if res.StatusCode >= 300 {
return nil
}
return ParsePS(raw)
}
func Run(c Client) Report {
if c.Device == "" {
c.Device = "cpu"
}
rep := Report{
Model: c.Model,
HF: HFFor(c.Model),
Device: c.Device,
}
for _, p := range BakePrompts {
r := RunPrompt(c, p)
rep.Prompts = append(rep.Prompts, r)
rep.ToolCallN++
if r.OK {
rep.ToolCallOK++
}
if r.XMLLeak {
rep.XMLLeak++
}
}
if mems := c.FetchPS(); len(mems) > 0 {
rep.RSSMB = mems[0].SizeMB
rep.VRAMMB = mems[0].VRAMMB
if c.Device == "cpu" && mems[0].VRAMMB > 0 {
rep.Device = "gpu"
}
for i := range rep.Prompts {
rep.Prompts[i].RSSMB = rep.RSSMB
}
}
return rep
}
+166
View File
@@ -0,0 +1,166 @@
package reasoner
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/eSlider/2dph/internal/httpapi"
)
func TestHFIdsAreRealAndNoQwen36Nine(t *testing.T) {
if HFQwen35_9B != "Qwen/Qwen3.5-9B" {
t.Fatalf("9B id = %s", HFQwen35_9B)
}
if HFQwen36_27B != "Qwen/Qwen3.6-27B" {
t.Fatalf("27B id = %s", HFQwen36_27B)
}
if HFBonsai27B != "prism-ml/Bonsai-27B-gguf" {
t.Fatalf("bonsai id = %s", HFBonsai27B)
}
}
func TestParseToolResponseOpenAI(t *testing.T) {
raw := []byte(`{"choices":[{"message":{"tool_calls":[{"function":{"name":"search","arguments":"{\"q\":\"LadybugDB\"}"}}]}}]}`)
tc, _, err := ParseToolResponse(raw)
if err != nil {
t.Fatal(err)
}
if tc.Name != "search" {
t.Fatalf("name=%s", tc.Name)
}
}
func TestParseToolResponseXMLIsFailure(t *testing.T) {
raw := []byte(`{"choices":[{"message":{"content":"<tool_call>search</tool_call>"}}]}`)
_, content, err := ParseToolResponse(raw)
if err == nil {
t.Fatal("expected no tool_calls")
}
if !XMLLeak(content) {
t.Fatal("xml leak not detected")
}
}
func TestBakePromptsWantMCPTools(t *testing.T) {
if len(BakePrompts) != 3 {
t.Fatalf("prompts=%d", len(BakePrompts))
}
names := map[string]bool{}
for _, p := range BakePrompts {
names[p.Want] = true
}
for _, n := range []string{"search", "get", "audit"} {
if !names[n] {
t.Fatalf("missing want %s", n)
}
}
}
func TestParseToolResponseObjectArgs(t *testing.T) {
raw := []byte(`{"choices":[{"message":{"tool_calls":[{"function":{"name":"get","arguments":{"id":"leaf-demo"}}}]}}]}`)
tc, _, err := ParseToolResponse(raw)
if err != nil {
t.Fatal(err)
}
if tc.Name != "get" {
t.Fatalf("name=%s", tc.Name)
}
if !strings.Contains(tc.Arguments, "leaf-demo") {
t.Fatalf("args=%s", tc.Arguments)
}
}
func TestParsePSCPUNotVRAM(t *testing.T) {
raw := []byte(`{"models":[{"name":"qwen3.5:9b","size":6900000000,"size_vram":0}]}`)
got := ParsePS(raw)
if len(got) != 1 {
t.Fatalf("n=%d", len(got))
}
if got[0].VRAMMB != 0 {
t.Fatalf("vram=%d", got[0].VRAMMB)
}
if got[0].SizeMB < 6000 {
t.Fatalf("rss=%d", got[0].SizeMB)
}
}
func TestMCPToolsArePicoClawSubset(t *testing.T) {
mcp := map[string]bool{}
for _, op := range httpapi.Ops {
if op.MCP {
mcp[op.ID] = true
}
}
seen := map[string]bool{}
for _, tool := range MCPTools() {
fn, _ := tool["function"].(map[string]any)
name, _ := fn["name"].(string)
if !mcp[name] {
t.Fatalf("%s is not an MCP op", name)
}
seen[name] = true
}
for _, n := range []string{"search", "get", "audit"} {
if !seen[n] {
t.Fatalf("bake-off missing %s", n)
}
}
}
func TestChatToolsHitsOpenAIPath(t *testing.T) {
var gotTools bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("path=%s", r.URL.Path)
}
body, _ := io.ReadAll(r.Body)
var payload map[string]any
_ = json.Unmarshal(body, &payload)
if _, ok := payload["tools"]; ok {
gotTools = true
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"choices":[{"message":{"tool_calls":[{"function":{"name":"search","arguments":"{\"q\":\"LadybugDB\"}"}}]}}]}`))
}))
defer srv.Close()
c := Client{BaseURL: srv.URL + "/v1", Model: "stub", HTTP: srv.Client(), Device: "cpu"}
r := RunPrompt(c, BakePrompts[0])
if !gotTools {
t.Fatal("tools not sent")
}
if !r.OK {
t.Fatalf("result=%+v", r)
}
}
func TestRunSamplesPSAfterPrompts(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if r.URL.Path == "/api/ps" {
_, _ = w.Write([]byte(`{"models":[{"name":"stub","size":6500000000,"size_vram":0}]}`))
return
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"tool_calls":[{"function":{"name":"search","arguments":"{}"}}]}}]}`))
}))
defer srv.Close()
c := Client{BaseURL: srv.URL + "/v1", Model: "stub", HTTP: srv.Client(), Device: "cpu"}
if mems := c.FetchPS(); len(mems) != 1 || mems[0].VRAMMB != 0 {
t.Fatalf("%+v", mems)
}
if mems := c.FetchPS(); mems[0].SizeMB < 6000 {
t.Fatalf("rss=%d", mems[0].SizeMB)
}
}
func TestHFForKnownOllamaTags(t *testing.T) {
if HFFor(OllamaRAM) != HFQwen35_9B {
t.Fatal(HFFor(OllamaRAM))
}
if HFFor(OllamaQuality) != HFBonsai27B {
t.Fatal(HFFor(OllamaQuality))
}
}
+63
View File
@@ -0,0 +1,63 @@
package websearch
import (
"fmt"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Query, Site, Lang, Fresh, Category, Engines string
Limit int
JSONOut, Refresh, Force bool
TTL float64
Timeout int
}
func NewCLI() CLI {
return CLI{Limit: DefaultLimit, TTL: float64(CacheTTL), Timeout: 25}
}
func Parser() *flaggy.Parser {
c := NewCLI()
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
p := cli.New("web-search")
p.Description = "SearXNG second source (throttled ≠ absence)"
p.Bool(&c.JSONOut, "", "json", "JSON output")
p.Bool(&c.Refresh, "", "refresh", "bypass cache")
p.Bool(&c.Force, "", "force", "allow PII in query")
p.Int(&c.Limit, "n", "limit", "max hits")
p.String(&c.Site, "", "site", "restrict to host")
p.String(&c.Lang, "", "lang", "language")
p.String(&c.Fresh, "", "fresh", "day|week|month|year")
p.String(&c.Category, "", "category", "searx category")
p.String(&c.Engines, "", "engines", "engine list")
p.Float64(&c.TTL, "", "ttl", "cache ttl seconds")
p.Int(&c.Timeout, "", "timeout", "http timeout seconds")
return p
}
func ParseArgs(args []string) (CLI, error) {
c := NewCLI()
p := Bind(&c)
var q string
p.AddPositionalValue(&q, "query", 1, false, "search query")
if err := cli.Parse(p, args); err != nil {
return c, err
}
c.Query = cli.Query(q, p.TrailingArguments)
if c.Query == "" {
return c, fmt.Errorf("query required")
}
if c.Limit < 0 {
return c, fmt.Errorf("--limit must be a non-negative integer")
}
if c.Timeout <= 0 {
return c, fmt.Errorf("--timeout must be a positive integer")
}
return c, nil
}
-1
View File
@@ -6,7 +6,6 @@ readme = "README.md"
requires-python = ">=3.12"
license = { text = "MIT" }
dependencies = [
"docling>=2.119.0",
"ladybug==0.19.1",
"markitdown[docx,epub,html,image-exif,pdf,pptx,xlsx,zip]>=0.1.7",
"mistune==3.3.4",
+257
View File
@@ -0,0 +1,257 @@
#!/usr/bin/env python3
"""System performance test: PicoClaw surface (brain MCP) + optional reasoner.
BRAIN_URL=http://127.0.0.1:8630 ./qa/system_perf.py --json
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \\
./qa/system_perf.py --reasoner --picoclaw --json
Does not write Ladybug. Search includes web (D17); expect ~10s+ per search.
Exit 1 if health/get/audit gates fail. Reasoner is measured, not gated.
"""
from __future__ import annotations
import argparse
import json
import os
import statistics
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor
DEFAULT_BRAIN = "http://127.0.0.1:8630"
DEFAULT_REASONER = "http://127.0.0.1:11435/v1"
DEFAULT_MODEL = "qwen3.5:9b"
DEFAULT_PICOCLAW = "http://127.0.0.1:18790"
GATE_HEALTH_MS = 500
GATE_GET_P50_MS = 50
GATE_AUDIT_P50_MS = 50
def _req(url: str, data: bytes | None = None, timeout: float = 90) -> bytes:
headers = {"Content-Type": "application/json"} if data is not None else {}
req = urllib.request.Request(url, data=data, headers=headers)
with urllib.request.urlopen(req, timeout=timeout) as res:
return res.read()
def timed(fn):
t0 = time.perf_counter()
out = fn()
return (time.perf_counter() - t0) * 1000.0, out
def stats(samples: list[float]) -> dict:
s = sorted(samples)
n = len(s)
return {
"n": n,
"min_ms": round(s[0], 1),
"p50_ms": round(s[n // 2], 1),
"p95_ms": round(s[min(n - 1, int(n * 0.95))], 1),
"max_ms": round(s[-1], 1),
"avg_ms": round(statistics.mean(s), 1),
}
def mcp(brain: str, method: str, params=None, timeout: float = 90) -> dict:
payload: dict = {"jsonrpc": "2.0", "id": 1, "method": method}
if params is not None:
payload["params"] = params
raw = _req(brain.rstrip("/") + "/mcp", json.dumps(payload).encode(), timeout=timeout)
return json.loads(raw.decode())
def mcp_call(brain: str, name: str, arguments: dict, timeout: float = 90) -> tuple[bool, str]:
d = mcp(brain, "tools/call", {"name": name, "arguments": arguments}, timeout=timeout)
res = d.get("result") or {}
text = ((res.get("content") or [{}])[0].get("text") or "")
return (not res.get("isError")), text
def reasoner_tool_call(base: str, model: str, user: str) -> str:
payload = {
"model": model,
"messages": [
{"role": "system", "content": "You are PicoClaw. Always call search before answering."},
{"role": "user", "content": user},
],
"tools": [
{
"type": "function",
"function": {
"name": "search",
"description": "deduction search",
"parameters": {
"type": "object",
"properties": {"q": {"type": "string"}},
"required": ["q"],
},
},
}
],
"tool_choice": "required",
}
raw = _req(
base.rstrip("/") + "/chat/completions",
json.dumps(payload).encode(),
timeout=600,
)
chat = json.loads(raw.decode())
tcs = chat["choices"][0]["message"].get("tool_calls") or []
if not tcs:
return ""
return tcs[0]["function"]["name"]
def run(args: argparse.Namespace) -> dict:
brain = args.brain.rstrip("/")
report: dict = {
"brain": brain,
"device": "cpu",
"ok": True,
"gates": {},
"mcp": {},
}
ms, _ = timed(lambda: _req(brain + "/health", timeout=5))
report["mcp"]["health"] = {"n": 1, "avg_ms": round(ms, 1)}
report["gates"]["health"] = ms <= GATE_HEALTH_MS
if ms > GATE_HEALTH_MS:
report["ok"] = False
list_ms = []
for _ in range(args.n):
ms, d = timed(lambda: mcp(brain, "tools/list", timeout=10))
names = [t["name"] for t in ((d.get("result") or {}).get("tools") or [])]
if "search" not in names:
report["ok"] = False
list_ms.append(ms)
report["mcp"]["tools_list"] = stats(list_ms)
audit_ms = []
for _ in range(args.n):
ms, (ok, _) = timed(lambda: mcp_call(brain, "audit", {}))
if not ok:
report["ok"] = False
audit_ms.append(ms)
report["mcp"]["audit"] = stats(audit_ms)
report["gates"]["audit_p50"] = report["mcp"]["audit"]["p50_ms"] <= GATE_AUDIT_P50_MS
if not report["gates"]["audit_p50"]:
report["ok"] = False
ok, text = mcp_call(brain, "search", {"q": "LadybugDB", "n": 2}, timeout=90)
inner = json.loads(text) if ok else {}
hits = inner.get("results") or []
leaf_id = hits[0]["id"] if hits else ""
report["mcp"]["search_seed"] = {
"ok": ok,
"count": inner.get("count"),
"web": (inner.get("web") or {}).get("status"),
}
get_ms = []
if leaf_id:
for _ in range(args.n):
ms, (ok, _) = timed(lambda: mcp_call(brain, "get", {"id": leaf_id, "body": True}))
if not ok:
report["ok"] = False
get_ms.append(ms)
report["mcp"]["get"] = stats(get_ms)
report["gates"]["get_p50"] = report["mcp"]["get"]["p50_ms"] <= GATE_GET_P50_MS
if not report["gates"]["get_p50"]:
report["ok"] = False
def one_get() -> float:
t0 = time.perf_counter()
mcp_call(brain, "get", {"id": leaf_id, "body": True})
return (time.perf_counter() - t0) * 1000.0
t0 = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as ex:
conc = list(ex.map(lambda _: one_get(), range(8)))
wall = (time.perf_counter() - t0) * 1000.0
report["mcp"]["get_concurrent_8"] = {**stats(conc), "wall_ms": round(wall, 1)}
search_ms = []
for q in ("LadybugDB", "model2vec"):
ms, (ok, text) = timed(lambda q=q: mcp_call(brain, "search", {"q": q, "n": 3}, timeout=90))
inner = json.loads(text) if ok else {}
search_ms.append(ms)
report.setdefault("mcp", {}).setdefault("search_samples", []).append(
{
"q": q,
"ms": round(ms, 1),
"ok": ok,
"count": inner.get("count"),
"web": (inner.get("web") or {}).get("status"),
}
)
if search_ms:
report["mcp"]["search"] = stats(search_ms)
if args.reasoner:
base = args.reasoner_url
model = args.model
report["reasoner"] = {"base_url": base, "model": model, "calls": []}
for user in (
"Use tools. Search the 2dph brain for LadybugDB. Call search.",
"Use tools. Search the 2dph brain for model2vec. Call search.",
):
ms, name = timed(lambda user=user: reasoner_tool_call(base, model, user))
report["reasoner"]["calls"].append({"ms": round(ms, 1), "tool": name})
tools = [c["tool"] for c in report["reasoner"]["calls"]]
report["gates"]["reasoner_tool_call"] = bool(tools) and all(t == "search" for t in tools)
if not report["gates"]["reasoner_tool_call"]:
report["ok"] = False
if args.picoclaw:
gw = args.picoclaw_url.rstrip("/")
ms, raw = timed(lambda: _req(gw + "/health", timeout=5))
body = json.loads(raw.decode())
report["picoclaw"] = {
"url": gw,
"health_ms": round(ms, 1),
"status": body.get("status"),
}
report["gates"]["picoclaw_health"] = body.get("status") == "ok" and ms <= GATE_HEALTH_MS
if not report["gates"]["picoclaw_health"]:
report["ok"] = False
return report
def main(argv: list[str]) -> int:
p = argparse.ArgumentParser(description="2dph system performance (MCP + optional reasoner)")
p.add_argument("--brain", default=os.environ.get("BRAIN_URL", DEFAULT_BRAIN))
p.add_argument("--n", type=int, default=20)
p.add_argument("--json", action="store_true")
p.add_argument("--reasoner", action="store_true")
p.add_argument("--picoclaw", action="store_true")
p.add_argument("--picoclaw-url", default=os.environ.get("PICOCLAW_URL", DEFAULT_PICOCLAW))
p.add_argument("--reasoner-url", default=os.environ.get("REASONER_BASE_URL", DEFAULT_REASONER))
p.add_argument("--model", default=os.environ.get("REASONER_MODEL", DEFAULT_MODEL))
args = p.parse_args(argv)
try:
report = run(args)
except (urllib.error.URLError, TimeoutError, OSError) as e:
print(f"system_perf: {e}", file=sys.stderr)
return 1
if args.json:
print(json.dumps(report, indent=2))
else:
print(f"ok={report['ok']} brain={report['brain']}")
for name, block in report.get("mcp", {}).items():
if isinstance(block, dict) and "p50_ms" in block:
print(f" {name}: p50={block['p50_ms']} p95={block['p95_ms']} n={block['n']}")
elif name == "health":
print(f" health: {block.get('avg_ms')} ms")
for k, v in report.get("gates", {}).items():
print(f" gate {k}: {v}")
for c in (report.get("reasoner") or {}).get("calls") or []:
print(f" reasoner {c['tool']}: {c['ms']} ms")
return 0 if report["ok"] else 1
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
+5 -2
View File
@@ -26,13 +26,14 @@ second independent source when local roots cannot confirm. An answer is
bin/brain/search.go "Matrix federation" # pointers + snippets, YAML
bin/brain/search.go "onlyoffice postgres" --root facts # restrict to confirmed
bin/brain/search.go "where is cs-lexicon" --json | yq '.[].ref'
bin/brain/add.go --text T --root facts --source "a.md x b.md"
bin/brain/get.go <id> --body # full chunk only when needed
bin/brain/stats.go # index health
bin/brain/eval.go # recall@5 >= 0.95 gate (Go; Python bin/kb/eval is CI fallback)
```
`bin/kb/search` is a deprecated wrapper. `--hop` errors (File/FROM_FILE edges
are not wired yet); do not treat it as a graph walk.
`bin/kb/search` is a deprecated wrapper. `--hop N` walks
`FROM_FILE` / `HAS_VERSION` / `AUTHORED` from each hit (1=File, 3=Person).
## Rules
@@ -44,6 +45,8 @@ are not wired yet); do not treat it as a graph walk.
are not evidence of absence. `--root facts|info` and `--no-web` skip the web.
- If recall looks wrong, run `bin/brain/eval.go`; it gates control questions and
should stay at or above 95% recall@5.
- Contradictions (≥2 yes vs ≥2 no) stay `(not confirmed)` until
`bin/facts/audit contradict` fires `temporal_freshness` or `authority_pairing`.
- Agents: `GET /openapi.json` and `POST /mcp` on `bin/brain/serve.go` (same
handlers; tool names match paths `search`/`get`/`stats`/`audit`). Generated
list: [tools.md](tools.md).
+1 -1
View File
@@ -8,4 +8,4 @@ Serve: `bin/brain/serve.go` (`GET /openapi.json`, `POST /mcp`).
- `get` — read one leaf by id
- `stats` — index health
- `audit` — facts confidence histogram
- `ingest`rebuild hint (write is v2)
- `ingest`add a leaf without rebuild
+38
View File
@@ -0,0 +1,38 @@
---
name: duckdb
description: >-
Use https://github.com/duckdb/duckdb-go in-process for columnar analytics
(quantiles, GROUP BY, JSON/CSV/Parquet/JSONL scans) when that is faster than
nested Go loops. Not Ladybug. Not the web-search sqlite cache. Use when
aggregating samples, counting JSONL, or SQL over tabular files.
---
# duckdb-go
Use https://github.com/duckdb/duckdb-go where it makes sense to get better performance in code.
In-process DuckDB (`internal/duckstats`, `database/sql` driver `duckdb`).
Vectorized SQL over tables, JSONL, CSV, Parquet. CGO with bundled libs
(linux/darwin amd64/arm64). Links with **gcc/g++** (libstdc++), not Zig.
D21 Zig (`bin/cgo/zcc`) is Ladybug/tokenizers only. After
`eval "$(bin/cgo/zig env)"`:
```bash
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= ./bin/qa/stats.go <<< '[1,2,3,4,5]'
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go test ./internal/duckstats
```
| Store | Job |
|-------|-----|
| Ladybug | graph + FTS + HNSW (facts/info) |
| modernc sqlite | web-search KV cache + throttle |
| duckdb-go | OLAP: quantiles, counts, scans of many rows/files |
| mikefarah/yq | small YAML/JSON/XML/CSV/TOML/HCL slice, not bulk |
```bash
./bin/qa/stats.go <<< '[1,2,3,4,5]'
./bin/qa/stats.go --jsonl path/to/rows.jsonl
```
Do not open Ladybug through DuckDB. Do not put secrets or client PII into
DuckDB files under the repo.

Some files were not shown because too many files have changed in this diff Show More