Compare commits

..
14 Commits
Author SHA1 Message Date
eSliderandGitHub fc2723c39f feat: D24 fact intervals (--as-of) and bin/stack assistant helpers. (#37)
Tests / Test (push) Skipped
Tests / OCR (tesseract fixture) (push) Skipped
Tests / Release (semver) (push) Skipped
Store valid_from/valid_to on leafs and filter search by calendar day without
overloading D16 source staleness; stack start/start-assistant wires brain + PicoClaw.
2026-08-14 15:48:26 +01:00
eSliderandGitHub 0c4bb87001 feat: parse all Go CLIs with flaggy; dump bash complete. (#36)
Tests / Test (push) Skipped
Tests / OCR (tesseract fixture) (push) Skipped
Tests / Release (semver) (push) Skipped
stdlib flag dropped --hop after the query. One wrapper in internal/cli,
source <(./bin/cli/complete.go bash). Gitea #34.
2026-08-14 12:12:38 +01:00
eSliderandGitHub 0065655e03 feat: D16 adjudication — 2v2 stays hypothesis until a rule fires. (#35)
Tests / Test (push) Skipped
Tests / OCR (tesseract fixture) (push) Skipped
Tests / Release (semver) (push) Skipped
temporal_freshness then authority_pairing; unresolved keeps (not confirmed).
bin/facts/audit contradict. Gitea #29.
2026-08-14 11:43:00 +01:00
eSliderandGitHub a331042488 feat: DuckDB quantiles in-process (gcc CGO), not Ladybug. (#34)
Tests / Test (push) Failing after 6s
Tests / OCR (tesseract fixture) (push) Failing after 4s
Tests / Release (semver) (push) Skipped
OQ3: internal/duckstats + bin/qa/stats.go. Zig stays Ladybug-only.
mikefarah/yq for small structured slices. Gitea #30.
2026-08-14 11:32:23 +01:00
eSliderandGitHub f99dfea104 feat: OCR scans with tesseract, drop docling from the default path. (#33)
Tests / Test (push) Failing after 5s
Tests / OCR (tesseract fixture) (push) Failing after 4s
Tests / Release (semver) (push) Skipped
pdftotext still wins on born-digital PDFs. Empty text layers go through
pdftoppm + tesseract eng+deu. Optional OCR_ENGINE=paddle. Gitea #6.
2026-08-14 11:26:04 +01:00
eSliderandGitHub bae1494258 ci: recall@5 SoT is Zig bin/brain/eval.go (#19) (#32)
Tests / Test (push) Failing after 6s
Tests / Release (semver) (push) Skipped
* ci: recall@5 SoT is Zig bin/brain/eval.go.

GitHub Actions rebuilds the repo corpus and runs the Go eval gate
instead of Python bin/kb/eval. Audit self is a hard fail (Gitea #19).

* ci: point eval at the checkout and keep DevOps in the corpus.

/tmp/brain-eval cannot walk up to var/kb.lbug; KB_ROOT is required.
Recall fragments must exist in README/docs so a repo-only rebuild gates.
2026-08-14 11:03:39 +01:00
eSliderandGitHub c5be3f19be feat: index facts extract and chats markdown on rebuild. (#31)
Tests / Test (push) Failing after 5s
Tests / Release (semver) (push) Skipped
--with-facts / --facts-json write root=facts leafs; --with-chats
picks up var/chats/md as info. WhatsApp sync stays out of v1 (Gitea #18).
2026-08-14 10:57:40 +01:00
eSliderandGitHub a88dbb490c feat: search --hop walks FROM_FILE to Person. (#30)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
Parser no longer errors; hop 1 returns File, hop 3 reaches Person.
Rebuild writes Leaf-[:FROM_FILE]->File so the walk is not empty
on a fresh index (Gitea #17).
2026-08-14 10:51:35 +01:00
eSliderandGitHub 9d1a3f3c70 feat: write leafs incrementally without rebuilding the graph. (#29)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
Ladybug 0.19 stays FTS/HNSW queryable on MERGE of new ids; DROP INDEX
was the fatal path. bin/brain/add.go and POST /ingest land facts+info
in one transaction so watch/mail/git can become leafs now (Gitea #14).
2026-08-14 10:46:59 +01:00
eSliderandGitHub 8b5be9b659 docs: record v1 detective epic and remaining gaps. (#28)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
Read path and MCP are in; write, hops, corpus, and CI eval are still open. Board is Gitea epic #16.
2026-08-14 10:32:19 +01:00
eSliderandGitHub 1edb158f35 docs: portable runbook and Diataxis index (#27)
Tests / Test (push) Failing after 5s
Tests / Release (semver) (push) Skipped
Public face is a product, not a laptop tool. Run steps live in
docs/runbook.md; decisions D3/D6/D14/D15/D17/D18 stay in the docs index.
2026-08-13 23:31:41 +01:00
eSliderandGitHub 85caff90b9 feat: markdown import splits H2 leafs in Go (#26)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Drop the Python ExecFile wrapper. Conversion still does not write
Ladybug; add a dry-run fixture check for the index adapter split.
2026-08-13 23:27:42 +01:00
eSliderandGitHub b317968a4c feat: CPU reasoner bake-off for Qwen3.5-9B vs Bonsai (#25)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
Measure OpenAI tool_calls (search/get/audit) on a CPU Ollama sidecar
instead of gating D18 on GPU or PicoClaw. Weights stay out of the image.
2026-08-13 23:19:51 +01:00
eSliderandGitHub 1f9bdb0bf6 feat: compile ladybug CGO with Zig, not gcc (#24)
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
API image is Go-only (compose target api). Python rebuild is profile
index. bin/cgo/zig pins Zig 0.14.1, liblbug, and tokenizers.
2026-08-13 21:54:31 +01:00
116 changed files with 6344 additions and 2368 deletions
+1
View File
@@ -3,6 +3,7 @@
var var
.git .git
.github .github
lib-ladybug
__pycache__ __pycache__
*.pyc *.pyc
*.lbug *.lbug
+38 -8
View File
@@ -37,31 +37,61 @@ jobs:
bash -n bin/db/ssh-tunnel bash -n bin/db/ssh-tunnel
bash -n bin/docker-entrypoint bash -n bin/docker-entrypoint
bash -n bin/kb/search bash -n bin/kb/search
bash -n bin/cgo/zig
sh -n bin/cgo/zcc
sh -n bin/cgo/zc++
- name: Python unit tests (offline, vendored tools) - name: Python unit tests (offline, vendored tools)
run: | run: |
uv run python -m unittest discover -s bin/tools -t . uv run python -m unittest discover -s bin/tools -t .
- name: Go tests (root module, no ladybug cgo) - name: Go tests (root module; duckdb-go CGO via gcc, no ladybug)
run: | run: |
go vet ./... CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go vet ./...
go test ./... -count=1 CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go test ./... -count=1
- name: brain ranking tests (no cgo / no ladybug) - name: brain ranking tests (no cgo / no ladybug)
run: go test ./internal/brain/rank -count=1 run: go test ./internal/brain/rank -count=1
- name: facts/audit self (lexicon consistency, no network) - name: facts/audit self (lexicon consistency, no network)
run: | run: ./bin/facts/audit self
./bin/facts/audit self 2>/dev/null || echo "audit: not yet implemented; gate skipped"
- name: kb/eval recall gate - name: CGO via Zig (compile brain/search + eval)
run: | run: |
./bin/kb/eval 2>/dev/null || echo "eval: not yet implemented; gate skipped" chmod +x bin/cgo/zig bin/cgo/zcc bin/cgo/zc++
bin/cgo/zig go build -tags system_ladybug -o /tmp/brain-search ./bin/brain/search.go
bin/cgo/zig go build -tags 'system_ladybug,brain_eval' -o /tmp/brain-eval ./bin/brain/eval.go
- uses: actions/cache@v4
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-hf-potion-multilingual-128M
- name: recall@5 SoT (Zig bin/brain/eval.go)
run: |
uv run python bin/kb/index --rebuild --json
KB_ROOT="$PWD" /tmp/brain-eval --json
ocr:
name: OCR (tesseract fixture)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Install tesseract + poppler
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu poppler-utils
- name: Go OCR tests (synthetic HELLO PNG)
run: go test ./internal/ocr -count=1
release: release:
name: Release (semver) name: Release (semver)
if: github.event_name == 'push' && github.ref == 'refs/heads/main' if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: test needs: [test, ocr]
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
contents: write contents: write
+3
View File
@@ -11,3 +11,6 @@ __pycache__/
.secrets/ .secrets/
lib-ladybug/ lib-ladybug/
go.work.local go.work.local
models/
# Purged from git history. Do not re-add.
docs/crm-associations-proof.md
+37 -17
View File
@@ -3,7 +3,8 @@
Evidence-first brain over the ops/eSlider stack. Facts need proof or they are Evidence-first brain over the ops/eSlider stack. Facts need proof or they are
`(not confirmed)`. `(not confirmed)`.
Read first: [PLAN](PLAN.md) → [docs](docs/). Read first: [PLAN](PLAN.md) → [docs](docs/) → [roadmap](docs/roadmap.md)
(epic [#16](https://git.produktor.io/eSlider/2dph/issues/16)).
## Method (detective, no fork) ## Method (detective, no fork)
@@ -39,19 +40,23 @@ PLAN.md decisions + execution + open questions
docs/ published docs docs/ published docs
skills/ in-project agent skills (vendored, no external links) skills/ in-project agent skills (vendored, no external links)
bin/ self-describing tools bin/{subject}/{method}.go (shebang) bin/ self-describing tools bin/{subject}/{method}.go (shebang)
bin/brain/ search.go serve.go index.go get.go stats.go eval.go watch.go bin/brain/ search.go serve.go index.go add.go get.go stats.go eval.go watch.go
bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats
bin/mail/ sync.go import.go (index_mail → brain/index.go) bin/mail/ sync.go import.go ocr.go (index_mail → brain/index.go)
bin/markdown/ import.go (mistune leafs) bin/markdown/ import.go (H2 leaf split; Python bin/md/import fallback)
bin/postgres/ query.go (read-only YAML) bin/postgres/ query.go (read-only YAML)
bin/git/ import.go (go-git history; Python shim execs it) bin/git/ import.go (go-git history; Python shim execs it)
bin/web/ search.go (SearXNG; Python shim execs it) bin/web/ search.go (SearXNG; Python shim execs it)
internal/ shared Go (brain/rank is cgo-free; chats parsers; gitlog; websearch) bin/reasoner/ bakeoff.go (D18 CPU OpenAI tool-call bake-off)
internal/ shared Go (brain/rank is cgo-free; facts D16; cli flaggy D23; chats; gitlog; websearch; reasoner; duckstats)
bin/qa/ stats.go (DuckDB quantiles / JSONL count; gcc CGO, not Zig)
bin/watch/ corpus watcher (used by bin/brain/watch.go) bin/watch/ corpus watcher (used by bin/brain/watch.go)
bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch) bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch)
bin/docker-entrypoint container entrypoint (brain index|search|serve|watch) bin/cgo/ zig zcc zc++ (CGO via zig cc, not gcc)
bin/stack/ start start-assistant stop status (compose + PicoClaw agent)
bin/docker-entrypoint container entrypoint (api: serve|search|watch; index: python)
compose.yaml docker composition (root level, not docker/) compose.yaml docker composition (root level, not docker/)
Dockerfile multi-stage: python deps + static Go binaries Dockerfile api (Zig CGO, no Python) + index (Python write)
var/ kb.lbug, var/mail/*, caches (gitignored) var/ kb.lbug, var/mail/*, caches (gitignored)
.venv/ ladybug + model2vec + mistune .venv/ ladybug + model2vec + mistune
``` ```
@@ -62,43 +67,58 @@ var/ kb.lbug, var/mail/*, caches (gitignored)
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments
bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox) bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox)
bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only) bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only)
bin/brain/index.go --rebuild # rebuild brain incl. all mail (fresh DB) bin/brain/index.go --rebuild --with-facts --with-chats
``` ```
- `sync` (Go) downloads messages + attachments; Gmail uses paginated list + - `sync` (Go) downloads messages + attachments; Gmail uses paginated list +
`body.attachmentId` (not partId) for attachments. `body.attachmentId` (not partId) for attachments.
- `import` converts body + attachments to markdown. PDFs use poppler - `import` converts body + attachments to markdown. PDFs use poppler
`pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to `pdftotext -layout` fast path (~15ms); textless/scanned PDFs use
docling (isolated subprocess — its native onnx can segfault the parent). `pdftoppm` + tesseract `eng+deu` (`bin/mail/ocr.go`). Optional
Conversion never touches the brain DB (crash safety). `OCR_ENGINE=paddle`. Conversion never touches the brain DB (crash safety).
- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Ladybug - `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Bulk
corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector rebuild still deletes `var/kb.lbug` and creates FTS/HNSW last. Single-leaf
indexes exist; a fresh DB with indexes created last is the only safe path. write is `bin/brain/add.go` (safe while indexes exist; do not DROP INDEX).
Keep conversion + indexing separate so a conversion crash can't leave the Keep conversion + indexing separate so a conversion crash can't leave the
DB mid-transaction. DB mid-transaction.
## Tools ## Tools
```bash ```bash
bin/facts/audit.go ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate bin/facts/audit.go ["self"|"db"|"contradict"] # 2-source + D16 adjudication
bin/facts/crm.go [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT) bin/facts/crm.go [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT)
bin/kb/search "query" [--repo X] # deprecated wrapper → bin/brain/search.go bin/kb/search "query" [--repo X] # deprecated wrapper → bin/brain/search.go
bin/brain/search.go "query" [--root facts|info] # deduction search → YAML bin/brain/search.go "query" [--root facts|info] # deduction search → YAML
bin/brain/search.go "query" --as-of 2025-01-01 # D24 fact intervals
bin/brain/search.go "query" --no-web # local graph only bin/brain/search.go "query" --no-web # local graph only
source <(./bin/cli/complete.go bash) # flaggy completions (D23)
eval "$(bin/cgo/zig env)" # Zig cc + liblbug (not gcc)
bin/brain/index.go --rebuild [--with-mail] [--with-facts] [--with-chats]
bin/brain/add.go --text T --root facts --source "a.md x b.md" # incremental write
bin/brain/add.go --json # stdin leaf or {leafs:[...]}
bin/brain/get.go <id> [--body] [--json] # Go read; Python bin/kb/get CI fallback bin/brain/get.go <id> [--body] [--json] # Go read; Python bin/kb/get CI fallback
bin/brain/stats.go [--json] bin/brain/stats.go [--json]
bin/brain/eval.go [--json] # recall@5; questions in internal/brain/rank bin/brain/eval.go [--json] # recall@5; questions in internal/brain/rank
bin/brain/serve.go # HTTP :8630; GET /openapi.json POST /mcp bin/brain/serve.go # HTTP :8630; GET /openapi.json POST /mcp
bin/markdown/import.go [dir] # mistune leaves → YAML bin/stack/start # brain HTTP/MCP (reuse healthy :8630)
bin/stack/start-assistant # + reasoner + PicoClaw agent
bin/stack/status # YAML health
bin/stack/stop # compose stop; volumes kept
bin/markdown/import.go [dir] # H2 leafs → YAML; Python bin/md/import fallback
bin/git/import.go [REPO] [--json] [--limit N] # go-git history → commit leafs bin/git/import.go [REPO] [--json] [--limit N] # go-git history → commit leafs
bin/web/search.go "query" [--json] # SearXNG; throttled ≠ absence bin/web/search.go "query" [--json] # SearXNG; throttled ≠ absence
bin/reasoner/bakeoff.go [--model ID] [--json] # D18 CPU tool-call bake-off
bin/postgres/query.go --profile onlyoffice -c 'SELECT 1' bin/postgres/query.go --profile onlyoffice -c 'SELECT 1'
bin/qa/stats.go # D22 DuckDB quantiles / JSONL (gcc CGO)
bin/mail/ocr.go <image|pdf> # tesseract eng+deu (scans)
bin/md/tables # what the graph holds → YAML bin/md/tables # what the graph holds → YAML
bin/brain/deduce "question" # thinking wrapper bin/brain/deduce "question" # thinking wrapper
``` ```
Never start a shell command with `cd` — use the tool working-directory Never start a shell command with `cd` — use the tool working-directory
parameter. Search before reading whole files. parameter. Search before reading whole files. For YAML/JSON/XML/CSV/TOML/HCL
prefer mikefarah/yq (`skills/yq/SKILL.md`). For bulk rows and quantiles use
duckdb-go (`internal/duckstats`, `skills/duckdb/SKILL.md`), not Ladybug.
## GitHub safety rules (ABSOLUTE — never violate) ## GitHub safety rules (ABSOLUTE — never violate)
+60 -16
View File
@@ -1,5 +1,13 @@
# syntax=docker/dockerfile:1 # syntax=docker/dockerfile:1
FROM python:3.12-slim AS base #
# docker build --target api -t 2dph:api .
# docker build --target index -t 2dph:index .
#
# API: Go + ladybug via Zig CGO (no CPython).
# Index: Python write path (profile `index` until brain/add is v2).
# --- Python sidecar (Ladybug write / rebuild) ---
FROM python:3.12-slim AS index
ENV PYTHONUNBUFFERED=1 \ ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \ PYTHONDONTWRITEBYTECODE=1 \
@@ -8,26 +16,16 @@ ENV PYTHONUNBUFFERED=1 \
WORKDIR /app WORKDIR /app
RUN id -u 2dph 2>/dev/null || useradd --create-home --uid 1001 2dph RUN id -u 2dph 2>/dev/null || useradd --create-home --uid 1001 2dph
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
poppler-utils tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu \
&& rm -rf /var/lib/apt/lists/*
# deps layer-first: rebuild only on dependency change
COPY requirements.lock.txt /tmp/requirements.lock.txt COPY requirements.lock.txt /tmp/requirements.lock.txt
RUN python -m pip install --no-cache-dir -r /tmp/requirements.lock.txt \ RUN python -m pip install --no-cache-dir -r /tmp/requirements.lock.txt \
&& rm /tmp/requirements.lock.txt && rm /tmp/requirements.lock.txt
# Go services: static binaries, no interpreter at runtime
FROM golang:1.25 AS go-build
WORKDIR /src
COPY go.mod ./
COPY bin/server ./bin/server
COPY bin/watch ./bin/watch
RUN CGO_ENABLED=0 go build -o /serve ./bin/server \
&& CGO_ENABLED=0 go build -o /watch ./bin/watch
# runtime: python toolchain + Go services
FROM base
COPY . . COPY . .
COPY --from=go-build /serve /app/bin/serve
COPY --from=go-build /watch /app/bin/watch
RUN chmod +x /app/bin/docker-entrypoint \ RUN chmod +x /app/bin/docker-entrypoint \
&& chown -R 2dph:2dph /app && chown -R 2dph:2dph /app
USER 2dph USER 2dph
@@ -37,5 +35,51 @@ ENV PATH="/app/bin:${PATH}" \
KB_ROOT=/app KB_ROOT=/app
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import model2vec, ladybug, mistune; print('ok')" || exit 1 CMD python -c "import model2vec, ladybug, mistune; print('ok')" || exit 1
ENTRYPOINT ["/app/bin/docker-entrypoint"] ENTRYPOINT ["/app/bin/docker-entrypoint"]
# --- Go API: CGO with Zig, not gcc ---
FROM golang:1.26-bookworm AS api-build
WORKDIR /src
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl xz-utils ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY bin/cgo ./bin/cgo
RUN chmod +x bin/cgo/zig bin/cgo/zcc bin/cgo/zc++ \
&& ./bin/cgo/zig env >/dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ENV CGO_RPATH=/usr/local/lib
RUN eval "$(./bin/cgo/zig env)" \
&& go build -tags brain_serve,system_ladybug -o /out/brain-serve ./bin/brain/serve.go \
&& go build -tags system_ladybug -o /out/brain-search ./bin/brain/search.go \
&& CGO_ENABLED=0 go build -tags brain_watch -o /out/brain-watch ./bin/brain/watch.go
FROM debian:bookworm-slim AS api
RUN apt-get update \
&& apt-get install -y --no-install-recommends libssl3 ca-certificates wget \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --uid 1001 2dph
COPY --from=api-build /out/brain-serve /usr/local/bin/brain-serve
COPY --from=api-build /out/brain-search /usr/local/bin/brain-search
COPY --from=api-build /out/brain-watch /usr/local/bin/brain-watch
COPY --from=api-build /src/lib-ladybug/liblbug.so.0.19.1 /usr/local/lib/liblbug.so.0.19.1
COPY bin/docker-entrypoint /usr/local/bin/docker-entrypoint
RUN chmod +x /usr/local/bin/docker-entrypoint \
&& ln -s liblbug.so.0.19.1 /usr/local/lib/liblbug.so.0 \
&& ln -s liblbug.so.0 /usr/local/lib/liblbug.so \
&& ldconfig
USER 2dph
ENV KB_ROOT=/data \
KB_PORT=8630 \
LD_LIBRARY_PATH=/usr/local/lib \
HF_HOME=/data/hf
WORKDIR /data
EXPOSE 8630
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:8630/health || exit 1
ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
CMD ["serve"]
+68 -24
View File
@@ -4,7 +4,14 @@ A brain that loves facts and deduction. Evidence-first knowledge graph + hybrid
RAG over the operational Brain/ops/eSlider stack. Built like Sherlock RAG over the operational Brain/ops/eSlider stack. Built like Sherlock
Holmes: nothing is asserted unless it has proof. Holmes: nothing is asserted unless it has proof.
Status: **in progress** — this file is the plan and the record of decisions. Status: **v1 in** (epic [#16](https://git.produktor.io/eSlider/2dph/issues/16) closed).
v2 board: milestone [v2](https://git.produktor.io/eSlider/2dph/milestone/13) —
OCR [#6](https://git.produktor.io/eSlider/2dph/issues/6) in,
[#29](https://git.produktor.io/eSlider/2dph/issues/29) OQ1 in,
[#30](https://git.produktor.io/eSlider/2dph/issues/30) OQ3 in,
[#34](https://git.produktor.io/eSlider/2dph/issues/34) D23 in,
[#36](https://git.produktor.io/eSlider/2dph/issues/36) OQ5/D24 in.
Gap: [docs/roadmap.md](docs/roadmap.md).
## What ## What
@@ -29,9 +36,9 @@ detective method: **a fact needs ≥2 independent sources or it is
| D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. | | D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. |
| D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. | | D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. |
| D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). | | D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). |
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path (`get.go` / `stats.go` / `eval.go`) is Go + cgo. Python `bin/kb/{get,stats,eval}` is the CI fallback (GitHub runners have no ladybug cgo). Index/write stays Python until the Go write path is safe. | | D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Incremental write is Python `bin/kb/add` (`bin/brain/add.go`). Bulk rebuild stays `compose --profile index` until the Go write path is safe. |
| D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). | | D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). |
| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. | | D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. 2-source auto-pair docker ps × compose × ssh-config × docs. |
| D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. | | D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. |
| D10 | versioning | everything is a leaf with `sha256 + observed_at + source_rev`; `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`. Stale = `source_rev` < git HEAD. | | D10 | versioning | everything is a leaf with `sha256 + observed_at + source_rev`; `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`. Stale = `source_rev` < git HEAD. |
| D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. | | D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. |
@@ -39,11 +46,15 @@ detective method: **a fact needs ≥2 independent sources or it is
| D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. | | D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. |
| D14 | tooling style | `bin/{subject}/{method}.go` shebang (e.g. `bin/brain/search.go`). Shared code in `internal/`. One root `go.mod` + `go.work`. No `bin/*/main.go`, no nested modules. | | D14 | tooling style | `bin/{subject}/{method}.go` shebang (e.g. `bin/brain/search.go`). Shared code in `internal/`. One root `go.mod` + `go.work`. No `bin/*/main.go`, no nested modules. |
| D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. | | D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. |
| D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. | | D16 | contradictions | ≥2 yes vs ≥2 no → hypothesis → `(not confirmed)` until a rule fires. Order: **temporal_freshness** (fresh ≥2 vs stale minority), then **authority_pairing** (runtime/config A×B beats narrative C). Store as `a x b vs c x d` on hypothesis leafs. `bin/facts/audit contradict`. [#29](https://git.produktor.io/eSlider/2dph/issues/29). |
| D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. | | D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. |
| D18 | reasoner | Pluggable OpenAI-compatible URL. RAM: Qwen3.5-9B. Quality: Bonsai-27B or Qwen3.6-27B. No official Qwen3.6-9B. | | D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is compose profile `picoclaw`; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. Agent lever/loop: [#15](https://git.produktor.io/eSlider/2dph/issues/15). |
| D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. | | D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. |
| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). | | D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`/`ingest`). |
| D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc``zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. |
| D22 | analytics | **duckdb-go** in-process (`internal/duckstats`, `bin/qa/stats.go`) for quantiles/JSONL. Links with **gcc/g++**, not Zig. Ladybug stays the graph; web-search cache stays modernc sqlite. Slice small structured docs with **mikefarah/yq**, not kislyuk/jq. [#30](https://git.produktor.io/eSlider/2dph/issues/30). |
| D23 | CLI | **flaggy** (`github.com/integrii/flaggy`, 0 deps). Flags at any position. Wrapper `internal/cli`. Bash complete: `source <(./bin/cli/complete.go bash)`. No cobra, no stdlib `flag` in Go tools. Search does not intercept the word `completion`. [#34](https://git.produktor.io/eSlider/2dph/issues/34). |
| D24 | fact intervals | Leaf `valid_from` / `valid_to` (YYYY-MM-DD, inclusive; empty = open/legacy). Search `--as-of` / MCP `as_of` keeps facts active that day. Not D16 `temporal_freshness` (source stale vs HEAD). Empty interval = always visible. [#36](https://git.produktor.io/eSlider/2dph/issues/36). |
## Architecture ## Architecture
@@ -54,21 +65,28 @@ detective method: **a fact needs ≥2 independent sources or it is
skills/ in-project skills (web-search, postgres, brain, picoclaw, diataxis-docs) skills/ in-project skills (web-search, postgres, brain, picoclaw, diataxis-docs)
bin/ bin/
facts/extract.go audit.go crm.go # D14 shebang; Python implementation facts/extract.go audit.go crm.go # D14 shebang; Python implementation
kb/index Python write path (called by bin/brain/index.go) kb/index Python bulk write (called by bin/brain/index.go)
kb/add Python incremental write (called by bin/brain/add.go)
brain/index.go rebuild FTS + HNSW (incl. --with-mail) brain/index.go rebuild FTS + HNSW (incl. --with-mail)
brain/add.go incremental leaf write (no rebuild)
brain/get.go stats.go eval.go # Go read (cgo); Python bin/kb/* CI fallback brain/get.go stats.go eval.go # Go read (cgo); Python bin/kb/* CI fallback
brain/watch.go brain/watch.go
brain/search.go deduction: facts → info → web-search brain/search.go deduction: facts → info → web
brain/serve.go HTTP API in-process + OpenAPI/MCP (D20); compose profile picoclaw cli/complete.go flaggy bash/zsh/fish complete (D23)
brain/serve.go HTTP API in-process + OpenAPI/MCP (D20); Zig CGO (D21)
cgo/zig zcc zc++ CGO toolchain (zig cc, not gcc)
mail/import.go JSON → markdown (no brain write) mail/import.go JSON → markdown (no brain write)
markdown/import.go mistune leaves markdown/import.go H2 leaf split (Go); Python bin/md/import fallback
postgres/query.go read-only YAML (wraps bin/db/psql-yq) postgres/query.go read-only YAML (wraps bin/db/psql-yq)
git/import.go go-git history (no git binary; conversion only) git/import.go go-git history (no git binary; conversion only)
web/search.go SearXNG client (throttled ≠ absence) web/search.go SearXNG client (throttled ≠ absence)
reasoner/bakeoff.go CPU tool-call bake-off (D18; OpenAI tools)
chats/sync.go import.go facts.go apply.go chats/sync.go import.go facts.go apply.go
(libs in internal/chats; no chats index) (libs in internal/chats; no chats index)
mail/ocr.go tesseract eng+deu (pdftoppm scans)
md/import (deprecated; bin/markdown/import.go) md/import (deprecated; bin/markdown/import.go)
brain/extract brain/audit brain/deduce (thinking wrapper) brain/extract brain/audit brain/deduce (thinking wrapper)
stack/start start-assistant stop status
web/search (deprecated shim → web/search.go) web/search (deprecated shim → web/search.go)
db/psql-yq (vendored) db/psql-yq (vendored)
ssh-tunnel onlyoffice pg tunnel 5433 ssh-tunnel onlyoffice pg tunnel 5433
@@ -80,10 +98,14 @@ detective method: **a fact needs ≥2 independent sources or it is
Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`. Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`.
`Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`. `Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`.
Edges: `RUNS / USES / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`. Edges: `RUNS / USES / FROM_FILE / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
`FROM_FILE` / `HAS_VERSION` / `AUTHORED`: `bin/brain/search.go --hop N` walks
them from each hit (1=File, 2=Commit, 3=Person). Rebuild writes
`Leaf-[:FROM_FILE]->File`; git import writes the rest.
Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
`where`, `when`, `source_rev`. `where`, `when`, `source_rev`. Leaf interval of truth (D24): `valid_from`,
`valid_to`.
## Config ## Config
@@ -98,26 +120,32 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
- `bin/{subject}/{method}` — line 2 is a usage comment (mirrors `psql-yq`). - `bin/{subject}/{method}` — line 2 is a usage comment (mirrors `psql-yq`).
- bash + python primary; golang via Go shebang when a compiled helper is right. - bash + python primary; golang via Go shebang when a compiled helper is right.
- YAML default output, `--json` for machines. Slice with `yq`. - YAML default output, `--json` for machines. Slice with mikefarah/yq.
- Everything that touches the network / DB is read-only, throttled, cached. - Everything that touches the network / DB is read-only, throttled, cached.
- Tests (TDD) gate every commit; `gh` + CI/CD on every push. - Tests (TDD) gate every commit; `gh` + CI/CD on every push.
## Open questions (v2) ## Open questions (v2)
- OQ1: mutually-contradicting evidence — how to resolve (authority weighting, - OQ1: **in** — D16 adjudication: `temporal_freshness` then `authority_pairing`.
temporal freshness, audit adjudication). Unresolved 2v2 stays hypothesis. [#29](https://git.produktor.io/eSlider/2dph/issues/29).
- OQ2: OCR pipeline for pdfs/images/docs — mostly solved: poppler pdftotext - OQ2: OCR **in**. `pdftotext -layout` first; scans `pdftoppm` + tesseract
fast-path for born-digital PDFs, docling fallback for the ~5% textless ones. `eng+deu` (`bin/mail/ocr.go`, `internal/ocr`). No gocv, no gosseract CGO
- OQ3: optional duckdb-md layer for `SELECT … FORMAT MARKDOWN` export/write-back. (D21 Zig owns Ladybug CGO). Optional `OCR_ENGINE=paddle` / compose profile
`ocr-paddle`. Docling left the default path. [#6](https://git.produktor.io/eSlider/2dph/issues/6).
- OQ3: **in** — duckdb-go (`internal/duckstats`, `bin/qa/stats.go`) for
quantiles / JSONL count. Not a second graph. [#30](https://git.produktor.io/eSlider/2dph/issues/30).
- OQ4: YAML-first storage for leafs — deferred: JSON is ~10x faster to - OQ4: YAML-first storage for leafs — deferred: JSON is ~10x faster to
serialize and unambiguous; YAML only where humans edit files. serialize and unambiguous; YAML only where humans edit files.
- OQ5: **in** — fact `valid_from` / `valid_to` + `--as-of` / MCP `as_of` (D24).
Not D16 `temporal_freshness`. [#36](https://git.produktor.io/eSlider/2dph/issues/36).
## Mail pipeline (done) ## Mail pipeline (done)
1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download. 1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download.
Gmail attachments key off `body.attachmentId`, not MIME `partId`. Gmail attachments key off `body.attachmentId`, not MIME `partId`.
2. `bin/mail/import.go --from-raw` — message.json → message.md; PDFs via 2. `bin/mail/import.go --from-raw` — message.json → message.md; PDFs via
`pdftotext -layout` (~15ms) with docling subprocess fallback; ICS sidecars `pdftotext -layout` (~15ms); textless/scanned PDFs `pdftoppm` + tesseract
`eng+deu`. ICS sidecars
Latin-1→UTF-8 normalized. Latin-1→UTF-8 normalized.
3. `bin/brain/index.go --rebuild` — fresh rebuild (repo corpus + mail) because ladybug 3. `bin/brain/index.go --rebuild` — fresh rebuild (repo corpus + mail) because ladybug
corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and
@@ -134,10 +162,9 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser) 2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser)
3. python -m unittest discover -s bin/tools (includes published-docs SoT) 3. python -m unittest discover -s bin/tools (includes published-docs SoT)
4. `bin/facts/audit self` (lexicon internal consistency; `bin/facts/audit.go` is the D14 wrapper) 4. `bin/facts/audit self` (lexicon internal consistency; `bin/facts/audit.go` is the D14 wrapper)
5. `bin/kb/eval` (recall@5 ≥ 0.95). Local SoT is `bin/brain/eval.go`; CI uses 5. `bin/brain/eval.go` via Zig (recall@5 ≥ 0.95). Python `bin/kb/eval` is an
the Python twin until the runner has ladybug cgo. Questions live in explicit fallback, not the CI SoT.
`internal/brain/rank`. 6. `bin/cgo/zig go build -tags system_ladybug` (compile search with zig cc; fetches pinned zig+libs).
6. md-docs build/lint if docs tooling arrives.
Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as
`db/tech-poc`: contract first where there is an OpenAPI/message shape. `db/tech-poc`: contract first where there is an OpenAPI/message shape.
@@ -150,5 +177,22 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
4. .venv: ladybug + model2vec + mistune 4. .venv: ladybug + model2vec + mistune
5. schema + tools with TDD (kb + md + facts + brain) 5. schema + tools with TDD (kb + md + facts + brain)
6. ~/.config/brain config 6. ~/.config/brain config
7. corpus extraction (facts/info) 7. corpus extraction (facts/info)**in**: [#18](https://git.produktor.io/eSlider/2dph/issues/18)
8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit 8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit
## Gap to v1 (epic #16)
Remaining: none for epic #16 (v1). Board:
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16),
milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
Narrative: [docs/roadmap.md](docs/roadmap.md).
| Order | Issue | Gap |
|-------|-------|-----|
| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in**`bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. |
| 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | **in**`--hop N` walks `FROM_FILE``HAS_VERSION``AUTHORED` (max 3). |
| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | **in**`--with-facts` / `--facts-json` land `root=facts`; `--with-chats` indexes `var/chats/md`. WhatsApp sync is out of v1. |
| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search``get``audit`). |
| 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | **in** — CI recall SoT is `bin/brain/eval.go` via Zig. Python `bin/kb/eval` stays as an explicit fallback. |
Does **not** block epic close: OQ4. OCR [#6](https://git.produktor.io/eSlider/2dph/issues/6), OQ1 [#29](https://git.produktor.io/eSlider/2dph/issues/29), OQ3 [#30](https://git.produktor.io/eSlider/2dph/issues/30) are **in**.
+44 -30
View File
@@ -7,14 +7,16 @@
[![Latest Release](https://img.shields.io/github/v/tag/eSlider/2dph?sort=semver&label=release)](https://github.com/eSlider/2dph/releases) [![Latest Release](https://img.shields.io/github/v/tag/eSlider/2dph?sort=semver&label=release)](https://github.com/eSlider/2dph/releases)
[![GitHub Stars](https://img.shields.io/github/stars/eSlider/2dph?style=social)](https://github.com/eSlider/2dph/stargazers) [![GitHub Stars](https://img.shields.io/github/stars/eSlider/2dph?style=social)](https://github.com/eSlider/2dph/stargazers)
An evidence-first brain over the operational eSlider stack. **Facts need two An evidence-first brain. **Facts need two independent sources, or they are
independent sources, or they are `(not confirmed)`.** `(not confirmed)`.** Cursor is not the runtime.
`2dph` is a single embedded knowledge graph (LadybugDB = Kuzu successor) with `2dph` is a single embedded knowledge graph (LadybugDB) with native **HNSW
native **HNSW vector** + **BM25 full-text** indexes, built from markdown, vector** + **BM25 full-text** indexes. Search is *deduction*: confirmed facts
compose files, ssh config, docker state, and git history. Search is first, supporting info second, `web-search` as the independent second source
*deduction*: confirmed facts first, supporting info second, `web-search` as when the local graph cannot confirm.
the independent second source when the local graph cannot confirm.
Run it: [docs/runbook.md](docs/runbook.md). Design: [docs/design.md](docs/design.md).
Docs index: [docs/README.md](docs/README.md).
## Architecture ## Architecture
@@ -31,7 +33,7 @@ graph TB
EX["bin/facts/extract.go<br/>2-source pairing"] EX["bin/facts/extract.go<br/>2-source pairing"]
AU["bin/facts/audit.go<br/>confidence + staleness"] AU["bin/facts/audit.go<br/>confidence + staleness"]
IDX["bin/brain/index.go<br/>chunk + embed"] IDX["bin/brain/index.go<br/>chunk + embed"]
MD["bin/markdown/import.go<br/>mistune leaves"] MD["bin/markdown/import.go<br/>H2 leaf split"]
SR["bin/brain/search.go<br/>deduction"] SR["bin/brain/search.go<br/>deduction"]
end end
@@ -74,7 +76,7 @@ graph TB
## The method ## The method
Every assertion is `Who / What / How / Where / When + evidence + confidence`, Every assertion is `Who / What / How / Where / When + evidence + confidence`,
mirroring the detective detective skill: **≥2 independent sources confirm a mirroring the detective method: **≥2 independent sources confirm a
fact; conflicting sources or a single source → `hypothesis``(not confirmed)`.** fact; conflicting sources or a single source → `hypothesis``(not confirmed)`.**
| root | meaning | used for answers | | root | meaning | used for answers |
@@ -94,7 +96,7 @@ bin/brain/stats.go # index health
bin/brain/eval.go # recall@5 gate bin/brain/eval.go # recall@5 gate
``` ```
`--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`. `--hop N` walks File/Commit/Person from each hit (max 3). `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary): Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary):
@@ -118,23 +120,24 @@ Mail is a first-class corpus (retrievable through the same search):
```bash ```bash
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go) bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go)
bin/mail/import.go --from-raw var/mail # JSON → markdown bin/mail/import.go --from-raw var/mail # JSON → markdown
bin/brain/add.go --text T --root facts --source "a.md x b.md"
bin/brain/index.go --rebuild --with-facts --with-chats # facts extract + chats md
bin/brain/index.go --rebuild # rebuild brain (incl. mail) bin/brain/index.go --rebuild # rebuild brain (incl. mail)
bin/brain/search.go "invoice from last week" # same search over mail leafs bin/brain/search.go "invoice from last week" # same search over mail leafs
``` ```
## Storage ## Storage
- **LadybugDB** — single `var/kb.lbug`, Cypher property graph, HNSW + BM25 - **LadybugDB** — single `var/kb.lbug`, Cypher + HNSW + BM25, embedded.
in one engine, embedded (no server), ACID, read-only-safe for concurrent Read tools (`get` / `stats` / `eval`) are Go + Zig CGO (`bin/cgo/zcc`).
readers. Read tools (`get` / `stats` / `eval`) are Go + cgo; Python Python fallbacks stay for CI until the runner fetches Zig. Incremental
`bin/kb/{get,stats,eval}` is the CI fallback. **Never `DROP INDEX` FTS/VECTOR** on Ladybug 0.19: DROP leaves write is `bin/brain/add.go` (Python `kblib.add_leafs`). Bulk rebuild is
ghost catalog tables (`_0_Leaf_vec_UPPER`) so recreate fails while Compose profile `index` (`bin/brain/index.go --rebuild`).
`SHOW_INDEXES` omits HNSW. Fresh indexes = delete `var/kb.lbug` + - **model2vec** — `potion-multilingual-128M` (256-dim), CPU, no Ollama
`bin/brain/index.go --rebuild`. Use `ensure_indexes()` after upserts. runtime dependency.
- **model2vec** — `potion-multilingual-128M` static embeddings (256-dim), - facts and info split by `root` but written in the same transaction.
CPU-fast, deterministic, no Ollama runtime dependency.
- facts and info split semantically by `root` column but written inside the Ladybug 0.19 DROP INDEX warning: [docs/runbook.md](docs/runbook.md).
same transaction.
## Tooling conventions ## Tooling conventions
@@ -145,32 +148,43 @@ machines. Tests gate every commit. HTTP: `bin/brain/serve.go` calls
## Development ## Development
See the portable runbook: [docs/runbook.md](docs/runbook.md).
```bash ```bash
uv venv .venv # Python 3.12, uv-managed uv venv .venv
uv pip install -r requirements.lock.txt # pinned toolchain uv pip install -r requirements.lock.txt
bin/facts/audit.go self # lexicon consistency gate bin/facts/audit.go self
go test ./... && python -m unittest discover -s bin/tools -t . go test ./... && uv run python -m unittest discover -s bin/tools -t .
``` ```
Docker (optional, cached model + var volumes): Docker (optional, cached model + var volumes):
```bash ```bash
docker compose run --rm brain index # (re)index corpus bin/stack/start # brain HTTP/MCP :8630
docker compose run --rm brain search "query" # one-shot query bin/stack/start-assistant # + qwen3.5:9b + PicoClaw agent
docker compose run --rm brain serve # bin/brain/serve.go bin/stack/status
bin/stack/stop
docker compose up -d brain # API (Zig CGO serve :8630)
docker compose --profile index run --rm index # Python Ladybug rebuild
docker compose --profile picoclaw up brain-mcp # MCP on 127.0.0.1:8630 docker compose --profile picoclaw up brain-mcp # MCP on 127.0.0.1:8630
docker compose --profile reasoner up -d reasoner # CPU Ollama 127.0.0.1:11435
docker compose up brain-watch # auto re-index on change docker compose up brain-watch # auto re-index on change
``` ```
## Related ## Related
eSlider DevOps engineer practice: ops, OnlyOffice, and mail feed the facts
root through `bin/facts/extract` (two-source pairing).
- [go-second-brain](https://github.com/eSlider/go-second-brain) — the earlier - [go-second-brain](https://github.com/eSlider/go-second-brain) — the earlier
Neo4j + Qdrant + Matrix RAG brain Neo4j + Qdrant + Matrix RAG brain
- [agent-skills](https://github.com/eSlider/agent-skills) — upstream - [agent-skills](https://github.com/eSlider/agent-skills) — upstream
skills (`web-search`, `postgres`, …) that 2dph integrates skills (`web-search`, `postgres`, …) that 2dph integrates
- detective method — the two-source method - detective method — the two-source method
Work board (issues): [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues). Work board (issues): [epic #16](https://git.produktor.io/eSlider/2dph/issues/16)
on [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph). PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).
See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions. See [PLAN.md](PLAN.md) for decisions, [docs/roadmap.md](docs/roadmap.md) for
the gap to v1, and v2 open questions.
+21
View File
@@ -0,0 +1,21 @@
//usr/bin/env go run -tags=brain_add "$0" "$@"; exit
//go:build brain_add
//
// bin/brain/add.go - incremental leaf write (Python kblib, no rebuild).
//
// ./bin/brain/add.go --text T --root facts --source "a.md x b.md"
// ./bin/brain/add.go --json
//
// D6: write stays Python. Does not delete var/kb.lbug.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/cmdbin"
)
func main() {
os.Exit(cmdbin.ExecFile("bin/kb/add", os.Args[1:]))
}
+1
View File
@@ -8,6 +8,7 @@
// ./bin/brain/get.go <id> --json // ./bin/brain/get.go <id> --json
// //
// Needs CGO + libladybug. Python bin/kb/get is the CI fallback (no cgo). // Needs CGO + libladybug. Python bin/kb/get is the CI fallback (no cgo).
// CGO compiler is Zig (`eval "$(bin/cgo/zig env)"`), not gcc.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang. // NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main package main
+3 -3
View File
@@ -3,12 +3,12 @@
// //
// bin/brain/index.go - rebuild the Ladybug graph (Python write path). // bin/brain/index.go - rebuild the Ladybug graph (Python write path).
// //
// ./bin/brain/index.go --rebuild // ./bin/brain/index.go --rebuild --with-facts --with-chats
// ./bin/brain/index.go --rebuild --with-mail // ./bin/brain/index.go --rebuild --with-mail
// ./bin/brain/index.go --dry-run --with-mail // ./bin/brain/index.go --dry-run --with-mail
// //
// v1 write is always a rebuild when mail is included (live FTS/HNSW + bulk // v1 write: bin/brain/add.go for one/few leafs (indexes may already exist).
// insert corrupts Ladybug 0.19 WAL). `add` is v2. // Bulk mail/corpus still --rebuild (fresh file, indexes last).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang. // NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main package main
+2 -2
View File
@@ -3,11 +3,11 @@
// //
// bin/brain/search.go - deduction search over the 2dph brain. // bin/brain/search.go - deduction search over the 2dph brain.
// //
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--json] [--no-web] // ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--hop N] [--json] [--no-web]
// ./bin/brain/search.go serve [port] // ./bin/brain/search.go serve [port]
// ./bin/brain/search.go --list-model // ./bin/brain/search.go --list-model
// //
// Needs CGO + libladybug (CGO_CFLAGS/CGO_LDFLAGS). Prefer the wrapper // Needs CGO + libladybug via Zig (`eval "$(bin/cgo/zig env)"`), not gcc.
// bin/kb/search which sets those and builds a binary for the embed daemon. // bin/kb/search which sets those and builds a binary for the embed daemon.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang. // NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main package main
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# bin/cgo/zc++ — CGO CXX. Zig, not g++.
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
case "$(uname -m)" in
x86_64|amd64) TARGET=x86_64-linux-gnu ;;
aarch64|arm64) TARGET=aarch64-linux-gnu ;;
*)
echo "zc++: unsupported arch $(uname -m)" >&2
exit 2
;;
esac
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
:
elif [ -x "$ROOT/var/zig/zig" ]; then
ZIG="$ROOT/var/zig/zig"
elif command -v zig >/dev/null 2>&1; then
ZIG="$(command -v zig)"
else
echo "zc++: zig missing; run bin/cgo/zig first" >&2
exit 127
fi
exec "$ZIG" c++ -target "$TARGET" "$@"
Executable
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
# bin/cgo/zcc — CGO CC. Zig, not gcc.
# Go invokes CC with many args; a wrapper avoids spaces in $CC.
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
case "$(uname -m)" in
x86_64|amd64) TARGET=x86_64-linux-gnu ;;
aarch64|arm64) TARGET=aarch64-linux-gnu ;;
*)
echo "zcc: unsupported arch $(uname -m)" >&2
exit 2
;;
esac
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
:
elif [ -x "$ROOT/var/zig/zig" ]; then
ZIG="$ROOT/var/zig/zig"
elif command -v zig >/dev/null 2>&1; then
ZIG="$(command -v zig)"
else
echo "zcc: zig missing; run bin/cgo/zig first" >&2
exit 127
fi
exec "$ZIG" cc -target "$TARGET" "$@"
Executable
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# bin/cgo/zig — CGO toolchain: zig cc (not gcc) + pinned liblbug + libtokenizers.
#
# eval "$(bin/cgo/zig env)" # export CC/CXX/CGO_*
# bin/cgo/zig go build ... # ensure, then exec with env
# bin/cgo/zig ./bin/brain/search.go "query"
#
# Pins live in this file. Downloads land in var/ (gitignored).
set -euo pipefail
ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
ZIG_VERSION=0.14.1
LBUG_VERSION=0.19.1
TOKENIZERS_VERSION=1.27.0
arch="$(uname -m)"
case "$arch" in
x86_64|amd64)
ZIG_ARCH=x86_64
LBUG_ARCH=x86_64
TOK_ARCH=x86_64
ZIG_SHA=24aeeec8af16c381934a6cd7d95c807a8cb2cf7df9fa40d359aa884195c4716c
LBUG_SHA=ed263ae913f68cb0ddba0b98548b58edaac49929766d03bdaaa83be46c68847d
TOK_SHA=72556cdca798dd4ea7cdaba308e5f0d68a8cb93b67c96edf485b7a0edd7b07f4
;;
aarch64|arm64)
ZIG_ARCH=aarch64
LBUG_ARCH=aarch64
TOK_ARCH=aarch64
ZIG_SHA=f7a654acc967864f7a050ddacfaa778c7504a0eca8d2b678839c21eea47c992b
LBUG_SHA=b07df2cd533c3976a2a3025866d6420a5f35514d0a822ecc4b2902d55b4725b7
TOK_SHA=e96545ad05930c26f51f63d932ee6d3bbd32bbed149e102c5290d587a2293067
;;
*)
echo "bin/cgo/zig: unsupported arch $arch" >&2
exit 2
;;
esac
CACHE="$ROOT/var/cache"
LIB="$ROOT/lib-ladybug"
ZIG_DIR="$ROOT/var/zig-dist"
ZIG_BIN="$ROOT/var/zig/zig"
sha256of() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{print $1}'
else
shasum -a 256 "$1" | awk '{print $1}'
fi
}
fetch() {
local url="$1" dest="$2" expect="$3"
if [ -f "$dest" ] && [ "$(sha256of "$dest")" = "$expect" ]; then
return 0
fi
mkdir -p "$(dirname "$dest")"
echo "fetch $url" >&2
curl -fsSL "$url" -o "$dest"
local got
got="$(sha256of "$dest")"
if [ "$got" != "$expect" ]; then
echo "checksum mismatch $dest: got $got want $expect" >&2
rm -f "$dest"
exit 1
fi
}
ensure_zig() {
if [ -n "${ZIG:-}" ] && [ -x "$ZIG" ]; then
return 0
fi
if [ -x "$ZIG_BIN" ]; then
export ZIG="$ZIG_BIN"
return 0
fi
if command -v zig >/dev/null 2>&1; then
export ZIG
ZIG="$(command -v zig)"
return 0
fi
local tar="$CACHE/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}.tar.xz"
fetch "https://ziglang.org/download/${ZIG_VERSION}/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}.tar.xz" \
"$tar" "$ZIG_SHA"
mkdir -p "$CACHE"
rm -rf "$ZIG_DIR"
tar -xJf "$tar" -C "$CACHE"
mv "$CACHE/zig-${ZIG_ARCH}-linux-${ZIG_VERSION}" "$ZIG_DIR"
mkdir -p "$ROOT/var/zig"
ln -sfn "$ZIG_DIR/zig" "$ZIG_BIN"
export ZIG="$ZIG_BIN"
}
ensure_libs() {
mkdir -p "$LIB"
if [ ! -f "$LIB/liblbug.so" ]; then
local tar="$CACHE/liblbug-linux-${LBUG_ARCH}.tar.gz"
fetch "https://github.com/LadybugDB/ladybug/releases/download/v${LBUG_VERSION}/liblbug-linux-${LBUG_ARCH}.tar.gz" \
"$tar" "$LBUG_SHA"
tar -xzf "$tar" -C "$LIB"
fi
if [ ! -f "$LIB/libtokenizers.a" ]; then
local tar="$CACHE/libtokenizers.linux-${TOK_ARCH}.tar.gz"
fetch "https://github.com/daulet/tokenizers/releases/download/v${TOKENIZERS_VERSION}/libtokenizers.linux-${TOK_ARCH}.tar.gz" \
"$tar" "$TOK_SHA"
tar -xzf "$tar" -C "$LIB"
fi
}
print_env() {
printf 'export ZIG=%q\n' "$ZIG"
printf 'export CC=%q\n' "$ROOT/bin/cgo/zcc"
printf 'export CXX=%q\n' "$ROOT/bin/cgo/zc++"
printf 'export CGO_ENABLED=1\n'
printf 'export CGO_CFLAGS=%q\n' "-I$LIB"
printf 'export CGO_LDFLAGS=%q\n' "-L$LIB -Wl,-rpath,${CGO_RPATH:-$LIB}"
}
ensure_zig
ensure_libs
cmd="${1:-env}"
if [ "$cmd" = "env" ]; then
print_env
exit 0
fi
eval "$(print_env)"
exec "$@"
+3 -2
View File
@@ -29,10 +29,11 @@ func main() {
case "linkedin": case "linkedin":
os.Exit(chats.RunSyncLinkedIn(args)) os.Exit(chats.RunSyncLinkedIn(args))
case "whatsapp": case "whatsapp":
fmt.Fprintln(os.Stderr, "chats: WhatsApp not implemented yet") fmt.Fprintln(os.Stderr, "chats: WhatsApp sync is out of v1")
os.Exit(1) os.Exit(1)
case "help", "-h", "--help": case "help", "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`) fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]
WhatsApp sync is out of v1.`)
return return
default: default:
fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform) fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform)
+91
View File
@@ -0,0 +1,91 @@
//usr/bin/env go run "$0" "$@"; exit
//
// bin/cli/complete.go - dump flaggy shell completions for all Go shebang tools (D23).
//
// source <(./bin/cli/complete.go bash)
// ./bin/cli/complete.go zsh|fish|powershell|nushell
//
// Search does not steal the word "completion"; this binary dumps scripts.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"strings"
mailsync "github.com/eSlider/2dph/bin/mail/sync"
"github.com/eSlider/2dph/internal/brain/rank"
"github.com/eSlider/2dph/internal/chats"
"github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/gitlog"
"github.com/eSlider/2dph/internal/mdleaves"
"github.com/eSlider/2dph/internal/ocr"
"github.com/eSlider/2dph/internal/reasoner"
"github.com/eSlider/2dph/internal/websearch"
"github.com/integrii/flaggy"
)
func tools() []cli.Tool {
return []cli.Tool{
{Path: "bin/brain/search.go", Name: "brain-search", New: rank.Parser},
{Path: "bin/brain/get.go", Name: "brain-get", New: func() *flaggy.Parser {
o := rank.GetOptions{}
return rank.GetParser(&o)
}},
{Path: "bin/brain/stats.go", Name: "brain-stats", New: rank.StatsParser},
{Path: "bin/brain/eval.go", Name: "brain-eval", New: rank.EvalParser},
{Path: "bin/web/search.go", Name: "web-search", New: websearch.Parser},
{Path: "bin/git/import.go", Name: "git-import", New: gitlog.Parser},
{Path: "bin/markdown/import.go", Name: "markdown-import", New: mdleaves.Parser},
{Path: "bin/qa/stats.go", Name: "qa-stats", New: cli.QAParser},
{Path: "bin/reasoner/bakeoff.go", Name: "reasoner-bakeoff", New: reasoner.Parser},
{Path: "bin/mail/ocr.go", Name: "mail-ocr", New: ocr.Parser},
{Path: "bin/mail/sync.go", Name: "mail-sync", New: mailsync.Parser},
{Path: "bin/chats/sync.go", Name: "chats-sync", New: chats.SyncParser},
{Path: "bin/chats/import.go", Name: "chats-import", New: chats.ImportParser},
{Path: "bin/chats/facts.go", Name: "chats-facts", New: chats.FactsParser},
{Path: "bin/chats/apply.go", Name: "chats-apply", New: chats.ApplyParser},
}
}
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
shell := "bash"
if len(args) > 0 {
switch args[0] {
case "bash", "zsh", "fish", "powershell", "nushell":
shell = args[0]
case "-h", "--help", "help":
fmt.Fprintln(os.Stderr, "usage: bin/cli/complete.go [bash|zsh|fish|powershell|nushell]")
return 0
default:
fmt.Fprintf(os.Stderr, "cli/complete: unknown shell %q\n", args[0])
return 2
}
}
if shell == "bash" {
fmt.Print(cli.BashScript(tools()))
return 0
}
var b strings.Builder
for _, t := range tools() {
p := t.New()
p.Name = t.Name
switch shell {
case "zsh":
b.WriteString(flaggy.GenerateZshCompletion(p))
case "fish":
b.WriteString(flaggy.GenerateFishCompletion(p))
case "powershell":
b.WriteString(flaggy.GeneratePowerShellCompletion(p))
case "nushell":
b.WriteString(flaggy.GenerateNushellCompletion(p))
}
}
fmt.Print(b.String())
return 0
}
+18 -7
View File
@@ -1,13 +1,10 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# bin/docker-entrypoint - run 2dph tools inside the container. # bin/docker-entrypoint - run 2dph tools inside the container.
# #
# brain shell (default) # API image (Zig CGO binaries):
# brain search <q> bin/brain/search.go # serve | search | watch
# brain index bin/kb/index --with-mail # Index image (Python write path, compose profile `index`):
# brain watch <dir> compiled /app/bin/watch (bin/brain/watch.go) # index | extract | audit | search (deprecated python wrapper)
# brain serve compiled /app/bin/serve (bin/brain/serve.go)
# brain extract bin/facts/extract (docker×compose pairing)
# brain audit bin/facts/audit
# #
# Usage comment starts at line 2 (self-describing convention). # Usage comment starts at line 2 (self-describing convention).
set -euo pipefail set -euo pipefail
@@ -15,6 +12,20 @@ set -euo pipefail
CMD="${1:-shell}" CMD="${1:-shell}"
shift || true shift || true
if [ -x /usr/local/bin/brain-serve ]; then
case "$CMD" in
shell) exec bash ;;
serve) exec /usr/local/bin/brain-serve "$@" ;;
search) exec /usr/local/bin/brain-search "$@" ;;
watch) exec /usr/local/bin/brain-watch "$@" ;;
index)
echo "index is the Python sidecar: docker compose --profile index run --rm index" >&2
exit 2
;;
*) echo "unknown command: $CMD (api: serve|search|watch)" >&2; exit 2 ;;
esac
fi
case "$CMD" in case "$CMD" in
shell) exec bash ;; shell) exec bash ;;
search) exec "$KB_PY" /app/bin/kb/search "$@" ;; search) exec "$KB_PY" /app/bin/kb/search "$@" ;;
+45 -18
View File
@@ -1,14 +1,15 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""facts/audit - evidence & lexicon checks for the 2dph brain. """facts/audit - evidence & lexicon checks for the 2dph brain.
bin/facts/audit self # lexicon: every fact in db has >=2 sources bin/facts/audit self # lexicon: docs + two-source rule
bin/facts/audit db # evidence gate: run against var/kb.lbug bin/facts/audit db # evidence gate against var/kb.lbug
bin/facts/audit contradict # D16 adjudication (JSON claim(s) on stdin)
`self` mode checks the repo itself (no network, no runtime deps). It greps `self` mode checks the repo itself (no network, no runtime deps).
for known-good two-source pairings and confirms the docs are consistent. `db` mode loads every Leaf with root=facts. Confirmed facts need ` x `;
`db` mode loads every Leaf with root=facts and asserts each has source_rev hypothesis contradictions need `a x b vs c x d` (both sides ≥2).
and a non-empty `loc` (the "where did you see it" evidence pointer) and that `contradict` applies temporal_freshness then authority_pairing; ≥2 vs ≥2
'confirmed' facts carry a two-source `source` field. with no rule stays hypothesis / `(not confirmed)`.
Exit 0 = all checks pass, 1 = audit failures, 2 = could not evaluate. Exit 0 = all checks pass, 1 = audit failures, 2 = could not evaluate.
""" """
@@ -22,6 +23,8 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[2] ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools")) sys.path.insert(0, str(ROOT / "bin" / "tools"))
from contradict import adjudicate, check_fact_row # noqa: E402
def audit_db() -> list[str]: def audit_db() -> list[str]:
from kblib import connect from kblib import connect
@@ -33,14 +36,8 @@ def audit_db() -> list[str]:
r = conn.execute("MATCH (l:Leaf {root:'facts'}) RETURN l.id, l.source, l.loc, l.how, l.confidence") r = conn.execute("MATCH (l:Leaf {root:'facts'}) RETURN l.id, l.source, l.loc, l.how, l.confidence")
problems: list[str] = [] problems: list[str] = []
for lid, source, loc, how, conf in r.get_all(): for lid, source, loc, how, conf in r.get_all():
if conf != "confirmed": problems.extend(check_fact_row(str(lid), str(source or ""), str(loc or ""),
problems.append(f"{lid}: facts require confidence='confirmed', got '{conf}'") str(how or ""), str(conf or "")))
if not source or " x " not in source:
problems.append(f"{lid}: needs 2-source evidence in source, got '{source}'")
if not loc:
problems.append(f"{lid}: missing loc (evidence pointer)")
if not how:
problems.append(f"{lid}: missing how")
conn.close() conn.close()
db.close() db.close()
return problems return problems
@@ -54,20 +51,50 @@ def audit_self() -> list[str]:
problems.append("PLAN.md missing recall@5 gate") problems.append("PLAN.md missing recall@5 gate")
if re.search(r"(?i)facts must have.*2 sources|2.source", plan) is None: if re.search(r"(?i)facts must have.*2 sources|2.source", plan) is None:
problems.append("PLAN.md missing the two-source evidence rule for facts") problems.append("PLAN.md missing the two-source evidence rule for facts")
if "temporal_freshness" not in plan or "authority_pairing" not in plan:
problems.append("PLAN.md missing D16 adjudication rules")
if re.search(r"(?i)HNSW|BM25|deduction", (ROOT / "README.md").read_text()) is None: if re.search(r"(?i)HNSW|BM25|deduction", (ROOT / "README.md").read_text()) is None:
problems.append("README.md missing search/retrieval description") problems.append("README.md missing search/retrieval description")
return problems return problems
def audit_contradict(raw: str) -> tuple[list[str], list[dict]]:
raw = raw.strip()
if not raw:
return ["contradict: empty stdin (JSON claim or {claims:[...]})"], []
try:
payload = json.loads(raw)
except json.JSONDecodeError as e:
return [f"contradict: invalid JSON: {e}"], []
if isinstance(payload, dict) and "claims" in payload:
claims = list(payload.get("claims") or [])
elif isinstance(payload, dict):
claims = [payload]
elif isinstance(payload, list):
claims = payload
else:
return ["contradict: expected object or list"], []
details = [adjudicate(c) for c in claims]
return [], details
def main(argv: list[str]) -> int: def main(argv: list[str]) -> int:
import argparse import argparse
p = argparse.ArgumentParser(description="evidence & lexicon audit") p = argparse.ArgumentParser(description="evidence & lexicon audit")
p.add_argument("mode", choices=("self", "db")) p.add_argument("mode", choices=("self", "db", "contradict"))
p.add_argument("--json", action="store_true") p.add_argument("--json", action="store_true")
a = p.parse_args(argv) a = p.parse_args(argv)
problems = audit_self() if a.mode == "self" else audit_db() details: list[dict] = []
out = {"mode": a.mode, "ok": not problems, "problems": problems} if a.mode == "self":
problems = audit_self()
elif a.mode == "db":
problems = audit_db()
else:
problems, details = audit_contradict(sys.stdin.read())
out: dict = {"mode": a.mode, "ok": not problems, "problems": problems}
if details:
out["contradictions"] = details
if a.json: if a.json:
print(json.dumps(out, indent=2)) print(json.dumps(out, indent=2))
else: else:
+1
View File
@@ -5,6 +5,7 @@
// //
// ./bin/facts/audit.go self // ./bin/facts/audit.go self
// ./bin/facts/audit.go db // ./bin/facts/audit.go db
// ./bin/facts/audit.go contradict --json < claim.json
// //
// Python bin/facts/audit is the implementation (CI runs it directly). // Python bin/facts/audit is the implementation (CI runs it directly).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang. // NOTE: never run `gofmt -w` on this file — it breaks the shebang.
+6 -49
View File
@@ -16,9 +16,8 @@ import (
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strconv"
"time"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/cmdbin" "github.com/eSlider/2dph/internal/cmdbin"
"github.com/eSlider/2dph/internal/gitlog" "github.com/eSlider/2dph/internal/gitlog"
) )
@@ -28,50 +27,17 @@ func main() {
} }
func run(args []string) int { func run(args []string) int {
var repo, root, since string c, err := gitlog.ParseArgs(args)
limit := 0 if err != nil {
jsonOut := false return cliparse.Fail(err)
i := 0
for i < len(args) {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--limit" && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 0 {
fmt.Fprintf(os.Stderr, "git/import: --limit must be a non-negative integer\n")
return 2
}
limit = n
case a == "--since" && i+1 < len(args):
i++
since = args[i]
case a == "--root" && i+1 < len(args):
i++
root = args[i]
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/git/import.go [REPO] [--json] [--limit N] [--since DATE] [--root DIR]`)
return 0
case len(a) > 0 && a[0] != '-':
repo = a
default:
fmt.Fprintf(os.Stderr, "git/import: unknown flag %s\n", a)
return 2
}
i++
} }
repo, root, since, limit, jsonOut := c.Repo, c.Root, c.Since, c.Limit, c.JSONOut
var sinceT time.Time sinceT, err := gitlog.ParseSince(since)
if since != "" {
var err error
sinceT, err = parseSince(since)
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "git/import: %v\n", err) fmt.Fprintf(os.Stderr, "git/import: %v\n", err)
return 2 return 2
} }
}
repos := []string{} repos := []string{}
if repo != "" { if repo != "" {
@@ -132,12 +98,3 @@ func run(args []string) int {
} }
return 0 return 0
} }
func parseSince(s string) (time.Time, error) {
for _, layout := range []string{time.RFC3339, "2006-01-02"} {
if t, err := time.Parse(layout, s); err == nil {
return t, nil
}
}
return time.Time{}, fmt.Errorf("cannot parse --since %q", s)
}
Executable
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""kb/add - incremental leaf write (no rebuild).
bin/kb/add --text T --root facts|info --source S
bin/kb/add --json # stdin: one object or {"leafs":[...]}
bin/kb/add --db PATH --json
Writes facts+info in one Ladybug transaction. Does not delete kb.lbug.
Embedding is used when provided; otherwise model2vec encodes the text.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402
EMBED_DIM,
add_leafs,
connect,
ensure_indexes,
init_schema,
)
def _as_leafs(payload: object) -> list[dict]:
if isinstance(payload, list):
return [dict(x) for x in payload]
if isinstance(payload, dict):
if "leafs" in payload:
return [dict(x) for x in payload["leafs"]]
return [dict(payload)]
raise ValueError("json must be an object, a list, or {leafs:[...]}")
def _embed_missing(leafs: list[dict]) -> None:
missing = [lf for lf in leafs if not lf.get("embedding")]
if not missing:
return
from model2vec import StaticModel
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
for lf in missing:
text = str(lf.get("text") or "")
vec = model.encode([text])[0].astype(float).tolist()
if len(vec) != EMBED_DIM:
vec = (vec + [0.0] * EMBED_DIM)[:EMBED_DIM]
lf["embedding"] = vec
def main(argv: list[str]) -> int:
import argparse
p = argparse.ArgumentParser(description="add leafs without rebuilding the brain")
p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)")
p.add_argument("--json", action="store_true", help="read leaf JSON from stdin")
p.add_argument("--text", default="", help="leaf text")
p.add_argument("--root", default="info", choices=("facts", "info"))
p.add_argument("--source", default="")
p.add_argument("--confidence", default="confirmed")
p.add_argument("--source-rev", default="working-tree")
p.add_argument("--how", default="brain/add")
p.add_argument("--loc", default="")
p.add_argument("--type", default="reference", dest="type_")
p.add_argument("--valid-from", default="", dest="valid_from",
help="fact interval start YYYY-MM-DD (D24)")
p.add_argument("--valid-to", default="", dest="valid_to",
help="fact interval end YYYY-MM-DD inclusive; empty=open (D24)")
args = p.parse_args(argv)
if args.json:
raw = sys.stdin.read()
if not raw.strip():
print("kb/add: empty stdin", file=sys.stderr)
return 2
leafs = _as_leafs(json.loads(raw))
else:
if not args.text or not args.source:
print("kb/add: --text and --source are required (or --json)", file=sys.stderr)
return 2
leafs = [{
"text": args.text,
"root": args.root,
"source": args.source,
"confidence": args.confidence,
"source_rev": args.source_rev,
"how": args.how,
"loc": args.loc or args.source,
"type": args.type_,
"valid_from": args.valid_from,
"valid_to": args.valid_to,
}]
for lf in leafs:
if not lf.get("text") or not lf.get("source"):
print("kb/add: each leaf needs text and source", file=sys.stderr)
return 2
_embed_missing(leafs)
from kblib import DB_PATH, VAR
dbpath = Path(args.db) if args.db else DB_PATH
dbpath.parent.mkdir(parents=True, exist_ok=True)
VAR.mkdir(exist_ok=True)
db, conn = connect(dbpath, read_only=False)
init_schema(conn)
ids = add_leafs(conn, leafs)
ensure_indexes(conn)
conn.close()
db.close()
print(json.dumps({"mode": "add", "ids": ids, "db": str(dbpath)}))
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+98 -14
View File
@@ -2,12 +2,15 @@
"""kb/index - build the 2dph brain from markdown + factual leafs. """kb/index - build the 2dph brain from markdown + factual leafs.
bin/kb/index [--corpus DIR] [--rebuild] [--limit N] bin/kb/index [--corpus DIR] [--rebuild] [--limit N]
bin/kb/index --rebuild --with-facts --with-chats
bin/kb/index --json # emit stats as JSON bin/kb/index --json # emit stats as JSON
Reads every .md under the corpus (default: repo root docs, skills, READMEs) Reads every .md under the corpus (default: repo root docs, skills, READMEs)
as `info` leafs, embeds them with model2vec (potion-multilingual-128M), and as `info` leafs, embeds them with model2vec (potion-multilingual-128M), and
writes them into var/kb.lbug with FTS + HNSW indexes. `facts` leafs come writes them into var/kb.lbug with FTS + HNSW indexes. `facts` leafs come
from bin/facts/extract (docker x compose x ssh-config pairing). from bin/facts/extract (docker × compose × ssh-config pairing) when
`--with-facts` is set. `--with-chats` indexes markdown under var/chats/md
(or a given dir) as info. WhatsApp sync stays out of v1.
--rebuild drops the database file and indexes from scratch. Without it a run --rebuild drops the database file and indexes from scratch. Without it a run
is idempotent (MERGE by (source,text) id). is idempotent (MERGE by (source,text) id).
@@ -22,7 +25,7 @@ ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "bin" / "tools")) sys.path.insert(0, str(ROOT / "bin" / "tools"))
from kblib import ( # noqa: E402 from kblib import ( # noqa: E402
connect, ensure_indexes, init_schema, upsert_leaf, add_leafs, connect, ensure_indexes, init_schema, upsert_leaf, link_from_file,
open_readonly, stats, open_readonly, stats,
) )
from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402 from mdleaves import read_markdown, to_all, walk_markdown # noqa: E402
@@ -81,10 +84,11 @@ def index_leafs(conn, leafs: list[dict], embed_fn, limit: int) -> tuple[int, int
for lf in leafs[:limit] if limit else leafs: for lf in leafs[:limit] if limit else leafs:
query = f"{lf['heading']}\n\n{lf['text']}" query = f"{lf['heading']}\n\n{lf['text']}"
emb = embed_fn(lf["text"]) if lf["text"] else None emb = embed_fn(lf["text"]) if lf["text"] else None
upsert_leaf(conn, text=query, root="info", confidence="confirmed", lid = upsert_leaf(conn, text=query, root="info", confidence="confirmed",
source=lf["source"], source_rev="working-tree", source=lf["source"], source_rev="working-tree",
how="kb/index", loc=lf["source"], type_=lf.get("type", "reference"), how="kb/index", loc=lf["source"], type_=lf.get("type", "reference"),
embedding=emb) embedding=emb)
link_from_file(conn, lid, lf["source"], repo=str(lf.get("repo") or ""))
count += 1 count += 1
return count, len(leafs) return count, len(leafs)
@@ -95,12 +99,65 @@ def embedder():
return lambda text: model.encode([text])[0].astype(float).tolist() return lambda text: model.encode([text])[0].astype(float).tolist()
def index_fact_dicts(conn, facts: list[dict], embed_fn) -> int:
"""Write extract-shaped dicts as root=facts leafs (2-source source field)."""
leafs = []
for f in facts:
text = str(f.get("text") or "")
source = str(f.get("source") or "")
if not text or not source:
continue
leafs.append({
"text": text,
"root": "facts",
"confidence": "confirmed",
"source": source,
"source_rev": f.get("source_rev") or "working-tree",
"how": f.get("how") or "facts/extract",
"loc": f.get("loc") or source,
"type": "fact",
"embedding": embed_fn(text) if text else None,
})
return len(add_leafs(conn, leafs))
def facts_from_extract() -> list[dict]:
import subprocess
proc = subprocess.run(
[sys.executable, str(ROOT / "bin" / "facts" / "extract"), "--json", "--dry-run"],
cwd=ROOT,
capture_output=True,
text=True,
check=False,
)
if proc.returncode != 0:
print(f"kb/index: facts/extract failed: {proc.stderr}", file=sys.stderr)
return []
try:
payload = json.loads(proc.stdout)
except json.JSONDecodeError:
print("kb/index: facts/extract produced non-JSON", file=sys.stderr)
return []
return list(payload.get("facts") or [])
def main(argv: list[str]) -> int: def main(argv: list[str]) -> int:
import argparse import argparse
p = argparse.ArgumentParser(description="build the 2dph brain index") p = argparse.ArgumentParser(description="build the 2dph brain index")
p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)") p.add_argument("--corpus", action="append", help="extra markdown dir/file to index (may repeat)")
p.add_argument("--rebuild", action="store_true", help="fresh db + indexes") p.add_argument("--rebuild", action="store_true", help="fresh db + indexes")
p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)")
p.add_argument("--no-defaults", action="store_true", help="do not index repo README/docs/skills")
p.add_argument("--with-mail", action="store_true", help="include var/mail message.md leafs") p.add_argument("--with-mail", action="store_true", help="include var/mail message.md leafs")
p.add_argument("--with-facts", action="store_true", help="run facts/extract into root=facts")
p.add_argument("--facts-json", default="", help="JSON list (or {facts:[...]}) of fact dicts")
p.add_argument(
"--with-chats",
nargs="?",
const=str(ROOT / "var" / "chats" / "md"),
default="",
help="index chat markdown as info (default var/chats/md)",
)
p.add_argument("--since", default="", help="with --with-mail, only messages dated >= YYYY-MM-DD") p.add_argument("--since", default="", help="with --with-mail, only messages dated >= YYYY-MM-DD")
p.add_argument("--dry-run", action="store_true", help="count leafs, write nothing") p.add_argument("--dry-run", action="store_true", help="count leafs, write nothing")
p.add_argument( p.add_argument(
@@ -114,44 +171,71 @@ def main(argv: list[str]) -> int:
from kblib import DB_PATH, VAR from kblib import DB_PATH, VAR
leafs = load_corpus(ROOT) dbpath = Path(a.db) if a.db else DB_PATH
leafs: list[dict] = [] if a.no_defaults else load_corpus(ROOT)
if a.corpus: if a.corpus:
for source in a.corpus: for source in a.corpus:
leafs.extend(load_corpus_glob(source)) leafs.extend(load_corpus_glob(source))
chat_n = 0
if a.with_chats:
chats = load_corpus_glob(a.with_chats)
chat_n = len(chats)
leafs.extend(chats)
mail_n = 0 mail_n = 0
if a.with_mail: if a.with_mail:
mail = from_mail_root(ROOT / "var" / "mail", since=a.since) mail = from_mail_root(ROOT / "var" / "mail", since=a.since)
mail_n = len(mail) mail_n = len(mail)
leafs.extend(mail) leafs.extend(mail)
facts: list[dict] = []
if a.facts_json:
raw = Path(a.facts_json).read_text(encoding="utf-8")
payload = json.loads(raw)
facts = list(payload.get("facts") if isinstance(payload, dict) else payload)
if a.with_facts:
facts.extend(facts_from_extract())
if a.dry_run: if a.dry_run:
msg = {"indexed": 0, "corpus_total": len(leafs), "mail_leafs": mail_n, "dry_run": True} msg = {
"indexed": 0,
"corpus_total": len(leafs),
"mail_leafs": mail_n,
"chat_leafs": chat_n,
"facts_leafs": len(facts),
"dry_run": True,
}
print(json.dumps(msg, indent=2) if a.json else print(json.dumps(msg, indent=2) if a.json else
f"brain/index: {len(leafs)} leafs would be indexed (mail={mail_n})") f"brain/index: {len(leafs)} info + {len(facts)} facts would be indexed")
return 0 return 0
VAR.mkdir(exist_ok=True) VAR.mkdir(exist_ok=True)
if a.rebuild and DB_PATH.exists(): dbpath.parent.mkdir(parents=True, exist_ok=True)
DB_PATH.unlink() if a.rebuild and dbpath.exists():
dbpath.unlink()
db, conn = connect(DB_PATH, read_only=False) db, conn = connect(dbpath, read_only=False)
init_schema(conn) init_schema(conn)
# Never DROP FTS/VECTOR (ghost catalog). Write leafs, then ensure indexes
# unless --skip-indexes (seed facts first — MERGE under live FTS corrupts it).
# --rebuild already deleted kb.lbug above, so CREATE runs on a clean DB.
embed = embedder() embed = embedder()
done, total = index_leafs(conn, leafs, embed, a.limit) done, total = index_leafs(conn, leafs, embed, a.limit)
fact_n = index_fact_dicts(conn, facts, embed) if facts else 0
if not a.skip_indexes: if not a.skip_indexes:
ensure_indexes(conn) ensure_indexes(conn)
s = stats(conn) s = stats(conn)
conn.close() conn.close()
db.close() db.close()
result = {"indexed": done, "corpus_total": total, **{k: v for k, v in s.items() if k in ("total", "by_root")}} result = {
"indexed": done,
"corpus_total": total,
"facts_leafs": fact_n,
"chat_leafs": chat_n,
**{k: v for k, v in s.items() if k in ("total", "by_root")},
}
if a.skip_indexes: if a.skip_indexes:
result["indexes"] = "skipped" result["indexes"] = "skipped"
print(json.dumps(result, indent=2) if a.json else f"indexed {done}/{total} leafs; db total {s['total']}") print(json.dumps(result, indent=2) if a.json else
f"indexed {done}/{total} info + {fact_n} facts; db total {s['total']}")
return 0 return 0
+4 -6
View File
@@ -1,10 +1,9 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# bin/kb/search — deprecated wrapper. Use bin/brain/search.go. # bin/kb/search — deprecated wrapper. Use bin/brain/search.go.
# Sets CGO for ladybug, builds a binary (embed daemon needs a real executable), # CGO via Zig (bin/cgo/zig), not gcc. Builds a binary then execs it.
# then execs it. Prints one deprecation line.
set -euo pipefail set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)" ROOT="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)"
BIN="$ROOT/var/bin/brain-search" BIN="$ROOT/var/bin/brain-search"
SRC="$ROOT/internal/brain" SRC="$ROOT/internal/brain"
CMD="$ROOT/bin/brain" CMD="$ROOT/bin/brain"
@@ -24,11 +23,10 @@ else
fi fi
if [ "$need_build" -eq 1 ]; then if [ "$need_build" -eq 1 ]; then
echo "Building brain/search..." >&2 echo "Building brain/search (zig cc)..." >&2
( (
cd "$ROOT" && cd "$ROOT" &&
CGO_CFLAGS="-I$ROOT/lib-ladybug" \ eval "$("$ROOT/bin/cgo/zig" env)" &&
CGO_LDFLAGS="-L$ROOT/lib-ladybug -Wl,-rpath,$ROOT/lib-ladybug" \
go build -tags system_ladybug -o "$BIN" ./bin/brain go build -tags system_ladybug -o "$BIN" ./bin/brain
) || exit 1 ) || exit 1
fi fi
+11 -77
View File
@@ -7,7 +7,7 @@
bin/mail/import --since 2026-01-01 only messages after a date bin/mail/import --since 2026-01-01 only messages after a date
bin/mail/import --limit 50 cap messages per run bin/mail/import --limit 50 cap messages per run
bin/mail/import --no-attachments body only, skip attachment conversion bin/mail/import --no-attachments body only, skip attachment conversion
bin/mail/import --ocr OCR scanned PDFs/images via docling bin/mail/import --ocr OCR images (PDFs OCR when textless)
bin/mail/import --dry-run list messages without writing anything bin/mail/import --dry-run list messages without writing anything
Writes one directory per message: var/mail/{folder}/{message_id}/ Writes one directory per message: var/mail/{folder}/{message_id}/
@@ -16,10 +16,11 @@ Writes one directory per message: var/mail/{folder}/{message_id}/
attachments/*.md converted attachment content attachments/*.md converted attachment content
Indexing is a separate step (`bin/brain/index.go --rebuild`): conversion can Indexing is a separate step (`bin/brain/index.go --rebuild`): conversion can
crash in native docling and must not leave the brain DB mid-transaction. crash and must not leave the brain DB mid-transaction.
Requires ONLYOFFICE_URL/USER/PASS in .env (or env). Idempotent: a message Requires ONLYOFFICE_URL/USER/PASS in .env (or env) except `--from-raw`.
already present (message.md exists) is skipped unless --force. Idempotent: a message already present (message.md exists) is skipped unless
--force.
""" """
from __future__ import annotations from __future__ import annotations
@@ -41,10 +42,10 @@ from mailconv import ( # noqa: E402
IMAGE_SUFFIXES, IMAGE_SUFFIXES,
LEGACY_OFFICE_SUFFIXES, LEGACY_OFFICE_SUFFIXES,
TEXT_SUFFIXES, TEXT_SUFFIXES,
convert_pdf,
html_to_markdown, html_to_markdown,
is_convertible,
normalize_markdown, normalize_markdown,
subject_to_filename, ocr_image,
zip_extract_safe, zip_extract_safe,
) )
@@ -147,9 +148,9 @@ def convert_file_to_md(path: Path, ocr: bool) -> str | None:
except Exception as e: except Exception as e:
return f"\n<!-- conversion failed: {e} -->\n" return f"\n<!-- conversion failed: {e} -->\n"
if suffix == ".pdf": if suffix == ".pdf":
return _convert_pdf(path, ocr) return convert_pdf(path, ocr)
if suffix in IMAGE_SUFFIXES and ocr: if suffix in IMAGE_SUFFIXES and ocr:
return _convert_pdf(path, ocr) return ocr_image(path) or "\n<!-- ocr unavailable -->\n"
if suffix in LEGACY_OFFICE_SUFFIXES: if suffix in LEGACY_OFFICE_SUFFIXES:
return _convert_legacy(path) return _convert_legacy(path)
if suffix in ARCHIVE_SUFFIXES: if suffix in ARCHIVE_SUFFIXES:
@@ -157,67 +158,6 @@ def convert_file_to_md(path: Path, ocr: bool) -> str | None:
return None return None
def _convert_pdf(path: Path, ocr: bool) -> str:
"""Convert one PDF to markdown.
Fast path: poppler's pdftotext (-layout) extracts exact text from
born-digital PDFs in ~15ms vs docling's 1-3s. Only textless PDFs (scanned
pages, layout-heavy) fall back to docling, which runs isolated in a
subprocess because its native onnx/RT-DETR has segfaulted the main process.
"""
text = _pdf_fast_text(path)
if ocr or text is None or not text.strip():
return _convert_pdf_docling(path, ocr)
return normalize_markdown(text)
def _pdf_fast_text(path: Path) -> str | None:
"""pdftotext -layout; None when poppler is unavailable (or the PDF has no text layer)."""
try:
proc = subprocess.run(
["pdftotext", "-layout", str(path), "-"],
capture_output=True, timeout=60)
except (OSError, subprocess.TimeoutExpired):
return None
if proc.returncode != 0:
return None
return proc.stdout.decode("utf-8", errors="replace")
def _convert_pdf_docling(path: Path, ocr: bool) -> str:
try:
proc = subprocess.run(
[sys.executable, os.path.abspath(__file__), "--pdf-worker", str(path),
"--ocr" if ocr else "--no-ocr"],
capture_output=True, text=True, timeout=600)
except subprocess.TimeoutExpired:
return "\n<!-- pdf conversion timed out -->\n"
if proc.returncode != 0:
tail = proc.stderr.strip().splitlines()[-3:]
return f"\n<!-- pdf conversion failed: {proc.returncode}: {' | '.join(tail)} -->\n"
return proc.stdout
def _pdf_worker(path: Path, ocr: bool) -> None:
"""docling worker entry: prints converted markdown on stdout, exits non-zero on error."""
try:
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.pipeline_options import PdfPipelineOptions
opts = PdfPipelineOptions()
opts.do_ocr = bool(ocr)
opts.do_table_structure = True
conv = DocumentConverter(format_options={"pdf": PdfFormatOption(pipeline_options=opts)})
res = conv.convert(str(path))
sys.stdout.write(normalize_markdown(res.document.export_to_markdown()))
sys.exit(0)
except Exception as e:
# errors/stacktraces to stderr; the caller only reports a one-liner
print(f"pdf-worker: {e}", file=sys.stderr)
import traceback
traceback.print_exc(file=sys.stderr)
sys.exit(1)
def _convert_legacy(path: Path) -> str: def _convert_legacy(path: Path) -> str:
"""Legacy .doc/.xls/.ppt -> md via pandoc (installed) or a stub.""" """Legacy .doc/.xls/.ppt -> md via pandoc (installed) or a stub."""
try: try:
@@ -356,19 +296,12 @@ def main(argv: list[str]) -> int:
p.add_argument("--from-raw", default="", p.add_argument("--from-raw", default="",
help="convert Go-synced dirs (var/mail/<folder>/<id>/message.json) to markdown") help="convert Go-synced dirs (var/mail/<folder>/<id>/message.json) to markdown")
p.add_argument("--no-attachments", action="store_true", help="skip attachment download+convert") p.add_argument("--no-attachments", action="store_true", help="skip attachment download+convert")
p.add_argument("--ocr", action="store_true", help="OCR scanned PDFs/images via docling") p.add_argument("--ocr", action="store_true", help="OCR images (PDFs OCR when textless)")
p.add_argument("--force", action="store_true", help="re-import even if message.md exists") p.add_argument("--force", action="store_true", help="re-import even if message.md exists")
p.add_argument("--dry-run", action="store_true", help="list messages, write nothing") p.add_argument("--dry-run", action="store_true", help="list messages, write nothing")
p.add_argument("--json", action="store_true") p.add_argument("--json", action="store_true")
p.add_argument("--pdf-worker", default="", help=argparse.SUPPRESS)
p.add_argument("--no-ocr", action="store_true", help=argparse.SUPPRESS)
a = p.parse_args(argv) a = p.parse_args(argv)
if a.pdf_worker:
_pdf_worker(Path(a.pdf_worker), ocr=not a.no_ocr)
return 0
conf = load_env()
fid = folder_id(a.folder) fid = folder_id(a.folder)
out_root = ROOT / "var" / "mail" out_root = ROOT / "var" / "mail"
summary: list[dict] = [] summary: list[dict] = []
@@ -394,6 +327,7 @@ def main(argv: list[str]) -> int:
target_dir=msg_dir.parent)) target_dir=msg_dir.parent))
summary.append(entry) summary.append(entry)
else: else:
conf = load_env()
OOCLIENT = OOClient(conf) OOCLIENT = OOClient(conf)
if a.id: if a.id:
messages = [{"id": i} for i in a.id] messages = [{"id": i} for i in a.id]
+46
View File
@@ -0,0 +1,46 @@
//usr/bin/env go run -tags=mail_ocr "$0" "$@"; exit
//go:build mail_ocr
//
// bin/mail/ocr.go - OCR an image or scanned PDF (tesseract eng+deu).
//
// ./bin/mail/ocr.go scan.png
// ./bin/mail/ocr.go scan.pdf
// OCR_ENGINE=paddle ./bin/mail/ocr.go scan.png
//
// PDFs try pdftotext -layout first; empty text layer uses pdftoppm + tesseract.
// No gocv. Tesseract CGO bindings are not used (D21 Zig owns Ladybug CGO).
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"fmt"
"os"
"strings"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/ocr"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := ocr.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
path := c.Path
var text string
if strings.HasSuffix(strings.ToLower(path), ".pdf") {
text, err = ocr.PDFFile(path)
} else {
text, err = ocr.ImageFile(path)
}
if err != nil {
fmt.Fprintf(os.Stderr, "mail/ocr: %v\n", err)
return 1
}
fmt.Println(text)
return 0
}
+68 -40
View File
@@ -5,12 +5,15 @@ package sync
import ( import (
"context" "context"
"flag" "errors"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
) )
// CLIConfig is a superset of SyncConfig plus flag parsing results. // CLIConfig is a superset of SyncConfig plus flag parsing results.
@@ -21,58 +24,84 @@ type CLIConfig struct {
Help bool Help bool
} }
// ParseCLI reads os.Args into a CLIConfig. Exit codes: 0 ok, 2 usage. type flagVals struct {
env, out, srcs, query string
workers, limit, offset int
force, dryRun bool
}
func Parser() *flaggy.Parser {
v := flagVals{workers: 4, query: "in:inbox", srcs: "onlyoffice"}
return bind(&v)
}
func bind(v *flagVals) *flaggy.Parser {
if v.workers == 0 {
v.workers = 4
}
if v.query == "" {
v.query = "in:inbox"
}
if v.srcs == "" {
v.srcs = "onlyoffice"
}
p := cliparse.New("mail-sync")
p.Description = "download mail to var/mail"
p.String(&v.env, "", "env", ".env file")
p.String(&v.out, "", "out", "var/mail root")
p.Int(&v.workers, "", "workers", "concurrent downloads")
p.Int(&v.limit, "", "limit", "max messages per source (0 = all)")
p.Int(&v.offset, "", "offset", "skip first N messages per source")
p.Bool(&v.force, "", "force", "overwrite existing message.json")
p.Bool(&v.dryRun, "", "dry-run", "list counts without writing")
p.String(&v.query, "", "query", "Gmail search query")
p.String(&v.srcs, "", "source", "comma list: onlyoffice,gmail")
return p
}
// ParseCLI reads args into a CLIConfig. Exit codes: 0 ok, 2 usage.
func ParseCLI(args []string) (CLIConfig, int, error) { func ParseCLI(args []string) (CLIConfig, int, error) {
fs := flag.NewFlagSet("mail/sync", flag.ContinueOnError) v := flagVals{workers: 4, query: "in:inbox", srcs: "onlyoffice"}
var ( p := bind(&v)
env = fs.String("env", "", ".env file (default: <cwd>/.env)") if err := cliparse.Parse(p, args); err != nil {
out = fs.String("out", "", "var/mail root (default: <cwd>/var/mail)") if errors.Is(err, cliparse.ErrHelp) {
workers = fs.Int("workers", 4, "concurrent downloads") return CLIConfig{Help: true}, 0, nil
limit = fs.Int("limit", 0, "max messages per source (0 = all)") }
offset = fs.Int("offset", 0, "skip first N messages per source")
force = fs.Bool("force", false, "overwrite existing message.json + attachments")
dryRun = fs.Bool("dry-run", false, "list message counts without writing")
query = fs.String("query", "in:inbox", "Gmail search query (gmail source only)")
srcs = fs.String("source", "onlyoffice", "comma list: onlyoffice,gmail (default onlyoffice)")
help = fs.Bool("help", false, "usage")
)
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return CLIConfig{}, 2, err return CLIConfig{}, 2, err
} }
if *help || fs.NArg() > 0 { if len(p.TrailingArguments) > 0 {
return CLIConfig{Help: true}, 0, nil return CLIConfig{Help: true}, 0, nil
} }
wd, err := os.Getwd() wd, err := os.Getwd()
if err != nil { if err != nil {
return CLIConfig{}, 2, err return CLIConfig{}, 2, err
} }
if *env == "" { if v.env == "" {
*env = filepath.Join(wd, ".env") v.env = filepath.Join(wd, ".env")
} }
if *out == "" { if v.out == "" {
*out = filepath.Join(wd, "var", "mail") v.out = filepath.Join(wd, "var", "mail")
} }
envVars := readEnv(*env) envVars := readEnv(v.env)
cfg := SyncConfig{ cfg := SyncConfig{
Out: *out, Out: v.out,
Workers: *workers, Workers: v.workers,
Limit: *limit, Limit: v.limit,
Offset: *offset, Offset: v.offset,
Force: *force, Force: v.force,
DryRun: *dryRun, DryRun: v.dryRun,
Query: *query, Query: v.query,
Policy: RetryPolicy{}, Policy: RetryPolicy{},
} }
cli := CLIConfig{Sync: cfg, Env: *env, Sources: *srcs} out := CLIConfig{Sync: cfg, Env: v.env, Sources: v.srcs}
for _, s := range strings.Split(*srcs, ",") { for _, s := range strings.Split(v.srcs, ",") {
switch strings.TrimSpace(s) { switch strings.TrimSpace(s) {
case "onlyoffice": case "onlyoffice":
u := pick(envVars["ONLYOFFICE_URL"], envVars["OO_URL"]) u := pick(envVars["ONLYOFFICE_URL"], envVars["OO_URL"])
user := pick(envVars["ONLYOFFICE_USER"], envVars["OO_USER"]) user := pick(envVars["ONLYOFFICE_USER"], envVars["OO_USER"])
pass := pick(envVars["ONLYOFFICE_PASS"], envVars["OO_PASSWORD"]) pass := pick(envVars["ONLYOFFICE_PASS"], envVars["OO_PASSWORD"])
if u == "" || user == "" || pass == "" { if u == "" || user == "" || pass == "" {
return CLIConfig{}, 2, fmt.Errorf("onlyoffice source needs ONLYOFFICE_URL/USER/PASS in %s", *env) return CLIConfig{}, 2, fmt.Errorf("onlyoffice source needs ONLYOFFICE_URL/USER/PASS in %s", v.env)
} }
cfg.OO = &OOConfig{URL: u, User: user, Password: pass} cfg.OO = &OOConfig{URL: u, User: user, Password: pass}
case "gmail": case "gmail":
@@ -85,30 +114,30 @@ func ParseCLI(args []string) (CLIConfig, int, error) {
return CLIConfig{}, 2, fmt.Errorf("unknown source %q", s) return CLIConfig{}, 2, fmt.Errorf("unknown source %q", s)
} }
} }
cli.Sync = cfg out.Sync = cfg
return cli, 0, nil return out, 0, nil
} }
// Main is the CLI entry: returns process exit code. // Main is the CLI entry: returns process exit code.
func Main(args []string) int { func Main(args []string) int {
cli, code, err := ParseCLI(args) cfg, code, err := ParseCLI(args)
if err != nil { if err != nil {
fmt.Fprintln(os.Stderr, "mail/sync:", err) fmt.Fprintln(os.Stderr, "mail/sync:", err)
return code return code
} }
if cli.Help { if cfg.Help {
fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--query GMAIL_Q] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]") fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--query GMAIL_Q] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]")
return 0 return 0
} }
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour) ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
defer cancel() defer cancel()
start := time.Now() start := time.Now()
stats, err := Run(ctx, cli.Sync) stats, err := Run(ctx, cfg.Sync)
if err != nil { if err != nil {
fmt.Fprintln(os.Stderr, "mail/sync:", err) fmt.Fprintln(os.Stderr, "mail/sync:", err)
return 1 return 1
} }
if cli.Sync.DryRun { if cfg.Sync.DryRun {
fmt.Printf("mail/sync: dry-run checked=%d (no writes)\n", stats.Checked) fmt.Printf("mail/sync: dry-run checked=%d (no writes)\n", stats.Checked)
return 0 return 0
} }
@@ -135,7 +164,6 @@ func readEnv(path string) map[string]string {
k, v, _ := strings.Cut(line, "=") k, v, _ := strings.Cut(line, "=")
out[strings.TrimSpace(k)] = strings.Trim(strings.TrimSpace(v), "\"'") out[strings.TrimSpace(k)] = strings.Trim(strings.TrimSpace(v), "\"'")
} }
// env overrides file
for _, kv := range os.Environ() { for _, kv := range os.Environ() {
k, v, ok := strings.Cut(kv, "=") k, v, ok := strings.Cut(kv, "=")
if !ok { if !ok {
+70 -5
View File
@@ -1,20 +1,85 @@
//usr/bin/env go run -tags=markdown_import "$0" "$@"; exit //usr/bin/env go run "$0" "$@"; exit
//go:build markdown_import
// //
// bin/markdown/import.go - split markdown into leafs (mistune). // bin/markdown/import.go - split markdown into leafs (H2 boundaries).
// //
// ./bin/markdown/import.go [dir] // ./bin/markdown/import.go [dir]
// ./bin/markdown/import.go --files a.md,b.md --json // ./bin/markdown/import.go --files a.md,b.md --json
// //
// Conversion only. Brain write is bin/brain/index.go.
// Python bin/md/import remains as a fallback.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang. // NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main package main
import ( import (
"fmt"
"os" "os"
"strings"
"github.com/eSlider/2dph/internal/cmdbin" cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/mdleaves"
) )
func main() { func main() {
os.Exit(cmdbin.ExecFile("bin/md/import", os.Args[1:])) os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := mdleaves.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
jsonOut := c.JSONOut
files := c.Files
root := c.Root
var paths []string
if files != "" {
for _, f := range strings.Split(files, ",") {
f = strings.TrimSpace(f)
if f != "" {
paths = append(paths, f)
}
}
} else {
st, err := os.Stat(root)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: no such path %s\n", root)
return 2
}
if !st.IsDir() {
paths = []string{root}
} else {
var err error
paths, err = mdleaves.WalkMarkdown(root)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: %v\n", err)
return 1
}
}
}
if len(paths) == 0 {
fmt.Fprintln(os.Stderr, "md/import: no markdown files")
return 1
}
var all []mdleaves.Leaf
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
fmt.Fprintf(os.Stderr, "md/import: %s: %v\n", p, err)
continue
}
all = append(all, mdleaves.ToAll(string(raw), p, "")...)
}
if jsonOut {
s, err := mdleaves.EncodeJSON(all)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Print(s)
return 0
}
fmt.Print(mdleaves.EncodeYAML(all))
return 0
} }
+61
View File
@@ -0,0 +1,61 @@
//usr/bin/env go run -tags=qa_stats "$0" "$@"; exit
//go:build qa_stats
//
// bin/qa/stats.go - DuckDB quantiles over a JSON number array or JSONL count.
//
// ./bin/qa/stats.go <<< '[1,2,3,4,5]'
// ./bin/qa/stats.go --jsonl rows.jsonl
//
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
// DuckDB CGO needs gcc/g++ (not Zig). After eval "$(bin/cgo/zig env)":
// CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= ./bin/qa/stats.go
package main
import (
"encoding/json"
"fmt"
"io"
"os"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/duckstats"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := cliparse.ParseQAStats(args)
if err != nil {
return cliparse.Fail(err)
}
jsonl := c.JSONL
if jsonl != "" {
n, err := duckstats.CountJSONL(jsonl)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Printf("n: %d\n", n)
return 0
}
raw, err := io.ReadAll(os.Stdin)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
var samples []float64
if err := json.Unmarshal(raw, &samples); err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
s, err := duckstats.Quantiles(samples)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
fmt.Printf("n: %d\nmin: %g\np50: %g\np95: %g\nmax: %g\navg: %g\n",
s.N, s.Min, s.P50, s.P95, s.Max, s.Avg)
return 0
}
+73
View File
@@ -0,0 +1,73 @@
//usr/bin/env go run -tags=reasoner_bakeoff "$0" "$@"; exit
//go:build reasoner_bakeoff
//
// bin/reasoner/bakeoff.go - CPU tool-call bake-off against an OpenAI-compatible URL (D18).
//
// REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b ./bin/reasoner/bakeoff.go
// ./bin/reasoner/bakeoff.go --model MichelRosselli/bonsai-27b:Q1_0 --json
//
// Measures OpenAI tool_calls (search/get/audit) and RSS from Ollama /api/ps, not VRAM.
// PicoClaw is compose profile picoclaw; tool names match internal/httpapi MCP ops.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"encoding/json"
"fmt"
"os"
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/duckstats"
"github.com/eSlider/2dph/internal/reasoner"
)
func main() {
os.Exit(run(os.Args[1:]))
}
func run(args []string) int {
c, err := reasoner.ParseArgs(args)
if err != nil {
return cliparse.Fail(err)
}
base, model, jsonOut, device := c.Base, c.Model, c.JSONOut, c.Device
client := reasoner.Client{BaseURL: base, Model: model, Device: device}
rep := reasoner.Run(client)
lat := make([]float64, 0, len(rep.Prompts))
for _, p := range rep.Prompts {
lat = append(lat, float64(p.LatencyMS))
}
if st, err := duckstats.Quantiles(lat); err == nil {
rep.LatencyP50MS = st.P50
rep.LatencyP95MS = st.P95
}
raw, err := json.MarshalIndent(rep, "", " ")
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
if jsonOut {
fmt.Println(string(raw))
} else {
fmt.Printf("model: %s\n", rep.Model)
fmt.Printf("hf_id: %s\n", rep.HF)
fmt.Printf("device: %s\n", rep.Device)
fmt.Printf("tool_call: %d/%d\n", rep.ToolCallOK, rep.ToolCallN)
fmt.Printf("xml_leak: %d\n", rep.XMLLeak)
fmt.Printf("rss_mb: %d\n", rep.RSSMB)
fmt.Printf("vram_mb: %d\n", rep.VRAMMB)
fmt.Printf("latency_p50_ms: %g\n", rep.LatencyP50MS)
fmt.Printf("latency_p95_ms: %g\n", rep.LatencyP95MS)
for _, p := range rep.Prompts {
status := "fail"
if p.OK {
status = "ok"
}
fmt.Printf(" %s: %s wanted=%s got=%s xml=%v %dms %s\n", p.WantedTool, status, p.WantedTool, p.ToolName, p.XMLLeak, p.LatencyMS, p.Err)
}
}
if rep.ToolCallN == 0 {
return 1
}
return 0
}
+2
View File
@@ -0,0 +1,2 @@
// Commands in this directory are shebang mains (bakeoff.go).
package main
+235
View File
@@ -0,0 +1,235 @@
# bin/stack/lib.sh — compose helpers for start / start-assistant / stop / status.
# Sourced, not executed. No secrets. No host-absolute paths.
BRAIN_URL="${BRAIN_URL:-http://127.0.0.1:8630}"
REASONER_URL="${REASONER_URL:-http://127.0.0.1:11435}"
PICOCLAW_URL="${PICOCLAW_URL:-http://127.0.0.1:18790}"
REASONER_MODEL="${REASONER_MODEL:-qwen3.5:9b}"
STACK_WAIT_SECS="${STACK_WAIT_SECS:-90}"
STACK_WAIT_INTERVAL="${STACK_WAIT_INTERVAL:-2}"
STACK_PULL_SECS="${STACK_PULL_SECS:-600}"
if [[ -z "${ROOT:-}" ]]; then
STACK_DIR="$(CDPATH= cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(CDPATH= cd -- "$STACK_DIR/../.." && pwd)"
fi
stack_usage() {
awk 'NR == 1 { next } /^#/ { sub(/^# ?/, ""); print; next } { exit }' "$1"
}
stack_die() {
echo "bin/stack: $*" >&2
return 1
}
compose() {
docker compose -f "$ROOT/compose.yaml" --project-directory "$ROOT" "$@"
}
http_get() {
local url=$1
local timeout=${2:-5}
curl -sS --max-time "$timeout" "$url" 2>/dev/null || return 1
}
health_ok() {
local url=$1
local timeout=${2:-5}
local body
body=$(http_get "$url" "$timeout") || return 1
printf '%s' "$body" | grep -q '"status":"ok"'
}
wait_health() {
local url=$1
local n=0
while ((n <= STACK_WAIT_SECS)); do
if health_ok "$url"; then
return 0
fi
n=$((n + 1))
if ((n <= STACK_WAIT_SECS)); then
sleep "$STACK_WAIT_INTERVAL"
fi
done
return 1
}
wait_http() {
local url=$1
local n=0
while ((n <= STACK_WAIT_SECS)); do
if http_get "$url" 5 >/dev/null; then
return 0
fi
n=$((n + 1))
if ((n <= STACK_WAIT_SECS)); then
sleep "$STACK_WAIT_INTERVAL"
fi
done
return 1
}
mcp_body() {
curl -sS --max-time 10 \
-H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \
"$BRAIN_URL/mcp" 2>/dev/null || return 1
}
mcp_ok() {
local body
body=$(mcp_body) || return 1
printf '%s' "$body" | grep -Eq '"name": ?"search"' || return 1
printf '%s' "$body" | grep -Eq '"name": ?"get"' || return 1
printf '%s' "$body" | grep -Eq '"name": ?"audit"' || return 1
return 0
}
reasoner_tags() {
http_get "$REASONER_URL/api/tags" 5
}
reasoner_has_model() {
local body
body=$(reasoner_tags) || return 1
printf '%s' "$body" | grep -Fq "$REASONER_MODEL"
}
ensure_mcp() {
mcp_ok || stack_die "MCP tools/list missing search/get/audit at $BRAIN_URL/mcp"
}
ensure_brain() {
if health_ok "$BRAIN_URL/health"; then
echo "brain: reuse $BRAIN_URL" >&2
else
echo "brain: compose up" >&2
compose up -d brain
wait_health "$BRAIN_URL/health" || stack_die "brain health failed at $BRAIN_URL/health"
fi
ensure_mcp
}
pull_reasoner_model() {
echo "reasoner: pulling $REASONER_MODEL (CPU, may take minutes)" >&2
curl -sS --max-time "$STACK_PULL_SECS" \
-H 'Content-Type: application/json' \
-d "{\"name\":\"$REASONER_MODEL\"}" \
"$REASONER_URL/api/pull" >/dev/null
}
ensure_reasoner() {
if reasoner_has_model; then
echo "reasoner: reuse $REASONER_URL model $REASONER_MODEL" >&2
return 0
fi
if ! reasoner_tags >/dev/null; then
echo "reasoner: compose up" >&2
compose --profile reasoner up -d reasoner
wait_http "$REASONER_URL/api/tags" || stack_die "reasoner not listening at $REASONER_URL"
fi
if reasoner_has_model; then
return 0
fi
pull_reasoner_model
reasoner_has_model || stack_die "reasoner missing model $REASONER_MODEL"
}
ensure_picoclaw() {
echo "picoclaw: compose up --no-deps (reuse healthy :8630/:11435)" >&2
compose --profile picoclaw up -d --no-deps picoclaw
wait_health "$PICOCLAW_URL/health" || stack_die "picoclaw health failed at $PICOCLAW_URL/health"
}
stack_status() {
local bh=down mcp=down ph=down present=false
health_ok "$BRAIN_URL/health" && bh=ok
mcp_ok && mcp=ok
reasoner_has_model && present=true
health_ok "$PICOCLAW_URL/health" && ph=ok
cat <<EOF
brain:
url: $BRAIN_URL
health: $bh
mcp: $mcp
reasoner:
url: $REASONER_URL
model: $REASONER_MODEL
present: $present
picoclaw:
url: $PICOCLAW_URL
health: $ph
EOF
}
stack_start() {
ensure_brain
}
stack_attach_agent() {
local opts=()
if [[ -t 0 && -t 1 ]]; then
opts+=(-it)
else
opts+=(-T)
fi
if [[ (! -t 0 || ! -t 1) && $# -eq 0 ]]; then
echo "picoclaw: no TTY. Attach with:" >&2
echo " $ROOT/bin/stack/start-assistant" >&2
echo " docker compose --profile picoclaw exec -it picoclaw picoclaw agent" >&2
return 0
fi
echo "picoclaw: agent (search → get → audit before a factual reply)" >&2
exec docker compose -f "$ROOT/compose.yaml" --project-directory "$ROOT" \
--profile picoclaw exec "${opts[@]}" picoclaw picoclaw agent "$@"
}
stack_start_assistant() {
local attach=1
local agent_args=()
while (($#)); do
case "$1" in
-h | --help)
stack_usage "$ROOT/bin/stack/start-assistant"
return 0
;;
--no-attach)
attach=0
shift
;;
--)
shift
agent_args+=("$@")
break
;;
*)
agent_args+=("$1")
shift
;;
esac
done
stack_start
ensure_reasoner
ensure_picoclaw
stack_status
if ((attach == 0)); then
echo "picoclaw: gateway $PICOCLAW_URL (agent not attached)" >&2
echo "ask the brain: $ROOT/bin/stack/start-assistant" >&2
echo "one-shot: $ROOT/bin/stack/start-assistant -- -m \"search the 2dph brain for LadybugDB\"" >&2
return 0
fi
stack_attach_agent "${agent_args[@]}"
}
stack_stop() {
case "${1:-}" in
-h | --help)
stack_usage "$ROOT/bin/stack/stop"
return 0
;;
esac
echo "stack: stop brain brain-mcp reasoner picoclaw (volumes kept)" >&2
compose --profile picoclaw --profile reasoner stop picoclaw brain-mcp reasoner brain
}
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# bin/stack/start - bring up brain HTTP/MCP and wait until search/get/audit respond.
#
# bin/stack/start
# bin/stack/status
#
# Reuses a healthy process on :8630 (host serve or compose). Does not start
# PicoClaw. Does not rebuild Ladybug.
set -euo pipefail
STACK_DIR="$(CDPATH= cd -- "$(dirname "$0")" && pwd)"
# shellcheck source=lib.sh
source "$STACK_DIR/lib.sh"
case "${1:-}" in
-h | --help)
stack_usage "$0"
exit 0
;;
esac
stack_start "$@"
stack_status
+15
View File
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
# bin/stack/start-assistant - start + CPU reasoner + PicoClaw, then attach agent.
#
# bin/stack/start-assistant
# bin/stack/start-assistant --no-attach
# bin/stack/start-assistant -- -m "search the 2dph brain for LadybugDB"
#
# Pulls qwen3.5:9b if missing. Gateway :18790. Agent uses MCP search → get → audit.
# --no-attach leaves the gateway up without exec.
set -euo pipefail
STACK_DIR="$(CDPATH= cd -- "$(dirname "$0")" && pwd)"
# shellcheck source=lib.sh
source "$STACK_DIR/lib.sh"
stack_start_assistant "$@"
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# bin/stack/status - YAML health for brain MCP, reasoner model, PicoClaw gateway.
#
# bin/stack/status
# bin/stack/status | yq '.picoclaw'
set -euo pipefail
STACK_DIR="$(CDPATH= cd -- "$(dirname "$0")" && pwd)"
# shellcheck source=lib.sh
source "$STACK_DIR/lib.sh"
case "${1:-}" in
-h | --help)
stack_usage "$0"
exit 0
;;
esac
stack_status
Executable
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# bin/stack/stop - stop compose brain / brain-mcp / reasoner / picoclaw.
#
# bin/stack/stop
#
# Volumes kept (kb, reasoner weights, picoclaw-home). Does not kill a host
# bin/brain/serve.go that is not a compose service.
set -euo pipefail
STACK_DIR="$(CDPATH= cd -- "$(dirname "$0")" && pwd)"
# shellcheck source=lib.sh
source "$STACK_DIR/lib.sh"
stack_stop "$@"
+103
View File
@@ -0,0 +1,103 @@
"""D16 contradiction adjudication (same rules as internal/facts)."""
from __future__ import annotations
from typing import Any
CONF_CONFIRMED = "confirmed"
CONF_HYPOTHESIS = "hypothesis"
RULE_UNRESOLVED = "unresolved"
RULE_TEMPORAL = "temporal_freshness"
RULE_AUTHORITY = "authority_pairing"
RULE_TWO_SOURCE = "two_source"
RULE_SINGLE = "single_source"
KIND_RUNTIME = "runtime"
KIND_CONFIG = "config"
KIND_NARRATIVE = "narrative"
def _independent(sources: list[dict]) -> int:
seen: set[str] = set()
for i, s in enumerate(sources):
sid = str(s.get("id") or "") or f"{s.get('kind', '')}#{i}"
seen.add(sid)
return len(seen)
def _fresh_n(sources: list[dict]) -> int:
return sum(1 for s in sources if not s.get("stale"))
def _strong_n(sources: list[dict]) -> int:
return sum(1 for s in sources if s.get("kind") in (KIND_RUNTIME, KIND_CONFIG))
def adjudicate(claim: dict[str, Any]) -> dict[str, Any]:
yes = list(claim.get("yes") or [])
no = list(claim.get("no") or [])
yes_n, no_n = _independent(yes), _independent(no)
text = str(claim.get("text") or "")
def out(conf: str, rule: str, winner: str = "") -> dict[str, Any]:
return {
"text": text,
"confidence": conf,
"confirmed": conf == CONF_CONFIRMED,
"rule": rule,
"winner": winner,
"yes": yes_n,
"no": no_n,
}
if yes_n < 2 or no_n < 2:
if yes_n >= 2:
return out(CONF_CONFIRMED, RULE_TWO_SOURCE, "yes")
if no_n >= 2:
return out(CONF_CONFIRMED, RULE_TWO_SOURCE, "no")
return out(CONF_HYPOTHESIS, RULE_SINGLE)
yf, nf = _fresh_n(yes), _fresh_n(no)
if yf >= 2 and nf < 2:
return out(CONF_CONFIRMED, RULE_TEMPORAL, "yes")
if nf >= 2 and yf < 2:
return out(CONF_CONFIRMED, RULE_TEMPORAL, "no")
ys, ns = _strong_n(yes), _strong_n(no)
if ys >= 2 and ns < 2:
return out(CONF_CONFIRMED, RULE_AUTHORITY, "yes")
if ns >= 2 and ys < 2:
return out(CONF_CONFIRMED, RULE_AUTHORITY, "no")
return out(CONF_HYPOTHESIS, RULE_UNRESOLVED)
def parse_source_field(source: str) -> tuple[str, str]:
"""Split `a x b vs c x d` into (yes, no). Empty no if no ` vs `."""
if " vs " not in source:
return source, ""
yes, _, no = source.partition(" vs ")
return yes.strip(), no.strip()
def check_fact_row(lid: str, source: str, loc: str, how: str, conf: str) -> list[str]:
"""Lexicon checks for one facts leaf (no Ladybug)."""
problems: list[str] = []
src = source or ""
if conf == CONF_CONFIRMED:
if " vs " in src:
problems.append(f"{lid}: confirmed fact cannot keep a vs-contradiction")
if " x " not in src:
problems.append(f"{lid}: needs 2-source evidence in source, got '{source}'")
elif conf == CONF_HYPOTHESIS:
yes, no = parse_source_field(src)
if not no or " x " not in yes or " x " not in no:
problems.append(
f"{lid}: hypothesis contradiction needs 'a x b vs c x d', got '{source}'"
)
elif conf == "partial":
pass
else:
problems.append(f"{lid}: unknown confidence '{conf}'")
if not loc:
problems.append(f"{lid}: missing loc (evidence pointer)")
if not how:
problems.append(f"{lid}: missing how")
return problems
+165 -12
View File
@@ -4,7 +4,7 @@ Single embedded graph `var/kb.lbug`. Two roots: facts (assertions backed by
>=2 independent sources) and info (narrative leafs). Hybrid retrieval: BM25 >=2 independent sources) and info (narrative leafs). Hybrid retrieval: BM25
(FTS extension) + HNSW cosine (VECTOR extension) + Cypher graph hops. (FTS extension) + HNSW cosine (VECTOR extension) + Cypher graph hops.
All access is read-only unless `--rebuild` is passed to kb/index. All access is read-only unless `--rebuild` (kb/index) or `kb/add`.
""" """
from __future__ import annotations from __future__ import annotations
@@ -62,7 +62,9 @@ def init_schema(conn: ladybug.Connection) -> None:
"CREATE NODE TABLE IF NOT EXISTS Leaf (" "CREATE NODE TABLE IF NOT EXISTS Leaf ("
" id STRING, text STRING, root STRING, confidence STRING, " " id STRING, text STRING, root STRING, confidence STRING, "
" sha256 STRING, source STRING, source_rev STRING, observed_at STRING, " " sha256 STRING, source STRING, source_rev STRING, observed_at STRING, "
" how STRING, loc STRING, type STRING, embedding FLOAT[256], " " how STRING, loc STRING, type STRING, "
" valid_from STRING, valid_to STRING, "
" embedding FLOAT[256], "
" PRIMARY KEY(id))" " PRIMARY KEY(id))"
) )
conn.execute( conn.execute(
@@ -91,6 +93,46 @@ def init_schema(conn: ladybug.Connection) -> None:
conn.execute( conn.execute(
"CREATE REL TABLE IF NOT EXISTS AUTHORED (FROM Commit TO Person)" "CREATE REL TABLE IF NOT EXISTS AUTHORED (FROM Commit TO Person)"
) )
ensure_interval_columns(conn)
def ensure_interval_columns(conn: ladybug.Connection) -> None:
"""D24: add valid_from/valid_to on older Leaf tables (idempotent ALTER)."""
for col in ("valid_from", "valid_to"):
try:
conn.execute(f"ALTER TABLE Leaf ADD {col} STRING")
except Exception:
pass
def normalize_day(s: str) -> str:
s = (s or "").strip()
if len(s) >= 10 and s[4] == "-" and s[7] == "-":
return s[:10]
return s
def active_at(valid_from: str, valid_to: str, as_of: str) -> bool:
"""D24: fact interval of truth. Empty ends = always; empty as_of = no filter."""
as_of = normalize_day(as_of)
if not as_of:
return True
fro = normalize_day(valid_from)
to = normalize_day(valid_to)
if fro and as_of < fro:
return False
if to and as_of > to:
return False
return True
def filter_as_of(hits: list[dict], as_of: str) -> list[dict]:
if not as_of:
return hits
return [
h for h in hits
if active_at(str(h.get("valid_from") or ""), str(h.get("valid_to") or ""), as_of)
]
def leaf_id(text: str, source: str) -> str: def leaf_id(text: str, source: str) -> str:
@@ -99,25 +141,124 @@ def leaf_id(text: str, source: str) -> str:
def upsert_leaf(conn: ladybug.Connection, *, text: str, root: str, confidence: str, def upsert_leaf(conn: ladybug.Connection, *, text: str, root: str, confidence: str,
source: str, source_rev: str, how: str, loc: str, type_: str, source: str, source_rev: str, how: str, loc: str, type_: str,
embedding: list[float] | None) -> str: embedding: list[float] | None,
valid_from: str = "", valid_to: str = "") -> str:
lid = leaf_id(text, source) lid = leaf_id(text, source)
obs = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) obs = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
vf = normalize_day(valid_from)
vt = normalize_day(valid_to)
conn.execute( conn.execute(
"MERGE (l:Leaf {id:$id}) " "MERGE (l:Leaf {id:$id}) "
"SET l.text=$text, l.root=$root, l.confidence=$confidence, " "SET l.text=$text, l.root=$root, l.confidence=$confidence, "
" l.sha256=$sha, l.source=$source, l.source_rev=$rev, l.observed_at=$obs, " " l.sha256=$sha, l.source=$source, l.source_rev=$rev, l.observed_at=$obs, "
" l.how=$how, l.loc=$location, l.type=$type" " l.how=$how, l.loc=$location, l.type=$type, "
" l.valid_from=$vf, l.valid_to=$vt"
+ (", l.embedding=$emb" if embedding else ""), + (", l.embedding=$emb" if embedding else ""),
parameters={ parameters={
"id": lid, "text": text, "root": root, "confidence": confidence, "id": lid, "text": text, "root": root, "confidence": confidence,
"sha": sha256_b64(text), "source": source, "rev": source_rev, "sha": sha256_b64(text), "source": source, "rev": source_rev,
"obs": obs, "how": how, "location": loc, "type": type_, "obs": obs, "how": how, "location": loc, "type": type_,
"vf": vf, "vt": vt,
"emb": (embedding if embedding else None), "emb": (embedding if embedding else None),
}, },
) )
return lid return lid
def add_leafs(conn: ladybug.Connection, leafs: list[dict]) -> list[str]:
"""Write facts+info leafs in one transaction. Safe while FTS/HNSW exist.
Each leaf dict: text, source, optional root/confidence/source_rev/how/loc/type/
embedding/valid_from/valid_to. Does not delete the database file. Measured on
Ladybug 0.19: MERGE of new ids (and updates) stays FTS+HNSW queryable; DROP
INDEX is the fatal path.
"""
if not leafs:
return []
started = False
try:
conn.execute("BEGIN TRANSACTION")
started = True
except Exception:
started = False
ids: list[str] = []
try:
for lf in leafs:
ids.append(
upsert_leaf(
conn,
text=str(lf["text"]),
root=str(lf.get("root") or ROOT_INFO),
confidence=str(lf.get("confidence") or CONF_CONFIRMED),
source=str(lf["source"]),
source_rev=str(lf.get("source_rev") or "working-tree"),
how=str(lf.get("how") or "brain/add"),
loc=str(lf.get("loc") or lf.get("source") or ""),
type_=str(lf.get("type") or lf.get("type_") or "reference"),
embedding=lf.get("embedding"),
valid_from=str(lf.get("valid_from") or ""),
valid_to=str(lf.get("valid_to") or ""),
)
)
if started:
conn.execute("COMMIT")
except Exception:
if started:
try:
conn.execute("ROLLBACK")
except Exception:
pass
raise
return ids
def file_id(repo: str, path: str) -> str:
"""Stable File.id matching gitimport (`repo:path`)."""
return f"{repo}:{path}" if repo else path
def link_from_file(conn: ladybug.Connection, leaf_id: str, path: str,
repo: str = "", mtime: str = "") -> str:
"""MERGE File and Leaf-[:FROM_FILE]->File so --hop 1 can walk."""
fid = file_id(repo, path)
conn.execute(
"MERGE (f:File {id:$id}) SET f.path=$path, f.repo=$repo, f.mtime=$mtime",
parameters={"id": fid, "path": path, "repo": repo, "mtime": mtime},
)
conn.execute(
"MATCH (l:Leaf {id:$lid}), (f:File {id:$fid}) "
"MERGE (l)-[:FROM_FILE]->(f)",
parameters={"lid": leaf_id, "fid": fid},
)
return fid
HOP_STMTS = {
1: "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File) RETURN f.id, f.path, 1",
2: ("MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit) "
"RETURN c.id, c.subject, 2"),
3: ("MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit)"
"-[:AUTHORED]->(p:Person) RETURN p.id, p.name, 3"),
}
HOP_LABELS = {1: "File", 2: "Commit", 3: "Person"}
def hop_walk(conn: ladybug.Connection, leaf_id: str, n: int) -> list[dict]:
"""Walk Leaf → File → Commit → Person up to n hops (max 3)."""
depth = min(max(int(n), 0), 3)
out: list[dict] = []
for d in range(1, depth + 1):
rows = conn.execute(HOP_STMTS[d], parameters={"id": leaf_id}).get_all()
for row in rows:
out.append({
"id": row[0],
"label": HOP_LABELS[d],
"name": row[1],
"depth": int(row[2]),
})
return out
def leaf_index_names(conn: ladybug.Connection) -> set[str]: def leaf_index_names(conn: ladybug.Connection) -> set[str]:
"""Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'}).""" """Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'})."""
rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all() rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()
@@ -194,29 +335,39 @@ def drop_indexes(conn: ladybug.Connection) -> None:
def query_fts(conn: ladybug.Connection, text: str, limit: int = 10) -> list[dict]: def query_fts(conn: ladybug.Connection, text: str, limit: int = 10) -> list[dict]:
r = conn.execute( r = conn.execute(
"CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " "CALL QUERY_FTS_INDEX('Leaf', 'id', $q) "
"RETURN node.id, node.text, node.root, score ORDER BY score DESC LIMIT $n", "RETURN node.id, node.text, node.root, score, node.valid_from, node.valid_to "
"ORDER BY score DESC LIMIT $n",
parameters={"q": text, "n": limit}, parameters={"q": text, "n": limit},
) )
return [{"id": row[0], "text": row[1], "root": row[2], "score": row[3]} for row in r.get_all()] return [
{
"id": row[0], "text": row[1], "root": row[2], "score": row[3],
"valid_from": row[4] or "", "valid_to": row[5] or "",
}
for row in r.get_all()
]
def query_vector(conn: ladybug.Connection, embedding: list[float], limit: int = 10) -> list[dict]: def query_vector(conn: ladybug.Connection, embedding: list[float], limit: int = 10) -> list[dict]:
r = conn.execute( r = conn.execute(
"CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) "
"RETURN node.id, node.text, node.root, distance ORDER BY distance LIMIT $n", "RETURN node.id, node.text, node.root, distance, node.valid_from, node.valid_to "
"ORDER BY distance LIMIT $n",
parameters={"q": embedding, "n": limit}, parameters={"q": embedding, "n": limit},
) )
out = [] out = []
for row in r.get_all(): for row in r.get_all():
# distance -> similarity reasonable for cosine
score = 1.0 - row[3] if row[3] is not None else 0.0 score = 1.0 - row[3] if row[3] is not None else 0.0
out.append({"id": row[0], "text": row[1], "root": row[2], "score": score}) out.append({
"id": row[0], "text": row[1], "root": row[2], "score": score,
"valid_from": row[4] or "", "valid_to": row[5] or "",
})
return out return out
def hybrid_search(conn: ladybug.Connection, embedding: list[float], fts_hits: list[dict], def hybrid_search(conn: ladybug.Connection, embedding: list[float], fts_hits: list[dict],
limit: int = 10) -> list[dict]: limit: int = 10, as_of: str = "") -> list[dict]:
"""Merge FTS + vector by reciprocal rank fusion.""" """Merge FTS + vector by reciprocal rank fusion; optional D24 as-of filter."""
fused: dict[str, dict] = {} fused: dict[str, dict] = {}
for rank, hit in enumerate(fts_hits): for rank, hit in enumerate(fts_hits):
fused.setdefault(hit["id"], {**hit, "rrf": 0.0})["rrf"] = 1.0 / (60 + rank + 1) fused.setdefault(hit["id"], {**hit, "rrf": 0.0})["rrf"] = 1.0 / (60 + rank + 1)
@@ -224,8 +375,10 @@ def hybrid_search(conn: ladybug.Connection, embedding: list[float], fts_hits: li
entry = fused.setdefault(hit["id"], {**hit, "rrf": 0.0}) entry = fused.setdefault(hit["id"], {**hit, "rrf": 0.0})
entry["rrf"] += 1.0 / (60 + rank + 1) entry["rrf"] += 1.0 / (60 + rank + 1)
entry.setdefault("score", hit.get("score", 0.0)) entry.setdefault("score", hit.get("score", 0.0))
entry.setdefault("valid_from", hit.get("valid_from") or "")
entry.setdefault("valid_to", hit.get("valid_to") or "")
ranked = sorted(fused.values(), key=lambda h: h.get("rrf", 0.0), reverse=True) ranked = sorted(fused.values(), key=lambda h: h.get("rrf", 0.0), reverse=True)
return ranked[:limit] return filter_as_of(ranked, as_of)[:limit]
def stats(conn: ladybug.Connection) -> dict: def stats(conn: ladybug.Connection) -> dict:
+84 -1
View File
@@ -7,7 +7,10 @@ offline against fixtures.
from __future__ import annotations from __future__ import annotations
import html import html
import os
import re import re
import subprocess
import tempfile
import zipfile import zipfile
from pathlib import Path from pathlib import Path
@@ -18,9 +21,10 @@ OFFICE_SUFFIXES = {".docx", ".pptx", ".xlsx", ".html", ".htm", ".epub", ".eml",
PDF_SUFFIXES = {".pdf"} PDF_SUFFIXES = {".pdf"}
IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".tif", ".webp"} IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".tif", ".webp"}
ARCHIVE_SUFFIXES = {".zip"} ARCHIVE_SUFFIXES = {".zip"}
# Legacy binary Office (doc/xls/ppt) — markitdown/docling skip them; we try # Legacy binary Office (doc/xls/ppt) — markitdown skip them; we try
# pandoc first, else leave a stub. # pandoc first, else leave a stub.
LEGACY_OFFICE_SUFFIXES = {".doc", ".xls", ".ppt"} LEGACY_OFFICE_SUFFIXES = {".doc", ".xls", ".ppt"}
TESS_LANG = "eng+deu"
CONVERTIBLE_SUFFIXES = ( CONVERTIBLE_SUFFIXES = (
TEXT_SUFFIXES | OFFICE_SUFFIXES | PDF_SUFFIXES | IMAGE_SUFFIXES | ARCHIVE_SUFFIXES | LEGACY_OFFICE_SUFFIXES TEXT_SUFFIXES | OFFICE_SUFFIXES | PDF_SUFFIXES | IMAGE_SUFFIXES | ARCHIVE_SUFFIXES | LEGACY_OFFICE_SUFFIXES
@@ -146,3 +150,82 @@ def zip_extract_safe(zip_path: Path, dest: Path) -> list[Path]:
def is_convertible(suffix: str) -> bool: def is_convertible(suffix: str) -> bool:
return suffix.lower() in CONVERTIBLE_SUFFIXES return suffix.lower() in CONVERTIBLE_SUFFIXES
def convert_pdf(path: Path, ocr: bool = False) -> str:
"""pdftotext -layout first; empty text layer → pdftoppm + tesseract.
`ocr` is unused for born-digital PDFs (text layer wins). Scans OCR
automatically. This path never execs an ONNX document converter.
"""
del ocr # scans OCR when the text layer is empty; flag is for images
text = pdf_fast_text(path)
if text and text.strip():
return normalize_markdown(text)
scanned = ocr_pdf(path)
if scanned and scanned.strip():
return normalize_markdown(scanned)
if text:
return normalize_markdown(text)
return "\n<!-- pdf has no text layer (ocr unavailable) -->\n"
def pdf_fast_text(path: Path) -> str | None:
"""pdftotext -layout; None when poppler is missing or the command fails."""
try:
proc = subprocess.run(
["pdftotext", "-layout", str(path), "-"],
capture_output=True, timeout=60)
except (OSError, subprocess.TimeoutExpired):
return None
if proc.returncode != 0:
return None
return proc.stdout.decode("utf-8", errors="replace")
def ocr_pdf(path: Path) -> str:
"""Rasterize with pdftoppm and OCR each page (tesseract or paddle)."""
try:
with tempfile.TemporaryDirectory(prefix="2dph-ocr-") as tmp:
prefix = str(Path(tmp) / "page")
proc = subprocess.run(
["pdftoppm", "-png", "-r", "200", str(path), prefix],
capture_output=True, timeout=120)
if proc.returncode != 0:
return ""
pages = sorted(Path(tmp).glob("page*.png"))
parts = [ocr_image(p) for p in pages]
return "\n\n".join(p for p in parts if p and p.strip())
except (OSError, subprocess.TimeoutExpired):
return ""
def ocr_image(path: Path) -> str:
engine = os.environ.get("OCR_ENGINE", "tesseract")
if engine == "paddle":
return _ocr_paddle(path)
return _ocr_tesseract(path)
def _ocr_tesseract(path: Path) -> str:
try:
proc = subprocess.run(
["tesseract", str(path), "stdout", "-l", TESS_LANG, "--psm", "6"],
capture_output=True, timeout=120)
except (OSError, subprocess.TimeoutExpired):
return ""
if proc.returncode != 0:
return ""
return proc.stdout.decode("utf-8", errors="replace").strip()
def _ocr_paddle(path: Path) -> str:
try:
proc = subprocess.run(
["paddleocr", "ocr", "-i", str(path)],
capture_output=True, timeout=180)
except (OSError, subprocess.TimeoutExpired):
return ""
if proc.returncode != 0:
return ""
return proc.stdout.decode("utf-8", errors="replace").strip()
+159 -2
View File
@@ -1,6 +1,7 @@
"""D14 layout: bin/{subject}/{method}.go, libs in internal/, one go.mod.""" """D14 layout: bin/{subject}/{method}.go, libs in internal/, one go.mod."""
from __future__ import annotations from __future__ import annotations
import os
import unittest import unittest
from pathlib import Path from pathlib import Path
@@ -74,9 +75,20 @@ class BinLayoutTest(unittest.TestCase):
) )
def test_brain_methods_are_shebangs(self) -> None: def test_brain_methods_are_shebangs(self) -> None:
for method in ("index.go", "get.go", "stats.go", "eval.go", "watch.go"): for method in ("index.go", "add.go", "get.go", "stats.go", "eval.go", "watch.go"):
self._assert_shebang(f"bin/brain/{method}") self._assert_shebang(f"bin/brain/{method}")
def test_brain_add_is_python_write_not_rebuild(self) -> None:
self._assert_shebang("bin/brain/add.go")
text = (ROOT / "bin" / "brain" / "add.go").read_text()
self.assertIn("cmdbin.ExecFile", text)
self.assertIn("bin/kb/add", text)
self.assertNotIn("--rebuild", text)
py = (ROOT / "bin" / "kb" / "add").read_text()
self.assertIn("add_leafs", py)
self.assertIn("--json", py)
self.assertNotIn("unlink", py.lower())
def test_brain_get_stats_eval_are_not_python_exec(self) -> None: def test_brain_get_stats_eval_are_not_python_exec(self) -> None:
for method in ("get.go", "stats.go", "eval.go"): for method in ("get.go", "stats.go", "eval.go"):
text = (ROOT / "bin" / "brain" / method).read_text() text = (ROOT / "bin" / "brain" / method).read_text()
@@ -115,6 +127,38 @@ class BinLayoutTest(unittest.TestCase):
self.assertIn("cmdbin.ExecFile", text) self.assertIn("cmdbin.ExecFile", text)
self.assertIn(f"bin/facts/{method.removesuffix('.go')}", text) self.assertIn(f"bin/facts/{method.removesuffix('.go')}", text)
def test_d16_adjudication_is_cgo_free(self) -> None:
self.assertTrue((ROOT / "internal" / "facts" / "contradict.go").is_file())
go = (ROOT / "internal" / "facts" / "contradict.go").read_text()
py = (ROOT / "bin" / "tools" / "contradict.py").read_text()
audit = (ROOT / "bin" / "facts" / "audit").read_text()
for token in ("temporal_freshness", "authority_pairing", "unresolved"):
self.assertIn(token, go)
self.assertIn(token, py)
self.assertIn("contradict", audit)
self.assertIn(" vs ", py)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("temporal_freshness", plan)
self.assertIn("authority_pairing", plan)
shebang = (ROOT / "bin" / "facts" / "audit.go").read_text()
self.assertIn("contradict", shebang)
def test_d23_flaggy_cli(self) -> None:
self.assertTrue((ROOT / "internal" / "cli" / "cli.go").is_file())
self.assertIn("github.com/integrii/flaggy", (ROOT / "go.mod").read_text())
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D23", plan)
self.assertIn("flaggy", plan)
complete = (ROOT / "bin" / "cli" / "complete.go").read_text()
first = complete.splitlines()[0]
self.assertTrue(first.startswith("//usr/bin/env go run"), first)
self.assertIn("complete.go bash", complete)
self.assertIn("brain-search", complete)
chats_import = (ROOT / "internal" / "chats" / "import.go").read_text()
self.assertNotIn("flag.NewFlagSet", chats_import)
args = (ROOT / "internal" / "brain" / "rank" / "args.go").read_text()
self.assertIn("internal/cli", args)
def test_mail_import_is_shebang_not_brain_write(self) -> None: def test_mail_import_is_shebang_not_brain_write(self) -> None:
self._assert_shebang("bin/mail/import.go") self._assert_shebang("bin/mail/import.go")
index_mail = (ROOT / "bin" / "mail" / "index_mail").read_text() index_mail = (ROOT / "bin" / "mail" / "index_mail").read_text()
@@ -124,8 +168,55 @@ class BinLayoutTest(unittest.TestCase):
"index_mail must point at bin/brain/index.go", "index_mail must point at bin/brain/index.go",
) )
def test_markdown_import_is_shebang(self) -> None: def test_mail_ocr_is_tesseract_not_docling(self) -> None:
self._assert_shebang("bin/mail/ocr.go")
ocr = (ROOT / "bin" / "mail" / "ocr.go").read_text()
self.assertIn("internal/ocr", ocr)
self.assertIn("mail_ocr", ocr)
self.assertNotIn("github.com/otiai10/gosseract", ocr)
py = (ROOT / "bin" / "mail" / "import").read_text()
self.assertNotIn("from docling", py)
self.assertNotIn("import docling", py)
self.assertIn("convert_pdf", py)
conv = (ROOT / "bin" / "tools" / "mailconv.py").read_text()
self.assertIn("pdftotext", conv)
self.assertIn("pdftoppm", conv)
self.assertIn("tesseract", conv)
self.assertIn("eng+deu", conv)
self.assertNotIn("from docling", conv)
self.assertNotIn("import docling", conv)
self.assertNotIn("gocv", conv.lower())
proj = (ROOT / "pyproject.toml").read_text()
self.assertNotIn("docling", proj)
ci = (ROOT / ".github" / "workflows" / "ci.yml").read_text()
self.assertIn("tesseract-ocr", ci)
self.assertIn("./internal/ocr", ci)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn("ocr-paddle", compose)
self.assertIn("OCR_ENGINE", compose)
def test_markdown_import_is_go_not_python_exec(self) -> None:
self._assert_shebang("bin/markdown/import.go") self._assert_shebang("bin/markdown/import.go")
text = (ROOT / "bin" / "markdown" / "import.go").read_text()
self.assertNotIn("ExecFile", text)
self.assertNotIn("cmdbin", text)
self.assertIn("internal/mdleaves", text)
self.assertNotIn("kb.lbug", text)
def test_import_adapters_do_not_write_ladybug(self) -> None:
for rel in (
"bin/mail/import.go",
"bin/mail/import",
"bin/markdown/import.go",
"bin/chats/import.go",
"bin/git/import.go",
):
text = (ROOT / rel).read_text()
self.assertNotIn("upsert_leaf", text, rel)
self.assertNotIn("kb.lbug", text, rel)
self.assertNotIn("var/brain.lbug", text, rel)
index = (ROOT / "bin" / "brain" / "index.go").read_text()
self.assertIn("bin/kb/index", index)
def test_postgres_query_is_shebang(self) -> None: def test_postgres_query_is_shebang(self) -> None:
self._assert_shebang("bin/postgres/query.go") self._assert_shebang("bin/postgres/query.go")
@@ -157,3 +248,69 @@ class BinLayoutTest(unittest.TestCase):
for line in first.splitlines(): for line in first.splitlines():
if "go-git/go-git" in line: if "go-git/go-git" in line:
self.assertNotIn("indirect", line) self.assertNotIn("indirect", line)
def test_duckdb_go_is_direct_require(self) -> None:
text = (ROOT / "go.mod").read_text()
first = text.split("require (")[1].split(")")[0]
self.assertRegex(first, r"github.com/duckdb/duckdb-go/v2\s+v")
for line in first.splitlines():
if "duckdb/duckdb-go" in line:
self.assertNotIn("indirect", line)
skill = (ROOT / "skills" / "duckdb" / "SKILL.md").read_text()
self.assertIn("github.com/duckdb/duckdb-go", skill)
self.assertIn("Ladybug", skill)
self.assertIn("sqlite", skill.lower())
self.assertIn("gcc", skill.lower())
self.assertIn("Zig", skill)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D22", plan)
self.assertIn("duckdb-go", plan)
self._assert_shebang("bin/qa/stats.go")
reasoner = (ROOT / "internal" / "reasoner" / "client.go").read_text()
self.assertNotIn("duckdb", reasoner)
self.assertNotIn("duckstats", reasoner)
bakeoff = (ROOT / "bin" / "reasoner" / "bakeoff.go").read_text()
self.assertIn("internal/duckstats", bakeoff)
webcache = (ROOT / "internal" / "websearch" / "cache.go").read_text()
self.assertNotIn("duckdb", webcache)
self.assertIn("modernc.org/sqlite", webcache)
def test_cgo_uses_zig_not_gcc(self) -> None:
for rel in ("bin/cgo/zig", "bin/cgo/zcc", "bin/cgo/zc++"):
p = ROOT / rel
self.assertTrue(p.is_file(), f"missing {rel}")
self.assertTrue(
os.access(p, os.X_OK),
f"{rel} must be executable",
)
zig = (ROOT / "bin" / "cgo" / "zig").read_text()
self.assertIn("zig cc", zig)
self.assertIn("0.14.1", zig)
zcc = (ROOT / "bin" / "cgo" / "zcc").read_text()
self.assertIn('exec "$ZIG" cc', zcc)
self.assertNotIn("command -v gcc", zcc)
search = (ROOT / "bin" / "kb" / "search").read_text()
self.assertIn("bin/cgo/zig", search)
self.assertNotIn("command -v gcc", search)
def test_ci_recall_sot_is_zig_brain_eval(self) -> None:
ci = (ROOT / ".github" / "workflows" / "ci.yml").read_text()
self.assertIn("bin/brain/eval.go", ci)
self.assertIn("system_ladybug,brain_eval", ci)
self.assertIn("/tmp/brain-eval", ci)
self.assertIn("KB_ROOT", ci)
self.assertNotIn("bin/kb/eval", ci)
self.assertNotIn("gate skipped", ci)
self.assertIn("./bin/facts/audit self", ci)
def test_eval_fragments_live_in_default_corpus(self) -> None:
"""CI --rebuild indexes README/PLAN/docs/skills; fragments must be there."""
corpus = []
for rel in ("README.md", "PLAN.md", "AGENTS.md"):
corpus.append((ROOT / rel).read_text())
for d in ("docs", "skills"):
for p in (ROOT / d).rglob("*.md"):
corpus.append(p.read_text())
blob = "\n".join(corpus)
for frag in ("BM25", "DevOps", "LadybugDB"):
self.assertIn(frag, blob, f"{frag} must appear in default index corpus")
+104
View File
@@ -0,0 +1,104 @@
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(__file__))
from contradict import ( # noqa: E402
RULE_AUTHORITY,
RULE_SINGLE,
RULE_TEMPORAL,
RULE_TWO_SOURCE,
RULE_UNRESOLVED,
adjudicate,
check_fact_row,
parse_source_field,
)
def src(i, kind, stale=False):
return {"id": i, "kind": kind, "stale": stale}
class TestContradict(unittest.TestCase):
def test_two_vs_two_stays_hypothesis(self):
r = adjudicate({
"text": "svc listens on 443",
"yes": [src("docker-ps", "runtime"), src("compose", "config")],
"no": [src("docker-old", "runtime"), src("compose-old", "config")],
})
self.assertFalse(r["confirmed"])
self.assertEqual(r["rule"], RULE_UNRESOLVED)
self.assertEqual(r["winner"], "")
def test_temporal_freshness(self):
r = adjudicate({
"text": "svc listens on 443",
"yes": [src("docker-ps", "runtime"), src("compose", "config")],
"no": [src("old-readme", "narrative", True), src("old-wiki", "narrative", True)],
})
self.assertTrue(r["confirmed"])
self.assertEqual(r["rule"], RULE_TEMPORAL)
self.assertEqual(r["winner"], "yes")
def test_authority_pairing(self):
r = adjudicate({
"text": "svc listens on 443",
"yes": [src("docker-ps", "runtime"), src("compose", "config")],
"no": [src("readme", "narrative"), src("wiki", "narrative")],
})
self.assertTrue(r["confirmed"])
self.assertEqual(r["rule"], RULE_AUTHORITY)
self.assertEqual(r["winner"], "yes")
def test_two_source_and_single(self):
two = adjudicate({
"text": "arc-1 runs Matrix",
"yes": [src("compose", "config"), src("docker-ps", "runtime")],
})
self.assertTrue(two["confirmed"])
self.assertEqual(two["rule"], RULE_TWO_SOURCE)
one = adjudicate({"text": "maybe", "yes": [src("readme", "narrative")]})
self.assertFalse(one["confirmed"])
self.assertEqual(one["rule"], RULE_SINGLE)
def test_parse_source_field(self):
yes, no = parse_source_field("docker ps x compose.yml vs old.md x wiki.md")
self.assertIn(" x ", yes)
self.assertIn(" x ", no)
def test_check_fact_row_allows_hypothesis_vs(self):
p = check_fact_row(
"L1", "a.md x b.md vs c.md x d.md", "var/", "audit", "hypothesis",
)
self.assertEqual(p, [])
p = check_fact_row("L2", "a.md x b.md", "var/", "audit", "confirmed")
self.assertEqual(p, [])
p = check_fact_row("L3", "a.md x b.md vs c.md x d.md", "var/", "audit", "confirmed")
self.assertTrue(any("vs-contradiction" in x for x in p))
p = check_fact_row("L4", "only-one.md", "var/", "audit", "hypothesis")
self.assertTrue(any("a x b vs" in x for x in p))
def test_audit_contradict_cli_unresolved(self):
import json
import subprocess
from pathlib import Path
root = Path(__file__).resolve().parents[2]
payload = json.dumps({
"text": "svc 443",
"yes": [src("a", "runtime"), src("b", "config")],
"no": [src("c", "runtime"), src("d", "config")],
})
proc = subprocess.run(
[sys.executable, str(root / "bin" / "facts" / "audit"), "contradict", "--json"],
input=payload, capture_output=True, text=True, check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
out = json.loads(proc.stdout)
self.assertTrue(out["ok"])
self.assertEqual(out["contradictions"][0]["rule"], RULE_UNRESOLVED)
self.assertFalse(out["contradictions"][0]["confirmed"])
if __name__ == "__main__":
unittest.main()
+97
View File
@@ -0,0 +1,97 @@
"""Import adapters write files only. Index rebuild is brain/index (D14 / Gitea #7)."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class IndexAdapterTest(unittest.TestCase):
def test_dry_run_fixture_corpus_does_not_write_lbug(self) -> None:
tmp = Path(tempfile.mkdtemp())
(tmp / "note.md").write_text("# Fixture\n\n## Leaf\n\nhello corpus\n", encoding="utf-8")
lbug = tmp / "kb.lbug"
try:
import ladybug # noqa: F401
except ImportError:
venv_py = ROOT / ".venv" / "bin" / "python"
if not venv_py.is_file():
self.skipTest("ladybug missing")
py = str(venv_py)
else:
py = sys.executable
proc = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "index"), "--dry-run", "--json", "--corpus", str(tmp)],
cwd=ROOT,
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
msg = json.loads(proc.stdout)
self.assertTrue(msg.get("dry_run"))
self.assertGreaterEqual(msg.get("corpus_total", 0), 1)
self.assertFalse(lbug.exists(), "dry-run must not create a Ladybug file")
def test_facts_json_and_chats_land_on_rebuild(self) -> None:
"""Gitea #18: facts (2-source) + chats markdown become leafs on rebuild."""
tmp = Path(tempfile.mkdtemp())
dbpath = tmp / "kb.lbug"
chats = tmp / "chats"
chats.mkdir()
(chats / "alice.md").write_text(
"# Chat\n\n## Alice and Bob\n\nhello from chats fixture unique-chat-token\n",
encoding="utf-8",
)
facts_path = tmp / "facts.json"
facts_path.write_text(json.dumps([{
"text": "container 'brain' unique-fact-token is running and declared in compose.yaml",
"source": "docker ps x compose.yaml",
"loc": "compose.yaml:brain",
"how": "facts/extract",
}]), encoding="utf-8")
venv_py = ROOT / ".venv" / "bin" / "python"
py = str(venv_py) if venv_py.is_file() else sys.executable
proc = subprocess.run(
[
py, str(ROOT / "bin" / "kb" / "index"),
"--rebuild", "--db", str(dbpath), "--no-defaults",
"--with-chats", str(chats),
"--facts-json", str(facts_path),
"--json",
],
cwd=ROOT,
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
msg = json.loads(proc.stdout)
self.assertGreaterEqual(msg.get("facts_leafs", 0), 1)
self.assertGreaterEqual(msg.get("chat_leafs", 0), 1)
self.assertTrue(dbpath.exists())
sys.path.insert(0, str(ROOT / "bin" / "tools"))
import kblib
db, conn = kblib.connect(dbpath, read_only=True)
try:
stats = kblib.stats(conn)
self.assertGreaterEqual(stats["by_root"].get("facts", 0), 1)
fts = kblib.query_fts(conn, "unique-chat-token", 5)
self.assertTrue(fts, "chats markdown must be FTS-searchable")
fact_hits = kblib.query_fts(conn, "unique-fact-token", 5)
self.assertTrue(any(h.get("root") == "facts" for h in fact_hits))
src = conn.execute(
"MATCH (l:Leaf {root:'facts'}) RETURN l.source"
).get_all()
self.assertTrue(any(" x " in str(r[0]) for r in src))
finally:
conn.close()
db.close()
+69
View File
@@ -0,0 +1,69 @@
"""Incremental add writes leafs without deleting kb.lbug."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class KbAddCLITest(unittest.TestCase):
def test_json_add_does_not_delete_db(self) -> None:
tmp = Path(tempfile.mkdtemp())
dbpath = tmp / "kb.lbug"
py = sys.executable
venv_py = ROOT / ".venv" / "bin" / "python"
if venv_py.is_file():
py = str(venv_py)
payload = {
"text": "cli zebra leaf",
"root": "info",
"source": "cli-test",
"confidence": "confirmed",
"how": "test",
"loc": str(tmp),
"type": "reference",
"embedding": [0.0] * 256,
}
payload["embedding"][0] = 0.3
proc = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "add"), "--db", str(dbpath), "--json"],
cwd=ROOT,
input=json.dumps(payload),
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(proc.returncode, 0, proc.stderr)
self.assertTrue(dbpath.exists(), "add must create the db, not skip write")
out = json.loads(proc.stdout)
self.assertEqual(out.get("mode"), "add")
self.assertEqual(len(out.get("ids") or []), 1)
again = subprocess.run(
[py, str(ROOT / "bin" / "kb" / "add"), "--db", str(dbpath), "--json"],
cwd=ROOT,
input=json.dumps({
**payload,
"text": "second moose leaf",
"source": "cli-test-2",
}),
capture_output=True,
text=True,
env=os.environ.copy(),
check=False,
)
self.assertEqual(again.returncode, 0, again.stderr)
self.assertTrue(dbpath.exists())
second = json.loads(again.stdout)
self.assertEqual(len(second.get("ids") or []), 1)
self.assertNotEqual(out["ids"][0], second["ids"][0])
if __name__ == "__main__":
unittest.main()
+128
View File
@@ -71,6 +71,69 @@ class KblibTest(unittest.TestCase):
self.assertTrue(hits) self.assertTrue(hits)
self.assertIn("Leaf_vec", kblib.leaf_index_names(self.conn)) self.assertIn("Leaf_vec", kblib.leaf_index_names(self.conn))
def test_add_after_indexes_keeps_fts_queryable(self):
"""Incremental add after FTS+HNSW must find the new leaf on both indexes."""
kblib.upsert_leaf(self.conn, text="seed fox leaf", root="info",
confidence="confirmed", source="s", source_rev="r1",
how="test", loc="/tmp", type_="reference",
embedding=make_emb(0.1))
kblib.ensure_indexes(self.conn)
ids = kblib.add_leafs(self.conn, [{
"text": "added zebra after index",
"root": "facts",
"confidence": "confirmed",
"source": "a.md x b.md",
"source_rev": "r1",
"how": "test",
"loc": "/tmp",
"type": "fact",
"embedding": make_emb(0.9),
}])
self.assertEqual(len(ids), 1)
fts = kblib.query_fts(self.conn, "zebra", 5)
self.assertTrue(fts)
self.assertIn("zebra", fts[0]["text"])
self.assertEqual(fts[0]["root"], "facts")
vec = kblib.query_vector(self.conn, make_emb(0.9), 5)
self.assertTrue(any("zebra" in h["text"] for h in vec))
fox = kblib.query_fts(self.conn, "fox", 5)
self.assertTrue(fox)
self.assertIn("fox", fox[0]["text"])
def test_add_facts_and_info_one_transaction(self):
"""D12: facts and info land in the same transaction."""
kblib.ensure_indexes(self.conn)
ids = kblib.add_leafs(self.conn, [
{
"text": "tx fact leaf two-source",
"root": "facts",
"confidence": "confirmed",
"source": "compose.yml x docker ps",
"source_rev": "r1",
"how": "test",
"loc": "/tmp",
"type": "fact",
"embedding": make_emb(0.4),
},
{
"text": "tx info narrative",
"root": "info",
"confidence": "confirmed",
"source": "note.md",
"source_rev": "r1",
"how": "test",
"loc": "/tmp",
"type": "reference",
"embedding": make_emb(0.5),
},
])
self.assertEqual(len(ids), 2)
stats = kblib.stats(self.conn)
self.assertEqual(stats["by_root"].get("facts"), 1)
self.assertEqual(stats["by_root"].get("info"), 1)
self.assertTrue(kblib.query_fts(self.conn, "two-source", 5))
self.assertTrue(kblib.query_fts(self.conn, "narrative", 5))
def test_drop_vector_then_create_raises_clear_error(self): def test_drop_vector_then_create_raises_clear_error(self):
"""DROP INDEX leaves ghost catalog; create_fts_and_vector must raise.""" """DROP INDEX leaves ghost catalog; create_fts_and_vector must raise."""
kblib.upsert_leaf(self.conn, text="seed", root="info", kblib.upsert_leaf(self.conn, text="seed", root="info",
@@ -98,6 +161,71 @@ class KblibTest(unittest.TestCase):
self.assertEqual(stats["total"], 2) self.assertEqual(stats["total"], 2)
self.assertEqual(stats["by_root"], {"facts": 1, "info": 1}) self.assertEqual(stats["by_root"], {"facts": 1, "info": 1})
def test_hop_1_returns_file_hop_3_reaches_person(self):
"""--hop walks FROM_FILE / HAS_VERSION / AUTHORED (Gitea #17)."""
import gitimport
lid = kblib.upsert_leaf(
self.conn, text="readme hop fixture", root="info",
confidence="confirmed", source="README.md", source_rev="r1",
how="test", loc="README.md", type_="reference",
embedding=make_emb(0.3),
)
kblib.link_from_file(self.conn, lid, "README.md", repo="sample-repo")
gitimport.index_commits(self.conn, [gitimport.Commit(
sha="a1b2c3d",
author="Ada Lovelace",
email="ada@example.com",
date="2026-08-10T12:00:00Z",
subject="feat: first commit",
files=["README.md"],
)], "sample-repo")
hop1 = kblib.hop_walk(self.conn, lid, 1)
self.assertEqual(len(hop1), 1)
self.assertEqual(hop1[0]["label"], "File")
self.assertEqual(hop1[0]["name"], "README.md")
self.assertEqual(hop1[0]["depth"], 1)
hop3 = kblib.hop_walk(self.conn, lid, 3)
labels = {n["label"] for n in hop3}
self.assertIn("File", labels)
self.assertIn("Commit", labels)
self.assertIn("Person", labels)
person = [n for n in hop3 if n["label"] == "Person"][0]
self.assertEqual(person["name"], "Ada Lovelace")
self.assertEqual(person["depth"], 3)
def test_as_of_keeps_x_drops_y(self) -> None:
"""OQ5/#36: as of 2025-01-01 → works-at-X, not works-at-Y."""
kblib.upsert_leaf(
self.conn, text="Andrey works at X", root="facts",
confidence="confirmed", source="crm.md x contract.md",
source_rev="r1", how="test", loc="/tmp", type_="fact",
embedding=make_emb(0.5),
valid_from="2024-03-01", valid_to="2025-07-15",
)
kblib.upsert_leaf(
self.conn, text="Andrey works at Y", root="facts",
confidence="confirmed", source="offer.md x payroll.md",
source_rev="r1", how="test", loc="/tmp", type_="fact",
embedding=make_emb(0.6),
valid_from="2025-07-16", valid_to="",
)
kblib.ensure_indexes(self.conn)
hits = kblib.query_fts(self.conn, "Andrey works", 10)
kept = kblib.filter_as_of(hits, "2025-01-01")
texts = [h["text"] for h in kept]
self.assertTrue(any("works at X" in t for t in texts), texts)
self.assertFalse(any("works at Y" in t for t in texts), texts)
later = kblib.filter_as_of(hits, "2025-08-01")
later_texts = [h["text"] for h in later]
self.assertTrue(any("works at Y" in t for t in later_texts), later_texts)
self.assertFalse(any("works at X" in t for t in later_texts), later_texts)
def test_active_at_pure(self) -> None:
self.assertTrue(kblib.active_at("2024-03-01", "2025-07-15", "2025-01-01"))
self.assertFalse(kblib.active_at("2025-07-16", "", "2025-01-01"))
self.assertTrue(kblib.active_at("", "", "2025-01-01"))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+89
View File
@@ -8,10 +8,13 @@ from pathlib import Path
sys.path.insert(0, os.path.dirname(__file__)) sys.path.insert(0, os.path.dirname(__file__))
from mailconv import ( # noqa: E402 from mailconv import ( # noqa: E402
TESS_LANG,
clean_email_address, clean_email_address,
convert_pdf,
html_to_markdown, html_to_markdown,
is_convertible, is_convertible,
normalize_markdown, normalize_markdown,
ocr_image,
split_zip_members, split_zip_members,
subject_to_filename, subject_to_filename,
zip_extract_safe, zip_extract_safe,
@@ -100,6 +103,92 @@ class TestMailConv(unittest.TestCase):
self.assertFalse(is_convertible(".exe")) self.assertFalse(is_convertible(".exe"))
self.assertFalse(is_convertible(".unknown")) self.assertFalse(is_convertible(".unknown"))
def test_convert_pdf_prefers_pdftotext(self):
import mailconv as mc
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(list(cmd))
class P:
returncode = 0
stdout = b"Invoice BM25 layout"
stderr = b""
return P()
self._patch_run(mc, fake_run)
out = convert_pdf(Path(self._tmp("born.pdf")))
self.assertIn("BM25", out)
self.assertEqual(calls[0][:2], ["pdftotext", "-layout"])
self.assertFalse(any(c[0] == "tesseract" for c in calls))
self.assertFalse(any(c[0] == "pdftoppm" for c in calls))
def test_convert_pdf_empty_layer_uses_pdftoppm_tesseract(self):
import mailconv as mc
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(list(cmd))
class P:
returncode = 0
stdout = b""
stderr = b""
if cmd[0] == "pdftotext":
P.stdout = b" \n"
return P()
if cmd[0] == "pdftoppm":
prefix = Path(cmd[-1])
(prefix.parent / "page-1.png").write_bytes(b"fake")
return P()
if cmd[0] == "tesseract":
P.stdout = b"scanned HELLO"
return P()
return P()
self._patch_run(mc, fake_run)
out = convert_pdf(Path(self._tmp("scan.pdf")))
self.assertIn("HELLO", out)
bins = [c[0] for c in calls]
self.assertIn("pdftotext", bins)
self.assertIn("pdftoppm", bins)
self.assertIn("tesseract", bins)
tess = next(c for c in calls if c[0] == "tesseract")
self.assertIn(TESS_LANG, tess)
self.assertNotIn("docling", " ".join(bins))
def test_ocr_image_paddle_engine(self):
import mailconv as mc
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(list(cmd))
class P:
returncode = 0
stdout = b"paddle text"
stderr = b""
return P()
self._patch_run(mc, fake_run)
os.environ["OCR_ENGINE"] = "paddle"
try:
out = ocr_image(Path(self._tmp("x.png")))
finally:
os.environ.pop("OCR_ENGINE", None)
self.assertEqual(out, "paddle text")
self.assertEqual(calls[0][:2], ["paddleocr", "ocr"])
def _patch_run(self, mod, fn) -> None:
self.addCleanup(setattr, mod.subprocess, "run", mod.subprocess.run)
mod.subprocess.run = fn
def _mk_zip(self, members): def _mk_zip(self, members):
zpath = Path(self._tmp("arc.zip")) zpath = Path(self._tmp("arc.zip"))
with zipfile.ZipFile(zpath, "w") as zf: with zipfile.ZipFile(zpath, "w") as zf:
+102 -10
View File
@@ -1,7 +1,6 @@
"""Published docs must match live commands (Gitea SoT, brain/search, no fake --hop).""" """Published docs must match live commands (Gitea SoT, brain/search)."""
from __future__ import annotations from __future__ import annotations
import re
import unittest import unittest
from pathlib import Path from pathlib import Path
@@ -89,6 +88,49 @@ class PublishedDocsTest(unittest.TestCase):
self.assertFalse((ROOT / "skills" / "db-yaml").exists()) self.assertFalse((ROOT / "skills" / "db-yaml").exists())
self.assertTrue((ROOT / "skills" / "postgres" / "SKILL.md").is_file()) self.assertTrue((ROOT / "skills" / "postgres" / "SKILL.md").is_file())
def test_cgo_zig_and_index_profile(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D21", plan)
self.assertIn("zig cc", plan)
dockerfile = (ROOT / "Dockerfile").read_text()
self.assertIn("bin/cgo/zcc", dockerfile)
self.assertIn("FROM debian:bookworm-slim AS api", dockerfile)
self.assertIn("FROM python:3.12-slim AS index", dockerfile)
api = dockerfile[dockerfile.index("FROM debian:bookworm-slim AS api") :]
self.assertNotIn("pip install", api)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('profiles: ["index"]', compose)
self.assertIn("target: api", compose)
def test_reasoner_docs_name_real_hf_ids_cpu_sidecar(self) -> None:
docs = (ROOT / "docs" / "reasoner.md").read_text()
for hf in (
"Qwen/Qwen3.5-9B",
"Qwen/Qwen3.6-27B",
"prism-ml/Bonsai-27B-gguf",
):
self.assertIn(hf, docs)
self.assertIn("no official qwen3.6-9b", docs.lower())
self.assertIn("OLLAMA_NUM_GPU", docs)
self.assertIn("rss_mb", docs)
self.assertIn("vram_mb", docs)
self.assertIn("3/3", docs)
self.assertIn("Do not claim 9B is better at tools", docs)
self.assertNotIn("Qwen/Qwen3.6-9B", docs)
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("D18", plan)
self.assertIn("Qwen/Qwen3.5-9B", plan)
compose = (ROOT / "compose.yaml").read_text()
self.assertIn('"reasoner"', compose)
self.assertIn("OLLAMA_NUM_GPU", compose)
self.assertIn("127.0.0.1:11435", compose)
dockerfile = (ROOT / "Dockerfile").read_text()
self.assertNotIn(".gguf", dockerfile.lower())
self.assertNotIn(".safetensors", dockerfile.lower())
api = dockerfile[dockerfile.index("FROM debian:bookworm-slim AS api") :]
self.assertNotIn("COPY models", api)
self.assertNotIn("qwen", api.lower())
def test_readme_search_escalates_web(self) -> None: def test_readme_search_escalates_web(self) -> None:
text = (ROOT / "README.md").read_text() text = (ROOT / "README.md").read_text()
self.assertIn("--no-web", text) self.assertIn("--no-web", text)
@@ -96,19 +138,69 @@ class PublishedDocsTest(unittest.TestCase):
skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text() skill = (ROOT / "skills" / "brain" / "SKILL.md").read_text()
self.assertIn("`web` block", skill) self.assertIn("`web` block", skill)
def test_docs_do_not_claim_hop_walks(self) -> None: def test_docs_say_hop_walks_from_file(self) -> None:
paths = [ paths = [
ROOT / "README.md", ROOT / "README.md",
ROOT / "docs" / "design.md", ROOT / "docs" / "design.md",
ROOT / "skills" / "brain" / "SKILL.md", ROOT / "skills" / "brain" / "SKILL.md",
ROOT / "skills" / "diataxis-docs" / "SKILL.md", ROOT / "docs" / "runbook.md",
ROOT / "docs" / "README.md",
] ]
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
# Honest "not implemented" notes must not match.
lie = re.compile(r"--hop (?:N|1).*(?:follow|walk)", re.I | re.S)
for path in paths: for path in paths:
text = path.read_text() text = path.read_text()
self.assertIsNone( self.assertIn("--hop", text, f"{path.relative_to(ROOT)} must document --hop")
lie.search(text), self.assertNotIn(
f"{path.relative_to(ROOT)} still claims --hop walks the graph", "not implemented",
text.lower(),
f"{path.relative_to(ROOT)} still says hop is not implemented",
) )
def test_docs_are_portable_diataxis(self) -> None:
index = (ROOT / "docs" / "README.md").read_text()
self.assertIn("type: reference", index)
for d in ("D3", "D6", "D14", "D15", "D17", "D18"):
self.assertIn(d, index)
runbook = (ROOT / "docs" / "runbook.md").read_text()
self.assertIn("type: howto", runbook)
self.assertIn("bin/brain/search.go", runbook)
self.assertIn("bin/brain/index.go", runbook)
self.assertNotIn("search.ops.io", runbook)
self.assertNotIn("/mnt/", runbook)
self.assertNotIn("/home/", runbook)
readme = (ROOT / "README.md").read_text()
self.assertIn("docs/runbook.md", readme)
self.assertNotIn("search.ops.io", readme)
def test_v1_epic_is_named_in_docs(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("Gap to v1", plan)
self.assertIn("eSlider/2dph/issues/16", plan)
self.assertIn("eSlider/2dph/issues/17", plan)
self.assertIn("eSlider/2dph/milestone/12", plan)
road = (ROOT / "docs" / "roadmap.md").read_text()
self.assertIn("type: explanation", road)
self.assertIn("issues/16", road)
self.assertIn("issues/14", road)
index = (ROOT / "docs" / "README.md").read_text()
self.assertIn("roadmap.md", index)
self.assertIn("epic #16", index)
agents = (ROOT / "AGENTS.md").read_text()
self.assertIn("roadmap.md", agents)
def test_oq5_intervals_are_not_d16_stale(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
design = (ROOT / "docs" / "design.md").read_text()
road = (ROOT / "docs" / "roadmap.md").read_text()
contradict = (ROOT / "internal" / "facts" / "contradict.go").read_text()
interval = (ROOT / "internal" / "facts" / "interval.go").read_text()
self.assertIn("OQ5", plan)
self.assertIn("D24", plan)
self.assertIn("issues/36", plan)
self.assertIn("valid_from", plan)
self.assertIn("--as-of", design)
self.assertIn("issues/36", road)
self.assertIn("**in**", road[road.index("OQ5"):road.index("OQ5") + 80])
self.assertIn("temporal_freshness", contradict)
self.assertNotIn("valid_from", contradict)
self.assertIn("ActiveAt", interval)
self.assertIn("NormalizeDay", interval)
+14
View File
@@ -47,3 +47,17 @@ class SkillsTest(unittest.TestCase):
self.assertIn("throttled", skill.lower()) self.assertIn("throttled", skill.lower())
self.assertIn("not a negative finding", agents) self.assertIn("not a negative finding", agents)
self.assertIn("Fact-check every", agents) self.assertIn("Fact-check every", agents)
def test_yq_is_mikefarah_for_structured_data(self) -> None:
skill = (ROOT / "skills" / "yq" / "SKILL.md").read_text()
self.assertIn("https://github.com/mikefarah/yq", skill)
for fmt in ("YAML", "JSON", "XML", "CSV", "TOML", "HCL"):
self.assertIn(fmt, skill)
self.assertIn("not kislyuk", skill.lower())
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("mikefarah/yq", plan)
agents = (ROOT / "AGENTS.md").read_text()
self.assertIn("mikefarah/yq", agents)
web = (ROOT / "skills" / "web-search" / "SKILL.md").read_text()
self.assertIn("| yq ", web)
self.assertNotIn("| jq ", web)
+213
View File
@@ -0,0 +1,213 @@
"""bin/stack/{start,start-assistant,stop,status} — offline contract + fake PATH."""
from __future__ import annotations
import os
import stat
import subprocess
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
METHODS = ("start", "start-assistant", "stop", "status")
class StackLayoutTest(unittest.TestCase):
def test_methods_are_bash_with_usage_comment(self) -> None:
lib = ROOT / "bin" / "stack" / "lib.sh"
self.assertTrue(lib.is_file(), "missing bin/stack/lib.sh")
for name in METHODS:
p = ROOT / "bin" / "stack" / name
self.assertTrue(p.is_file(), f"missing bin/stack/{name}")
self.assertTrue(os.access(p, os.X_OK), f"bin/stack/{name} must be executable")
lines = p.read_text().splitlines()
self.assertEqual(lines[0], "#!/usr/bin/env bash", name)
self.assertTrue(lines[1].startswith("# bin/stack/"), name)
text = "\n".join(lines)
self.assertIn("lib.sh", text, name)
self.assertNotIn("/mnt/", text, name)
self.assertNotIn("/home/", text, name)
def test_lib_has_no_host_paths_or_secrets(self) -> None:
lib = (ROOT / "bin" / "stack" / "lib.sh").read_text()
self.assertIn("stack_start", lib)
self.assertIn("stack_start_assistant", lib)
self.assertIn("stack_stop", lib)
self.assertIn("stack_status", lib)
self.assertIn("qwen3.5:9b", lib)
self.assertIn("picoclaw agent", lib)
self.assertIn("--no-deps", lib)
self.assertIn("tools/list", lib)
self.assertNotIn("/mnt/", lib)
self.assertNotIn("/home/", lib)
self.assertNotIn("password", lib.lower())
self.assertNotIn("GITEA_TOKEN", lib)
def test_start_does_not_launch_picoclaw(self) -> None:
start = (ROOT / "bin" / "stack" / "start").read_text()
self.assertIn("stack_start", start)
self.assertNotIn("stack_start_assistant", start)
self.assertNotIn("picoclaw agent", start)
def test_start_assistant_attaches_agent(self) -> None:
src = (ROOT / "bin" / "stack" / "start-assistant").read_text()
self.assertIn("stack_start_assistant", src)
self.assertIn("--no-attach", src)
def test_stop_does_not_down_volumes(self) -> None:
lib = (ROOT / "bin" / "stack" / "lib.sh").read_text()
self.assertIn(" compose ", lib)
self.assertRegex(lib, r"\bstop\b")
self.assertNotIn(" compose down", lib)
self.assertNotIn("compose down", lib)
def test_docs_name_stack_commands(self) -> None:
runbook = (ROOT / "docs" / "runbook.md").read_text()
pico = (ROOT / "docs" / "picoclaw.md").read_text()
agents = (ROOT / "AGENTS.md").read_text()
for text in (runbook, pico, agents):
self.assertIn("bin/stack/start", text)
self.assertIn("bin/stack/start-assistant", text)
self.assertIn("bin/stack/status", text)
self.assertIn("bin/stack/stop", text)
def test_help_prints_comments_not_source(self) -> None:
r = subprocess.run(
[str(ROOT / "bin" / "stack" / "start"), "--help"],
cwd=str(ROOT),
capture_output=True,
text=True,
check=False,
)
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn("bin/stack/start", r.stdout)
self.assertNotIn("set -euo pipefail", r.stdout)
self.assertNotIn("source ", r.stdout)
class StackFakePathTest(unittest.TestCase):
def _fake_bin(self, tmp: Path, *, health_ok: bool) -> Path:
bindir = tmp / "bin"
bindir.mkdir()
curl = bindir / "curl"
docker = bindir / "docker"
log = tmp / "docker.log"
curl.write_text(
f"""#!/usr/bin/env bash
url=""
for a in "$@"; do
case "$a" in http*) url=$a ;;
esac
done
if [ "{int(health_ok)}" = "0" ] && [[ "$url" == */health ]]; then
echo '{{"status":"down"}}'
exit 7
fi
case "$url" in
*/mcp)
echo '{{"jsonrpc":"2.0","id":1,"result":{{"tools":[{{"name":"search"}},{{"name":"get"}},{{"name":"audit"}}]}}}}'
;;
*/api/tags)
echo '{{"models":[{{"name":"qwen3.5:9b"}}]}}'
;;
*/api/pull)
echo '{{"status":"success"}}'
;;
*)
echo '{{"status":"ok"}}'
;;
esac
"""
)
docker.write_text(
f"""#!/usr/bin/env bash
echo "$*" >> "{log}"
exit 0
"""
)
curl.chmod(curl.stat().st_mode | stat.S_IEXEC)
docker.chmod(docker.stat().st_mode | stat.S_IEXEC)
return bindir
def _env(self, bindir: Path) -> dict[str, str]:
env = os.environ.copy()
env["PATH"] = f"{bindir}:{env.get('PATH', '')}"
env["STACK_WAIT_SECS"] = "1"
env["STACK_WAIT_INTERVAL"] = "0"
return env
def test_start_skips_compose_when_brain_healthy(self) -> None:
with tempfile.TemporaryDirectory() as raw:
tmp = Path(raw)
bindir = self._fake_bin(tmp, health_ok=True)
log = tmp / "docker.log"
r = subprocess.run(
[str(ROOT / "bin" / "stack" / "start")],
cwd=str(ROOT),
env=self._env(bindir),
capture_output=True,
text=True,
check=False,
)
self.assertEqual(r.returncode, 0, r.stderr)
self.assertFalse(log.exists(), "healthy brain must not docker compose up")
def test_start_ups_brain_when_unhealthy(self) -> None:
with tempfile.TemporaryDirectory() as raw:
tmp = Path(raw)
bindir = self._fake_bin(tmp, health_ok=False)
log = tmp / "docker.log"
r = subprocess.run(
[str(ROOT / "bin" / "stack" / "start")],
cwd=str(ROOT),
env=self._env(bindir),
capture_output=True,
text=True,
check=False,
)
self.assertNotEqual(r.returncode, 0, "unhealthy brain without recovering compose must fail")
self.assertTrue(log.exists(), r.stderr)
logged = log.read_text()
self.assertIn("up -d", logged)
self.assertIn("brain", logged)
self.assertNotIn("picoclaw", logged)
def test_stop_stops_named_services(self) -> None:
with tempfile.TemporaryDirectory() as raw:
tmp = Path(raw)
bindir = self._fake_bin(tmp, health_ok=True)
log = tmp / "docker.log"
r = subprocess.run(
[str(ROOT / "bin" / "stack" / "stop")],
cwd=str(ROOT),
env=self._env(bindir),
capture_output=True,
text=True,
check=False,
)
self.assertEqual(r.returncode, 0, r.stderr)
logged = log.read_text()
self.assertIn("stop", logged)
self.assertNotIn(" down", logged)
for svc in ("brain", "brain-mcp", "reasoner", "picoclaw"):
self.assertIn(svc, logged)
def test_start_assistant_no_attach_starts_picoclaw(self) -> None:
with tempfile.TemporaryDirectory() as raw:
tmp = Path(raw)
bindir = self._fake_bin(tmp, health_ok=True)
log = tmp / "docker.log"
r = subprocess.run(
[str(ROOT / "bin" / "stack" / "start-assistant"), "--no-attach"],
cwd=str(ROOT),
env=self._env(bindir),
capture_output=True,
text=True,
check=False,
)
self.assertEqual(r.returncode, 0, r.stderr + r.stdout)
logged = log.read_text() if log.exists() else ""
self.assertIn("picoclaw", logged)
self.assertIn("--no-deps", logged)
self.assertNotIn("picoclaw agent", logged)
+36
View File
@@ -0,0 +1,36 @@
"""qa/system_perf.py is an offline-gated system test (no live brain in CI)."""
from __future__ import annotations
import ast
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
class SystemPerfScriptTest(unittest.TestCase):
def test_script_compiles_and_is_read_only(self) -> None:
path = ROOT / "qa" / "system_perf.py"
src = path.read_text()
compile(src, str(path), "exec")
self.assertIn("--json", src)
self.assertIn("qwen3.5:9b", src)
self.assertIn("--picoclaw", src)
self.assertIn("BRAIN_URL", src)
self.assertIn("tools/list", src)
self.assertIn("tools/call", src)
self.assertIn("GATE_HEALTH_MS", src)
self.assertIn("GATE_GET_P50_MS", src)
self.assertNotIn("kb.lbug", src)
self.assertNotIn("password", src.lower())
self.assertNotIn("token", src.lower())
def test_script_does_not_write_ladybug(self) -> None:
tree = ast.parse((ROOT / "qa" / "system_perf.py").read_text())
writes = [
n.func.attr
for n in ast.walk(tree)
if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)
and n.func.attr in {"write_text", "write_bytes", "dump"}
]
self.assertEqual(writes, [], f"system_perf must not write files: {writes}")
+8 -70
View File
@@ -16,9 +16,9 @@ import (
"fmt" "fmt"
"net/http" "net/http"
"os" "os"
"strconv"
"time" "time"
"github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/websearch" "github.com/eSlider/2dph/internal/websearch"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
) )
@@ -28,77 +28,15 @@ func main() {
} }
func run(args []string) int { func run(args []string) int {
var ( c, err := websearch.ParseArgs(args)
query, site, lang, fresh, category, engines string
limit = websearch.DefaultLimit
jsonOut, refresh, force bool
ttl = float64(websearch.CacheTTL)
timeout = 25
)
i := 0
for i < len(args) {
a := args[i]
switch {
case a == "--json":
jsonOut = true
case a == "--refresh":
refresh = true
case a == "--force":
force = true
case (a == "-n" || a == "--limit") && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 0 {
fmt.Fprintln(os.Stderr, "web/search: --limit must be a non-negative integer")
return 2
}
limit = n
case a == "--site" && i+1 < len(args):
i++
site = args[i]
case a == "--lang" && i+1 < len(args):
i++
lang = args[i]
case a == "--fresh" && i+1 < len(args):
i++
fresh = args[i]
case a == "--category" && i+1 < len(args):
i++
category = args[i]
case a == "--engines" && i+1 < len(args):
i++
engines = args[i]
case a == "--ttl" && i+1 < len(args):
i++
v, err := strconv.ParseFloat(args[i], 64)
if err != nil { if err != nil {
fmt.Fprintln(os.Stderr, "web/search: --ttl must be a number") return cli.Fail(err)
return 2
}
ttl = v
case a == "--timeout" && i+1 < len(args):
i++
n, err := strconv.Atoi(args[i])
if err != nil || n <= 0 {
fmt.Fprintln(os.Stderr, "web/search: --timeout must be a positive integer")
return 2
}
timeout = n
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/web/search.go QUERY [--json] [-n N] [--site HOST] [--lang LANG] [--fresh day|week|month|year] [--category CAT] [--engines LIST] [--refresh] [--force]`)
return 0
case len(a) > 0 && a[0] != '-' && query == "":
query = a
default:
fmt.Fprintf(os.Stderr, "web/search: unknown flag %s\n", a)
return 2
}
i++
}
if query == "" {
fmt.Fprintln(os.Stderr, "web/search: query required")
return 2
} }
query, site, lang, fresh, category, engines := c.Query, c.Site, c.Lang, c.Fresh, c.Category, c.Engines
limit := c.Limit
jsonOut, refresh, force := c.JSONOut, c.Refresh, c.Force
ttl := c.TTL
timeout := c.Timeout
if site != "" { if site != "" {
query = "site:" + site + " " + query query = "site:" + site + " " + query
} }
+99 -22
View File
@@ -1,66 +1,97 @@
# 2dph — docker composition # 2dph — docker composition
# #
# docker compose run --rm brain index # rebuild graph # bin/stack/start / start-assistant / status / stop
# docker compose run --rm brain search "Matrix fed" # one-shot query # docker compose up -d brain # API (Zig CGO serve)
# docker compose run --rm brain serve # async Go server # docker compose --profile index run --rm index # Python rebuild
# docker compose up brain-watch # auto re-index # docker compose --profile picoclaw up -d # brain-mcp + CPU reasoner + PicoClaw gateway
# docker compose --profile reasoner up -d reasoner # CPU Ollama :11435
# docker compose --profile searxng up -d
# OCR_ENGINE=paddle docker compose --profile ocr-paddle run --rm ocr-paddle
# #
# Caching: the 128M model (HF_HOME) and kb.lbug (VAR_DIR) live in named # Secrets never baked in: search.env + db-profiles.yml from ~/.config/brain.
# volumes, so rebuilds never redownload the model or re-derive the graph.
# Secrets are never baked into the image: search.env + db-profiles.yml mount
# read-only from ~/.config/brain.
name: 2dph name: 2dph
networks:
default:
name: 2dph_sys
driver: bridge
ipam:
config:
- subnet: 10.23.42.0/24
services: services:
brain: brain:
image: ghcr.io/eslider/2dph:latest image: ghcr.io/eslider/2dph:api
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
target: api
cache_from: cache_from:
- ghcr.io/eslider/2dph:cache - ghcr.io/eslider/2dph:cache
command: ["brain", "search", "help"] command: ["serve"]
environment: &env environment: &env
HF_HOME: /data/hf HF_HOME: /data/hf
BRAIN_SEARCH_CACHE: /data/cache/web-search.sqlite BRAIN_SEARCH_CACHE: /data/cache/web-search.sqlite
BRAIN_DB_PROFILES: /secret/db-profiles.yml BRAIN_DB_PROFILES: /secret/db-profiles.yml
BRAIN_SEARCH_ENV: /secret/search.env BRAIN_SEARCH_ENV: /secret/search.env
KB_SEARCH_CMD: /app/bin/kb/search KB_ROOT: /data
KB_WORKERS: "4" KB_WORKERS: "4"
KB_PORT: "8630"
volumes: volumes:
- kb-model:/data/hf - kb-model:/data/hf
- kb-var:/data - kb-var:/data
# corpus is read-only on the host, never written from the container
- ..:/corpus:ro
- ~/.config/brain:/secret:ro - ~/.config/brain:/secret:ro
ports:
- "127.0.0.1:8630:8630"
read_only: true read_only: true
tmpfs: tmpfs:
- /tmp - /tmp
healthcheck: healthcheck:
test: ["CMD", "python3", "-c", "import ladybug, model2vec, mistune; print('ok')"] test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8630/health"]
interval: 30s interval: 30s
timeout: 5s timeout: 5s
retries: 3 retries: 3
restart: unless-stopped restart: unless-stopped
stop_grace_period: 20s stop_grace_period: 20s
# watcher: re-index on corpus file change (watchdog script)
brain-watch: brain-watch:
image: ghcr.io/eslider/2dph:latest image: ghcr.io/eslider/2dph:api
environment: *env environment: *env
volumes: volumes:
- kb-model:/data/hf - kb-model:/data/hf
- kb-var:/data - kb-var:/data
- ..:/corpus:ro - ..:/corpus:ro
- ~/.config/brain:/secret:ro - ~/.config/brain:/secret:ro
command: ["brain", "watch", "/corpus"] command: ["watch", "/corpus"]
read_only: true read_only: true
tmpfs: tmpfs:
- /tmp - /tmp
restart: unless-stopped restart: unless-stopped
stop_grace_period: 20s stop_grace_period: 20s
# Python write path (Ladybug rebuild). Not in the API image.
# docker compose --profile index run --rm index
index:
profiles: ["index"]
image: ghcr.io/eslider/2dph:index
build:
context: .
dockerfile: Dockerfile
target: index
environment:
HF_HOME: /data/hf
KB_PY: python3
volumes:
- kb-model:/data/hf
- kb-var:/app/var
- ..:/corpus:ro
- ~/.config/brain:/secret:ro
command: ["index"]
read_only: true
tmpfs:
- /tmp
# Optional local SearXNG (D3). Skip if BRAIN_SEARCH_URL already points at a # Optional local SearXNG (D3). Skip if BRAIN_SEARCH_URL already points at a
# live instance — do not run a second copy on that host. # live instance — do not run a second copy on that host.
# SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d # SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d
@@ -76,18 +107,17 @@ services:
- ./deploy/searxng/limiter.toml:/etc/searxng/limiter.toml:ro - ./deploy/searxng/limiter.toml:/etc/searxng/limiter.toml:ro
restart: unless-stopped restart: unless-stopped
# MCP endpoint for an external agent (PicoClaw is not shipped here). # MCP endpoint for PicoClaw (and any MCP client).
# docker compose --profile picoclaw up brain-mcp # docker compose --profile picoclaw up -d
# Point the agent at http://127.0.0.1:8630/mcp (see deploy/picoclaw/).
brain-mcp: brain-mcp:
profiles: ["picoclaw"] profiles: ["picoclaw"]
image: ghcr.io/eslider/2dph:latest image: ghcr.io/eslider/2dph:api
environment: *env environment: *env
volumes: volumes:
- kb-model:/data/hf - kb-model:/data/hf
- kb-var:/data - kb-var:/data
- ~/.config/brain:/secret:ro - ~/.config/brain:/secret:ro
command: ["brain", "serve"] command: ["serve"]
ports: ports:
- "127.0.0.1:8630:8630" - "127.0.0.1:8630:8630"
read_only: true read_only: true
@@ -95,6 +125,53 @@ services:
- /tmp - /tmp
restart: unless-stopped restart: unless-stopped
# CPU OpenAI-compatible sidecar (D18). Weights are pulled at runtime, not
# baked into the 2dph image. Does not touch host Ollama on :11434.
# docker compose --profile reasoner up -d reasoner
# docker compose --profile reasoner exec reasoner ollama pull qwen3.5:9b
reasoner:
profiles: ["reasoner", "picoclaw"]
image: docker.io/ollama/ollama:latest
environment:
OLLAMA_NUM_GPU: "0"
OLLAMA_HOST: "0.0.0.0:11434"
ports:
- "127.0.0.1:11435:11434"
volumes:
- reasoner-ollama:/root/.ollama
restart: unless-stopped
# Official PicoClaw gateway. Config has no secrets (Ollama + HTTP MCP).
# Host network: brain/reasoner bind 127.0.0.1 only, so host.docker.internal
# (docker0) cannot reach them. Gateway 127.0.0.1:18790 (not the 18800 launcher).
# If :8630/:11435 are already bound, do not start brain-mcp/reasoner:
# docker compose --profile picoclaw up -d --no-deps picoclaw
picoclaw:
profiles: ["picoclaw"]
image: docker.io/sipeed/picoclaw:v0.3.1
network_mode: host
depends_on:
- brain-mcp
- reasoner
environment:
PICOCLAW_GATEWAY_HOST: "127.0.0.1"
entrypoint: ["picoclaw", "gateway"]
volumes:
- picoclaw-home:/root/.picoclaw
- ./deploy/picoclaw/config.json:/root/.picoclaw/config.json:ro
restart: unless-stopped
# Optional PP-OCRv5 (not default). Default OCR is tesseract eng+deu.
# OCR_ENGINE=paddle docker compose --profile ocr-paddle run --rm ocr-paddle
ocr-paddle:
profiles: ["ocr-paddle"]
image: python:3.12-slim
environment:
OCR_ENGINE: paddle
command: ["python", "-c", "print('OCR_ENGINE=paddle; install paddleocr on PATH')"]
volumes: volumes:
kb-model: kb-model:
kb-var: kb-var:
reasoner-ollama:
picoclaw-home:
+33
View File
@@ -0,0 +1,33 @@
{
"agents": {
"defaults": {
"model_name": "qwen3.5-9b",
"max_tool_iterations": 8,
"max_tokens": 512,
"context_window": 8192
}
},
"model_list": [
{
"model_name": "qwen3.5-9b",
"model": "ollama/qwen3.5:9b",
"api_base": "http://127.0.0.1:11435/v1",
"request_timeout": 600
}
],
"tools": {
"web": {
"enabled": false
},
"mcp": {
"enabled": true,
"servers": {
"2dph": {
"enabled": true,
"type": "http",
"url": "http://127.0.0.1:8630/mcp"
}
}
}
}
}
+32 -9
View File
@@ -1,15 +1,38 @@
# 2dph (deductionphile) ---
type: reference
status: current
related:
- docs/runbook.md
- docs/design.md
- PLAN.md
- docs/roadmap.md
---
Evidence-first knowledge graph + hybrid RAG over the operational # 2dph docs (Diataxis)
Brain/ops/eSlider stack. Facts need proof or they are
Evidence-first knowledge graph. Facts need proof or they are
`(not confirmed)`. `(not confirmed)`.
- [PLAN.md](../PLAN.md) — decisions, execution order, open questions (v2) | Type | Doc |
- [design](design.md) — schema, deduction model, sources |------|-----|
- [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) — work board (origin) | tutorial / howto | [runbook](runbook.md) — run anywhere (uv, Go, Docker) |
| explanation | [design](design.md) — two roots, deduction, D17/D20/D18 |
| explanation | [roadmap](roadmap.md) — gap to v1 (epic #16) |
| howto | [picoclaw](picoclaw.md) — MCP agent (`bin/stack/start-assistant`) |
| howto | [reasoner](reasoner.md) — CPU bake-off (D18) |
| reference | [PLAN.md](../PLAN.md) — decisions D1D24 |
Decisions the public face must name: **D3** SearXNG compose, **D6** Go service /
Python write sidecar, **D14** `bin/{subject}/{method}.go`, **D15** Gitea origin,
**D17** assertion gate (facts → info → web), **D18** pluggable reasoner.
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go` Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is `/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop N` walks
not a walk; the flag errors until File/FROM_FILE edges exist. `FROM_FILE` → Commit → Person from each hit (max 3). Rebuild writes
File edges ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
Published docs live here and mirror the project state. Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues)
([epic #16](https://git.produktor.io/eSlider/2dph/issues/16)).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).
Published docs live here and match live commands.
+2 -1
View File
@@ -21,4 +21,5 @@ OO_CLI (default: $HOME/go/bin/oo)
./bin/chats/apply.go --dry-run ./bin/chats/apply.go --dry-run
``` ```
JSONL → markdown only. Brain ingest is `bin/brain/index.go` (not a `chats index`). JSONL → markdown only. Brain ingest is `bin/brain/index.go --with-chats`
(default `var/chats/md`). WhatsApp sync is out of v1.
+47 -7
View File
@@ -1,3 +1,12 @@
---
type: explanation
status: current
related:
- docs/README.md
- docs/runbook.md
- docs/roadmap.md
---
# Design — facts, info, deduction # Design — facts, info, deduction
## Two roots, one transaction ## Two roots, one transaction
@@ -25,8 +34,13 @@ bin/brain/search.go "question"
is not evidence of absence; `--no-web` / `--root` skip it) is not evidence of absence; `--no-web` / `--root` skip it)
``` ```
`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an `--hop N` walks `Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`
error; it is not a graph walk. from each hit (1=File, 2=Commit, 3=Person). Rebuild writes FROM_FILE;
git import writes HAS_VERSION/AUTHORED ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
Go CLIs parse with **flaggy** via `internal/cli` (D23). Flags may appear
after positionals (`search q --hop 1`). Completions:
`source <(./bin/cli/complete.go bash)`.
## Who / What / How / Where / When + evidence ## Who / What / How / Where / When + evidence
@@ -54,6 +68,15 @@ binary); conversion prints leafs, brain write is `bin/brain/index.go`.
`bin/facts/audit stale` flags leafs whose observed revision is behind the `bin/facts/audit stale` flags leafs whose observed revision is behind the
corpus HEAD. corpus HEAD.
Fact **interval of truth** (D24 / OQ5): leaf props `valid_from` /
`valid_to` (YYYY-MM-DD, inclusive; empty end = open; both empty = legacy
always-active). `bin/brain/search.go --as-of YYYY-MM-DD` and MCP/HTTP
`as_of` keep hits whose interval covers that day. This is not D16
`temporal_freshness` (source freshness vs HEAD). [#36](https://git.produktor.io/eSlider/2dph/issues/36).
Existing `kb.lbug` without the columns: open/search/add runs an idempotent
`ALTER TABLE Leaf ADD …` (no full rebuild required). Fresh `--rebuild` still
creates them in `CREATE NODE TABLE`.
## Sources (auto-pairing) ## Sources (auto-pairing)
- A: runtime state — `docker ps` (container running), ports actually bound - A: runtime state — `docker ps` (container running), ports actually bound
@@ -61,19 +84,36 @@ corpus HEAD.
- C: narrative — READMEs, AGENTS.md, docs - C: narrative — READMEs, AGENTS.md, docs
Confirmed = A×B or B×C agreement. Single source = hypothesis + `(not confirmed)`. Confirmed = A×B or B×C agreement. Single source = hypothesis + `(not confirmed)`.
Conflicting pairings (≥2 yes vs ≥2 no) = hypothesis (OQ1 → v2 resolution). Conflicting pairings (≥2 yes vs ≥2 no) stay hypothesis until
`temporal_freshness` or `authority_pairing` fires (`bin/facts/audit contradict`,
[#29](https://git.produktor.io/eSlider/2dph/issues/29)).
## Read path ## Read path
`bin/brain/get.go`, `stats.go`, and `eval.go` call `internal/brain` with cgo `bin/brain/get.go`, `stats.go`, and `eval.go` call `internal/brain` with cgo
(`system_ladybug`). They do not exec Python. Control questions for recall@5 (`system_ladybug`), compiled by **Zig** (`bin/cgo/zcc`, D21), not gcc.
live in `internal/brain/rank` so CI can test the table without libladybug. They do not exec Python. Control questions for recall@5 live in
`internal/brain/rank` so CI can test the table without libladybug.
Python `bin/kb/{get,stats,eval}` remain for GitHub Actions until the runner Python `bin/kb/{get,stats,eval}` remain for GitHub Actions until the runner
has ladybug cgo. Index/write is still `bin/kb/index`. fetches Zig + libs (`bin/cgo/zig`). Incremental write is `bin/kb/add`
(`bin/brain/add.go`). Bulk index/write is still `bin/kb/index`
(`docker compose --profile index`).
## Agent API (D20) ## Agent API (D20)
`bin/brain/serve.go` exposes the same `internal/httpapi.Ops` table as OpenAPI `bin/brain/serve.go` exposes the same `internal/httpapi.Ops` table as OpenAPI
(`GET /openapi.json`) and MCP (`POST /mcp` JSON-RPC `tools/list` + (`GET /openapi.json`) and MCP (`POST /mcp` JSON-RPC `tools/list` +
`tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`. `tools/call`). Tool names match paths: `search`, `get`, `stats`, `audit`,
`ingest` (add a leaf; omit body for the CLI hint).
Agents should use these endpoints instead of shebang CLIs. Agents should use these endpoints instead of shebang CLIs.
## Reasoner (D18)
Pluggable OpenAI-compatible URL. RAM: `Qwen/Qwen3.5-9B`. Quality:
`prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B.
CPU sidecar: compose profile `reasoner` (`OLLAMA_NUM_GPU=0`,
`127.0.0.1:11435`). Bake-off: `bin/reasoner/bakeoff.go`. Weights stay out
of the 2dph image. See [docs/reasoner.md](reasoner.md).
Gap to v1 (hops, corpus, CI eval): [roadmap](roadmap.md),
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
+34 -6
View File
@@ -1,18 +1,46 @@
# PicoClaw profile (reference agent) # PicoClaw profile (reference agent)
2dph is the memory/fact gate. PicoClaw (or any MCP client) is the agent loop 2dph is the memory/fact gate. Compose profile `picoclaw` runs the official
and is **not** shipped in this repo. PicoClaw gateway (`docker.io/sipeed/picoclaw:v0.3.1`) plus `brain-mcp` and the
CPU reasoner. Default agent model is `qwen3.5:9b` (RAM path, D18). Weights stay
in the reasoner volume, not in the 2dph image.
No secrets in git: Ollama needs no key; MCP is local HTTP.
```bash ```bash
docker compose --profile picoclaw up brain-mcp bin/stack/start-assistant
bin/stack/start-assistant --no-attach
bin/stack/start-assistant -- -m "search the 2dph brain for LadybugDB"
bin/stack/status
bin/stack/stop
``` ```
The API listens on `127.0.0.1:8630`. Point the agent at `start-assistant` reuses a healthy brain on `:8630`, starts the CPU reasoner,
`http://127.0.0.1:8630/mcp` using [deploy/picoclaw/mcp.json.example](../deploy/picoclaw/mcp.json.example). pulls `qwen3.5:9b` if missing, brings up the gateway with `--no-deps picoclaw`,
then `picoclaw agent` (MCP `search``get``audit`). Gateway-only Compose:
```bash
docker compose --profile picoclaw up -d
# already serving :8630 / :11435:
docker compose --profile picoclaw up -d --no-deps picoclaw
```
Gateway: `127.0.0.1:18790`. Brain MCP: `http://127.0.0.1:8630/mcp`.
Cursor-style clients can use [deploy/picoclaw/mcp.json.example](../deploy/picoclaw/mcp.json.example).
PicoClaw itself uses [deploy/picoclaw/config.json](../deploy/picoclaw/config.json)
(`127.0.0.1` + host network — loopback publishes are not reachable via docker0).
OpenAPI: `GET http://127.0.0.1:8630/openapi.json`. OpenAPI: `GET http://127.0.0.1:8630/openapi.json`.
Before a factual reply: `search``get``audit`. `throttled` is not a Before a factual reply: `search``get``audit`. `throttled` is not a
negative finding. See `skills/picoclaw/SKILL.md`. negative finding. See `skills/picoclaw/SKILL.md`.
No Cursor required. A live PicoClaw binary/image is an operator choice. System performance (MCP gates + qwen3.5:9b tool_call + PicoClaw gateway):
```bash
./qa/system_perf.py --json | yq '.gates'
REASONER_MODEL=qwen3.5:9b ./qa/system_perf.py --reasoner --picoclaw --json | yq '.reasoner'
```
The default agent model is `qwen3.5:9b`. PicoClaw `context_window` is 8192
(heuristic `max_tokens*4` at 512 is 2048, too small for MCP tool schemas).
`request_timeout` is 600s for a CPU turn (tool_call + MCP search + answer).
+71
View File
@@ -0,0 +1,71 @@
# Reasoner bake-off (D18)
Pluggable OpenAI-compatible URL. 2dph does not ship weights. PicoClaw is
compose profile `picoclaw` (`sipeed/picoclaw`); the bake-off hits the same
tool names (`search``get``audit` from `internal/httpapi.Ops`).
```bash
docker compose --profile reasoner up -d reasoner
docker compose --profile reasoner exec reasoner ollama pull qwen3.5:9b
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \
./bin/reasoner/bakeoff.go --json
```
JSON includes `latency_p50_ms` / `latency_p95_ms` from DuckDB (`internal/duckstats`, D22).
Host Ollama on `:11434` is left alone. This sidecar binds `127.0.0.1:11435`
with `OLLAMA_NUM_GPU=0` (CPU). Measure RSS (`/api/ps` `size`), not VRAM.
If Compose cannot allocate a project network (Docker IPAM pool exhausted),
the same sidecar is:
```bash
docker run -d --name 2dph-reasoner \
-e OLLAMA_NUM_GPU=0 \
-p 127.0.0.1:11435:11434 \
-v 2dph-reasoner-ollama:/root/.ollama \
ollama/ollama:latest
```
## Real Hugging Face ids
| Role | HF id | Ollama tag (this bake-off) |
|------|-------|----------------------------|
| RAM / 9B | `Qwen/Qwen3.5-9B` | `qwen3.5:9b` |
| Quality 27B (CPU) | `prism-ml/Bonsai-27B-gguf` (derived from Qwen3.6-27B) | `MichelRosselli/bonsai-27b:Q1_0` |
| Quality 27B (full) | `Qwen/Qwen3.6-27B` | not pulled on this CPU box |
There is **no official Qwen3.6-9B**. Do not invent that id.
Qwen3.5-9B has documented upstream tool-call XML bugs. A 9B win on tools
is only claimed if this bake-off records OpenAI `tool_calls` (not
`<tool_call>` XML in `content`).
The 2dph API image does not `COPY` GGUF/safetensors. Pull at runtime into
the `reasoner-ollama` volume.
## Live CPU run
Host sidecar: Ollama **0.32.9**, `OLLAMA_NUM_GPU=0`, `127.0.0.1:11435`,
`device: cpu`, `vram_mb: 0`. Date: 2026-08-13. Same three prompts
(`search` / `get` / `audit`). PicoClaw binary was not used; the OpenAI
tools payload is the surface it would send.
| Model | HF id | tool_call | xml_leak | rss_mb | latency_ms (search/get/audit) |
|-------|-------|-----------|----------|--------|-------------------------------|
| `qwen3.5:9b` | `Qwen/Qwen3.5-9B` | 3/3 | 0 | 5790 | 50059 / 51678 / 31980 |
| `MichelRosselli/bonsai-27b:Q1_0` | `prism-ml/Bonsai-27B-gguf` | 3/3 | 0 | 21951 | 327702 / 166830 / 119808 |
| `Qwen/Qwen3.6-27B` | `Qwen/Qwen3.6-27B` | not loaded | — | — | too heavy for this CPU box |
Both loaded models emitted OpenAI `tool_calls` (not `<tool_call>` XML) on
this runtime. **Do not claim 9B is better at tools** — the score is tied
at 3/3. 9B is smaller and faster. Bonsai RSS includes weights + KV
(`size` from `/api/ps`); first Bonsai prompt includes cold load.
Re-run:
```bash
REASONER_BASE_URL=http://127.0.0.1:11435/v1 REASONER_MODEL=qwen3.5:9b \
./bin/reasoner/bakeoff.go --json
REASONER_MODEL=MichelRosselli/bonsai-27b:Q1_0 ./bin/reasoner/bakeoff.go --json
```
+71
View File
@@ -0,0 +1,71 @@
---
type: explanation
status: current
related:
- PLAN.md
- docs/design.md
- docs/runbook.md
---
# Gap to v1 — detective brain
Goal: a brain that does not assert without proof. Search is deduction
(`facts` ≥2 sources → `info``web`). `confirmed` only from the facts root.
**v1 is a living graph the agent can write and walk**, not “more RAG”.
Epic: [Gitea #16](https://git.produktor.io/eSlider/2dph/issues/16).
Milestone: [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
Decisions: [PLAN.md](../PLAN.md).
## In (do not reopen)
Read path Go + Zig CGO (D21). HTTP + OpenAPI + MCP (D20). PicoClaw compose
profile + CPU reasoner (D18). Mail sync → import → rebuild. D14 shebangs.
Compose `api` (no CPython) / `index` (Python write). Issues #1#5, #7#13.
[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop.
[#14](https://git.produktor.io/eSlider/2dph/issues/14) `bin/brain/add.go` /
`POST /ingest` (Python `kblib.add_leafs`; no Go upsert port).
[#17](https://git.produktor.io/eSlider/2dph/issues/17) `--hop N` walks
FROM_FILE / HAS_VERSION / AUTHORED.
[#18](https://git.produktor.io/eSlider/2dph/issues/18) `--with-facts` /
`--with-chats` on rebuild (WhatsApp out of v1).
[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI recall SoT =
`bin/brain/eval.go` via Zig.
Epic [#16](https://git.produktor.io/eSlider/2dph/issues/16) closed.
## v2
[#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR — **in**.
[#30](https://git.produktor.io/eSlider/2dph/issues/30) OQ3 duckdb-go — **in**.
[#29](https://git.produktor.io/eSlider/2dph/issues/29) OQ1 contradiction
resolution — **in** (`temporal_freshness`, `authority_pairing`).
[#34](https://git.produktor.io/eSlider/2dph/issues/34) D23 flaggy CLI — **in**.
[#36](https://git.produktor.io/eSlider/2dph/issues/36) OQ5/D24 fact intervals /
as-of — **in** (not D16 `temporal_freshness`).
## Blockers
None for epic #16 (closed). Remaining v2: OQ4 (deferred).
```
question
├─ FTS + HNSW ← in
├─ facts / info roots ← in
├─ web (D17) ← in
├─ brain/add ACID ← in
├─ Cypher hop ← in
└─ facts+chats corpus ← in
```
## Not v1
OQ4 YAML-first leafs.
OQ5/D24 fact intervals + as-of — **in**
([#36](https://git.produktor.io/eSlider/2dph/issues/36)).
OCR (OQ2), duckdb-go (OQ3/D22), and D16 adjudication (OQ1) are in.
## Close epic #16 when
Children #14, #15, #17, #18, #19 are closed. MCP tool order stays gated by tests.
+100
View File
@@ -0,0 +1,100 @@
---
type: howto
status: current
related:
- docs/README.md
- PLAN.md
---
# Run 2dph (portable)
No laptop-absolute paths. Config lives in env files under `$HOME/.config/brain/`
(mode 0600), not in git.
## Toolchain
- Go (see `go.mod`)
- Python 3.12 + [uv](https://docs.astral.sh/uv)
- Optional: Docker, Zig CGO via `bin/cgo/zig` (not gcc)
- Optional: poppler (`pdftotext`/`pdftoppm`) + tesseract `eng+deu` for mail OCR
```bash
uv venv .venv
uv pip install -r requirements.lock.txt
eval "$(bin/cgo/zig env)" # when compiling Ladybug read tools
go test ./...
uv run python -m unittest discover -s bin/tools -t .
```
## Config
| File / env | Purpose |
|------------|---------|
| `$BRAIN_SEARCH_ENV` (default `$HOME/.config/brain/search.env`) | `BRAIN_SEARCH_URL` (SearXNG). Optional Basic Auth. |
| `$HOME/.config/brain/db-profiles.yml` | read-only Postgres profiles (OnlyOffice via tunnel) |
If the host already runs SearXNG, point `BRAIN_SEARCH_URL` at it. Do not start
a second copy (D3). Optional Compose instance:
```bash
SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d
```
That binds `127.0.0.1:8888`. JSON format must stay enabled.
## Index then search
Write path is `bin/brain/add.go` for a leaf (or `POST /ingest`). Bulk
corpus rebuild remains `bin/brain/index.go --rebuild` (Compose profile
`index`). Do not DROP INDEX on Ladybug 0.19.
```bash
bin/brain/add.go --text "arc-1 runs Matrix" --root facts --source "compose.yml x docker ps"
bin/brain/index.go --rebuild --with-facts --with-chats
bin/brain/search.go "LadybugDB vector index" # facts → info → web (D17)
bin/brain/search.go "upstream flag" --no-web
bin/brain/search.go "who works where" --as-of 2025-01-01 # D24 intervals
source <(./bin/cli/complete.go bash) # D23 flaggy complete
bin/brain/get.go <id> --body
bin/brain/stats.go
```
`--hop N` walks File → Commit → Person from each hit. `--as-of YYYY-MM-DD`
keeps leafs whose `valid_from`/`valid_to` cover that day (empty interval =
legacy always-on). Empty web results are `throttled`, not absence.
Gap to v1: [roadmap](roadmap.md) / [epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
Ladybug 0.19: never `DROP INDEX` FTS/VECTOR (ghost catalog). Fresh indexes =
delete `var/kb.lbug` then `--rebuild`.
## HTTP / MCP
```bash
bin/stack/start # brain :8630, wait until MCP search/get/audit
bin/stack/status # YAML: brain / reasoner / picoclaw
bin/stack/start-assistant # + qwen3.5:9b + PicoClaw agent (ask the brain)
bin/stack/start-assistant --no-attach
bin/stack/stop # compose stop; volumes kept
```
Same Compose services by hand:
```bash
docker compose up -d brain # :8630 Zig CGO serve
docker compose --profile index run --rm index # rebuild
docker compose --profile picoclaw up brain-mcp # MCP 127.0.0.1:8630
```
`GET /openapi.json`, `POST /mcp`. Agent tool order: `search``get``audit`.
See [picoclaw.md](picoclaw.md).
## Reasoner (optional, D18)
CPU sidecar on `127.0.0.1:11435`. Weights are not in the 2dph image.
```bash
docker compose --profile reasoner up -d reasoner
REASONER_BASE_URL=http://127.0.0.1:11435/v1 ./bin/reasoner/bakeoff.go --json
```
See [reasoner.md](reasoner.md).
+9
View File
@@ -7,7 +7,9 @@ require (
github.com/arran4/golang-ical v0.3.5 github.com/arran4/golang-ical v0.3.5
github.com/chewxy/math32 v1.11.2 github.com/chewxy/math32 v1.11.2
github.com/daulet/tokenizers v1.27.0 github.com/daulet/tokenizers v1.27.0
github.com/duckdb/duckdb-go/v2 v2.10505.0
github.com/go-git/go-git/v5 v5.19.2 github.com/go-git/go-git/v5 v5.19.2
github.com/integrii/flaggy v1.8.0
golang.org/x/sys v0.47.0 golang.org/x/sys v0.47.0
golang.org/x/text v0.40.0 golang.org/x/text v0.40.0
modernc.org/sqlite v1.56.0 modernc.org/sqlite v1.56.0
@@ -20,10 +22,17 @@ require (
github.com/apache/arrow-go/v18 v18.6.0 // indirect github.com/apache/arrow-go/v18 v18.6.0 // indirect
github.com/cloudflare/circl v1.6.3 // indirect github.com/cloudflare/circl v1.6.3 // indirect
github.com/cyphar/filepath-securejoin v0.6.1 // indirect github.com/cyphar/filepath-securejoin v0.6.1 // indirect
github.com/duckdb/duckdb-go-bindings v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emirpasic/gods v1.18.1 // indirect github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/goccy/go-json v0.10.6 // indirect github.com/goccy/go-json v0.10.6 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect github.com/google/flatbuffers v25.12.19+incompatible // indirect
+18
View File
@@ -31,6 +31,20 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/duckdb/duckdb-go-bindings v0.10505.0 h1:/0pPsTLrcCsTGxT0VrHgJWnOcPe1tQL1vrki1v3jbAI=
github.com/duckdb/duckdb-go-bindings v0.10505.0/go.mod h1:HoD5xePkDj3VZbBnVVfxVVYIljZ9khCprWA7FgwIiC4=
github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0 h1:FrMqquFBQlMsi34h2KZgCku54rqA8xEbXZ0NLVDKwYs=
github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10505.0/go.mod h1:EnAvZh1kNJHp5yF+M1ZHNEvapnmt6anq1xXHVrAGqMo=
github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0 h1:lbRbpQwT1MmUhh/VTwukV9K8bxKByV3UghAP3MvsbBo=
github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10505.0/go.mod h1:IGLSeEcFhNeZF16aVjQCULD7TsFZKG5G7SyKJAXKp5c=
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0 h1:nrsaVYj3XYCRbS2FpdOMD/KHE7egRMr+/NR1IHmjT84=
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10505.0/go.mod h1:KAIynZ0GHCS7X5fRyuFnQMg/SZBPK/bS9OCOVojClxw=
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0 h1:qM6oGDgwXBILJGbTY4fCy6QOczLpucUA6yn6g3ORjh4=
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10505.0/go.mod h1:81SGOYoEUs8qaAfSk1wRfM5oobrIJ5KI7AzYhK6/bvQ=
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0 h1:DjqZl9rYreHkSOqnqLmkrqH5T8UdQNcxZLJVZzGmXXA=
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10505.0/go.mod h1:K25pJL26ARblGDeuAkrdblFvUen92+CwksLtPEHRqqQ=
github.com/duckdb/duckdb-go/v2 v2.10505.0 h1:SWwvLn2Qx/RQSnQNupwgIF8VbnJ5A6OQU9lYb/mDETI=
github.com/duckdb/duckdb-go/v2 v2.10505.0/go.mod h1:m0PW4J4FG9hlFlVdXi6Ds9owpyIDaBdE2jyce00fGcE=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o= github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o=
@@ -47,6 +61,8 @@ github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399 h1:eMj
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII= github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII=
github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY= github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY=
github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk= github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU= github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ= github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
@@ -61,6 +77,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/integrii/flaggy v1.8.0 h1:tC1qWwg4fhF2Qdaj+MpPK04cxlOSq0+HoMZqAW6Arao=
github.com/integrii/flaggy v1.8.0/go.mod h1:QS4c80m87SXG0pmVUT/Lx2RY5EbkLvLp7IKBD2jwcFA=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4= github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4=
+9
View File
@@ -85,9 +85,18 @@ func openWithSandbox(epsv string) error {
closeBrain() closeBrain()
return fmt.Errorf("LOAD EXTENSION VECTOR: %w", err) return fmt.Errorf("LOAD EXTENSION VECTOR: %w", err)
} }
migrateIntervalColumns()
return nil return nil
} }
// migrateIntervalColumns adds D24 valid_from/valid_to on existing Leaf tables.
// Fresh CREATE already has them; ALTER is a no-op when the column exists.
func migrateIntervalColumns() {
for _, col := range []string{"valid_from", "valid_to"} {
_, _ = conn.Query("ALTER TABLE Leaf ADD " + col + " STRING")
}
}
func closeBrain() { func closeBrain() {
if conn != nil { if conn != nil {
conn.Close() conn.Close()
+19 -7
View File
@@ -7,6 +7,8 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"os/exec"
"path/filepath"
"github.com/eSlider/2dph/internal/brain/rank" "github.com/eSlider/2dph/internal/brain/rank"
) )
@@ -19,8 +21,8 @@ func Ready() error {
// HTTP is the in-process API used by bin/brain/serve.go. // HTTP is the in-process API used by bin/brain/serve.go.
type HTTP struct{} type HTTP struct{}
func (HTTP) Search(ctx context.Context, query string, limit int) ([]byte, error) { func (HTTP) Search(ctx context.Context, query string, limit int, asOf string) ([]byte, error) {
hits, err := searchHits(query, "", "", limit) hits, err := searchHits(query, "", "", limit, asOf)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -39,7 +41,7 @@ func (HTTP) Search(ctx context.Context, query string, limit int) ([]byte, error)
var buf bytes.Buffer var buf bytes.Buffer
enc := json.NewEncoder(&buf) enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false) enc.SetEscapeHTML(false)
if err := enc.Encode(toJSONOut(hits, query, "", webOut)); err != nil { if err := enc.Encode(toJSONOut(hits, query, "", asOf, webOut)); err != nil {
return nil, err return nil, err
} }
return buf.Bytes(), nil return buf.Bytes(), nil
@@ -137,12 +139,22 @@ func (HTTP) Audit(context.Context) ([]byte, error) {
return json.Marshal(map[string]any{"status": "ok", "by_confidence": rows}) return json.Marshal(map[string]any{"status": "ok", "by_confidence": rows})
} }
func (HTTP) Ingest(context.Context) ([]byte, error) { func (HTTP) Ingest(ctx context.Context, body []byte) ([]byte, error) {
if len(bytes.TrimSpace(body)) == 0 {
return json.Marshal(map[string]any{ return json.Marshal(map[string]any{
"mode": "rebuild", "mode": "add",
"command": "bin/brain/index.go --rebuild", "command": "bin/brain/add.go",
"add": "v2", "rebuild": "bin/brain/index.go --rebuild",
}) })
}
cmd := exec.CommandContext(ctx, filepath.Join(repoRoot(), "bin", "kb", "add"), "--json")
cmd.Stdin = bytes.NewReader(body)
cmd.Dir = repoRoot()
out, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("add: %w", err)
}
return out, nil
} }
func asInt(v any) int64 { func asInt(v any) int64 {
+55 -40
View File
@@ -3,73 +3,88 @@ package rank
import ( import (
"fmt" "fmt"
"strconv" "strconv"
"strings"
"github.com/eSlider/2dph/internal/cli"
"github.com/eSlider/2dph/internal/facts"
"github.com/integrii/flaggy"
) )
const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--json] [--no-web] const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--hop N] [--as-of YYYY-MM-DD] [--json] [--no-web]
bin/brain/search.go serve [port] bin/brain/search.go serve [port]
bin/brain/search.go --list-model` bin/brain/search.go --list-model
source <(./bin/cli/complete.go bash)`
type Options struct { type Options struct {
Query string Query string
Root string Root string
Repo string Repo string
Limit int Limit int
Hop int
AsOf string
JSONOut bool JSONOut bool
ListModel bool ListModel bool
NoWeb bool NoWeb bool
} }
// NewParser is the flaggy schema for search (also used by bin/cli/complete.go).
func NewParser(opt *Options) *flaggy.Parser {
if opt.Limit == 0 {
opt.Limit = 20
}
p := cli.New("brain-search")
p.Description = "deduction search: facts → info → web"
p.String(&opt.Root, "", "root", "facts or info")
p.String(&opt.Repo, "", "repo", "filter by repo")
p.Int(&opt.Limit, "n", "n", "max hits")
p.Int(&opt.Hop, "", "hop", "walk FROM_FILE depth 1-3")
p.String(&opt.AsOf, "", "as-of", "keep facts active on YYYY-MM-DD (D24)")
p.Bool(&opt.JSONOut, "", "json", "JSON output")
p.Bool(&opt.NoWeb, "", "no-web", "stay local")
p.Bool(&opt.ListModel, "", "list-model", "print embedding model")
return p
}
// ParseArgs reads flags. Unknown flags are an error: silently dropping them // ParseArgs reads flags. Unknown flags are an error: silently dropping them
// meant `--hop 1` vanished and its argument `1` was appended to the query. // meant `--hop 1` vanished and its argument `1` was appended to the query.
// --hop is recognised so it cannot be swallowed; it is not implemented until
// File/FROM_FILE edges exist.
func ParseArgs(args []string) (Options, error) { func ParseArgs(args []string) (Options, error) {
opt := Options{Limit: 20} opt := Options{Limit: 20}
var queryArgs []string p := NewParser(&opt)
var q string
for i := 0; i < len(args); i++ { p.AddPositionalValue(&q, "query", 1, false, "search query")
arg := args[i] if err := cli.Parse(p, args); err != nil {
wantsValue := arg == "--root" || arg == "--repo" || arg == "-n" || arg == "--hop" return opt, err
if wantsValue && i+1 >= len(args) {
return opt, fmt.Errorf("%s needs a value", arg)
} }
switch arg { opt.Query = cli.Query(q, p.TrailingArguments)
case "--root": if opt.Root != "" && opt.Root != "facts" && opt.Root != "info" {
i++
opt.Root = args[i]
if opt.Root != "facts" && opt.Root != "info" {
return opt, fmt.Errorf("--root must be facts or info, got %q", opt.Root) return opt, fmt.Errorf("--root must be facts or info, got %q", opt.Root)
} }
case "--repo": if opt.Limit < 1 {
i++ return opt, fmt.Errorf("-n must be a positive integer, got %q", strconv.Itoa(opt.Limit))
opt.Repo = args[i]
case "-n":
i++
n, err := strconv.Atoi(args[i])
if err != nil || n < 1 {
return opt, fmt.Errorf("-n must be a positive integer, got %q", args[i])
} }
opt.Limit = n if opt.Hop < 0 {
case "--hop": return opt, fmt.Errorf("--hop must be a positive integer, got %q", strconv.Itoa(opt.Hop))
return opt, fmt.Errorf("--hop is not implemented yet (needs File/FROM_FILE edges)")
case "--json":
opt.JSONOut = true
case "--no-web":
opt.NoWeb = true
case "--list-model":
opt.ListModel = true
default:
if strings.HasPrefix(arg, "-") {
return opt, fmt.Errorf("unknown flag %q", arg)
} }
queryArgs = append(queryArgs, arg) if opt.Hop > 3 {
return opt, fmt.Errorf("--hop max is 3 (File → Commit → Person)")
} }
if opt.AsOf != "" {
day := facts.NormalizeDay(opt.AsOf)
if len(day) != 10 || day[4] != '-' || day[7] != '-' {
return opt, fmt.Errorf("--as-of must be YYYY-MM-DD, got %q", opt.AsOf)
}
opt.AsOf = day
} }
opt.Query = strings.TrimSpace(strings.Join(queryArgs, " "))
if opt.Query == "" && !opt.ListModel { if opt.Query == "" && !opt.ListModel {
return opt, fmt.Errorf("no query given") return opt, fmt.Errorf("no query given")
} }
return opt, nil return opt, nil
} }
// Parser is the search schema for bin/cli/complete.go.
func Parser() *flaggy.Parser {
opt := Options{Limit: 20}
p := NewParser(&opt)
var q string
p.AddPositionalValue(&q, "query", 1, false, "search query")
return p
}
+34
View File
@@ -0,0 +1,34 @@
package rank
import "testing"
func TestFilterAsOfKeepsXDropsY(t *testing.T) {
hits := []Hit{
{ID: "x", Text: "Andrey works at X", ValidFrom: "2024-03-01", ValidTo: "2025-07-15"},
{ID: "y", Text: "Andrey works at Y", ValidFrom: "2025-07-16", ValidTo: ""},
{ID: "legacy", Text: "always true claim", ValidFrom: "", ValidTo: ""},
}
out := FilterAsOf(hits, "2025-01-01")
if len(out) != 2 {
t.Fatalf("len=%d want 2: %+v", len(out), out)
}
if out[0].ID != "x" || out[1].ID != "legacy" {
t.Fatalf("got %+v", out)
}
if FilterAsOf(hits, "") == nil || len(FilterAsOf(hits, "")) != 3 {
t.Fatal("empty as-of must keep all")
}
}
func TestParseArgsAsOf(t *testing.T) {
opt, err := ParseArgs([]string{"who works where", "--as-of", "2025-01-01", "--json"})
if err != nil {
t.Fatal(err)
}
if opt.AsOf != "2025-01-01" {
t.Fatalf("AsOf=%q", opt.AsOf)
}
if _, err := ParseArgs([]string{"q", "--as-of", "not-a-date"}); err == nil {
t.Fatal("expected bad as-of error")
}
}
+17 -2
View File
@@ -19,20 +19,35 @@ type SecondSourceHit struct {
type WebFn func(query string) SecondSource type WebFn func(query string) SecondSource
// ShouldEscalate is true when the default deduction path has no facts hit. // ShouldEscalate is true when the default deduction path has no confirmed
// facts hit. Hypothesis/partial facts are `(not confirmed)` (D16).
// `--root facts|info` is a single-root ask: do not mix in the web. // `--root facts|info` is a single-root ask: do not mix in the web.
func ShouldEscalate(hits []Hit, rootFilter string) bool { func ShouldEscalate(hits []Hit, rootFilter string) bool {
if rootFilter != "" { if rootFilter != "" {
return false return false
} }
for _, h := range hits { for _, h := range hits {
if h.Root == "facts" { if ConfirmedFact(h) {
return false return false
} }
} }
return true return true
} }
// ConfirmedFact is a facts-root hit that is not hypothesis/partial.
// Empty confidence is treated as confirmed (legacy leafs).
func ConfirmedFact(h Hit) bool {
if h.Root != "facts" {
return false
}
switch h.Confidence {
case "hypothesis", "partial":
return false
default:
return true
}
}
// Deduce returns the second-source block, or nil when web must not run. // Deduce returns the second-source block, or nil when web must not run.
func Deduce(hits []Hit, query, rootFilter string, noWeb bool, web WebFn) *SecondSource { func Deduce(hits []Hit, query, rootFilter string, noWeb bool, web WebFn) *SecondSource {
if noWeb || web == nil || !ShouldEscalate(hits, rootFilter) { if noWeb || web == nil || !ShouldEscalate(hits, rootFilter) {
+10
View File
@@ -14,6 +14,16 @@ func TestShouldEscalateWhenNoFacts(t *testing.T) {
} }
} }
func TestShouldEscalateWhenHypothesisFacts(t *testing.T) {
hyp := Hit{ID: "c", Root: "facts", Confidence: "hypothesis", Source: "a x b vs c x d"}
if !ShouldEscalate([]Hit{hyp}, "") {
t.Fatal("hypothesis facts are (not confirmed); escalate")
}
if ConfirmedFact(hyp) {
t.Fatal("hypothesis is not confirmed")
}
}
func TestShouldNotEscalateWhenFactsConfirm(t *testing.T) { func TestShouldNotEscalateWhenFactsConfirm(t *testing.T) {
hits := []Hit{h("f", "facts", "docker ps x compose"), h("i", "info", "docs/a.md")} hits := []Hit{h("f", "facts", "docker ps x compose"), h("i", "info", "docs/a.md")}
if ShouldEscalate(hits, "") { if ShouldEscalate(hits, "") {
+31 -2
View File
@@ -3,7 +3,36 @@ package rank
// BM25 ranks best-first, so the top hits are the *highest* scores; cosine // BM25 ranks best-first, so the top hits are the *highest* scores; cosine
// distance ranks best-first ascending. Both mirror kblib.py. // distance ranks best-first ascending. Both mirror kblib.py.
const FTSStmt = "CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " + const FTSStmt = "CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " +
"RETURN node.id, node.text, node.root, node.source, score ORDER BY score DESC LIMIT $n" "RETURN node.id, node.text, node.root, node.source, score, node.confidence, " +
"node.valid_from, node.valid_to ORDER BY score DESC LIMIT $n"
const VecStmt = "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " + const VecStmt = "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n" "RETURN node.id, node.text, node.root, node.source, distance, node.confidence, " +
"node.valid_from, node.valid_to ORDER BY distance LIMIT $n"
// HopStmt is the Cypher walk from a search hit. Depth 1 = File, 2 = Commit, 3 = Person.
func HopStmt(depth int) string {
switch depth {
case 1:
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File) RETURN f.id, f.path, 1"
case 2:
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit) RETURN c.id, c.subject, 2"
case 3:
return "MATCH (l:Leaf {id:$id})-[:FROM_FILE]->(f:File)-[:HAS_VERSION]->(c:Commit)-[:AUTHORED]->(p:Person) RETURN p.id, p.name, 3"
default:
return ""
}
}
func HopLabel(depth int) string {
switch depth {
case 1:
return "File"
case 2:
return "Commit"
case 3:
return "Person"
default:
return ""
}
}
+40 -4
View File
@@ -5,26 +5,44 @@ package rank
import ( import (
"sort" "sort"
"strings" "strings"
"github.com/eSlider/2dph/internal/facts"
) )
type HopNode struct {
ID string `json:"id"`
Label string `json:"label"`
Name string `json:"name"`
Depth int `json:"depth"`
}
// Hit is one search result, mirroring the python script's dict shape. // Hit is one search result, mirroring the python script's dict shape.
type Hit struct { type Hit struct {
ID string `json:"id"` ID string `json:"id"`
Text string `json:"text"` Text string `json:"text"`
Root string `json:"root"` Root string `json:"root"`
Confidence string `json:"confidence,omitempty"`
Source string `json:"-"` Source string `json:"-"`
Score float64 `json:"score"` Score float64 `json:"score"`
Snippet string `json:"snippet,omitempty"` Snippet string `json:"snippet,omitempty"`
ValidFrom string `json:"valid_from,omitempty"`
ValidTo string `json:"valid_to,omitempty"`
Hops []HopNode `json:"hops,omitempty"`
} }
// rrfK dampens the contribution of low ranks; same constant as kblib.py. // rrfK dampens the contribution of low ranks; same constant as kblib.py.
const rrfK = 60 const rrfK = 60
// RankAndFilter fuses the two hit lists, applies --root/--repo, then cuts to // RankAndFilter fuses the two hit lists, applies --root/--repo/--as-of, then
// limit. Cutting first dropped every matching leaf ranked below the cut, so // cuts to limit. Cutting first dropped every matching leaf ranked below the
// `--root facts` came back empty whenever info leafs filled the top N. // cut, so `--root facts` came back empty whenever info leafs filled the top N.
// limit <= 0 keeps everything. // limit <= 0 keeps everything. asOf empty skips interval filter (D24).
func RankAndFilter(fts, vec []Hit, root, repo string, limit int) []Hit { func RankAndFilter(fts, vec []Hit, root, repo string, limit int) []Hit {
return RankAndFilterAsOf(fts, vec, root, repo, "", limit)
}
// RankAndFilterAsOf is RankAndFilter with D24 fact-interval filter.
func RankAndFilterAsOf(fts, vec []Hit, root, repo, asOf string, limit int) []Hit {
out := Hybrid(fts, vec, 0) out := Hybrid(fts, vec, 0)
if root != "" { if root != "" {
out = FilterRoot(out, root) out = FilterRoot(out, root)
@@ -32,12 +50,30 @@ func RankAndFilter(fts, vec []Hit, root, repo string, limit int) []Hit {
if repo != "" { if repo != "" {
out = FilterRepo(out, repo) out = FilterRepo(out, repo)
} }
if asOf != "" {
out = FilterAsOf(out, asOf)
}
if limit > 0 && len(out) > limit { if limit > 0 && len(out) > limit {
out = out[:limit] out = out[:limit]
} }
return out return out
} }
// FilterAsOf keeps hits whose [valid_from, valid_to] covers asOf (D24).
// Empty intervals stay (legacy leafs). Empty asOf keeps all.
func FilterAsOf(hits []Hit, asOf string) []Hit {
if asOf == "" {
return hits
}
var out []Hit
for _, h := range hits {
if facts.ActiveAt(h.ValidFrom, h.ValidTo, asOf) {
out = append(out, h)
}
}
return out
}
// Hybrid merges FTS and vector hits by reciprocal rank fusion. // Hybrid merges FTS and vector hits by reciprocal rank fusion.
// limit <= 0 returns the full fused list. // limit <= 0 returns the full fused list.
func Hybrid(fts, vec []Hit, limit int) []Hit { func Hybrid(fts, vec []Hit, limit int) []Hit {
+36 -7
View File
@@ -91,15 +91,41 @@ func TestHybridKeepsVectorScoreForSharedHit(t *testing.T) {
} }
// The old parser dropped unknown flags and appended their arguments to the // The old parser dropped unknown flags and appended their arguments to the
// query, so `search "q" --hop 1` searched for "q 1". --hop is not implemented // query, so `search "q" --hop 1` searched for "q 1". --hop must stay a flag.
// here (needs File edges); it must still fail closed instead of changing q.
func TestParseHopIsNotSwallowedIntoTheQuery(t *testing.T) { func TestParseHopIsNotSwallowedIntoTheQuery(t *testing.T) {
_, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"}) opt, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"})
if err == nil { if err != nil {
t.Fatal("expected --hop to error (not implemented), not be swallowed") t.Fatalf("unexpected error: %v", err)
} }
if !strings.Contains(err.Error(), "--hop") { if opt.Query != "what runs on arc-2" {
t.Fatalf("error should name --hop, got %v", err) t.Fatalf("query swallowed hop arg: %q", opt.Query)
}
if opt.Hop != 1 {
t.Fatalf("hop = %d, want 1", opt.Hop)
}
}
func TestParseHopMaxIsThree(t *testing.T) {
if _, err := ParseArgs([]string{"q", "--hop", "4"}); err == nil {
t.Fatal("expected --hop 4 to error")
}
opt, err := ParseArgs([]string{"q", "--hop", "3"})
if err != nil || opt.Hop != 3 {
t.Fatalf("hop 3: %+v err=%v", opt, err)
}
}
func TestHopStmtWalksFromFile(t *testing.T) {
s := HopStmt(1)
if !strings.Contains(s, "FROM_FILE") || !strings.Contains(s, "File") {
t.Fatalf("hop 1 must walk FROM_FILE, got %q", s)
}
s3 := HopStmt(3)
if !strings.Contains(s3, "HAS_VERSION") || !strings.Contains(s3, "AUTHORED") || !strings.Contains(s3, "Person") {
t.Fatalf("hop 3 must reach Person, got %q", s3)
}
if HopLabel(1) != "File" || HopLabel(3) != "Person" {
t.Fatal("hop labels")
} }
} }
@@ -146,4 +172,7 @@ func TestFTSQueryOrdersByScoreDescending(t *testing.T) {
if !strings.Contains(FTSStmt, "ORDER BY score DESC") { if !strings.Contains(FTSStmt, "ORDER BY score DESC") {
t.Fatalf("FTS query must order by score DESC, got:\n%s", FTSStmt) t.Fatalf("FTS query must order by score DESC, got:\n%s", FTSStmt)
} }
if !strings.Contains(FTSStmt, "node.confidence") {
t.Fatal("FTS must return confidence for D16")
}
} }
+59
View File
@@ -0,0 +1,59 @@
package rank
import (
"fmt"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type GetOptions struct {
ID string
Body bool
JSONOut bool
}
func GetParser(opt *GetOptions) *flaggy.Parser {
p := cli.New("brain-get")
p.Description = "read one leaf"
p.Bool(&opt.Body, "", "body", "full text instead of snippet")
p.Bool(&opt.JSONOut, "", "json", "JSON output")
p.AddPositionalValue(&opt.ID, "id", 1, false, "leaf id")
return p
}
func ParseGet(args []string) (GetOptions, error) {
var opt GetOptions
if err := cli.Parse(GetParser(&opt), args); err != nil {
return opt, err
}
if opt.ID == "" {
return opt, fmt.Errorf("id required")
}
return opt, nil
}
type JSONFlag struct {
JSONOut bool
}
func bindJSON(name string, opt *JSONFlag) *flaggy.Parser {
p := cli.New(name)
p.Bool(&opt.JSONOut, "", "json", "JSON output")
return p
}
func StatsParser() *flaggy.Parser {
opt := JSONFlag{}
return bindJSON("brain-stats", &opt)
}
func EvalParser() *flaggy.Parser {
opt := JSONFlag{}
return bindJSON("brain-eval", &opt)
}
func ParseJSONFlag(name string, args []string) (JSONFlag, error) {
var opt JSONFlag
return opt, cli.Parse(bindJSON(name, &opt), args)
}
+13 -48
View File
@@ -11,30 +11,15 @@ import (
"unicode/utf8" "unicode/utf8"
"github.com/eSlider/2dph/internal/brain/rank" "github.com/eSlider/2dph/internal/brain/rank"
"github.com/eSlider/2dph/internal/cli"
) )
func MainGet(args []string) int { func MainGet(args []string) int {
id, body, jsonOut := "", false, false opt, err := rank.ParseGet(args)
for _, a := range args { if err != nil {
switch { return cli.Fail(err)
case a == "--body":
body = true
case a == "--json":
jsonOut = true
case a == "-h" || a == "--help":
fmt.Fprintln(os.Stderr, `usage: bin/brain/get.go <id> [--body] [--json]`)
return 0
case strings.HasPrefix(a, "-"):
fmt.Fprintf(os.Stderr, "brain/get: unknown flag %s\n", a)
return 2
default:
id = a
}
}
if id == "" {
fmt.Fprintln(os.Stderr, "brain/get: id required")
return 2
} }
id, body, jsonOut := opt.ID, opt.Body, opt.JSONOut
if err := openBrain(); err != nil { if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err) fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1 return 1
@@ -72,21 +57,11 @@ func MainGet(args []string) int {
} }
func MainStats(args []string) int { func MainStats(args []string) int {
jsonOut := false opt, err := rank.ParseJSONFlag("brain-stats", args)
for _, a := range args { if err != nil {
switch a { return cli.Fail(err)
case "--json":
jsonOut = true
case "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/brain/stats.go [--json]`)
return 0
default:
if strings.HasPrefix(a, "-") {
fmt.Fprintf(os.Stderr, "brain/stats: unknown flag %s\n", a)
return 2
}
}
} }
jsonOut := opt.JSONOut
if err := openBrain(); err != nil { if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err) fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1 return 1
@@ -124,21 +99,11 @@ func MainStats(args []string) int {
} }
func MainEval(args []string) int { func MainEval(args []string) int {
jsonOut := false opt, err := rank.ParseJSONFlag("brain-eval", args)
for _, a := range args { if err != nil {
switch a { return cli.Fail(err)
case "--json":
jsonOut = true
case "-h", "--help":
fmt.Fprintln(os.Stderr, `usage: bin/brain/eval.go [--json]`)
return 0
default:
if strings.HasPrefix(a, "-") {
fmt.Fprintf(os.Stderr, "brain/eval: unknown flag %s\n", a)
return 2
}
}
} }
jsonOut := opt.JSONOut
if err := openBrain(); err != nil { if err := openBrain(); err != nil {
fmt.Fprintf(os.Stderr, "open brain: %v\n", err) fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
return 1 return 1
+109 -6
View File
@@ -20,6 +20,7 @@ import (
lbug "github.com/LadybugDB/go-ladybug" lbug "github.com/LadybugDB/go-ladybug"
"github.com/eSlider/2dph/internal/brain/rank" "github.com/eSlider/2dph/internal/brain/rank"
"github.com/eSlider/2dph/internal/cli"
) )
const defaultPort = 17830 const defaultPort = 17830
@@ -29,6 +30,9 @@ const healthPath = "/health"
func runSearch(args []string) int { func runSearch(args []string) int {
opt, err := rank.ParseArgs(args) opt, err := rank.ParseArgs(args)
if err != nil { if err != nil {
if errors.Is(err, cli.ErrHelp) {
return 0
}
fmt.Fprintf(os.Stderr, "brain/search: %v\n%s\n", err, rank.Usage) fmt.Fprintf(os.Stderr, "brain/search: %v\n%s\n", err, rank.Usage)
return 2 return 2
} }
@@ -51,11 +55,17 @@ func runSearch(args []string) int {
} }
defer closeBrain() defer closeBrain()
hits, err := searchHits(query, root, repo, limit) hits, err := searchHits(query, root, repo, limit, opt.AsOf)
if err != nil { if err != nil {
fmt.Fprintf(os.Stderr, "search: %v\n", err) fmt.Fprintf(os.Stderr, "search: %v\n", err)
return 1 return 1
} }
if opt.Hop > 0 {
if err := attachHops(hits, opt.Hop); err != nil {
fmt.Fprintf(os.Stderr, "hop: %v\n", err)
return 1
}
}
results := hits results := hits
for i := range results { for i := range results {
@@ -75,6 +85,7 @@ func runSearch(args []string) int {
out := Dict{ out := Dict{
{"query", query}, {"query", query},
{"root_filter", root}, {"root_filter", root},
{"as_of", opt.AsOf},
{"count", len(results)}, {"count", len(results)},
{"results", resultsToDicts(results)}, {"results", resultsToDicts(results)},
} }
@@ -86,13 +97,13 @@ func runSearch(args []string) int {
enc := json.NewEncoder(os.Stdout) enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ") enc.SetIndent("", " ")
enc.SetEscapeHTML(false) enc.SetEscapeHTML(false)
return b2i(enc.Encode(toJSONOut(results, query, root, webOut))) return b2i(enc.Encode(toJSONOut(results, query, root, opt.AsOf, webOut)))
} }
fmt.Print(toYAML(out, 0)) fmt.Print(toYAML(out, 0))
return 0 return 0
} }
func searchHits(query, root, repo string, limit int) ([]Hit, error) { func searchHits(query, root, repo string, limit int, asOf string) ([]Hit, error) {
emb, err := embedQuery(query) emb, err := embedQuery(query)
if err != nil { if err != nil {
return nil, fmt.Errorf("embed: %w", err) return nil, fmt.Errorf("embed: %w", err)
@@ -105,7 +116,45 @@ func searchHits(query, root, repo string, limit int) ([]Hit, error) {
if vec, err = queryVector(emb, limit*3); err != nil { if vec, err = queryVector(emb, limit*3); err != nil {
fmt.Fprintf(os.Stderr, "vec: %v\n", err) fmt.Fprintf(os.Stderr, "vec: %v\n", err)
} }
return rank.RankAndFilter(fts, vec, root, repo, limit), nil return rank.RankAndFilterAsOf(fts, vec, root, repo, asOf, limit), nil
}
func attachHops(hits []Hit, n int) error {
if conn == nil {
return fmt.Errorf("brain not open")
}
for i := range hits {
var hops []rank.HopNode
for d := 1; d <= n; d++ {
stmt, err := conn.Prepare(rank.HopStmt(d))
if err != nil {
return err
}
res, err := conn.Execute(stmt, map[string]any{"id": hits[i].ID})
stmt.Close()
if err != nil {
return err
}
for res.HasNext() {
row, err := res.Next()
if err != nil {
return err
}
vals, err := row.GetAsSlice()
if err != nil || len(vals) < 3 {
continue
}
hops = append(hops, rank.HopNode{
ID: fmt.Sprint(vals[0]),
Label: rank.HopLabel(d),
Name: fmt.Sprint(vals[1]),
Depth: int(asInt(vals[2])),
})
}
}
hits[i].Hops = hops
}
return nil
} }
func b2i(err error) int { func b2i(err error) int {
@@ -168,15 +217,39 @@ func rowsToHits(res *lbug.QueryResult) ([]Hit, error) {
root := fmt.Sprint(vals[2]) root := fmt.Sprint(vals[2])
source := fmt.Sprint(vals[3]) source := fmt.Sprint(vals[3])
score := float64(vals[4].(float64)) score := float64(vals[4].(float64))
hits = append(hits, Hit{ID: id, Text: text, Root: root, Source: source, Score: score}) conf := ""
if len(vals) >= 6 {
conf = fmt.Sprint(vals[5])
}
vf, vt := "", ""
if len(vals) >= 8 {
vf = nullStr(vals[6])
vt = nullStr(vals[7])
}
hits = append(hits, Hit{
ID: id, Text: text, Root: root, Source: source, Score: score,
Confidence: conf, ValidFrom: vf, ValidTo: vt,
})
} }
return hits, nil return hits, nil
} }
func nullStr(v any) string {
if v == nil {
return ""
}
s := fmt.Sprint(v)
if s == "<nil>" {
return ""
}
return s
}
// JSON output types // JSON output types
type jsonOut struct { type jsonOut struct {
Query string `json:"query"` Query string `json:"query"`
RootFilter string `json:"root_filter"` RootFilter string `json:"root_filter"`
AsOf string `json:"as_of,omitempty"`
Count int `json:"count"` Count int `json:"count"`
Results []jsonHit `json:"results"` Results []jsonHit `json:"results"`
Web *rank.SecondSource `json:"web,omitempty"` Web *rank.SecondSource `json:"web,omitempty"`
@@ -186,24 +259,33 @@ type jsonHit struct {
ID string `json:"id"` ID string `json:"id"`
Text string `json:"text"` Text string `json:"text"`
Root string `json:"root"` Root string `json:"root"`
Confidence string `json:"confidence,omitempty"`
Score float64 `json:"score"` Score float64 `json:"score"`
Snippet string `json:"snippet,omitempty"` Snippet string `json:"snippet,omitempty"`
ValidFrom string `json:"valid_from,omitempty"`
ValidTo string `json:"valid_to,omitempty"`
Hops []rank.HopNode `json:"hops,omitempty"`
} }
func toJSONOut(hits []Hit, query, rootFilter string, web *rank.SecondSource) *jsonOut { func toJSONOut(hits []Hit, query, rootFilter, asOf string, web *rank.SecondSource) *jsonOut {
out := make([]jsonHit, len(hits)) out := make([]jsonHit, len(hits))
for i, h := range hits { for i, h := range hits {
out[i] = jsonHit{ out[i] = jsonHit{
ID: h.ID, ID: h.ID,
Text: h.Text, Text: h.Text,
Root: h.Root, Root: h.Root,
Confidence: h.Confidence,
Score: h.Score, Score: h.Score,
Snippet: h.Snippet, Snippet: h.Snippet,
ValidFrom: h.ValidFrom,
ValidTo: h.ValidTo,
Hops: h.Hops,
} }
} }
return &jsonOut{ return &jsonOut{
Query: query, Query: query,
RootFilter: rootFilter, RootFilter: rootFilter,
AsOf: asOf,
Count: len(hits), Count: len(hits),
Results: out, Results: out,
Web: web, Web: web,
@@ -219,9 +301,30 @@ func resultsToDicts(hits []Hit) []any {
{"root", h.Root}, {"root", h.Root},
{"score", h.Score}, {"score", h.Score},
} }
if h.Confidence != "" {
d = append(d, KV{"confidence", h.Confidence})
}
if h.ValidFrom != "" {
d = append(d, KV{"valid_from", h.ValidFrom})
}
if h.ValidTo != "" {
d = append(d, KV{"valid_to", h.ValidTo})
}
if h.Snippet != "" { if h.Snippet != "" {
d = append(d, KV{"snippet", h.Snippet}) d = append(d, KV{"snippet", h.Snippet})
} }
if len(h.Hops) > 0 {
nodes := make([]any, len(h.Hops))
for j, n := range h.Hops {
nodes[j] = Dict{
{"id", n.ID},
{"label", n.Label},
{"name", n.Name},
{"depth", n.Depth},
}
}
d = append(d, KV{"hops", nodes})
}
out[i] = d out[i] = d
} }
return out return out
+6 -12
View File
@@ -3,12 +3,13 @@ package chats
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"flag"
"fmt" "fmt"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"strings" "strings"
cliparse "github.com/eSlider/2dph/internal/cli"
) )
type ooContact struct { type ooContact struct {
@@ -25,16 +26,9 @@ type ooContact struct {
} }
func RunApply(args []string) int { func RunApply(args []string) int {
fs := flag.NewFlagSet("chats apply", flag.ContinueOnError) dryRun, err := parseApplyFlags(args)
dryRun := fs.Bool("dry-run", false, "show what would be done without writing") if err != nil {
help := fs.Bool("help", false, "") return cliparse.Fail(err)
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats apply [--dry-run]")
return 0
} }
ooCLI := findOO() ooCLI := findOO()
@@ -128,7 +122,7 @@ func RunApply(args []string) int {
fmt.Printf("\nchats apply: %d actions to apply\n", len(resolved)) fmt.Printf("\nchats apply: %d actions to apply\n", len(resolved))
if *dryRun { if dryRun {
for _, r := range resolved { for _, r := range resolved {
switch r.Action { switch r.Action {
case "info-add": case "info-add":
+74
View File
@@ -0,0 +1,74 @@
package chats
import (
cliparse "github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type syncTelegramFlags struct {
Limit int
Phone string
}
type syncLinkedInFlags struct {
Limit int
Refresh bool
}
func SyncParser() *flaggy.Parser {
p := cliparse.New("chats-sync")
p.Description = "download chats to var/chats"
tg := flaggy.NewSubcommand("telegram")
li := flaggy.NewSubcommand("linkedin")
var limit int
var phone string
var refresh bool
tg.Int(&limit, "", "limit", "max messages per chat")
tg.String(&phone, "", "phone", "phone (default TELEGRAM_PHONE)")
li.Int(&limit, "", "limit", "max messages per conversation")
li.Bool(&refresh, "", "refresh", "refresh webtop session")
p.AttachSubcommand(tg, 1)
p.AttachSubcommand(li, 1)
return p
}
func ImportParser() *flaggy.Parser {
return cliparse.New("chats-import")
}
func FactsParser() *flaggy.Parser {
return cliparse.New("chats-facts")
}
func ApplyParser() *flaggy.Parser {
p := cliparse.New("chats-apply")
dry := false
p.Bool(&dry, "", "dry-run", "show without writing")
return p
}
func parseTelegramFlags(args []string) (syncTelegramFlags, error) {
var f syncTelegramFlags
p := cliparse.New("chats-sync-telegram")
p.Int(&f.Limit, "", "limit", "max messages per chat")
p.String(&f.Phone, "", "phone", "phone (default TELEGRAM_PHONE)")
return f, cliparse.Parse(p, args)
}
func parseLinkedInFlags(args []string) (syncLinkedInFlags, error) {
var f syncLinkedInFlags
p := cliparse.New("chats-sync-linkedin")
p.Int(&f.Limit, "", "limit", "max messages per conversation")
p.Bool(&f.Refresh, "", "refresh", "refresh webtop session")
return f, cliparse.Parse(p, args)
}
func parseApplyFlags(args []string) (dryRun bool, err error) {
p := cliparse.New("chats-apply")
p.Bool(&dryRun, "", "dry-run", "show without writing")
return dryRun, cliparse.Parse(p, args)
}
func parseNoFlags(name string, args []string) error {
return cliparse.Parse(cliparse.New(name), args)
}
+4 -10
View File
@@ -3,12 +3,13 @@ package chats
import ( import (
"bufio" "bufio"
"encoding/json" "encoding/json"
"flag"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"regexp" "regexp"
"strings" "strings"
cliparse "github.com/eSlider/2dph/internal/cli"
) )
var ( var (
@@ -76,15 +77,8 @@ type ExtractedFact struct {
} }
func RunFacts(args []string) int { func RunFacts(args []string) int {
fs := flag.NewFlagSet("chats facts", flag.ContinueOnError) if err := parseNoFlags("chats-facts", args); err != nil {
help := fs.Bool("help", false, "") return cliparse.Fail(err)
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats facts")
return 0
} }
root := Dir() root := Dir()
+4 -10
View File
@@ -4,25 +4,19 @@ import (
"bufio" "bufio"
"bytes" "bytes"
"encoding/json" "encoding/json"
"flag"
"fmt" "fmt"
"html" "html"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strings" "strings"
cliparse "github.com/eSlider/2dph/internal/cli"
) )
func RunImport(args []string) int { func RunImport(args []string) int {
fs := flag.NewFlagSet("chats import", flag.ContinueOnError) if err := parseNoFlags("chats-import", args); err != nil {
help := fs.Bool("help", false, "") return cliparse.Fail(err)
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats import")
return 0
} }
root := Dir() root := Dir()
+9 -14
View File
@@ -2,12 +2,13 @@ package chats
import ( import (
"context" "context"
"flag"
"fmt" "fmt"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"time" "time"
cliparse "github.com/eSlider/2dph/internal/cli"
) )
func checkLinkedInSession(userDataDir string) (bool, error) { func checkLinkedInSession(userDataDir string) (bool, error) {
@@ -29,18 +30,12 @@ func checkLinkedInSession(userDataDir string) (bool, error) {
} }
func RunSyncLinkedIn(args []string) int { func RunSyncLinkedIn(args []string) int {
fs := flag.NewFlagSet("chats sync linkedin", flag.ContinueOnError) f, err := parseLinkedInFlags(args)
limit := fs.Int("limit", 0, "max messages per conversation (0 = all)") if err != nil {
refresh := fs.Bool("refresh", false, "refresh session from live webtop browser before sync") return cliparse.Fail(err)
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats sync linkedin [--limit N] [--refresh]")
return 0
} }
limit := f.Limit
refresh := f.Refresh
userDataDir := envVar("LINKEDIN_USER_DATA_DIR", "") userDataDir := envVar("LINKEDIN_USER_DATA_DIR", "")
if userDataDir == "" { if userDataDir == "" {
@@ -48,7 +43,7 @@ func RunSyncLinkedIn(args []string) int {
userDataDir = home + "/.linkedin-mcp/profile" userDataDir = home + "/.linkedin-mcp/profile"
} }
if *refresh { if refresh {
if code := refreshLinkedInSession(userDataDir); code != 0 { if code := refreshLinkedInSession(userDataDir); code != 0 {
return code return code
} }
@@ -72,7 +67,7 @@ func RunSyncLinkedIn(args []string) int {
defer cancel() defer cancel()
start := time.Now() start := time.Now()
if err := src.Sync(ctx, Dir(), *limit); err != nil { if err := src.Sync(ctx, Dir(), limit); err != nil {
fmt.Fprintf(os.Stderr, "chats sync linkedin: %v\n", err) fmt.Fprintf(os.Stderr, "chats sync linkedin: %v\n", err)
return 1 return 1
} }
+9 -14
View File
@@ -2,33 +2,28 @@ package chats
import ( import (
"context" "context"
"flag"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strconv" "strconv"
"strings" "strings"
"time" "time"
cliparse "github.com/eSlider/2dph/internal/cli"
) )
func RunSyncTelegram(args []string) int { func RunSyncTelegram(args []string) int {
fs := flag.NewFlagSet("chats sync telegram", flag.ContinueOnError) f, err := parseTelegramFlags(args)
limit := fs.Int("limit", 0, "max messages per chat (0 = all)") if err != nil {
phone := fs.String("phone", "", "phone number (default env TELEGRAM_PHONE)") return cliparse.Fail(err)
help := fs.Bool("help", false, "")
fs.SetOutput(os.Stderr)
if err := fs.Parse(args); err != nil {
return 2
}
if *help {
fmt.Fprintln(os.Stderr, "usage: chats sync telegram [--limit N] [--phone PHONE]")
return 0
} }
limit := f.Limit
phone := f.Phone
apiIDStr := envVar("TELEGRAM_API_ID", "") apiIDStr := envVar("TELEGRAM_API_ID", "")
apiHash := envVar("TELEGRAM_API_HASH", "") apiHash := envVar("TELEGRAM_API_HASH", "")
sessionStr := envVar("TELEGRAM_SESSION_STRING", "") sessionStr := envVar("TELEGRAM_SESSION_STRING", "")
phoneNum := *phone phoneNum := phone
if phoneNum == "" { if phoneNum == "" {
phoneNum = envVar("TELEGRAM_PHONE", "") phoneNum = envVar("TELEGRAM_PHONE", "")
} }
@@ -78,7 +73,7 @@ func RunSyncTelegram(args []string) int {
defer cancel() defer cancel()
start := time.Now() start := time.Now()
if err := src.Sync(ctx, Dir(), *limit); err != nil { if err := src.Sync(ctx, Dir(), limit); err != nil {
fmt.Fprintf(os.Stderr, "chats sync telegram: %v\n", err) fmt.Fprintf(os.Stderr, "chats sync telegram: %v\n", err)
return 1 return 1
} }
+190
View File
@@ -0,0 +1,190 @@
// Package cli is the shared flaggy wrapper (D23).
//
// flaggy: zero deps, flags at any position, shell completion scripts.
// Individual tools keep ShowCompletion off so a query like "completion" is
// not stolen; dump scripts with bin/cli/complete.go.
package cli
import (
"errors"
"fmt"
"os"
"strings"
"sync"
"github.com/integrii/flaggy"
)
// ErrHelp means -h/--help was requested (exit 0).
var ErrHelp = errors.New("help")
var parseMu sync.Mutex
// New returns a per-call parser. Never reuse: flaggy parses once.
func New(name string) *flaggy.Parser {
p := flaggy.NewParser(name)
p.ShowVersionWithVersionFlag = false
p.ShowCompletion = false
// Extra positionals become TrailingArguments (search "two words --json").
// Unknown dash tokens are rejected in Parse after flaggy returns.
p.ShowHelpOnUnexpected = false
p.ShowHelpWithHFlag = true
return p
}
// Parse runs p.ParseArgs and turns flaggy's os.Exit into an error.
// Not safe to call in parallel (flaggy.PanicInsteadOfExit is process-global).
func Parse(p *flaggy.Parser, args []string) error {
parseMu.Lock()
defer parseMu.Unlock()
prev := flaggy.PanicInsteadOfExit
flaggy.PanicInsteadOfExit = true
defer func() { flaggy.PanicInsteadOfExit = prev }()
var exitMsg string
err := func() error {
defer func() {
if r := recover(); r != nil {
exitMsg = fmt.Sprint(r)
}
}()
return p.ParseArgs(args)
}()
if err != nil {
return err
}
if exitMsg != "" {
if strings.Contains(exitMsg, "code: 0") {
return ErrHelp
}
return errors.New(exitMsg)
}
if u := unknownFlags(p, args); len(u) > 0 {
return fmt.Errorf("unknown flag %q", u[0])
}
return nil
}
func unknownFlags(p *flaggy.Parser, args []string) []string {
flags := collectFlags(&p.Subcommand)
var out []string
skipNext := false
for _, a := range args {
if skipNext {
skipNext = false
continue
}
if a == "--" {
break
}
name, inline := flagName(a)
if name == "" {
continue
}
if name == "h" || name == "help" {
continue
}
f := findFlag(flags, name)
if f == nil {
out = append(out, a)
continue
}
if !inline && !isBoolFlag(f) {
skipNext = true
}
}
return out
}
func flagName(a string) (name string, inline bool) {
if a == "-" || !strings.HasPrefix(a, "-") {
return "", false
}
rest := strings.TrimLeft(a, "-")
name, _, inline = strings.Cut(rest, "=")
return name, inline
}
func collectFlags(sc *flaggy.Subcommand) []*flaggy.Flag {
out := append([]*flaggy.Flag{}, sc.Flags...)
for _, sub := range sc.Subcommands {
out = append(out, collectFlags(sub)...)
}
return out
}
func findFlag(flags []*flaggy.Flag, name string) *flaggy.Flag {
for _, f := range flags {
if f.HasName(name) {
return f
}
}
return nil
}
func isBoolFlag(f *flaggy.Flag) bool {
_, ok := f.AssignmentVar.(*bool)
return ok
}
// Query joins the first positional with leftover trailing words.
func Query(first string, trailing []string) string {
parts := make([]string, 0, 1+len(trailing))
if s := strings.TrimSpace(first); s != "" {
parts = append(parts, s)
}
for _, t := range trailing {
if s := strings.TrimSpace(t); s != "" {
parts = append(parts, s)
}
}
return strings.Join(parts, " ")
}
// Code maps parse errors to process exit codes (0 help, 2 usage).
func Code(err error) int {
if err == nil || errors.Is(err, ErrHelp) {
return 0
}
return 2
}
// Fail prints err unless it is help or a flaggy exit that already wrote stderr.
func Fail(err error) int {
if err == nil || errors.Is(err, ErrHelp) {
return 0
}
if strings.HasPrefix(err.Error(), "Panic instead of exit") {
return 2
}
fmt.Fprintln(os.Stderr, err)
return 2
}
// Tool is one shebang CLI for completion dump.
type Tool struct {
Path string
Name string
New func() *flaggy.Parser
}
// BashScript concatenates flaggy bash complete scripts and binds each
// function to the shebang path (./bin/subject/method.go).
func BashScript(tools []Tool) string {
var b strings.Builder
b.WriteString("# 2dph flaggy completions (D23). source <(./bin/cli/complete.go bash)\n")
for _, t := range tools {
p := t.New()
p.Name = t.Name
script := flaggy.GenerateBashCompletion(p)
b.WriteString(script)
fn := "_" + strings.ReplaceAll(t.Name, "-", "_") + "_complete"
if t.Path != "" && t.Path != t.Name {
fmt.Fprintf(&b, "complete -F %s %s\n", fn, t.Path)
if !strings.HasPrefix(t.Path, "./") {
fmt.Fprintf(&b, "complete -F %s ./%s\n", fn, t.Path)
}
}
}
return b.String()
}
+77
View File
@@ -0,0 +1,77 @@
package cli
import (
"errors"
"strings"
"testing"
"github.com/integrii/flaggy"
)
func TestParseBoolAndIntAnyPosition(t *testing.T) {
p := New("t")
jsonOut := false
n := 20
q := ""
p.Bool(&jsonOut, "", "json", "JSON")
p.Int(&n, "n", "n", "limit")
p.AddPositionalValue(&q, "query", 1, false, "q")
if err := Parse(p, []string{"two", "words", "--json", "-n", "5"}); err != nil {
t.Fatal(err)
}
got := Query(q, p.TrailingArguments)
if got != "two words" || !jsonOut || n != 5 {
t.Fatalf("q=%q json=%v n=%d", got, jsonOut, n)
}
}
func TestParseUnknownFlagIsError(t *testing.T) {
p := New("t")
jsonOut := false
p.Bool(&jsonOut, "", "json", "JSON")
if err := Parse(p, []string{"--nope"}); err == nil {
t.Fatal("unknown flag accepted")
}
}
func TestParseHelpIsErrHelp(t *testing.T) {
p := New("t")
jsonOut := false
p.Bool(&jsonOut, "", "json", "JSON")
err := Parse(p, []string{"--help"})
if !errors.Is(err, ErrHelp) {
t.Fatalf("got %v", err)
}
}
func TestParseMissingFlagValueIsError(t *testing.T) {
p := New("t")
n := 0
p.Int(&n, "", "hop", "hop")
if err := Parse(p, []string{"--hop"}); err == nil {
t.Fatal("expected missing value error")
}
}
func TestBashScriptNamesShebangPath(t *testing.T) {
out := BashScript([]Tool{{
Path: "bin/brain/search.go",
Name: "brain-search",
New: newSearchLike,
}})
if !strings.Contains(out, "--json") || !strings.Contains(out, "--hop") {
t.Fatalf("flags missing:\n%s", out)
}
if !strings.Contains(out, "complete -F") || !strings.Contains(out, "bin/brain/search.go") {
t.Fatalf("shebang complete missing:\n%s", out)
}
}
func newSearchLike() *flaggy.Parser {
p := New("brain-search")
jsonOut := false
hop := 0
p.Bool(&jsonOut, "", "json", "JSON")
p.Int(&hop, "", "hop", "graph hop")
return p
}
+24
View File
@@ -0,0 +1,24 @@
package cli
import "github.com/integrii/flaggy"
type QAStats struct {
JSONL string
}
func QAParser() *flaggy.Parser {
c := QAStats{}
return BindQA(&c)
}
func BindQA(c *QAStats) *flaggy.Parser {
p := New("qa-stats")
p.Description = "DuckDB quantiles / JSONL count"
p.String(&c.JSONL, "", "jsonl", "JSONL file (else stdin JSON [float,…])")
return p
}
func ParseQAStats(args []string) (QAStats, error) {
var c QAStats
return c, Parse(BindQA(&c), args)
}
+47
View File
@@ -0,0 +1,47 @@
// Package duckstats runs in-process DuckDB for columnar aggregates.
// Graph facts stay in Ladybug. Web-search KV cache stays modernc sqlite.
package duckstats
import (
"database/sql"
"fmt"
_ "github.com/duckdb/duckdb-go/v2"
)
type Stats struct {
N int `json:"n"`
Min float64 `json:"min"`
P50 float64 `json:"p50"`
P95 float64 `json:"p95"`
Max float64 `json:"max"`
Avg float64 `json:"avg"`
}
func Quantiles(samples []float64) (Stats, error) {
if len(samples) == 0 {
return Stats{}, fmt.Errorf("duckstats: empty samples")
}
db, err := sql.Open("duckdb", "")
if err != nil {
return Stats{}, err
}
defer db.Close()
var s Stats
err = db.QueryRow(`
SELECT count(v), min(v), quantile_cont(v, 0.5), quantile_cont(v, 0.95), max(v), avg(v)
FROM (SELECT unnest(?) AS v)`, samples).Scan(
&s.N, &s.Min, &s.P50, &s.P95, &s.Max, &s.Avg)
return s, err
}
func CountJSONL(path string) (int64, error) {
db, err := sql.Open("duckdb", "")
if err != nil {
return 0, err
}
defer db.Close()
var n int64
err = db.QueryRow(`SELECT count(*) FROM read_json_auto(?)`, path).Scan(&n)
return n, err
}
+51
View File
@@ -0,0 +1,51 @@
package duckstats
import (
"os"
"testing"
)
func TestQuantilesEmpty(t *testing.T) {
_, err := Quantiles(nil)
if err == nil {
t.Fatal("empty slice must error")
}
}
func TestQuantilesOdd(t *testing.T) {
s, err := Quantiles([]float64{1, 2, 3, 4, 5})
if err != nil {
t.Fatal(err)
}
if s.N != 5 {
t.Fatalf("n=%d", s.N)
}
if s.Min != 1 || s.Max != 5 {
t.Fatalf("min=%v max=%v", s.Min, s.Max)
}
if s.P50 != 3 {
t.Fatalf("p50=%v want 3", s.P50)
}
if s.Avg != 3 {
t.Fatalf("avg=%v want 3", s.Avg)
}
if s.P95 < 4.5 || s.P95 > 5 {
t.Fatalf("p95=%v want in [4.5,5]", s.P95)
}
}
func TestCountJSONL(t *testing.T) {
dir := t.TempDir()
p := dir + "/rows.jsonl"
body := "{\"ms\":1}\n{\"ms\":2}\n{\"ms\":3}\n"
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
n, err := CountJSONL(p)
if err != nil {
t.Fatal(err)
}
if n != 3 {
t.Fatalf("count=%d want 3", n)
}
}
+120
View File
@@ -0,0 +1,120 @@
// Package facts is cgo-free evidence rules (D16 contradictions).
package facts
import "strconv"
const (
ConfConfirmed = "confirmed"
ConfHypothesis = "hypothesis"
RuleUnresolved = "unresolved"
RuleTemporalFreshness = "temporal_freshness"
RuleAuthorityPairing = "authority_pairing"
RuleTwoSource = "two_source"
RuleSingleSource = "single_source"
KindRuntime = "runtime"
KindConfig = "config"
KindNarrative = "narrative"
)
// Source is one independent pointer on a yes or no side.
type Source struct {
ID string `json:"id"`
Kind string `json:"kind"`
When string `json:"when,omitempty"`
Stale bool `json:"stale,omitempty"`
}
// Claim is one assertion with yes/no evidence lists.
type Claim struct {
Text string `json:"text"`
Yes []Source `json:"yes"`
No []Source `json:"no"`
}
// Result is audit output. Confirmed=false means `(not confirmed)`.
type Result struct {
Text string `json:"text"`
Confidence string `json:"confidence"`
Confirmed bool `json:"confirmed"`
Rule string `json:"rule"`
Winner string `json:"winner,omitempty"`
YesN int `json:"yes"`
NoN int `json:"no"`
}
func independent(ss []Source) int {
seen := map[string]struct{}{}
for i, s := range ss {
id := s.ID
if id == "" {
id = s.Kind + "#" + strconv.Itoa(i)
}
seen[id] = struct{}{}
}
return len(seen)
}
func freshN(ss []Source) int {
n := 0
for _, s := range ss {
if !s.Stale {
n++
}
}
return n
}
func strongN(ss []Source) int {
n := 0
for _, s := range ss {
if s.Kind == KindRuntime || s.Kind == KindConfig {
n++
}
}
return n
}
func out(c Claim, conf, rule, winner string) Result {
return Result{
Text: c.Text,
Confidence: conf,
Confirmed: conf == ConfConfirmed,
Rule: rule,
Winner: winner,
YesN: independent(c.Yes),
NoN: independent(c.No),
}
}
// Adjudicate applies D16: ≥2 yes vs ≥2 no stays hypothesis until a rule fires.
// Order: temporal_freshness, then authority_pairing (A/B beats narrative C).
func Adjudicate(c Claim) Result {
yesN := independent(c.Yes)
noN := independent(c.No)
if yesN < 2 || noN < 2 {
if yesN >= 2 {
return out(c, ConfConfirmed, RuleTwoSource, "yes")
}
if noN >= 2 {
return out(c, ConfConfirmed, RuleTwoSource, "no")
}
return out(c, ConfHypothesis, RuleSingleSource, "")
}
yf, nf := freshN(c.Yes), freshN(c.No)
if yf >= 2 && nf < 2 {
return out(c, ConfConfirmed, RuleTemporalFreshness, "yes")
}
if nf >= 2 && yf < 2 {
return out(c, ConfConfirmed, RuleTemporalFreshness, "no")
}
ys, ns := strongN(c.Yes), strongN(c.No)
if ys >= 2 && ns < 2 {
return out(c, ConfConfirmed, RuleAuthorityPairing, "yes")
}
if ns >= 2 && ys < 2 {
return out(c, ConfConfirmed, RuleAuthorityPairing, "no")
}
return out(c, ConfHypothesis, RuleUnresolved, "")
}
+86
View File
@@ -0,0 +1,86 @@
package facts
import "testing"
func src(id, kind string, stale bool) Source {
return Source{ID: id, Kind: kind, Stale: stale}
}
func TestTwoVsTwoStaysHypothesis(t *testing.T) {
c := Claim{
Text: "svc listens on 443",
Yes: []Source{
src("docker-ps", KindRuntime, false),
src("compose", KindConfig, false),
},
No: []Source{
src("docker-ps-old", KindRuntime, false),
src("compose-old", KindConfig, false),
},
}
r := Adjudicate(c)
if r.Confirmed || r.Confidence != ConfHypothesis || r.Rule != RuleUnresolved {
t.Fatalf("2v2 must stay (not confirmed): %+v", r)
}
if r.Winner != "" {
t.Fatalf("unresolved must not name a winner: %+v", r)
}
}
func TestTemporalFreshnessResolvesStaleSide(t *testing.T) {
c := Claim{
Text: "svc listens on 443",
Yes: []Source{
src("docker-ps", KindRuntime, false),
src("compose", KindConfig, false),
},
No: []Source{
src("old-readme", KindNarrative, true),
src("old-wiki", KindNarrative, true),
},
}
r := Adjudicate(c)
if !r.Confirmed || r.Rule != RuleTemporalFreshness || r.Winner != "yes" {
t.Fatalf("fresh yes vs stale no: %+v", r)
}
}
func TestAuthorityPairingBeatsNarrative(t *testing.T) {
c := Claim{
Text: "svc listens on 443",
Yes: []Source{
src("docker-ps", KindRuntime, false),
src("compose", KindConfig, false),
},
No: []Source{
src("readme", KindNarrative, false),
src("wiki", KindNarrative, false),
},
}
r := Adjudicate(c)
if !r.Confirmed || r.Rule != RuleAuthorityPairing || r.Winner != "yes" {
t.Fatalf("A×B vs C×C: %+v", r)
}
}
func TestTwoSourceYesIsConfirmed(t *testing.T) {
c := Claim{
Text: "arc-1 runs Matrix",
Yes: []Source{
src("compose", KindConfig, false),
src("docker-ps", KindRuntime, false),
},
}
r := Adjudicate(c)
if !r.Confirmed || r.Rule != RuleTwoSource || r.Winner != "yes" {
t.Fatalf("%+v", r)
}
}
func TestSingleSourceIsHypothesis(t *testing.T) {
c := Claim{Text: "maybe", Yes: []Source{src("readme", KindNarrative, false)}}
r := Adjudicate(c)
if r.Confirmed || r.Rule != RuleSingleSource {
t.Fatalf("%+v", r)
}
}
+44
View File
@@ -0,0 +1,44 @@
package facts
// Interval of truth for a fact leaf (D24 / OQ5). Not D16 source staleness.
//
// Empty valid_from and valid_to means "always" (legacy leafs). Empty asOf
// means "do not filter". Dates compare as YYYY-MM-DD (lexicographic).
// NormalizeDay keeps the calendar day from ISO-8601 or bare dates.
func NormalizeDay(s string) string {
s = trimSpace(s)
if len(s) >= 10 && s[4] == '-' && s[7] == '-' {
return s[:10]
}
return s
}
func trimSpace(s string) string {
i, j := 0, len(s)
for i < j && (s[i] == ' ' || s[i] == '\t' || s[i] == '\n' || s[i] == '\r') {
i++
}
for j > i && (s[j-1] == ' ' || s[j-1] == '\t' || s[j-1] == '\n' || s[j-1] == '\r') {
j--
}
return s[i:j]
}
// ActiveAt reports whether a fact with [validFrom, validTo] holds at asOf.
// validTo empty = open-ended. Both ends inclusive.
func ActiveAt(validFrom, validTo, asOf string) bool {
asOf = NormalizeDay(asOf)
if asOf == "" {
return true
}
from := NormalizeDay(validFrom)
to := NormalizeDay(validTo)
if from != "" && asOf < from {
return false
}
if to != "" && asOf > to {
return false
}
return true
}
+73
View File
@@ -0,0 +1,73 @@
package facts
import "testing"
func TestActiveAtOpenEnded(t *testing.T) {
// works at Y from 2025-07-16, no end
if !ActiveAt("2025-07-16", "", "2025-07-16") {
t.Fatal("inclusive valid_from")
}
if !ActiveAt("2025-07-16", "", "2026-01-01") {
t.Fatal("open-ended valid_to")
}
if ActiveAt("2025-07-16", "", "2025-07-15") {
t.Fatal("before valid_from must be inactive")
}
}
func TestActiveAtClosedInterval(t *testing.T) {
// works at X 2024-03-01 .. 2025-07-15
if !ActiveAt("2024-03-01", "2025-07-15", "2025-01-01") {
t.Fatal("mid interval")
}
if !ActiveAt("2024-03-01", "2025-07-15", "2024-03-01") {
t.Fatal("inclusive start")
}
if !ActiveAt("2024-03-01", "2025-07-15", "2025-07-15") {
t.Fatal("inclusive end")
}
if ActiveAt("2024-03-01", "2025-07-15", "2025-07-16") {
t.Fatal("day after end")
}
if ActiveAt("2024-03-01", "2025-07-15", "2024-02-28") {
t.Fatal("day before start")
}
}
func TestActiveAtEmptyIntervalAlwaysTrue(t *testing.T) {
// legacy leafs without intervals stay visible for any as-of
if !ActiveAt("", "", "2025-01-01") {
t.Fatal("empty interval must remain active")
}
if !ActiveAt("", "", "") {
t.Fatal("no as-of means all active")
}
}
func TestActiveAtEmptyAsOfKeepsAll(t *testing.T) {
if !ActiveAt("2099-01-01", "2099-12-31", "") {
t.Fatal("empty as-of must not filter")
}
}
func TestAsOfPickXNotY(t *testing.T) {
// Acceptance from #36: as of 2025-01-01 → X, not Y
xFrom, xTo := "2024-03-01", "2025-07-15"
yFrom, yTo := "2025-07-16", ""
asOf := "2025-01-01"
if !ActiveAt(xFrom, xTo, asOf) {
t.Fatal("X must be active as of 2025-01-01")
}
if ActiveAt(yFrom, yTo, asOf) {
t.Fatal("Y must be inactive as of 2025-01-01")
}
}
func TestNormalizeDayTrimsTime(t *testing.T) {
if NormalizeDay("2025-01-01T12:00:00Z") != "2025-01-01" {
t.Fatalf("got %q", NormalizeDay("2025-01-01T12:00:00Z"))
}
if NormalizeDay("2025-01-01") != "2025-01-01" {
t.Fatalf("got %q", NormalizeDay("2025-01-01"))
}
}
+51
View File
@@ -0,0 +1,51 @@
package gitlog
import (
"fmt"
"time"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Repo, Root, Since string
Limit int
JSONOut bool
}
func Parser() *flaggy.Parser {
c := CLI{}
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
p := cli.New("git-import")
p.Description = "go-git history → commit leafs"
p.Bool(&c.JSONOut, "", "json", "JSON output")
p.Int(&c.Limit, "", "limit", "max commits (0 = all)")
p.String(&c.Since, "", "since", "RFC3339 or YYYY-MM-DD")
p.String(&c.Root, "", "root", "scan dir for git repos")
p.AddPositionalValue(&c.Repo, "repo", 1, false, "git repo path")
return p
}
func ParseArgs(args []string) (CLI, error) {
var c CLI
if err := cli.Parse(Bind(&c), args); err != nil {
return c, err
}
return c, nil
}
func ParseSince(s string) (time.Time, error) {
if s == "" {
return time.Time{}, nil
}
for _, layout := range []string{time.RFC3339, "2006-01-02"} {
if t, err := time.Parse(layout, s); err == nil {
return t, nil
}
}
return time.Time{}, fmt.Errorf("cannot parse --since %q", s)
}
+17 -2
View File
@@ -105,11 +105,18 @@ func (s *Server) mcpCall(r *http.Request, params json.RawMessage) (any, error) {
if limit < 1 || limit > 100 { if limit < 1 || limit > 100 {
return mcpText(`{"error":"n must be int 1..100"}`, true), nil return mcpText(`{"error":"n must be int 1..100"}`, true), nil
} }
asOf := ""
if raw, ok := p.Arguments["as_of"]; ok {
asOf = strings.TrimSpace(fmt.Sprint(raw))
if asOf == "<nil>" {
asOf = ""
}
}
if !s.tryAcquire(r) { if !s.tryAcquire(r) {
return nil, fmt.Errorf("cancelled") return nil, fmt.Errorf("cancelled")
} }
defer s.release() defer s.release()
body, err = s.api.Search(r.Context(), q, limit) body, err = s.api.Search(r.Context(), q, limit, asOf)
case "get": case "get":
id := strings.TrimSpace(fmt.Sprint(p.Arguments["id"])) id := strings.TrimSpace(fmt.Sprint(p.Arguments["id"]))
if id == "" || id == "<nil>" { if id == "" || id == "<nil>" {
@@ -144,7 +151,15 @@ func (s *Server) mcpCall(r *http.Request, params json.RawMessage) (any, error) {
return nil, fmt.Errorf("cancelled") return nil, fmt.Errorf("cancelled")
} }
defer s.release() defer s.release()
body, err = s.api.Ingest(r.Context()) var payload []byte
text := strings.TrimSpace(fmt.Sprint(p.Arguments["text"]))
if text != "" && text != "<nil>" {
payload, err = json.Marshal(p.Arguments)
if err != nil {
return nil, err
}
}
body, err = s.api.Ingest(r.Context(), payload)
default: default:
return nil, fmt.Errorf("unknown tool %s", p.Name) return nil, fmt.Errorf("unknown tool %s", p.Name)
} }
+53 -9
View File
@@ -11,6 +11,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"io"
"log" "log"
"net/http" "net/http"
"os" "os"
@@ -23,11 +24,11 @@ import (
// API is the in-process brain surface. Production serve.go wires internal/brain. // API is the in-process brain surface. Production serve.go wires internal/brain.
type API interface { type API interface {
Search(ctx context.Context, query string, limit int) ([]byte, error) Search(ctx context.Context, query string, limit int, asOf string) ([]byte, error)
Get(ctx context.Context, id string, body bool) ([]byte, error) Get(ctx context.Context, id string, body bool) ([]byte, error)
Stats(ctx context.Context) ([]byte, error) Stats(ctx context.Context) ([]byte, error)
Audit(ctx context.Context) ([]byte, error) Audit(ctx context.Context) ([]byte, error)
Ingest(ctx context.Context) ([]byte, error) Ingest(ctx context.Context, body []byte) ([]byte, error)
} }
type Server struct { type Server struct {
@@ -59,7 +60,7 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
case PathAudit: case PathAudit:
s.handleJSON(w, r, s.api.Audit) s.handleJSON(w, r, s.api.Audit)
case PathIngest: case PathIngest:
s.handleJSON(w, r, s.api.Ingest) s.handleIngest(w, r)
case PathOpenAPI: case PathOpenAPI:
s.handleOpenAPI(w, r) s.handleOpenAPI(w, r)
case PathMCP: case PathMCP:
@@ -84,11 +85,12 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
} }
limit = n limit = n
} }
asOf := strings.TrimSpace(r.URL.Query().Get("as_of"))
if !s.acquire(w, r) { if !s.acquire(w, r) {
return return
} }
defer s.release() defer s.release()
body, err := s.api.Search(r.Context(), q, limit) body, err := s.api.Search(r.Context(), q, limit, asOf)
writeAPI(w, body, err) writeAPI(w, body, err)
} }
@@ -116,6 +118,24 @@ func (s *Server) handleJSON(w http.ResponseWriter, r *http.Request, fn func(cont
writeAPI(w, body, err) writeAPI(w, body, err)
} }
func (s *Server) handleIngest(w http.ResponseWriter, r *http.Request) {
var raw []byte
if r.Method == http.MethodPost {
b, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err != nil {
writeJSON(w, http.StatusBadRequest, map[string]any{"error": "read body"})
return
}
raw = b
}
if !s.acquire(w, r) {
return
}
defer s.release()
body, err := s.api.Ingest(r.Context(), raw)
writeAPI(w, body, err)
}
func (s *Server) tryAcquire(r *http.Request) bool { func (s *Server) tryAcquire(r *http.Request) bool {
return s.acquire(nopWriter{}, r) return s.acquire(nopWriter{}, r)
} }
@@ -168,13 +188,18 @@ type ExecSearcher struct {
Timeout time.Duration Timeout time.Duration
} }
func (b ExecSearcher) Search(ctx context.Context, query string, limit int) ([]byte, error) { func (b ExecSearcher) Search(ctx context.Context, query string, limit int, asOf string) ([]byte, error) {
if b.Timeout == 0 { if b.Timeout == 0 {
b.Timeout = 60 * time.Second b.Timeout = 60 * time.Second
} }
ctx, cancel := context.WithTimeout(ctx, b.Timeout) ctx, cancel := context.WithTimeout(ctx, b.Timeout)
defer cancel() defer cancel()
cmd := exec.CommandContext(ctx, b.CmdPath, "--json", "-n", strconv.Itoa(limit), query) args := []string{"--json", "-n", strconv.Itoa(limit)}
if asOf != "" {
args = append(args, "--as-of", asOf)
}
args = append(args, query)
cmd := exec.CommandContext(ctx, b.CmdPath, args...)
out, err := cmd.Output() out, err := cmd.Output()
if err != nil { if err != nil {
var exitErr *exec.ExitError var exitErr *exec.ExitError
@@ -191,11 +216,30 @@ func (ExecSearcher) Get(context.Context, string, bool) ([]byte, error) {
} }
func (ExecSearcher) Stats(context.Context) ([]byte, error) { return nil, errUnimplemented } func (ExecSearcher) Stats(context.Context) ([]byte, error) { return nil, errUnimplemented }
func (ExecSearcher) Audit(context.Context) ([]byte, error) { return nil, errUnimplemented } func (ExecSearcher) Audit(context.Context) ([]byte, error) { return nil, errUnimplemented }
func (ExecSearcher) Ingest(context.Context) ([]byte, error) { func (b ExecSearcher) Ingest(ctx context.Context, body []byte) ([]byte, error) {
if len(strings.TrimSpace(string(body))) == 0 {
return json.Marshal(map[string]any{ return json.Marshal(map[string]any{
"mode": "rebuild", "mode": "add",
"command": "bin/brain/index.go --rebuild", "command": "bin/brain/add.go",
"rebuild": "bin/brain/index.go --rebuild",
}) })
}
root := os.Getenv("KB_ROOT")
if root == "" {
root = "."
}
cmd := exec.CommandContext(ctx, filepath.Join(root, "bin/kb/add"), "--json")
cmd.Stdin = strings.NewReader(string(body))
cmd.Dir = root
out, err := cmd.Output()
if err != nil {
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
return nil, errors.New("add failed: " + strings.TrimSpace(string(exitErr.Stderr)))
}
return nil, err
}
return out, nil
} }
func defaultSearchCmd(root string) string { func defaultSearchCmd(root string) string {
+32 -7
View File
@@ -1,6 +1,7 @@
package httpapi package httpapi
import ( import (
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"net/http" "net/http"
@@ -20,10 +21,10 @@ type fakeSearcher struct {
calls int calls int
active atomic.Int32 active atomic.Int32
maxSeen atomic.Int32 maxSeen atomic.Int32
callback func(q string, limit int) ([]byte, error) callback func(q string, limit int, asOf string) ([]byte, error)
} }
func (f *fakeSearcher) Search(ctx context.Context, query string, limit int) ([]byte, error) { func (f *fakeSearcher) Search(ctx context.Context, query string, limit int, asOf string) ([]byte, error) {
f.mu.Lock() f.mu.Lock()
f.calls++ f.calls++
f.mu.Unlock() f.mu.Unlock()
@@ -43,7 +44,7 @@ func (f *fakeSearcher) Search(ctx context.Context, query string, limit int) ([]b
} }
} }
if f.callback != nil { if f.callback != nil {
return f.callback(query, limit) return f.callback(query, limit, asOf)
} }
return []byte(`{"query":"` + query + `","count":0,"results":[]}`), nil return []byte(`{"query":"` + query + `","count":0,"results":[]}`), nil
} }
@@ -64,8 +65,11 @@ func (f *fakeSearcher) Audit(context.Context) ([]byte, error) {
return []byte(`{"status":"ok"}`), nil return []byte(`{"status":"ok"}`), nil
} }
func (f *fakeSearcher) Ingest(context.Context) ([]byte, error) { func (f *fakeSearcher) Ingest(_ context.Context, body []byte) ([]byte, error) {
return []byte(`{"mode":"rebuild","command":"bin/brain/index.go --rebuild"}`), nil if len(bytes.TrimSpace(body)) == 0 {
return []byte(`{"mode":"add","command":"bin/brain/add.go"}`), nil
}
return []byte(`{"mode":"add","ids":["fake-leaf"]}`), nil
} }
func (f *fakeSearcher) count() int { func (f *fakeSearcher) count() int {
@@ -105,7 +109,7 @@ func TestSearchMissingQuery(t *testing.T) {
} }
func TestSearchReturnsSearcherResult(t *testing.T) { func TestSearchReturnsSearcherResult(t *testing.T) {
fs := &fakeSearcher{callback: func(q string, limit int) ([]byte, error) { fs := &fakeSearcher{callback: func(q string, limit int, asOf string) ([]byte, error) {
return []byte(`{"query":"` + q + `","count":1,"results":[{"id":"x"}]}`), nil return []byte(`{"query":"` + q + `","count":1,"results":[{"id":"x"}]}`), nil
}} }}
h := NewServer(fs, 1) h := NewServer(fs, 1)
@@ -164,7 +168,7 @@ func TestSearchRejectsBadLimit(t *testing.T) {
} }
func TestGetLeaf(t *testing.T) { func TestGetLeaf(t *testing.T) {
fs := &fakeSearcher{callback: func(q string, limit int) ([]byte, error) { fs := &fakeSearcher{callback: func(q string, limit int, asOf string) ([]byte, error) {
return []byte(`{}`), nil return []byte(`{}`), nil
}} }}
h := NewServer(fs, 1) h := NewServer(fs, 1)
@@ -193,6 +197,27 @@ func TestStatsAuditIngest(t *testing.T) {
} }
} }
func TestIngestIsAddNotRebuildHint(t *testing.T) {
h := NewServer(&fakeSearcher{}, 1)
code, body := get(t, h, "/ingest")
if code != http.StatusOK {
t.Fatalf("GET /ingest code = %d body=%s", code, body)
}
if strings.Contains(string(body), `"add":"v2"`) || strings.Contains(string(body), "write is v2") {
t.Fatalf("GET /ingest still a v2 hint: %s", body)
}
if !strings.Contains(string(body), "bin/brain/add.go") {
t.Fatalf("GET /ingest should name add.go: %s", body)
}
code, body = postJSON(t, h, "/ingest", `{"text":"hello","root":"info","source":"t"}`)
if code != http.StatusOK {
t.Fatalf("POST /ingest code = %d body=%s", code, body)
}
if !strings.Contains(string(body), "fake-leaf") {
t.Fatalf("POST /ingest should add: %s", body)
}
}
func TestHTTPPackageDoesNotExecPython(t *testing.T) { func TestHTTPPackageDoesNotExecPython(t *testing.T) {
raw, err := os.ReadFile("server.go") raw, err := os.ReadFile("server.go")
if err != nil { if err != nil {
+12 -1
View File
@@ -35,6 +35,7 @@ var Ops = []Op{
Params: []Param{ Params: []Param{
{Name: "q", In: "query", Type: "string", Description: "search query", Required: true}, {Name: "q", In: "query", Type: "string", Description: "search query", Required: true},
{Name: "n", In: "query", Type: "integer", Description: "hit limit 1..100 (default 10)"}, {Name: "n", In: "query", Type: "integer", Description: "hit limit 1..100 (default 10)"},
{Name: "as_of", In: "query", Type: "string", Description: "YYYY-MM-DD; keep facts active on that day (D24)"},
}, },
}, },
{ {
@@ -47,7 +48,17 @@ var Ops = []Op{
}, },
{Path: PathStats, Method: "get", ID: "stats", Summary: "index health", MCP: true}, {Path: PathStats, Method: "get", ID: "stats", Summary: "index health", MCP: true},
{Path: PathAudit, Method: "get", ID: "audit", Summary: "facts confidence histogram", MCP: true}, {Path: PathAudit, Method: "get", ID: "audit", Summary: "facts confidence histogram", MCP: true},
{Path: PathIngest, Method: "get", ID: "ingest", Summary: "rebuild hint (write is v2)", MCP: true}, {
Path: PathIngest, Method: "post", ID: "ingest", Summary: "add a leaf without rebuild",
MCP: true,
Params: []Param{
{Name: "text", In: "query", Type: "string", Description: "leaf text (omit for CLI hint)"},
{Name: "root", In: "query", Type: "string", Description: "facts or info (default info)"},
{Name: "source", In: "query", Type: "string", Description: "evidence pointer; facts need two sources"},
{Name: "valid_from", In: "query", Type: "string", Description: "fact interval start YYYY-MM-DD (D24)"},
{Name: "valid_to", In: "query", Type: "string", Description: "fact interval end YYYY-MM-DD inclusive (D24)"},
},
},
{Path: PathOpenAPI, Method: "get", ID: "openapi", Summary: "OpenAPI 3 document for this server"}, {Path: PathOpenAPI, Method: "get", ID: "openapi", Summary: "OpenAPI 3 document for this server"},
} }
+14 -1
View File
@@ -38,11 +38,24 @@ func TestMCPToolsMatchOpenAPIPaths(t *testing.T) {
t.Fatalf("MCP tool %s has no OpenAPI path %s", tool.Name, path) t.Fatalf("MCP tool %s has no OpenAPI path %s", tool.Name, path)
} }
} }
for _, need := range []string{"search", "get", "stats", "audit"} { for _, need := range []string{"search", "get", "stats", "audit", "ingest"} {
if !names[need] { if !names[need] {
t.Fatalf("MCP tools missing %s: %v", need, names) t.Fatalf("MCP tools missing %s: %v", need, names)
} }
} }
var ingest MCPTool
for _, tool := range tools {
if tool.Name == "ingest" {
ingest = tool
break
}
}
if strings.Contains(ingest.Description, "v2") {
t.Fatalf("ingest still a v2 hint: %s", ingest.Description)
}
if !strings.Contains(ingest.Description, "add") {
t.Fatalf("ingest should describe add: %s", ingest.Description)
}
} }
func TestOpenAPIHTTP(t *testing.T) { func TestOpenAPIHTTP(t *testing.T) {
+41
View File
@@ -0,0 +1,41 @@
package mdleaves
import (
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Root string
Files string
JSONOut bool
}
func Parser() *flaggy.Parser {
c := CLI{Root: "."}
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
if c.Root == "" {
c.Root = "."
}
p := cli.New("markdown-import")
p.Description = "split markdown H2 leafs"
p.Bool(&c.JSONOut, "", "json", "JSON output")
p.String(&c.Files, "", "files", "comma-separated paths")
p.AddPositionalValue(&c.Root, "dir", 1, false, "markdown root")
return p
}
func ParseArgs(args []string) (CLI, error) {
c := CLI{Root: "."}
p := Bind(&c)
if err := cli.Parse(p, args); err != nil {
return c, err
}
if extra := cli.Query("", p.TrailingArguments); extra != "" && c.Root == "." {
c.Root = extra
}
return c, nil
}
+146
View File
@@ -0,0 +1,146 @@
package mdleaves
import (
"encoding/json"
"os"
"path/filepath"
"regexp"
"strings"
)
type Leaf struct {
Source string `json:"source"`
Repo string `json:"repo"`
Heading string `json:"heading"`
Text string `json:"text"`
Type string `json:"type"`
Status string `json:"status"`
Related string `json:"related"`
}
type chunk struct {
Heading string
Text string
}
var (
h1 = regexp.MustCompile(`^# \S`)
h2 = regexp.MustCompile(`^## \S`)
)
func ExtractFrontmatter(text string) (map[string]string, string) {
if !strings.HasPrefix(text, "---") {
return map[string]string{}, text
}
end := strings.Index(text[3:], "\n---")
if end == -1 {
return map[string]string{}, text
}
fm := strings.TrimSpace(text[3 : 3+end])
body := text[3+end+4:]
meta := map[string]string{}
for _, line := range strings.Split(fm, "\n") {
key, value, ok := strings.Cut(line, ":")
if !ok {
continue
}
meta[strings.TrimSpace(key)] = strings.Trim(strings.TrimSpace(value), `"'`)
}
return meta, body
}
func SplitLeafs(meta map[string]string, body string) []chunk {
lines := strings.Split(body, "\n")
title := ""
type hdr struct {
heading string
start int
}
var headers []hdr
for i, line := range lines {
switch {
case h1.MatchString(line):
title = strings.TrimSpace(strings.TrimLeft(line, "#"))
case h2.MatchString(line):
headers = append(headers, hdr{strings.TrimSpace(strings.TrimLeft(line, "#")), i})
}
}
if len(headers) == 0 {
var kept []string
for _, l := range lines {
if strings.TrimSpace(l) != "" {
kept = append(kept, l)
}
}
return []chunk{{Heading: title, Text: strings.TrimSpace(strings.Join(kept, "\n"))}}
}
out := make([]chunk, 0, len(headers))
for idx, h := range headers {
end := len(lines)
if idx+1 < len(headers) {
end = headers[idx+1].start
}
var kept []string
for _, l := range lines[h.start:end] {
if strings.TrimSpace(l) != "" {
kept = append(kept, l)
}
}
text := strings.Join(kept, "\n")
if idx == 0 && title != "" {
text = title + "\n\n" + text
}
out = append(out, chunk{Heading: h.heading, Text: strings.TrimSpace(text)})
}
return out
}
func ToAll(text, path, repo string) []Leaf {
meta, body := ExtractFrontmatter(text)
if meta["type"] == "" {
meta["type"] = "reference"
}
if meta["status"] == "" {
meta["status"] = "current"
}
chunks := SplitLeafs(meta, body)
out := make([]Leaf, 0, len(chunks))
for _, c := range chunks {
out = append(out, Leaf{
Source: path,
Repo: repo,
Heading: c.Heading,
Text: c.Text,
Type: meta["type"],
Status: meta["status"],
Related: meta["related"],
})
}
return out
}
func WalkMarkdown(root string) ([]string, error) {
var out []string
err := filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
ext := strings.ToLower(filepath.Ext(p))
if ext == ".md" || ext == ".markdown" {
out = append(out, p)
}
return nil
})
return out, err
}
func EncodeJSON(leafs []Leaf) (string, error) {
raw, err := json.MarshalIndent(leafs, "", " ")
if err != nil {
return "", err
}
return string(raw) + "\n", nil
}
+73
View File
@@ -0,0 +1,73 @@
package mdleaves
import (
"strings"
"testing"
)
func TestSplitLeafsOnH2(t *testing.T) {
body := "# Title\n\n## One\n\nalpha\n\n## Two\n\nbeta\n"
leafs := SplitLeafs(map[string]string{}, body)
if len(leafs) != 2 {
t.Fatalf("n=%d", len(leafs))
}
if leafs[0].Heading != "One" || !strings.Contains(leafs[0].Text, "Title") {
t.Fatalf("first=%+v", leafs[0])
}
if leafs[1].Heading != "Two" || strings.Contains(leafs[1].Text, "Title") {
t.Fatalf("second=%+v", leafs[1])
}
}
func TestSplitLeafsNoH2IsWholeDoc(t *testing.T) {
body := "# Title\n\njust a paragraph\n"
leafs := SplitLeafs(nil, body)
if len(leafs) != 1 {
t.Fatalf("n=%d", len(leafs))
}
if leafs[0].Heading != "Title" {
t.Fatalf("heading=%q", leafs[0].Heading)
}
if !strings.Contains(leafs[0].Text, "just a paragraph") {
t.Fatalf("text=%q", leafs[0].Text)
}
}
func TestFrontmatterAndToAll(t *testing.T) {
raw := "---\ntype: howto\nstatus: current\nrelated: docs/design.md\n---\n# Doc\n\n## Step\n\ndo it\n"
got := ToAll(raw, "docs/x.md", "eSlider/2dph")
if len(got) != 1 {
t.Fatalf("n=%d", len(got))
}
if got[0].Type != "howto" || got[0].Status != "current" {
t.Fatalf("%+v", got[0])
}
if got[0].Source != "docs/x.md" || got[0].Repo != "eSlider/2dph" {
t.Fatalf("%+v", got[0])
}
if got[0].Related != "docs/design.md" {
t.Fatalf("related=%q", got[0].Related)
}
}
func TestEncodeJSONAndYAML(t *testing.T) {
leafs := ToAll("# Hi\n\nbody\n", "a.md", "")
js, err := EncodeJSON(leafs)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(js, `"heading": "Hi"`) {
t.Fatalf("json=%s", js)
}
y := EncodeYAML(leafs)
if !strings.Contains(y, "heading: Hi") {
t.Fatalf("yaml=%s", y)
}
}
func TestDefaultsTypeAndStatus(t *testing.T) {
got := ToAll("# Hi\n\nbody\n", "a.md", "")
if got[0].Type != "reference" || got[0].Status != "current" {
t.Fatalf("%+v", got[0])
}
}
+36
View File
@@ -0,0 +1,36 @@
package mdleaves
import (
"fmt"
"strconv"
"strings"
)
func EncodeYAML(leafs []Leaf) string {
if len(leafs) == 0 {
return "[]\n"
}
var b strings.Builder
for _, lf := range leafs {
b.WriteString("-\n")
writeKV(&b, "source", lf.Source)
writeKV(&b, "repo", lf.Repo)
writeKV(&b, "heading", lf.Heading)
writeKV(&b, "text", lf.Text)
writeKV(&b, "type", lf.Type)
writeKV(&b, "status", lf.Status)
writeKV(&b, "related", lf.Related)
}
return b.String()
}
func writeKV(b *strings.Builder, k, v string) {
fmt.Fprintf(b, " %s: %s\n", k, yamlScalar(v))
}
func yamlScalar(s string) string {
if strings.Contains(s, "\n") || s == "" || strings.ContainsAny(s, ":#'\"[]{}&*!|>%@`") || s != strings.TrimSpace(s) {
return strconv.Quote(s)
}
return s
}
+35
View File
@@ -0,0 +1,35 @@
package ocr
import (
"fmt"
"github.com/eSlider/2dph/internal/cli"
"github.com/integrii/flaggy"
)
type CLI struct {
Path string
}
func Parser() *flaggy.Parser {
c := CLI{}
return Bind(&c)
}
func Bind(c *CLI) *flaggy.Parser {
p := cli.New("mail-ocr")
p.Description = "tesseract eng+deu on image or scanned PDF"
p.AddPositionalValue(&c.Path, "file", 1, false, "image or pdf")
return p
}
func ParseArgs(args []string) (CLI, error) {
var c CLI
if err := cli.Parse(Bind(&c), args); err != nil {
return c, err
}
if c.Path == "" {
return c, fmt.Errorf("usage: bin/mail/ocr.go <image|pdf>")
}
return c, nil
}
+162
View File
@@ -0,0 +1,162 @@
// Package ocr runs Tesseract (eng+deu) on images and scanned PDFs.
//
// Default engine is the tesseract CLI, not gosseract CGO: Ladybug CGO stays
// Zig-only (D21). Same engine, no gocv. OCR_ENGINE=paddle selects paddleocr
// when that binary is on PATH (compose profile ocr-paddle).
package ocr
import (
"fmt"
"image"
"image/color"
"image/png"
"os"
"os/exec"
"path/filepath"
"strings"
)
const TessLang = "eng+deu"
func ImageFile(path string) (string, error) {
engine := os.Getenv("OCR_ENGINE")
if engine == "paddle" {
return runPaddle(path)
}
return runTesseract(path)
}
func PDFFile(path string) (string, error) {
text, err := pdfToText(path)
if err == nil && strings.TrimSpace(text) != "" {
return strings.TrimSpace(text), nil
}
ocr, oerr := pdfPages(path)
if oerr != nil {
if err != nil {
return "", err
}
return "", oerr
}
if strings.TrimSpace(ocr) != "" {
return strings.TrimSpace(ocr), nil
}
if text != "" {
return strings.TrimSpace(text), nil
}
return "", fmt.Errorf("pdf has no text layer (ocr unavailable)")
}
func pdfToText(path string) (string, error) {
cmd := exec.Command("pdftotext", "-layout", path, "-")
out, err := cmd.Output()
if err != nil {
return "", err
}
return string(out), nil
}
func pdfPages(path string) (string, error) {
dir, err := os.MkdirTemp("", "2dph-ocr-")
if err != nil {
return "", err
}
defer os.RemoveAll(dir)
prefix := filepath.Join(dir, "page")
cmd := exec.Command("pdftoppm", "-png", "-r", "200", path, prefix)
if err := cmd.Run(); err != nil {
return "", err
}
matches, err := filepath.Glob(prefix + "*.png")
if err != nil {
return "", err
}
var parts []string
for _, img := range matches {
t, err := ImageFile(img)
if err != nil {
continue
}
if s := strings.TrimSpace(t); s != "" {
parts = append(parts, s)
}
}
return strings.Join(parts, "\n\n"), nil
}
func runTesseract(path string) (string, error) {
pre, err := preprocessFile(path)
if err != nil {
pre = path
} else {
defer os.Remove(pre)
}
cmd := exec.Command("tesseract", pre, "stdout", "-l", TessLang, "--psm", "6")
out, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(out)), nil
}
func runPaddle(path string) (string, error) {
cmd := exec.Command("paddleocr", "ocr", "-i", path)
out, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(out)), nil
}
func preprocessFile(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
img, err := png.Decode(f)
if err != nil {
return "", err
}
out := filepath.Join(os.TempDir(), filepath.Base(path)+".gray.png")
w, err := os.Create(out)
if err != nil {
return "", err
}
defer w.Close()
if err := png.Encode(w, GrayContrast(img)); err != nil {
os.Remove(out)
return "", err
}
return out, nil
}
// GrayContrast is a stdlib preprocess (no gocv): grayscale + stretch.
func GrayContrast(src image.Image) image.Image {
b := src.Bounds()
dst := image.NewGray(b)
var minL, maxL uint8 = 255, 0
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
g := color.GrayModel.Convert(src.At(x, y)).(color.Gray)
if g.Y < minL {
minL = g.Y
}
if g.Y > maxL {
maxL = g.Y
}
}
}
span := int(maxL) - int(minL)
if span < 1 {
span = 1
}
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
g := color.GrayModel.Convert(src.At(x, y)).(color.Gray)
v := uint8((int(g.Y) - int(minL)) * 255 / span)
dst.SetGray(x, y, color.Gray{Y: v})
}
}
return dst
}
+54
View File
@@ -0,0 +1,54 @@
package ocr
import (
"image"
"image/color"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestGrayContrastStretches(t *testing.T) {
img := image.NewGray(image.Rect(0, 0, 2, 2))
img.SetGray(0, 0, color.Gray{Y: 64})
img.SetGray(0, 1, color.Gray{Y: 64})
img.SetGray(1, 0, color.Gray{Y: 64})
img.SetGray(1, 1, color.Gray{Y: 192})
out := GrayContrast(img).(*image.Gray)
if out.GrayAt(0, 0).Y != 0 {
t.Fatalf("min should map to 0, got %d", out.GrayAt(0, 0).Y)
}
if out.GrayAt(1, 1).Y != 255 {
t.Fatalf("max should map to 255, got %d", out.GrayAt(1, 1).Y)
}
}
func TestHelloPNGFixtureOCR(t *testing.T) {
if _, err := exec.LookPath("tesseract"); err != nil {
t.Skip("tesseract not installed")
}
path := filepath.Join("testdata", "hello.png")
got, err := ImageFile(path)
if err != nil {
t.Fatal(err)
}
up := strings.ToUpper(got)
if !strings.Contains(up, "HELLO") {
t.Fatalf("ocr %q missing HELLO", got)
}
}
func TestPaddleEngineUsesPaddleocrBinary(t *testing.T) {
t.Setenv("OCR_ENGINE", "paddle")
_, err := ImageFile(filepath.Join("testdata", "hello.png"))
if _, look := exec.LookPath("paddleocr"); look != nil {
if err == nil {
t.Fatal("expected error when paddleocr is missing")
}
return
}
if err != nil {
t.Fatal(err)
}
}

Some files were not shown because too many files have changed in this diff Show More