Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  eSliderandGitHub
|
cec0161ff6 | ||
|
eSliderandGitHub
|
46310f8773 | ||
|
eSliderandGitHub
|
7e0f3c9e06 | ||
|
eSliderandGitHub
|
f14025304e | ||
|
eSliderandGitHub
|
dd6d7e9395 | ||
|
eSliderandGitHub
|
68d478224f | ||
|
eSliderandGitHub
|
117f3c2cfd | ||
|
eSliderandGitHub
|
140d86a4b9 | ||
|
eSlider
|
c96c393a4a | ||
|
eSlider
|
3d0d95cf00 |
@@ -19,6 +19,10 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
@@ -32,16 +36,20 @@ jobs:
|
||||
bash -n bin/db/psql-yq
|
||||
bash -n bin/db/ssh-tunnel
|
||||
bash -n bin/docker-entrypoint
|
||||
bash -n bin/kb/search
|
||||
|
||||
- name: Python unit tests (offline, vendored tools)
|
||||
run: |
|
||||
uv run python -m unittest discover -s bin/tools -t .
|
||||
|
||||
- name: Go tests (server + watch packages)
|
||||
- name: Go tests (root module, no ladybug cgo)
|
||||
run: |
|
||||
go vet ./...
|
||||
go test ./... -count=1
|
||||
|
||||
- name: brain ranking tests (no cgo / no ladybug)
|
||||
run: go test ./internal/brain/rank -count=1
|
||||
|
||||
- name: facts/audit self (lexicon consistency, no network)
|
||||
run: |
|
||||
./bin/facts/audit self 2>/dev/null || echo "audit: not yet implemented; gate skipped"
|
||||
|
||||
@@ -10,3 +10,4 @@ __pycache__/
|
||||
.env
|
||||
.secrets/
|
||||
lib-ladybug/
|
||||
go.work.local
|
||||
|
||||
@@ -24,7 +24,7 @@ Read first: [PLAN](PLAN.md) → [docs](docs/).
|
||||
2. **Read-only data sources.** Ladybug `var/kb.lbug` and Postgres are opened
|
||||
read-only for queries. Index rebuilds write to `var/` (gitignored).
|
||||
3. **PII.** `brain-test`, `cs_brain` client data is never read or quoted.
|
||||
4. **No main pushes.** Feature branches + PR via `gh`; CI must be green.
|
||||
4. **No main pushes.** Feature branches + GitHub PR (`gh`); CI (Actions) must be green. Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues).
|
||||
5. **TDD.** Failing test before tool code. Unit tests run offline against
|
||||
fixtures; network/db calls are wrapped.
|
||||
6. **docs reflect behaviour.** Any change updates `docs/` + `PLAN.md` status.
|
||||
@@ -35,10 +35,11 @@ Read first: [PLAN](PLAN.md) → [docs](docs/).
|
||||
PLAN.md decisions + execution + open questions
|
||||
docs/ published docs
|
||||
skills/ in-project agent skills (vendored, no external links)
|
||||
bin/ self-describing tools bin/{subject}/{method} (shebang)
|
||||
bin/serve.go async Go HTTP server entry (self-executing go run shebang)
|
||||
bin/watch/ corpus watcher Go package (mtimes, no inotify deps)
|
||||
bin/server/ async Go HTTP server (goroutines, bounded worker pool)
|
||||
bin/ self-describing tools bin/{subject}/{method}.go (shebang)
|
||||
bin/brain/ search.go, serve.go; libs in internal/brain and internal/httpapi
|
||||
bin/chats/ sync.go import.go facts.go apply.go; libs in internal/chats
|
||||
internal/ shared Go (brain/rank is cgo-free; chats parsers too)
|
||||
bin/watch/ corpus watcher (internal via bin/brain/watch later)
|
||||
bin/mail/ mail pipeline: sync (Go), import (md), index_mail (rebuild)
|
||||
bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch)
|
||||
bin/docker-entrypoint container entrypoint (brain index|search|serve|watch)
|
||||
@@ -52,6 +53,7 @@ var/ kb.lbug, var/mail/*, caches (gitignored)
|
||||
|
||||
```bash
|
||||
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments
|
||||
bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox)
|
||||
bin/mail/import --from-raw var/mail # message.json → message.md (convert only)
|
||||
bin/mail/index_mail # rebuild brain incl. all mail (fresh DB)
|
||||
```
|
||||
@@ -73,7 +75,8 @@ bin/mail/index_mail # rebuil
|
||||
```bash
|
||||
bin/facts/audit ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate
|
||||
bin/facts/crm [--dry-run] # proof person↔company/company↔project (ooCRM × corpus SoT)
|
||||
bin/kb/search "query" [--hop N] [--repo X] # deduction search → YAML
|
||||
bin/kb/search "query" [--repo X] # deprecated wrapper → bin/brain/search.go
|
||||
bin/brain/search.go "query" [--root facts|info] # deduction search → YAML
|
||||
bin/md/tables # what the graph holds → YAML
|
||||
bin/brain/deduce "question" # thinking wrapper
|
||||
```
|
||||
@@ -89,7 +92,7 @@ parameter. Search before reading whole files.
|
||||
Test data must be synthetic (Alice, Bob, Charlie, Diana, example.com).
|
||||
3. **No credentials/secrets in commits.** API keys, tokens, passwords, session
|
||||
strings, phone numbers only in gitignored `.env` files, referenced by path.
|
||||
4. **Curasoft — no files, no mentions.** Remove all traces if found.
|
||||
4. **Curasoft, edelweiss — no files, no mentions.** Remove all traces if found.
|
||||
5. **Check git history before push.** If any commit contains leaks, rewrite
|
||||
history (rebase + force push) AND delete affected GitHub releases/tags.
|
||||
6. **`docs/chat-import-plan.md`** — reference Gitea issue, never embed secrets.
|
||||
|
||||
@@ -26,10 +26,10 @@ detective method: **a fact needs ≥2 independent sources or it is
|
||||
|---|----------|--------|
|
||||
| D1 | RAG corpus | ops stack (chat, onlyoffice, gitea/NPM, searchxng, observability, ai-bot, mcp-servers, `~/.ssh/config`) + portfolio. Exclude `office.dev` + jobs/applications. |
|
||||
| D2 | skill merging | integrate skills **in this project** `skills/`; skip gitea / brain-dependent skills. |
|
||||
| D3 | web search | import `web-search`, retire local `searxng-ops`. Vendored here, no remote link. |
|
||||
| D3 | web search | Vendored client; SearXNG URL is config. Optional Compose instance (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. |
|
||||
| D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. |
|
||||
| D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). |
|
||||
| D6 | graph engine | **LadybugDB** (Kuzu successor, MIT, embedded, native FTS+vector+Cypher). Python binding for `bin/*`; Go shebang for golang tools. |
|
||||
| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `internal/brain`); Python remains for index/write until the Go write path is safe. |
|
||||
| D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). |
|
||||
| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. |
|
||||
| D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. |
|
||||
@@ -37,9 +37,11 @@ detective method: **a fact needs ≥2 independent sources or it is
|
||||
| D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. |
|
||||
| D12 | transactional | facts and info split by root but **written in the same Ladybug transaction (ACID)** on every write. |
|
||||
| D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. |
|
||||
| D14 | tooling style | `bin/{subject}/{method}` self-describing: shebang line 1, usage comment from line 2. Go shebang: `///usr/bin/env go run "$0" "$@"; exit`. |
|
||||
| D15 | repo | GitHub `eSlider/2dph`, public (like sibling repos), push/commit via `gh`, TDD + commit every change, CI/CD. |
|
||||
| D14 | tooling style | `bin/{subject}/{method}.go` shebang (e.g. `bin/brain/search.go`). Shared code in `internal/`. One root `go.mod` + `go.work`. No `bin/*/main.go`, no nested modules. |
|
||||
| D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. |
|
||||
| D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. |
|
||||
| D17 | assertion gate | Fact-check every *claim* (facts → info → live sources → web), not every edit. Missing graph ≠ “does not exist”. |
|
||||
| D18 | reasoner | Pluggable OpenAI-compatible URL. RAM: Qwen3.5-9B. Quality: Bonsai-27B or Qwen3.6-27B. No official Qwen3.6-9B. |
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -51,9 +53,12 @@ detective method: **a fact needs ≥2 independent sources or it is
|
||||
bin/
|
||||
facts/extract auto-pair 2 sources → lexicon yaml + graph
|
||||
facts/audit ["self"|"facts"|"info"|"stale"] 2-source + staleness gate
|
||||
kb/index build FTS + HNSW from corpus
|
||||
kb/search deduction: facts → info → web-search; --hop N
|
||||
kb/index build FTS + HNSW from corpus (Python, for now)
|
||||
brain/search.go deduction: facts → info → web-search
|
||||
kb/get kb/stats kb/eval
|
||||
brain/serve.go HTTP API (internal/httpapi)
|
||||
chats/sync.go import.go facts.go apply.go
|
||||
(libs in internal/chats; no chats index)
|
||||
md/import md/select md/tables md/gaps (mistune)
|
||||
brain/extract brain/audit brain/deduce (thinking wrapper)
|
||||
web/search (vendored)
|
||||
@@ -110,17 +115,18 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
|
||||
corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and
|
||||
indexing stay separate for crash safety.
|
||||
4. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable
|
||||
via `bin/kb/search`.
|
||||
via `bin/brain/search.go`.
|
||||
|
||||
## CI/CD pipeline (D15)
|
||||
|
||||
`.github/workflows/ci.yml`:
|
||||
|
||||
1. go vet + go test ./... (Go tools)
|
||||
2. python -m unittest discover + pytest (Py tools)
|
||||
3. bin/facts/audit self (lexicon internal consistency)
|
||||
4. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions)
|
||||
5. md-docs build/lint if docs tooling arrives.
|
||||
1. go vet + go test ./... (root module; packages without ladybug cgo)
|
||||
2. `go test ./internal/brain/rank` (cgo-free ranking + flag parser)
|
||||
3. python -m unittest discover -s bin/tools (includes published-docs SoT)
|
||||
4. bin/facts/audit self (lexicon internal consistency)
|
||||
5. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions)
|
||||
6. md-docs build/lint if docs tooling arrives.
|
||||
|
||||
Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as
|
||||
`db/tech-poc`: contract first where there is an OpenAPI/message shape.
|
||||
|
||||
@@ -32,7 +32,7 @@ graph TB
|
||||
AU["bin/facts/audit<br/>confidence + staleness"]
|
||||
IDX["bin/kb/index<br/>chunk + embed"]
|
||||
MD["bin/md/import<br/>mistune leaves"]
|
||||
SR["bin/kb/search<br/>deduction + --hop"]
|
||||
SR["bin/brain/search.go<br/>deduction"]
|
||||
end
|
||||
|
||||
subgraph store["Ladybug var/kb.lbug"]
|
||||
@@ -85,21 +85,23 @@ fact; conflicting sources or a single source → `hypothesis` → `(not confirme
|
||||
## Deduction search
|
||||
|
||||
```bash
|
||||
bin/kb/search "Matrix federation over HTTPS" # facts → info → web-search
|
||||
bin/kb/search "what runs on arc-2" --hop 1 # walk graph edges
|
||||
bin/kb/search "where is cs-lexicon" --json | yq '.' # YAML by default
|
||||
bin/kb/get <id> --body # full chunk on demand
|
||||
bin/kb/stats # index health
|
||||
bin/kb/eval # recall@5 gate
|
||||
bin/brain/search.go "Matrix federation over HTTPS" # facts → info → web-search
|
||||
bin/brain/search.go "onlyoffice postgres" --root facts
|
||||
bin/brain/search.go "where is cs-lexicon" --json | yq '.'
|
||||
bin/kb/get <id> --body # full chunk on demand
|
||||
bin/kb/stats # index health
|
||||
bin/kb/eval # recall@5 gate
|
||||
```
|
||||
|
||||
`--hop` is not implemented (needs File/FROM_FILE edges); the flag errors instead of walking. `bin/kb/search` is a deprecated wrapper around `bin/brain/search.go`.
|
||||
|
||||
Mail is a first-class corpus (retrievable through the same search):
|
||||
|
||||
```bash
|
||||
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go)
|
||||
bin/mail/import --from-raw var/mail # JSON → markdown
|
||||
bin/mail/index_mail # rebuild brain incl. mail
|
||||
bin/kb/search "Mietwagen Nürnberg invoice" # now answers from mail
|
||||
bin/brain/search.go "invoice from last week" # same search over mail leafs
|
||||
```
|
||||
|
||||
## Storage
|
||||
@@ -117,10 +119,10 @@ bin/kb/search "Mietwagen Nürnberg invoice" # now a
|
||||
|
||||
## Tooling conventions
|
||||
|
||||
`bin/{subject}/{method}` — self-describing: shebang on line 1, usage comment
|
||||
from line 2. bash + python primary; golang via the Go shebang when a compiled
|
||||
helper is right. YAML default output, `--json` for machines. Everything that
|
||||
touches network/db is read-only, throttled, cached. Tests gate every commit.
|
||||
`bin/{subject}/{method}.go` — self-describing: shebang on line 1, usage comment
|
||||
from line 2. Shared code in `internal/`. YAML default output, `--json` for
|
||||
machines. Tests gate every commit. HTTP: `bin/brain/serve.go` (default search
|
||||
binary `var/bin/brain-search`, not Python).
|
||||
|
||||
## Development
|
||||
|
||||
@@ -136,7 +138,7 @@ Docker (optional, cached model + var volumes):
|
||||
```bash
|
||||
docker compose run --rm brain index # (re)index corpus
|
||||
docker compose run --rm brain search "query" # one-shot query
|
||||
docker compose run --rm brain serve # async Go HTTP server
|
||||
docker compose run --rm brain serve # bin/brain/serve.go
|
||||
docker compose up brain-watch # auto re-index on change
|
||||
```
|
||||
|
||||
@@ -148,4 +150,7 @@ docker compose up brain-watch # auto re-index on change
|
||||
skills (`web-search`, `db-yaml`, …) that 2dph integrates
|
||||
- detective method — the two-source method
|
||||
|
||||
See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions.
|
||||
Work board (issues): [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
|
||||
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).
|
||||
|
||||
See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
// Commands in this directory are shebang mains (search.go).
|
||||
// search.go is behind the system_ladybug build tag (cgo).
|
||||
package main
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
//usr/bin/env go run -tags=system_ladybug "$0" "$@"; exit
|
||||
//go:build cgo && system_ladybug
|
||||
//
|
||||
// bin/brain/search.go - deduction search over the 2dph brain.
|
||||
//
|
||||
// ./bin/brain/search.go "query" [--root facts|info] [--repo P] [-n N] [--json]
|
||||
// ./bin/brain/search.go serve [port]
|
||||
// ./bin/brain/search.go --list-model
|
||||
//
|
||||
// Needs CGO + libladybug (CGO_CFLAGS/CGO_LDFLAGS). Prefer the wrapper
|
||||
// bin/kb/search which sets those and builds a binary for the embed daemon.
|
||||
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/brain"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(brain.Main(os.Args[1:]))
|
||||
}
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
//usr/bin/env go run -tags=brain_serve "$0" "$@"; exit
|
||||
//go:build brain_serve
|
||||
//
|
||||
// bin/brain/serve.go - HTTP API for the 2dph brain.
|
||||
//
|
||||
// KB_ROOT=/path/to/2dph ./bin/brain/serve.go
|
||||
// KB_SEARCH_CMD=... KB_WORKERS=4 KB_PORT=8630 ./bin/brain/serve.go
|
||||
//
|
||||
// Default search backend is var/bin/brain-search (Go), not Python.
|
||||
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/httpapi"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if os.Getenv("KB_ROOT") == "" {
|
||||
if wd, err := os.Getwd(); err == nil {
|
||||
os.Setenv("KB_ROOT", wd)
|
||||
}
|
||||
}
|
||||
httpapi.Run()
|
||||
}
|
||||
Executable
+19
@@ -0,0 +1,19 @@
|
||||
//usr/bin/env go run -tags=chats_apply "$0" "$@"; exit
|
||||
//go:build chats_apply
|
||||
//
|
||||
// bin/chats/apply.go - push extracted chat facts to OnlyOffice CRM.
|
||||
//
|
||||
// ./bin/chats/apply.go [--dry-run]
|
||||
//
|
||||
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/chats"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(chats.RunApply(os.Args[1:]))
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
// Commands in this directory are shebang mains (sync.go, import.go, facts.go,
|
||||
// apply.go), each behind an exclusive build tag so `go build ./bin/chats`
|
||||
// does not see two mains. Shared code lives in internal/chats.
|
||||
package main
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
//usr/bin/env go run -tags=chats_facts "$0" "$@"; exit
|
||||
//go:build chats_facts
|
||||
//
|
||||
// bin/chats/facts.go - extract phone/email/linkedin facts from JSONL.
|
||||
//
|
||||
// ./bin/chats/facts.go
|
||||
//
|
||||
// Writes var/chats/facts/. Does not index the brain.
|
||||
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/chats"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(chats.RunFacts(os.Args[1:]))
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
module github.com/eSlider/2dph/bin/chats
|
||||
|
||||
go 1.25.0
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
//usr/bin/env go run -tags=chats_import "$0" "$@"; exit
|
||||
//go:build chats_import
|
||||
//
|
||||
// bin/chats/import.go - JSONL → markdown under var/chats/md/.
|
||||
//
|
||||
// ./bin/chats/import.go
|
||||
//
|
||||
// Conversion only. Brain ingest is bin/brain/index.go, not this command.
|
||||
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/chats"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(chats.RunImport(os.Args[1:]))
|
||||
}
|
||||
@@ -1,56 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func runIndex(args []string) int {
|
||||
fs := flag.NewFlagSet("chats index", flag.ContinueOnError)
|
||||
help := fs.Bool("help", false, "")
|
||||
fs.SetOutput(os.Stderr)
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return 2
|
||||
}
|
||||
if *help {
|
||||
fmt.Fprintln(os.Stderr, "usage: chats index")
|
||||
return 0
|
||||
}
|
||||
|
||||
root := repoRoot()
|
||||
mdDir := filepath.Join(chatsDir(), "md")
|
||||
|
||||
_, err := os.Stat(mdDir)
|
||||
if os.IsNotExist(err) {
|
||||
fmt.Fprintf(os.Stderr, "chats index: no chat markdown at %s; run 'chats import' first\n", mdDir)
|
||||
return 1
|
||||
}
|
||||
|
||||
indexScript := filepath.Join(root, "bin", "kb", "index")
|
||||
if _, err := os.Stat(indexScript); os.IsNotExist(err) {
|
||||
fmt.Fprintf(os.Stderr, "chats index: %s not found\n", indexScript)
|
||||
return 1
|
||||
}
|
||||
|
||||
cmd := exec.Command(indexScript, "--corpus", mdDir)
|
||||
var outBuf, errBuf bytes.Buffer
|
||||
cmd.Stdout = &outBuf
|
||||
cmd.Stderr = &errBuf
|
||||
cmd.Dir = root
|
||||
|
||||
if err := cmd.Run(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats index: %v\n%s", err, errBuf.String())
|
||||
return 1
|
||||
}
|
||||
result := strings.TrimSpace(outBuf.String())
|
||||
if result == "" {
|
||||
result = strings.TrimSpace(errBuf.String())
|
||||
}
|
||||
fmt.Printf("chats index: %s\n", result)
|
||||
return 0
|
||||
}
|
||||
@@ -1,116 +0,0 @@
|
||||
// bin/chats - sync, import, index, extract facts, and apply chat data
|
||||
// from Telegram, WhatsApp, LinkedIn into the brain and OnlyOffice CRM.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// chats sync telegram [--limit N] [--since DATE] [--phone PHONE]
|
||||
// chats sync whatsapp [--qr] [--limit N]
|
||||
// chats sync linkedin [--limit N]
|
||||
// chats import # JSONL → MD (all sources)
|
||||
// chats index # rebuild var/kb.lbug with chats
|
||||
// chats facts # extract + cross-check
|
||||
// chats apply [--dry-run] # push to OnlyOffice CRM
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 {
|
||||
usage()
|
||||
os.Exit(2)
|
||||
}
|
||||
cmd := os.Args[1]
|
||||
args := os.Args[2:]
|
||||
switch cmd {
|
||||
case "sync":
|
||||
if len(args) < 1 {
|
||||
usage()
|
||||
os.Exit(2)
|
||||
}
|
||||
platform := args[0]
|
||||
platformArgs := args[1:]
|
||||
switch platform {
|
||||
case "telegram":
|
||||
os.Exit(runSyncTelegram(platformArgs))
|
||||
case "whatsapp":
|
||||
fmt.Fprintf(os.Stderr, "chats: WhatsApp not implemented yet\n")
|
||||
os.Exit(1)
|
||||
case "linkedin":
|
||||
fmt.Fprintf(os.Stderr, "chats: LinkedIn not implemented yet\n")
|
||||
os.Exit(1)
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform)
|
||||
os.Exit(2)
|
||||
}
|
||||
case "import":
|
||||
os.Exit(runImport(args))
|
||||
case "index":
|
||||
os.Exit(runIndex(args))
|
||||
case "facts":
|
||||
os.Exit(runFacts(args))
|
||||
case "apply":
|
||||
os.Exit(runApply(args))
|
||||
case "help", "-h", "--help":
|
||||
usage()
|
||||
return
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "chats: unknown command %q\n", cmd)
|
||||
usage()
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func usage() {
|
||||
w := os.Stderr
|
||||
fmt.Fprintln(w, `Usage: chats <command> [args]
|
||||
|
||||
Commands:
|
||||
sync telegram [--limit N] [--since DATE] [--phone PHONE]
|
||||
sync whatsapp [--qr] [--limit N]
|
||||
sync linkedin [--limit N]
|
||||
import JSONL → MD (all sources)
|
||||
index rebuild var/kb.lbug with chats
|
||||
facts extract + cross-check facts
|
||||
apply [--dry-run] push to OnlyOffice CRM
|
||||
|
||||
Output layout:
|
||||
var/chats/<platform>/<chat_id>/messages.jsonl
|
||||
var/chats/md/<platform>/<chat_name>/messages.md`)
|
||||
}
|
||||
|
||||
// repoRoot locates the 2dph project root by walking up from the binary.
|
||||
func repoRoot() string {
|
||||
if v := os.Getenv("KB_ROOT"); v != "" {
|
||||
return v
|
||||
}
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
return "."
|
||||
}
|
||||
for i := 0; i < 10; i++ {
|
||||
if _, err := os.Stat(wd + "/var"); err == nil {
|
||||
return wd
|
||||
}
|
||||
if _, err := os.Stat(wd + "/.git"); err == nil {
|
||||
return wd
|
||||
}
|
||||
parent := wd
|
||||
if idx := strings.LastIndex(wd, "/"); idx >= 0 {
|
||||
parent = wd[:idx]
|
||||
}
|
||||
if parent == wd {
|
||||
break
|
||||
}
|
||||
wd = parent
|
||||
}
|
||||
return "."
|
||||
}
|
||||
|
||||
// chatsDir returns var/chats under the repo root.
|
||||
func chatsDir() string {
|
||||
return repoRoot() + "/var/chats"
|
||||
}
|
||||
Executable
+155
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chats/refresh-linkedin-session - refresh LinkedIn MCP session from webtop CDP.
|
||||
|
||||
bin/chats/refresh-linkedin-session [--cdp URL] [--root DIR]
|
||||
"""
|
||||
|
||||
Reads the current LinkedIn cookies out of the running Thorium browser in the
|
||||
work-webtop container via CDP (Network.getAllCookies), copies the live browser
|
||||
profile onto the source profile directory, and rewrites the portable
|
||||
cookies.json + source-state.json that mcp-server-linkedin requires.
|
||||
|
||||
Usage:
|
||||
refresh-linkedin-session [--cdp http://127.0.0.1:9222] [--root /var/tmp/liprofile]
|
||||
[--container work-webtop] [--profile thorium-profile]
|
||||
|
||||
After the headless driver uses a copied profile, LinkedIn rotates the session
|
||||
in that copy, so this must run before every sync.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.request
|
||||
|
||||
import websockets
|
||||
|
||||
|
||||
def cdp_tab(ws_json):
|
||||
for t in ws_json:
|
||||
if t.get("webSocketDebuggerUrl"):
|
||||
return t["webSocketDebuggerUrl"]
|
||||
return None
|
||||
|
||||
|
||||
async def get_cookies(ws_url):
|
||||
async with websockets.connect(ws_url, max_size=50_000_000) as ws:
|
||||
await ws.send(json.dumps({"id": 1, "method": "Network.getAllCookies", "params": {}}))
|
||||
resp = await ws.recv()
|
||||
return json.loads(resp).get("result", {}).get("cookies", [])
|
||||
|
||||
|
||||
def write_source_state(root, profile_dir):
|
||||
# Reuse the linkedin-mcp-server session_state module to write a valid
|
||||
# source-state.json (same schema the daemon reads).
|
||||
try:
|
||||
from linkedin_mcp_server.session_state import canonical, write_source_state
|
||||
|
||||
write_source_state(canonical(__import__("pathlib").Path(profile_dir)))
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback: minimal schema-compatible state.
|
||||
import uuid
|
||||
|
||||
state = {
|
||||
"version": 1,
|
||||
"source_runtime_id": "linux-amd64-host",
|
||||
"login_generation": str(uuid.uuid4()),
|
||||
"created_at": None,
|
||||
"profile_path": profile_dir,
|
||||
"cookies_path": os.path.join(root, "cookies.json"),
|
||||
}
|
||||
from datetime import datetime, timezone
|
||||
|
||||
state["created_at"] = datetime.now(timezone.utc).isoformat()
|
||||
with open(os.path.join(root, "source-state.json"), "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
|
||||
|
||||
def main():
|
||||
args = sys.argv[1:]
|
||||
cdp = "http://127.0.0.1:9222"
|
||||
root = "/var/tmp/liprofile"
|
||||
container = "work-webtop"
|
||||
cprofile = "thorium-profile"
|
||||
for i in range(0, len(args), 2):
|
||||
k = args[i]
|
||||
v = args[i + 1] if i + 1 < len(args) else ""
|
||||
if k == "--cdp":
|
||||
cdp = v
|
||||
elif k == "--root":
|
||||
root = v
|
||||
elif k == "--container":
|
||||
container = v
|
||||
elif k == "--profile":
|
||||
cprofile = v
|
||||
|
||||
profile_dir = os.path.join(root, "profile")
|
||||
os.makedirs(profile_dir, exist_ok=True)
|
||||
|
||||
# 1. Clear stale daemon/browser locks so the server can claim the profile.
|
||||
for lock in ("profile-claim.lock", "profile.lock", "daemon.lock", "lease.lock"):
|
||||
p = os.path.join(root, lock)
|
||||
if os.path.exists(p):
|
||||
os.remove(p)
|
||||
for name in os.listdir(profile_dir):
|
||||
if name.startswith("Singleton"):
|
||||
os.remove(os.path.join(profile_dir, name))
|
||||
for name in os.listdir(root):
|
||||
if name.startswith("invalid-state-"):
|
||||
shutil.rmtree(os.path.join(root, name), ignore_errors=True)
|
||||
|
||||
# 1. Copy the live browser profile (cookies DB + Local State) so the
|
||||
# session the driver launches carries the current login.
|
||||
subprocess.run(
|
||||
["docker", "cp", f"{container}:/config/{cprofile}/Default", os.path.join(profile_dir, "Default")],
|
||||
check=True, capture_output=True,
|
||||
)
|
||||
subprocess.run(
|
||||
["docker", "cp", f"{container}:/config/{cprofile}/Local State", os.path.join(profile_dir, "Local State")],
|
||||
check=True, capture_output=True,
|
||||
)
|
||||
for lock in ("SingletonLock", "SingletonCookie", "SingletonSocket"):
|
||||
p = os.path.join(profile_dir, lock)
|
||||
if os.path.exists(p):
|
||||
os.remove(p)
|
||||
|
||||
# 2. Pull the live cookies out of the running browser.
|
||||
with urllib.request.urlopen(f"{cdp}/json", timeout=5) as r:
|
||||
tabs = json.loads(r.read())
|
||||
ws_url = cdp_tab(tabs)
|
||||
if not ws_url:
|
||||
sys.stderr.write("refresh-linkedin-session: no CDP tab\n")
|
||||
sys.exit(1)
|
||||
cookies = asyncio.run(get_cookies(ws_url))
|
||||
|
||||
li = [c for c in cookies if "linkedin" in c.get("domain", "")]
|
||||
out = []
|
||||
for c in li:
|
||||
domain = c.get("domain", "")
|
||||
if domain in (".www.linkedin.com", "www.linkedin.com"):
|
||||
domain = ".linkedin.com"
|
||||
out.append({
|
||||
"name": c["name"],
|
||||
"value": c["value"].strip('"'),
|
||||
"domain": domain,
|
||||
"path": c.get("path", "/"),
|
||||
"expires": c.get("expires", -1),
|
||||
"httpOnly": c.get("httpOnly", False),
|
||||
"secure": c.get("secure", False),
|
||||
"sameSite": c.get("sameSite", "None"),
|
||||
})
|
||||
with open(os.path.join(root, "cookies.json"), "w") as f:
|
||||
json.dump(out, f, indent=2)
|
||||
|
||||
write_source_state(root, profile_dir)
|
||||
sys.stderr.write(f"refresh-linkedin-session: {len(out)} cookies, profile refreshed\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
//usr/bin/env go run -tags=chats_sync "$0" "$@"; exit
|
||||
//go:build chats_sync
|
||||
//
|
||||
// bin/chats/sync.go - download chat messages to var/chats/<platform>/.
|
||||
//
|
||||
// ./bin/chats/sync.go telegram [--limit N] [--phone PHONE]
|
||||
// ./bin/chats/sync.go linkedin [--limit N] [--refresh]
|
||||
//
|
||||
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/chats"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 {
|
||||
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`)
|
||||
os.Exit(2)
|
||||
}
|
||||
platform := os.Args[1]
|
||||
args := os.Args[2:]
|
||||
switch platform {
|
||||
case "telegram":
|
||||
os.Exit(chats.RunSyncTelegram(args))
|
||||
case "linkedin":
|
||||
os.Exit(chats.RunSyncLinkedIn(args))
|
||||
case "whatsapp":
|
||||
fmt.Fprintln(os.Stderr, "chats: WhatsApp not implemented yet")
|
||||
os.Exit(1)
|
||||
case "help", "-h", "--help":
|
||||
fmt.Fprintln(os.Stderr, `usage: bin/chats/sync.go telegram|linkedin [flags]`)
|
||||
return
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "chats: unknown platform %q\n", platform)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
// Deprecated shebang mains at bin root (serve.go is tagged brain_serve).
|
||||
package main
|
||||
+24
-21
@@ -1,34 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# bin/kb/search - Go deduction search over the brain (model served by daemon).
|
||||
# Builds the kbsearch binary on first run / when source changes, then execs it.
|
||||
# bin/kb/search — deprecated wrapper. Use bin/brain/search.go.
|
||||
# Sets CGO for ladybug, builds a binary (embed daemon needs a real executable),
|
||||
# then execs it. Prints one deprecation line.
|
||||
set -euo pipefail
|
||||
|
||||
KB="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
BIN="$KB/var/bin/kbsearch"
|
||||
SRC="$KB/bin/kbsearch"
|
||||
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
BIN="$ROOT/var/bin/brain-search"
|
||||
SRC="$ROOT/internal/brain"
|
||||
CMD="$ROOT/bin/brain"
|
||||
|
||||
mkdir -p "$KB/var/bin"
|
||||
mkdir -p "$ROOT/var/bin"
|
||||
|
||||
# Rebuild if binary missing or any .go source newer
|
||||
need_build=0
|
||||
if [ ! -x "$BIN" ]; then
|
||||
need_build=1
|
||||
need_build=1
|
||||
else
|
||||
# Check if any .go in kbsearch is newer than binary
|
||||
while IFS= read -r -d '' f; do
|
||||
if [ "$f" -nt "$BIN" ]; then
|
||||
need_build=1
|
||||
break
|
||||
fi
|
||||
done < <(find "$SRC" -name '*.go' -print0 2>/dev/null)
|
||||
while IFS= read -r -d '' f; do
|
||||
if [ "$f" -nt "$BIN" ]; then
|
||||
need_build=1
|
||||
break
|
||||
fi
|
||||
done < <(find "$SRC" "$CMD" -name '*.go' -print0 2>/dev/null)
|
||||
fi
|
||||
|
||||
if [ "$need_build" -eq 1 ]; then
|
||||
echo "Building kbsearch..." >&2
|
||||
(cd "$SRC" && \
|
||||
CGO_CFLAGS="-I$KB/lib-ladybug" \
|
||||
CGO_LDFLAGS="-L$KB/lib-ladybug -Wl,-rpath,$KB/lib-ladybug" \
|
||||
go build -tags system_ladybug -o "$BIN" .) || exit 1
|
||||
echo "Building brain/search..." >&2
|
||||
(
|
||||
cd "$ROOT" &&
|
||||
CGO_CFLAGS="-I$ROOT/lib-ladybug" \
|
||||
CGO_LDFLAGS="-L$ROOT/lib-ladybug -Wl,-rpath,$ROOT/lib-ladybug" \
|
||||
go build -tags system_ladybug -o "$BIN" ./bin/brain
|
||||
) || exit 1
|
||||
fi
|
||||
|
||||
exec "$BIN" "$@"
|
||||
echo "bin/kb/search is deprecated; use bin/brain/search.go" >&2
|
||||
exec "$BIN" "$@"
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
module github.com/eSlider/2dph/bin/kbsearch
|
||||
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
github.com/LadybugDB/go-ladybug v0.17.0
|
||||
github.com/chewxy/math32 v1.11.2
|
||||
github.com/daulet/tokenizers v1.27.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/apache/arrow-go/v18 v18.6.0 // indirect
|
||||
github.com/goccy/go-json v0.10.6 // indirect
|
||||
github.com/google/flatbuffers v25.12.19+incompatible // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/klauspost/compress v1.18.5 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.26 // indirect
|
||||
github.com/shopspring/decimal v1.4.0 // indirect
|
||||
github.com/zeebo/xxh3 v1.1.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect
|
||||
golang.org/x/sys v0.43.0 // indirect
|
||||
)
|
||||
@@ -1,44 +0,0 @@
|
||||
github.com/LadybugDB/go-ladybug v0.17.0 h1:RXDbkBjrbRmLdEbhGl4CLOIEzSt09gbP0n9UbKDEfwI=
|
||||
github.com/LadybugDB/go-ladybug v0.17.0/go.mod h1:GeIXmE8XyF5TFS94NAuTag7vgCC+no/HTBMRA6Rd5Cs=
|
||||
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
|
||||
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/apache/arrow-go/v18 v18.6.0 h1:GX/Jyd3R7mCLiECAwY9FWbbaYblie2WXBSz4Sw8fNpM=
|
||||
github.com/apache/arrow-go/v18 v18.6.0/go.mod h1:gm3MiPpY82fLYK5VKPB3WoJbsiLVDfT7flD5/vHReKw=
|
||||
github.com/apache/thrift v0.22.0 h1:r7mTJdj51TMDe6RtcmNdQxgn9XcyfGDOzegMDRg47uc=
|
||||
github.com/apache/thrift v0.22.0/go.mod h1:1e7J/O1Ae6ZQMTYdy9xa3w9k+XHWPfRvdPyJeynQ+/g=
|
||||
github.com/chewxy/math32 v1.11.2 h1:IufN08Zwr1NKuWfY+4Tz55BcwKmyKKNdOP7KtumehnM=
|
||||
github.com/chewxy/math32 v1.11.2/go.mod h1:dOB2rcuFrCn6UHrze36WSLVPKtzPMRAQvBvUwkSsLqs=
|
||||
github.com/daulet/tokenizers v1.27.0 h1:MmFYAEDFz69s/nNQfHg59DWqHz3v94m99kEZ/JbL+s4=
|
||||
github.com/daulet/tokenizers v1.27.0/go.mod h1:YjFY1o1HGMyWkQgbXJDghhvke/yFDp2vGdIO2hYs4MQ=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
|
||||
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs=
|
||||
github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
|
||||
github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
|
||||
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
|
||||
github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
|
||||
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU=
|
||||
golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU=
|
||||
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
|
||||
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -1,44 +0,0 @@
|
||||
// bin/kbsearch - the Go implementation of bin/kb/search (nested module so the
|
||||
// root `go test ./...` and CI never compile it against native ladyships).
|
||||
//
|
||||
// Usage (built/run by ./bin/kb/search):
|
||||
//
|
||||
// kbsearch "query" [--root facts|info] [--repo P] [-n N] [--json]
|
||||
// kbsearch serve [port] start the embedding daemon
|
||||
// kbsearch --list-model print the resolved model dir
|
||||
//
|
||||
// The potion-multilingual model is loaded only in `serve`; a CLI reuses the
|
||||
// daemon over localhost HTTP (falling back to in-process embedding).
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 && os.Args[1] == "serve" {
|
||||
port := 17830
|
||||
if len(os.Args) > 2 {
|
||||
if p, err := strconv.Atoi(os.Args[2]); err == nil {
|
||||
port = p
|
||||
}
|
||||
}
|
||||
if err := serve(port); err != nil {
|
||||
log.Fatalf("kbsearch serve: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if len(os.Args) > 1 && os.Args[1] == "--list-model" {
|
||||
dir, err := modelDir()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Println(dir)
|
||||
return
|
||||
}
|
||||
os.Exit(runSearch(os.Args[1:]))
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
// Common types and helpers for kbsearch.
|
||||
package main
|
||||
|
||||
import "os"
|
||||
|
||||
func eps() string { return os.Getenv("KBTEST_EPS") }
|
||||
|
||||
// Hit is one search result, mirroring the python script's dict shape.
|
||||
type Hit struct {
|
||||
ID string `json:"id"`
|
||||
Text string `json:"text"`
|
||||
Root string `json:"root"`
|
||||
Source string `json:"-"` // for repo filtering, not in output
|
||||
Score float64 `json:"score"`
|
||||
Snippet string `json:"snippet,omitempty"`
|
||||
}
|
||||
@@ -32,6 +32,7 @@ func ParseCLI(args []string) (CLIConfig, int, error) {
|
||||
offset = fs.Int("offset", 0, "skip first N messages per source")
|
||||
force = fs.Bool("force", false, "overwrite existing message.json + attachments")
|
||||
dryRun = fs.Bool("dry-run", false, "list message counts without writing")
|
||||
query = fs.String("query", "in:inbox", "Gmail search query (gmail source only)")
|
||||
srcs = fs.String("source", "onlyoffice", "comma list: onlyoffice,gmail (default onlyoffice)")
|
||||
help = fs.Bool("help", false, "usage")
|
||||
)
|
||||
@@ -60,6 +61,7 @@ func ParseCLI(args []string) (CLIConfig, int, error) {
|
||||
Offset: *offset,
|
||||
Force: *force,
|
||||
DryRun: *dryRun,
|
||||
Query: *query,
|
||||
Policy: RetryPolicy{},
|
||||
}
|
||||
cli := CLIConfig{Sync: cfg, Env: *env, Sources: *srcs}
|
||||
@@ -95,7 +97,7 @@ func Main(args []string) int {
|
||||
return code
|
||||
}
|
||||
if cli.Help {
|
||||
fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]")
|
||||
fmt.Fprintln(os.Stderr, "usage: bin/mail/sync.go [--source onlyoffice,gmail] [--query GMAIL_Q] [--limit N] [--offset N] [--workers N] [--force] [--dry-run]")
|
||||
return 0
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
|
||||
|
||||
+17
-4
@@ -112,6 +112,7 @@ type SyncConfig struct {
|
||||
Offset int // skip first N messages per source
|
||||
Force bool // overwrite existing message.json + attachments
|
||||
DryRun bool // list without writing
|
||||
Query string // Gmail search query; default in:inbox
|
||||
Policy RetryPolicy
|
||||
}
|
||||
|
||||
@@ -136,9 +137,17 @@ type ooSource struct {
|
||||
c *OOClient
|
||||
page int
|
||||
}
|
||||
// gmailAPI is the Gmail client surface gmailSource needs. *GmailClient implements it.
|
||||
type gmailAPI interface {
|
||||
ListIDs(ctx context.Context, q string, maxIDs int, pageToken string) ([]string, string, error)
|
||||
GetMessage(ctx context.Context, id string) (*Message, error)
|
||||
DownloadAttachment(ctx context.Context, msgID, attID string) ([]byte, error)
|
||||
}
|
||||
|
||||
type gmailSource struct {
|
||||
c *GmailClient
|
||||
cur string
|
||||
c gmailAPI
|
||||
cur string
|
||||
query string
|
||||
}
|
||||
|
||||
func (s *ooSource) Folder() string { return "inbox" }
|
||||
@@ -171,7 +180,11 @@ func (s *ooSource) DownloadAttachment(ctx context.Context, msg *Message, att Att
|
||||
}
|
||||
|
||||
func (s *gmailSource) ListIDs(ctx context.Context, limit int, cursor string) ([]string, string, error) {
|
||||
ids, next, err := s.c.ListIDs(ctx, "in:inbox", limit, cursor)
|
||||
q := s.query
|
||||
if q == "" {
|
||||
q = "in:inbox"
|
||||
}
|
||||
ids, next, err := s.c.ListIDs(ctx, q, limit, cursor)
|
||||
return ids, next, err
|
||||
}
|
||||
|
||||
@@ -207,7 +220,7 @@ func Run(ctx context.Context, cfg SyncConfig) (*SyncStats, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("gmail init: %w", err)
|
||||
}
|
||||
sources = append(sources, &gmailSource{c: gm})
|
||||
sources = append(sources, &gmailSource{c: gm, query: cfg.Query})
|
||||
}
|
||||
if len(sources) == 0 {
|
||||
return nil, errors.New("sync: no source configured (need OO, Gmail, or both)")
|
||||
|
||||
@@ -221,6 +221,71 @@ func TestCollectParts(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
type fakeGmailAPI struct {
|
||||
lastQ string
|
||||
lastLimit int
|
||||
ids []string
|
||||
}
|
||||
|
||||
func (f *fakeGmailAPI) ListIDs(_ context.Context, q string, maxIDs int, _ string) ([]string, string, error) {
|
||||
f.lastQ = q
|
||||
f.lastLimit = maxIDs
|
||||
return f.ids, "", nil
|
||||
}
|
||||
func (f *fakeGmailAPI) GetMessage(context.Context, string) (*Message, error) {
|
||||
return nil, errors.New("unused")
|
||||
}
|
||||
func (f *fakeGmailAPI) DownloadAttachment(context.Context, string, string) ([]byte, error) {
|
||||
return nil, errors.New("unused")
|
||||
}
|
||||
|
||||
func TestGmailSourcePassesQueryToListIDs(t *testing.T) {
|
||||
fake := &fakeGmailAPI{ids: []string{"m1"}}
|
||||
src := &gmailSource{c: fake, query: "from:alice@example.com"}
|
||||
ids, _, err := src.ListIDs(context.Background(), 10, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fake.lastQ != "from:alice@example.com" {
|
||||
t.Fatalf("ListIDs q=%q, want from:alice@example.com", fake.lastQ)
|
||||
}
|
||||
if fake.lastLimit != 10 {
|
||||
t.Fatalf("ListIDs limit=%d, want 10", fake.lastLimit)
|
||||
}
|
||||
if len(ids) != 1 || ids[0] != "m1" {
|
||||
t.Fatalf("ids=%v", ids)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGmailSourceEmptyQueryDefaultsToInbox(t *testing.T) {
|
||||
fake := &fakeGmailAPI{}
|
||||
src := &gmailSource{c: fake, query: ""}
|
||||
if _, _, err := src.ListIDs(context.Background(), 5, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fake.lastQ != "in:inbox" {
|
||||
t.Fatalf("empty query q=%q, want in:inbox", fake.lastQ)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCLIGmailQuery(t *testing.T) {
|
||||
cli, code, err := ParseCLI([]string{
|
||||
"--source", "gmail",
|
||||
"--query", "from:letrado@example.com",
|
||||
"--out", t.TempDir(),
|
||||
"--dry-run",
|
||||
})
|
||||
if err != nil || code != 0 {
|
||||
t.Fatalf("ParseCLI: code=%d err=%v", code, err)
|
||||
}
|
||||
if cli.Sync.Query != "from:letrado@example.com" {
|
||||
t.Fatalf("query=%q", cli.Sync.Query)
|
||||
}
|
||||
if cli.Sync.Gmail == nil {
|
||||
t.Fatal("gmail source not configured")
|
||||
}
|
||||
}
|
||||
|
||||
func b64(s string) string {
|
||||
return base64.URLEncoding.EncodeToString([]byte(s))
|
||||
}
|
||||
|
||||
+8
-13
@@ -1,27 +1,22 @@
|
||||
//usr/bin/env go run "$0" "$@"; exit
|
||||
// bin/serve.go - async Go HTTP server for the 2dph brain (see bin/server).
|
||||
//usr/bin/env go run -tags=brain_serve "$0" "$@"; exit
|
||||
//go:build brain_serve
|
||||
//
|
||||
// KB_ROOT=/path/to/2dph ./bin/serve.go # serve the brain
|
||||
// KB_SEARCH_CMD=... KB_WORKERS=4 KB_PORT=8630 ./bin/serve.go
|
||||
//
|
||||
// Shebang trick: the first line is a Go `//` comment; when executed, env runs
|
||||
// `go run "$0"` so this file doubles as an executable script. The real code
|
||||
// lives in the importable package (module path, never a relative import).
|
||||
// NOTE: never run `gofmt -w` on this file - it rewrites `//usr/bin/env` to
|
||||
// `// usr/...` and breaks the shebang.
|
||||
// bin/serve.go — deprecated; use bin/brain/serve.go.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/bin/server"
|
||||
"github.com/eSlider/2dph/internal/httpapi"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if env := os.Getenv("KB_ROOT"); env == "" {
|
||||
fmt.Fprintln(os.Stderr, "bin/serve.go is deprecated; use bin/brain/serve.go")
|
||||
if os.Getenv("KB_ROOT") == "" {
|
||||
if wd, err := os.Getwd(); err == nil {
|
||||
os.Setenv("KB_ROOT", wd)
|
||||
}
|
||||
}
|
||||
server.Run()
|
||||
httpapi.Run()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
"""D14 layout: bin/{subject}/{method}.go, libs in internal/, one go.mod."""
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
class BinLayoutTest(unittest.TestCase):
|
||||
def test_brain_search_shebang_exists(self) -> None:
|
||||
p = ROOT / "bin" / "brain" / "search.go"
|
||||
self.assertTrue(p.is_file(), "missing bin/brain/search.go")
|
||||
first = p.read_text().splitlines()[0]
|
||||
self.assertTrue(
|
||||
first.startswith("//usr/bin/env go run"),
|
||||
f"shebang first line, got {first!r}",
|
||||
)
|
||||
|
||||
def test_no_nested_go_mod_under_bin(self) -> None:
|
||||
nested = list((ROOT / "bin").rglob("go.mod"))
|
||||
self.assertEqual(nested, [], f"nested go.mod files: {nested}")
|
||||
|
||||
def test_rank_lives_in_internal_brain(self) -> None:
|
||||
self.assertTrue(
|
||||
(ROOT / "internal" / "brain" / "rank" / "rank.go").is_file(),
|
||||
"ranking must live in internal/brain/rank (cgo-free)",
|
||||
)
|
||||
self.assertFalse(
|
||||
(ROOT / "bin" / "kbsearch").exists(),
|
||||
"bin/kbsearch nested module must be gone",
|
||||
)
|
||||
|
||||
def test_no_main_go_under_bin_brain(self) -> None:
|
||||
main = ROOT / "bin" / "brain" / "main.go"
|
||||
self.assertFalse(main.exists(), "bin/brain/main.go is not a method")
|
||||
|
||||
def test_chats_methods_are_shebangs_not_main(self) -> None:
|
||||
chats = ROOT / "bin" / "chats"
|
||||
self.assertFalse(
|
||||
(chats / "main.go").exists(),
|
||||
"bin/chats/main.go is a dispatcher, not a method",
|
||||
)
|
||||
self.assertFalse(
|
||||
(chats / "index_cmd.go").exists(),
|
||||
"chats index is a brain write hiding under the wrong subject",
|
||||
)
|
||||
for method in ("sync.go", "import.go", "facts.go", "apply.go"):
|
||||
p = chats / method
|
||||
self.assertTrue(p.is_file(), f"missing bin/chats/{method}")
|
||||
first = p.read_text().splitlines()[0]
|
||||
self.assertTrue(
|
||||
first.startswith("//usr/bin/env go run"),
|
||||
f"{method} shebang, got {first!r}",
|
||||
)
|
||||
|
||||
def test_chats_lib_lives_in_internal(self) -> None:
|
||||
self.assertTrue(
|
||||
(ROOT / "internal" / "chats" / "linkedin.go").is_file(),
|
||||
"LinkedIn parser must live in internal/chats",
|
||||
)
|
||||
self.assertFalse(
|
||||
(ROOT / "bin" / "chats" / "linkedin.go").exists(),
|
||||
"parser must not stay under bin/chats as a second main",
|
||||
)
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Published docs must match live commands (Gitea SoT, brain/search, no fake --hop)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
class PublishedDocsTest(unittest.TestCase):
|
||||
def test_readme_points_issues_at_gitea(self) -> None:
|
||||
text = (ROOT / "README.md").read_text()
|
||||
self.assertIn(
|
||||
"https://git.produktor.io/eSlider/2dph/issues",
|
||||
text,
|
||||
"README must point issues at Gitea",
|
||||
)
|
||||
|
||||
def test_plan_d15_names_gitea_origin(self) -> None:
|
||||
text = (ROOT / "PLAN.md").read_text()
|
||||
self.assertIn("D15", text)
|
||||
self.assertIn("git.produktor.io/eSlider/2dph", text)
|
||||
|
||||
def test_readme_primary_search_is_brain(self) -> None:
|
||||
text = (ROOT / "README.md").read_text()
|
||||
self.assertIn(
|
||||
"bin/brain/search.go",
|
||||
text,
|
||||
"README deduction search must name bin/brain/search.go",
|
||||
)
|
||||
|
||||
def test_docs_do_not_claim_hop_walks(self) -> None:
|
||||
paths = [
|
||||
ROOT / "README.md",
|
||||
ROOT / "docs" / "design.md",
|
||||
ROOT / "skills" / "kb-search" / "SKILL.md",
|
||||
ROOT / "skills" / "diataxis-docs" / "SKILL.md",
|
||||
]
|
||||
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
|
||||
# Honest "not implemented" notes must not match.
|
||||
lie = re.compile(r"--hop (?:N|1).*(?:follow|walk)", re.I | re.S)
|
||||
for path in paths:
|
||||
text = path.read_text()
|
||||
self.assertIsNone(
|
||||
lie.search(text),
|
||||
f"{path.relative_to(ROOT)} still claims --hop walks the graph",
|
||||
)
|
||||
+5
-1
@@ -6,5 +6,9 @@ Brain/ops/eSlider stack. Facts need proof or they are
|
||||
|
||||
- [PLAN.md](../PLAN.md) — decisions, execution order, open questions (v2)
|
||||
- [design](design.md) — schema, deduction model, sources
|
||||
- [Gitea issues](https://git.produktor.io/eSlider/2dph/issues) — work board (origin)
|
||||
|
||||
Published docs live here and mirror the project state.
|
||||
Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go`). `--hop` is
|
||||
not a walk; the flag errors until File/FROM_FILE edges exist.
|
||||
|
||||
Published docs live here and mirror the project state.
|
||||
|
||||
@@ -15,9 +15,10 @@ OO_CLI (default: $HOME/go/bin/oo)
|
||||
## Quick reference
|
||||
|
||||
```
|
||||
./bin/chat sync telegram --limit 100
|
||||
./bin/chat import
|
||||
./bin/chat index
|
||||
./bin/chat facts
|
||||
./bin/chat apply --dry-run
|
||||
./bin/chats/sync.go telegram --limit 100
|
||||
./bin/chats/import.go
|
||||
./bin/chats/facts.go
|
||||
./bin/chats/apply.go --dry-run
|
||||
```
|
||||
|
||||
JSONL → markdown only. Brain ingest is `bin/brain/index.go` (not a `chats index`).
|
||||
|
||||
+3
-3
@@ -17,14 +17,14 @@ as one consistent state.
|
||||
## Deduction search
|
||||
|
||||
```
|
||||
bin/kb/search "question"
|
||||
bin/brain/search.go "question"
|
||||
1. facts root — confirmed answers only → return with evidence links
|
||||
2. info root — supporting narrative → snippets, marked (not confirmed)
|
||||
3. web-search — second independent source → upgrade hypothesis to confirmed
|
||||
```
|
||||
|
||||
`--hop N` follows graph edges (sibling leaves under a heading, owning file,
|
||||
`related:` files, vector-neighbour leaves) — the deduction walk.
|
||||
`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an
|
||||
error; it is not a graph walk.
|
||||
|
||||
## Who / What / How / Where / When + evidence
|
||||
|
||||
|
||||
@@ -1,8 +1,25 @@
|
||||
module github.com/eSlider/2dph
|
||||
|
||||
go 1.25.0
|
||||
go 1.26
|
||||
|
||||
require (
|
||||
github.com/LadybugDB/go-ladybug v0.17.0
|
||||
github.com/arran4/golang-ical v0.3.5
|
||||
github.com/chewxy/math32 v1.11.2
|
||||
github.com/daulet/tokenizers v1.27.0
|
||||
golang.org/x/text v0.40.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/apache/arrow-go/v18 v18.6.0 // indirect
|
||||
github.com/goccy/go-json v0.10.6 // indirect
|
||||
github.com/google/flatbuffers v25.12.19+incompatible // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/klauspost/compress v1.18.5 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.26 // indirect
|
||||
github.com/shopspring/decimal v1.4.0 // indirect
|
||||
github.com/zeebo/xxh3 v1.1.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect
|
||||
golang.org/x/sys v0.43.0 // indirect
|
||||
)
|
||||
|
||||
@@ -1,14 +1,50 @@
|
||||
github.com/LadybugDB/go-ladybug v0.17.0 h1:RXDbkBjrbRmLdEbhGl4CLOIEzSt09gbP0n9UbKDEfwI=
|
||||
github.com/LadybugDB/go-ladybug v0.17.0/go.mod h1:GeIXmE8XyF5TFS94NAuTag7vgCC+no/HTBMRA6Rd5Cs=
|
||||
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
|
||||
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/apache/arrow-go/v18 v18.6.0 h1:GX/Jyd3R7mCLiECAwY9FWbbaYblie2WXBSz4Sw8fNpM=
|
||||
github.com/apache/arrow-go/v18 v18.6.0/go.mod h1:gm3MiPpY82fLYK5VKPB3WoJbsiLVDfT7flD5/vHReKw=
|
||||
github.com/apache/thrift v0.22.0 h1:r7mTJdj51TMDe6RtcmNdQxgn9XcyfGDOzegMDRg47uc=
|
||||
github.com/apache/thrift v0.22.0/go.mod h1:1e7J/O1Ae6ZQMTYdy9xa3w9k+XHWPfRvdPyJeynQ+/g=
|
||||
github.com/arran4/golang-ical v0.3.5 h1:bbz6ld4dC+MmCKiFfOd6SkmIGnhNMBACZ485ULh7p9A=
|
||||
github.com/arran4/golang-ical v0.3.5/go.mod h1:OnguFgjN0Hmx8jzpmWcC+AkHio94ujmLHKoaef7xQh8=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/chewxy/math32 v1.11.2 h1:IufN08Zwr1NKuWfY+4Tz55BcwKmyKKNdOP7KtumehnM=
|
||||
github.com/chewxy/math32 v1.11.2/go.mod h1:dOB2rcuFrCn6UHrze36WSLVPKtzPMRAQvBvUwkSsLqs=
|
||||
github.com/daulet/tokenizers v1.27.0 h1:MmFYAEDFz69s/nNQfHg59DWqHz3v94m99kEZ/JbL+s4=
|
||||
github.com/daulet/tokenizers v1.27.0/go.mod h1:YjFY1o1HGMyWkQgbXJDghhvke/yFDp2vGdIO2hYs4MQ=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
|
||||
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs=
|
||||
github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
|
||||
github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
|
||||
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
|
||||
github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
|
||||
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU=
|
||||
golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU=
|
||||
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
|
||||
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
// Brain connection management using go-ladybug.
|
||||
package main
|
||||
//go:build cgo && system_ladybug
|
||||
|
||||
package brain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
lbug "github.com/LadybugDB/go-ladybug"
|
||||
)
|
||||
@@ -44,10 +46,10 @@ func dbPath() string {
|
||||
}
|
||||
|
||||
func openBrain() error {
|
||||
return openWithOpts(2, eps())
|
||||
return openWithSandbox(eps())
|
||||
}
|
||||
|
||||
func openWithOpts(allow int, epsv string) error {
|
||||
func openWithSandbox(epsv string) error {
|
||||
cfg := lbug.DefaultSystemConfig()
|
||||
cfg.MaxNumThreads = 8
|
||||
cfg.BufferPoolSize = 1 << 30 // 1GB
|
||||
@@ -57,20 +59,30 @@ func openWithOpts(allow int, epsv string) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("OpenDatabase: %w", err)
|
||||
}
|
||||
if epsv != "" {
|
||||
if _, err := conn.Query("SET STREAM_SANDBOX = '" + epsv + "'"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
conn, err = lbug.OpenConnection(db)
|
||||
if err != nil {
|
||||
closeBrain()
|
||||
return fmt.Errorf("OpenConnection: %w", err)
|
||||
}
|
||||
// Session settings need a live connection; running this before
|
||||
// OpenConnection dereferenced a nil *Connection.
|
||||
if epsv != "" {
|
||||
if strings.ContainsAny(epsv, "'\\") {
|
||||
closeBrain()
|
||||
return fmt.Errorf("SET STREAM_SANDBOX: invalid value")
|
||||
}
|
||||
if _, err := conn.Query("SET STREAM_SANDBOX = '" + epsv + "'"); err != nil {
|
||||
closeBrain()
|
||||
return fmt.Errorf("SET STREAM_SANDBOX: %w", err)
|
||||
}
|
||||
}
|
||||
if _, err := conn.Query("LOAD EXTENSION FTS"); err != nil {
|
||||
closeBrain()
|
||||
return fmt.Errorf("LOAD EXTENSION FTS: %w", err)
|
||||
}
|
||||
if _, err := conn.Query("LOAD EXTENSION VECTOR"); err != nil {
|
||||
closeBrain()
|
||||
return fmt.Errorf("LOAD EXTENSION VECTOR: %w", err)
|
||||
}
|
||||
return nil
|
||||
@@ -85,4 +97,4 @@ func closeBrain() {
|
||||
db.Close()
|
||||
db = nil
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
// Package brain is deduction search over Ladybug (FTS + HNSW).
|
||||
// Query/embed code that needs cgo lives behind the system_ladybug tag.
|
||||
package brain
|
||||
@@ -1,9 +1,11 @@
|
||||
//go:build cgo && system_ladybug
|
||||
|
||||
// StaticModel wraps the potion-multilingual-128m embedding model.
|
||||
//
|
||||
// Mirrors model2vec.StaticModel: tokenizer (daulet Unigram) + safetensors matrix.
|
||||
// Embed(text) applies the same preprocessing: median_token_length pre-truncation,
|
||||
// add_special_tokens=false, drop unk (id=1), truncate to 512, mean pool, L2 normalize +1e-32.
|
||||
package main
|
||||
package brain
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
@@ -1,5 +1,4 @@
|
||||
// modelDir returns the resolved potion-multilingual-128m model directory.
|
||||
package main
|
||||
package brain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -0,0 +1,72 @@
|
||||
package rank
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const Usage = `usage: bin/brain/search.go "query" [--root facts|info] [--repo REPO] [-n N] [--json]
|
||||
bin/brain/search.go serve [port]
|
||||
bin/brain/search.go --list-model`
|
||||
|
||||
type Options struct {
|
||||
Query string
|
||||
Root string
|
||||
Repo string
|
||||
Limit int
|
||||
JSONOut bool
|
||||
ListModel bool
|
||||
}
|
||||
|
||||
// ParseArgs reads flags. Unknown flags are an error: silently dropping them
|
||||
// meant `--hop 1` vanished and its argument `1` was appended to the query.
|
||||
// --hop is recognised so it cannot be swallowed; it is not implemented until
|
||||
// File/FROM_FILE edges exist.
|
||||
func ParseArgs(args []string) (Options, error) {
|
||||
opt := Options{Limit: 20}
|
||||
var queryArgs []string
|
||||
|
||||
for i := 0; i < len(args); i++ {
|
||||
arg := args[i]
|
||||
wantsValue := arg == "--root" || arg == "--repo" || arg == "-n" || arg == "--hop"
|
||||
if wantsValue && i+1 >= len(args) {
|
||||
return opt, fmt.Errorf("%s needs a value", arg)
|
||||
}
|
||||
switch arg {
|
||||
case "--root":
|
||||
i++
|
||||
opt.Root = args[i]
|
||||
if opt.Root != "facts" && opt.Root != "info" {
|
||||
return opt, fmt.Errorf("--root must be facts or info, got %q", opt.Root)
|
||||
}
|
||||
case "--repo":
|
||||
i++
|
||||
opt.Repo = args[i]
|
||||
case "-n":
|
||||
i++
|
||||
n, err := strconv.Atoi(args[i])
|
||||
if err != nil || n < 1 {
|
||||
return opt, fmt.Errorf("-n must be a positive integer, got %q", args[i])
|
||||
}
|
||||
opt.Limit = n
|
||||
case "--hop":
|
||||
return opt, fmt.Errorf("--hop is not implemented yet (needs File/FROM_FILE edges)")
|
||||
case "--json":
|
||||
opt.JSONOut = true
|
||||
case "--list-model":
|
||||
opt.ListModel = true
|
||||
default:
|
||||
if strings.HasPrefix(arg, "-") {
|
||||
return opt, fmt.Errorf("unknown flag %q", arg)
|
||||
}
|
||||
queryArgs = append(queryArgs, arg)
|
||||
}
|
||||
}
|
||||
|
||||
opt.Query = strings.TrimSpace(strings.Join(queryArgs, " "))
|
||||
if opt.Query == "" && !opt.ListModel {
|
||||
return opt, fmt.Errorf("no query given")
|
||||
}
|
||||
return opt, nil
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
package rank
|
||||
|
||||
// BM25 ranks best-first, so the top hits are the *highest* scores; cosine
|
||||
// distance ranks best-first ascending. Both mirror kblib.py.
|
||||
const FTSStmt = "CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " +
|
||||
"RETURN node.id, node.text, node.root, node.source, score ORDER BY score DESC LIMIT $n"
|
||||
|
||||
const VecStmt = "CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
|
||||
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n"
|
||||
@@ -0,0 +1,100 @@
|
||||
// Package rank is the cgo-free ranking and CLI parsing for brain search.
|
||||
// CI can `go test ./rank` without the native ladybug library.
|
||||
package rank
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Hit is one search result, mirroring the python script's dict shape.
|
||||
type Hit struct {
|
||||
ID string `json:"id"`
|
||||
Text string `json:"text"`
|
||||
Root string `json:"root"`
|
||||
Source string `json:"-"`
|
||||
Score float64 `json:"score"`
|
||||
Snippet string `json:"snippet,omitempty"`
|
||||
}
|
||||
|
||||
// rrfK dampens the contribution of low ranks; same constant as kblib.py.
|
||||
const rrfK = 60
|
||||
|
||||
// RankAndFilter fuses the two hit lists, applies --root/--repo, then cuts to
|
||||
// limit. Cutting first dropped every matching leaf ranked below the cut, so
|
||||
// `--root facts` came back empty whenever info leafs filled the top N.
|
||||
// limit <= 0 keeps everything.
|
||||
func RankAndFilter(fts, vec []Hit, root, repo string, limit int) []Hit {
|
||||
out := Hybrid(fts, vec, 0)
|
||||
if root != "" {
|
||||
out = FilterRoot(out, root)
|
||||
}
|
||||
if repo != "" {
|
||||
out = FilterRepo(out, repo)
|
||||
}
|
||||
if limit > 0 && len(out) > limit {
|
||||
out = out[:limit]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Hybrid merges FTS and vector hits by reciprocal rank fusion.
|
||||
// limit <= 0 returns the full fused list.
|
||||
func Hybrid(fts, vec []Hit, limit int) []Hit {
|
||||
byID := make(map[string]Hit, len(fts)+len(vec))
|
||||
rrf := make(map[string]float64, len(fts)+len(vec))
|
||||
|
||||
for i, h := range fts {
|
||||
byID[h.ID] = h
|
||||
rrf[h.ID] += 1.0 / (rrfK + float64(i+1))
|
||||
}
|
||||
for i, h := range vec {
|
||||
if existing, ok := byID[h.ID]; !ok {
|
||||
byID[h.ID] = h
|
||||
} else if existing.Score == 0 {
|
||||
existing.Score = h.Score
|
||||
byID[h.ID] = existing
|
||||
}
|
||||
rrf[h.ID] += 1.0 / (rrfK + float64(i+1))
|
||||
}
|
||||
|
||||
ids := make([]string, 0, len(rrf))
|
||||
for id := range rrf {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Slice(ids, func(i, j int) bool {
|
||||
if rrf[ids[i]] != rrf[ids[j]] {
|
||||
return rrf[ids[i]] > rrf[ids[j]]
|
||||
}
|
||||
return ids[i] < ids[j]
|
||||
})
|
||||
if limit > 0 && len(ids) > limit {
|
||||
ids = ids[:limit]
|
||||
}
|
||||
|
||||
out := make([]Hit, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
out = append(out, byID[id])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func FilterRoot(hits []Hit, root string) []Hit {
|
||||
var out []Hit
|
||||
for _, h := range hits {
|
||||
if h.Root == root {
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func FilterRepo(hits []Hit, repo string) []Hit {
|
||||
var out []Hit
|
||||
for _, h := range hits {
|
||||
if strings.Contains(h.Source, repo) {
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
// Unit tests for ranking/filtering and CLI parsing (no db, no model, offline).
|
||||
package rank
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func h(id, root, source string) Hit {
|
||||
return Hit{ID: id, Text: id, Root: root, Source: source}
|
||||
}
|
||||
|
||||
func ids(hits []Hit) []string {
|
||||
out := make([]string, len(hits))
|
||||
for i, hit := range hits {
|
||||
out[i] = hit.ID
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func eq(t *testing.T, got []Hit, want ...string) {
|
||||
t.Helper()
|
||||
g := ids(got)
|
||||
if len(g) != len(want) {
|
||||
t.Fatalf("got %v, want %v", g, want)
|
||||
}
|
||||
for i := range want {
|
||||
if g[i] != want[i] {
|
||||
t.Fatalf("got %v, want %v", g, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A facts leaf that ranks below the limit in the unfiltered list must still
|
||||
// be returned for --root facts. Filtering after truncation loses it.
|
||||
func TestRankAndFilterFiltersBeforeLimit(t *testing.T) {
|
||||
fts := []Hit{
|
||||
h("i1", "info", "docs/a.md"),
|
||||
h("i2", "info", "docs/b.md"),
|
||||
h("i3", "info", "docs/c.md"),
|
||||
h("f1", "facts", "docker ps x compose"),
|
||||
}
|
||||
eq(t, RankAndFilter(fts, nil, "facts", "", 2), "f1")
|
||||
}
|
||||
|
||||
func TestRankAndFilterRepoFiltersBeforeLimit(t *testing.T) {
|
||||
fts := []Hit{
|
||||
h("a", "info", "eSlider/2dph:README.md"),
|
||||
h("b", "info", "eSlider/2dph:PLAN.md"),
|
||||
h("c", "info", "eSlider/ops:compose.yaml"),
|
||||
}
|
||||
eq(t, RankAndFilter(fts, nil, "", "ops", 2), "c")
|
||||
}
|
||||
|
||||
func TestRankAndFilterTruncatesToLimit(t *testing.T) {
|
||||
fts := []Hit{h("a", "info", "x"), h("b", "info", "x"), h("c", "info", "x")}
|
||||
eq(t, RankAndFilter(fts, nil, "", "", 2), "a", "b")
|
||||
}
|
||||
|
||||
func TestRankAndFilterLimitZeroKeepsAll(t *testing.T) {
|
||||
fts := []Hit{h("a", "info", "x"), h("b", "info", "x")}
|
||||
eq(t, RankAndFilter(fts, nil, "", "", 0), "a", "b")
|
||||
}
|
||||
|
||||
func TestHybridFusesBothRetrievers(t *testing.T) {
|
||||
fts := []Hit{h("only-fts", "info", "x"), h("both", "info", "x")}
|
||||
vec := []Hit{h("only-vec", "info", "x"), h("both", "info", "x")}
|
||||
eq(t, Hybrid(fts, vec, 0), "both", "only-fts", "only-vec")
|
||||
}
|
||||
|
||||
func TestHybridTiesAreDeterministic(t *testing.T) {
|
||||
fts := []Hit{h("b", "info", "x"), h("a", "info", "x")}
|
||||
first := ids(Hybrid(fts, nil, 0))
|
||||
for i := 0; i < 50; i++ {
|
||||
got := ids(Hybrid(fts, nil, 0))
|
||||
for j := range first {
|
||||
if got[j] != first[j] {
|
||||
t.Fatalf("unstable order: %v then %v", first, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHybridKeepsVectorScoreForSharedHit(t *testing.T) {
|
||||
fts := []Hit{{ID: "x", Root: "info", Score: 0}}
|
||||
vec := []Hit{{ID: "x", Root: "info", Score: 0.87}}
|
||||
got := Hybrid(fts, vec, 0)
|
||||
if len(got) != 1 || got[0].Score != 0.87 {
|
||||
t.Fatalf("got %+v, want score 0.87", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The old parser dropped unknown flags and appended their arguments to the
|
||||
// query, so `search "q" --hop 1` searched for "q 1". --hop is not implemented
|
||||
// here (needs File edges); it must still fail closed instead of changing q.
|
||||
func TestParseHopIsNotSwallowedIntoTheQuery(t *testing.T) {
|
||||
_, err := ParseArgs([]string{"what runs on arc-2", "--hop", "1"})
|
||||
if err == nil {
|
||||
t.Fatal("expected --hop to error (not implemented), not be swallowed")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "--hop") {
|
||||
t.Fatalf("error should name --hop, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRejectsUnknownFlags(t *testing.T) {
|
||||
if _, err := ParseArgs([]string{"query", "--nope"}); err == nil {
|
||||
t.Fatal("unknown flag accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRejectsBadValues(t *testing.T) {
|
||||
for _, args := range [][]string{
|
||||
{"q", "-n", "zero"},
|
||||
{"q", "-n", "0"},
|
||||
{"q", "--root", "nonsense"},
|
||||
{"q", "--hop"},
|
||||
{"--json"},
|
||||
} {
|
||||
if _, err := ParseArgs(args); err == nil {
|
||||
t.Errorf("accepted %v", args)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDefaults(t *testing.T) {
|
||||
opt, err := ParseArgs([]string{"two", "words", "--json"})
|
||||
if err != nil || opt.Query != "two words" || opt.Limit != 20 || !opt.JSONOut {
|
||||
t.Fatalf("got %+v err=%v", opt, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListModelNeedsNoQuery(t *testing.T) {
|
||||
if _, err := ParseArgs([]string{"--list-model"}); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsageNamesBrainSearch(t *testing.T) {
|
||||
if !strings.Contains(Usage, "bin/brain/search.go") {
|
||||
t.Fatalf("usage must name bin/brain/search.go, got:\n%s", Usage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFTSQueryOrdersByScoreDescending(t *testing.T) {
|
||||
if !strings.Contains(FTSStmt, "ORDER BY score DESC") {
|
||||
t.Fatalf("FTS query must order by score DESC, got:\n%s", FTSStmt)
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
// Hybrid FTS + vector search implementation, plus daemon client/server.
|
||||
package main
|
||||
//go:build cgo && system_ladybug
|
||||
|
||||
package brain
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -13,12 +14,12 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
lbug "github.com/LadybugDB/go-ladybug"
|
||||
"github.com/eSlider/2dph/internal/brain/rank"
|
||||
)
|
||||
|
||||
const defaultPort = 17830
|
||||
@@ -26,45 +27,15 @@ const daemonPath = "/embed"
|
||||
const healthPath = "/health"
|
||||
|
||||
func runSearch(args []string) int {
|
||||
// Manual flag parsing to allow flags after query (like Python argparse)
|
||||
root := ""
|
||||
repo := ""
|
||||
limit := 20
|
||||
jsonOut := false
|
||||
listModel := false
|
||||
|
||||
var queryArgs []string
|
||||
for i := 0; i < len(args); i++ {
|
||||
switch args[i] {
|
||||
case "--root":
|
||||
if i+1 < len(args) {
|
||||
root = args[i+1]
|
||||
i++
|
||||
}
|
||||
case "--repo":
|
||||
if i+1 < len(args) {
|
||||
repo = args[i+1]
|
||||
i++
|
||||
}
|
||||
case "-n":
|
||||
if i+1 < len(args) {
|
||||
if n, err := strconv.Atoi(args[i+1]); err == nil {
|
||||
limit = n
|
||||
}
|
||||
i++
|
||||
}
|
||||
case "--json":
|
||||
jsonOut = true
|
||||
case "--list-model":
|
||||
listModel = true
|
||||
default:
|
||||
if !strings.HasPrefix(args[i], "-") {
|
||||
queryArgs = append(queryArgs, args[i])
|
||||
}
|
||||
}
|
||||
opt, err := rank.ParseArgs(args)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "brain/search: %v\n%s\n", err, rank.Usage)
|
||||
return 2
|
||||
}
|
||||
root, repo, limit, query := opt.Root, opt.Repo, opt.Limit, opt.Query
|
||||
jsonOut := opt.JSONOut
|
||||
|
||||
if listModel {
|
||||
if opt.ListModel {
|
||||
dir, err := modelDir()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
@@ -74,12 +45,6 @@ func runSearch(args []string) int {
|
||||
return 0
|
||||
}
|
||||
|
||||
query := strings.TrimSpace(strings.Join(queryArgs, " "))
|
||||
if query == "" {
|
||||
fmt.Fprintln(os.Stderr, "usage: kbsearch \"query\" [--root facts|info] [--repo REPO] [-n N] [--json]")
|
||||
return 1
|
||||
}
|
||||
|
||||
if err := openBrain(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "open brain: %v\n", err)
|
||||
return 1
|
||||
@@ -103,17 +68,7 @@ func runSearch(args []string) int {
|
||||
fmt.Fprintf(os.Stderr, "vec: %v\n", err)
|
||||
}
|
||||
|
||||
results := hybrid(fts, vec, limit)
|
||||
|
||||
if root != "" {
|
||||
results = filterRoot(results, root)
|
||||
}
|
||||
if repo != "" {
|
||||
results = filterRepo(results, repo)
|
||||
}
|
||||
if len(results) > limit {
|
||||
results = results[:limit]
|
||||
}
|
||||
results := rank.RankAndFilter(fts, vec, root, repo, limit)
|
||||
|
||||
for i := range results {
|
||||
if results[i].Text != "" {
|
||||
@@ -150,10 +105,7 @@ func b2i(err error) int {
|
||||
}
|
||||
|
||||
func queryFTS(text string, limit int) ([]Hit, error) {
|
||||
stmt, err := conn.Prepare(
|
||||
"CALL QUERY_FTS_INDEX('Leaf', 'id', $q) " +
|
||||
"RETURN node.id, node.text, node.root, node.source, score ORDER BY score LIMIT $n",
|
||||
)
|
||||
stmt, err := conn.Prepare(rank.FTSStmt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -170,10 +122,7 @@ func queryVector(emb []float64, limit int) ([]Hit, error) {
|
||||
for i, v := range emb {
|
||||
embList[i] = v
|
||||
}
|
||||
stmt, err := conn.Prepare(
|
||||
"CALL QUERY_VECTOR_INDEX('Leaf', 'Leaf_vec', $q, $n) " +
|
||||
"RETURN node.id, node.text, node.root, node.source, distance ORDER BY distance LIMIT $n",
|
||||
)
|
||||
stmt, err := conn.Prepare(rank.VecStmt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -215,10 +164,10 @@ func rowsToHits(res *lbug.QueryResult) ([]Hit, error) {
|
||||
|
||||
// JSON output types
|
||||
type jsonOut struct {
|
||||
Query string `json:"query"`
|
||||
RootFilter string `json:"root_filter"`
|
||||
Count int `json:"count"`
|
||||
Results []jsonHit `json:"results"`
|
||||
Query string `json:"query"`
|
||||
RootFilter string `json:"root_filter"`
|
||||
Count int `json:"count"`
|
||||
Results []jsonHit `json:"results"`
|
||||
}
|
||||
|
||||
type jsonHit struct {
|
||||
@@ -248,70 +197,6 @@ func toJSONOut(hits []Hit, query, rootFilter string) *jsonOut {
|
||||
}
|
||||
}
|
||||
|
||||
func hybrid(fts, vec []Hit, limit int) []Hit {
|
||||
byID := make(map[string]Hit)
|
||||
rrf := make(map[string]float64)
|
||||
|
||||
for rank, h := range fts {
|
||||
byID[h.ID] = h
|
||||
rrf[h.ID] += 1.0 / (60 + float64(rank+1))
|
||||
}
|
||||
for rank, h := range vec {
|
||||
if _, ok := byID[h.ID]; !ok {
|
||||
byID[h.ID] = h
|
||||
} else {
|
||||
existing := byID[h.ID]
|
||||
if existing.Score == 0 {
|
||||
existing.Score = h.Score
|
||||
byID[h.ID] = existing
|
||||
}
|
||||
}
|
||||
rrf[h.ID] += 1.0 / (60 + float64(rank+1))
|
||||
}
|
||||
|
||||
type scored struct {
|
||||
id string
|
||||
rrf float64
|
||||
}
|
||||
var scoredList []scored
|
||||
for id, v := range rrf {
|
||||
scoredList = append(scoredList, scored{id, v})
|
||||
}
|
||||
sort.Slice(scoredList, func(i, j int) bool {
|
||||
return scoredList[i].rrf > scoredList[j].rrf
|
||||
})
|
||||
|
||||
var out []Hit
|
||||
for i, s := range scoredList {
|
||||
if i >= limit {
|
||||
break
|
||||
}
|
||||
h := byID[s.id]
|
||||
out = append(out, h)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func filterRoot(hits []Hit, root string) []Hit {
|
||||
var out []Hit
|
||||
for _, h := range hits {
|
||||
if h.Root == root {
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func filterRepo(hits []Hit, repo string) []Hit {
|
||||
var out []Hit
|
||||
for _, h := range hits {
|
||||
if strings.Contains(h.Source, repo) {
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func resultsToDicts(hits []Hit) []any {
|
||||
out := make([]any, len(hits))
|
||||
for i, h := range hits {
|
||||
@@ -363,7 +248,7 @@ func serve(port int) error {
|
||||
})
|
||||
|
||||
addr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
log.Printf("kbsearch daemon listening on %s", addr)
|
||||
log.Printf("brain search daemon listening on %s", addr)
|
||||
return http.ListenAndServe(addr, mux)
|
||||
}
|
||||
|
||||
@@ -382,10 +267,16 @@ func embedQuery(text string) ([]float64, error) {
|
||||
port = p
|
||||
}
|
||||
}
|
||||
emb, err := tryDaemon(text, port)
|
||||
if err == nil {
|
||||
if emb, err := tryDaemon(text, port); err == nil {
|
||||
return emb, nil
|
||||
}
|
||||
if os.Getenv("KBSEARCH_NO_DAEMON") == "" {
|
||||
if err := ensureDaemon(port); err == nil {
|
||||
if emb, err := tryDaemon(text, port); err == nil {
|
||||
return emb, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
model, err := loadModel()
|
||||
if err != nil {
|
||||
@@ -447,9 +338,13 @@ func ensureDaemon(port int) error {
|
||||
cmd.Dir, _ = filepath.Split(self)
|
||||
cmd.Stdout = nil
|
||||
cmd.Stderr = nil
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
if cmd.Process != nil {
|
||||
_ = cmd.Process.Release()
|
||||
}
|
||||
|
||||
for i := 0; i < 40; i++ {
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
@@ -462,4 +357,22 @@ func ensureDaemon(port int) error {
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("daemon failed to start on port %d", port)
|
||||
}
|
||||
}
|
||||
|
||||
// Main is the bin/brain/search.go entry: search, serve, or --list-model.
|
||||
func Main(args []string) int {
|
||||
if len(args) > 0 && args[0] == "serve" {
|
||||
port := defaultPort
|
||||
if len(args) > 1 {
|
||||
if p, err := strconv.Atoi(args[1]); err == nil {
|
||||
port = p
|
||||
}
|
||||
}
|
||||
if err := serve(port); err != nil {
|
||||
log.Printf("brain/search serve: %v", err)
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
return runSearch(args)
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package brain
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/eSlider/2dph/internal/brain/rank"
|
||||
)
|
||||
|
||||
func eps() string { return os.Getenv("KBTEST_EPS") }
|
||||
|
||||
// Hit is the search hit type; ranking lives in package rank so CI can test
|
||||
// it without the native ladybug library.
|
||||
type Hit = rank.Hit
|
||||
@@ -1,5 +1,5 @@
|
||||
// YAML emitter ported from bin/kb/yamlout.py — preserves insertion order.
|
||||
package main
|
||||
package brain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -24,7 +24,7 @@ type ooContact struct {
|
||||
} `json:"commonData"`
|
||||
}
|
||||
|
||||
func runApply(args []string) int {
|
||||
func RunApply(args []string) int {
|
||||
fs := flag.NewFlagSet("chats apply", flag.ContinueOnError)
|
||||
dryRun := fs.Bool("dry-run", false, "show what would be done without writing")
|
||||
help := fs.Bool("help", false, "")
|
||||
@@ -176,7 +176,7 @@ func runApply(args []string) int {
|
||||
}
|
||||
|
||||
func loadFacts() ([]ExtractedFact, error) {
|
||||
factsPath := filepath.Join(chatsDir(), "facts", "chat-facts.json")
|
||||
factsPath := filepath.Join(Dir(), "facts", "chat-facts.json")
|
||||
data, err := os.ReadFile(factsPath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
@@ -3,7 +3,7 @@
|
||||
// These are integration tests using real data and real Telegram API (when
|
||||
// credentials are available). They follow the TDD workflow pattern:
|
||||
// sync → import → facts → verify.
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
@@ -48,7 +48,7 @@ func TestChatsImport(t *testing.T) {
|
||||
t.Cleanup(func() { os.Chdir(cwd) })
|
||||
t.Setenv("KB_ROOT", dir)
|
||||
|
||||
exitCode := runImport([]string{})
|
||||
exitCode := RunImport([]string{})
|
||||
if exitCode != 0 {
|
||||
t.Fatalf("import exit code %d", exitCode)
|
||||
}
|
||||
@@ -140,7 +140,7 @@ func TestChatsImportEmpty(t *testing.T) {
|
||||
t.Cleanup(func() { os.Chdir(cwd) })
|
||||
t.Setenv("KB_ROOT", dir)
|
||||
|
||||
exitCode := runImport([]string{})
|
||||
exitCode := RunImport([]string{})
|
||||
if exitCode == 0 {
|
||||
t.Fatal("expected non-zero exit for empty data dir")
|
||||
}
|
||||
@@ -171,7 +171,7 @@ func TestChatsRoundTrip(t *testing.T) {
|
||||
t.Cleanup(func() { os.Chdir(cwd) })
|
||||
t.Setenv("KB_ROOT", dir)
|
||||
|
||||
if code := runImport([]string{}); code != 0 {
|
||||
if code := RunImport([]string{}); code != 0 {
|
||||
t.Fatalf("import exit %d", code)
|
||||
}
|
||||
|
||||
@@ -1,13 +1,11 @@
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
@@ -77,7 +75,7 @@ type ExtractedFact struct {
|
||||
MessageID string `json:"message_id"`
|
||||
}
|
||||
|
||||
func runFacts(args []string) int {
|
||||
func RunFacts(args []string) int {
|
||||
fs := flag.NewFlagSet("chats facts", flag.ContinueOnError)
|
||||
help := fs.Bool("help", false, "")
|
||||
fs.SetOutput(os.Stderr)
|
||||
@@ -89,7 +87,7 @@ func runFacts(args []string) int {
|
||||
return 0
|
||||
}
|
||||
|
||||
root := chatsDir()
|
||||
root := Dir()
|
||||
telegramDir := filepath.Join(root, "telegram")
|
||||
|
||||
entries, err := os.ReadDir(telegramDir)
|
||||
@@ -149,7 +147,7 @@ func runFacts(args []string) int {
|
||||
}
|
||||
fmt.Printf("chats facts: saved to %s\n", factsPath)
|
||||
|
||||
writeFactsToBrain(root, allFacts)
|
||||
writeFactsMarkdown(allFacts)
|
||||
|
||||
return 0
|
||||
}
|
||||
@@ -273,14 +271,10 @@ func filterFacts(facts []ExtractedFact, factType string) []ExtractedFact {
|
||||
return result
|
||||
}
|
||||
|
||||
func writeFactsToBrain(root string, facts []ExtractedFact) {
|
||||
indexScript := filepath.Join(root, "bin", "kb", "index")
|
||||
if _, err := os.Stat(indexScript); os.IsNotExist(err) {
|
||||
fmt.Fprintf(os.Stderr, "chats facts: kb/index not found, skipping brain write\n")
|
||||
return
|
||||
}
|
||||
|
||||
mdDir := filepath.Join(chatsDir(), "facts")
|
||||
// writeFactsMarkdown stores a sidecar for humans. Brain ingest is
|
||||
// bin/brain/index.go (not this subject).
|
||||
func writeFactsMarkdown(facts []ExtractedFact) {
|
||||
mdDir := filepath.Join(Dir(), "facts")
|
||||
if err := os.MkdirAll(mdDir, 0755); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats facts: mkdir %s: %v\n", mdDir, err)
|
||||
return
|
||||
@@ -288,7 +282,7 @@ func writeFactsToBrain(root string, facts []ExtractedFact) {
|
||||
|
||||
var sb strings.Builder
|
||||
sb.WriteString("---\n")
|
||||
sb.WriteString("root: facts\n")
|
||||
sb.WriteString("root: info\n")
|
||||
sb.WriteString("---\n\n")
|
||||
sb.WriteString("# Chat-Derived Facts\n\n")
|
||||
for _, f := range facts {
|
||||
@@ -302,15 +296,5 @@ func writeFactsToBrain(root string, facts []ExtractedFact) {
|
||||
fmt.Fprintf(os.Stderr, "chats facts: write %s: %v\n", factsMD, err)
|
||||
return
|
||||
}
|
||||
|
||||
cmd := exec.Command(indexScript, "--corpus", mdDir, "--skip-indexes")
|
||||
var outBuf, errBuf bytes.Buffer
|
||||
cmd.Stdout = &outBuf
|
||||
cmd.Stderr = &errBuf
|
||||
cmd.Dir = root
|
||||
if err := cmd.Run(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats facts: brain index: %v\n%s", err, errBuf.String())
|
||||
return
|
||||
}
|
||||
fmt.Printf("chats facts: written to brain (%s)\n", strings.TrimSpace(outBuf.String()))
|
||||
fmt.Printf("chats facts: markdown %s (index via brain, not chats)\n", factsMD)
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
@@ -13,7 +13,7 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
func runImport(args []string) int {
|
||||
func RunImport(args []string) int {
|
||||
fs := flag.NewFlagSet("chats import", flag.ContinueOnError)
|
||||
help := fs.Bool("help", false, "")
|
||||
fs.SetOutput(os.Stderr)
|
||||
@@ -25,7 +25,7 @@ func runImport(args []string) int {
|
||||
return 0
|
||||
}
|
||||
|
||||
root := chatsDir()
|
||||
root := Dir()
|
||||
mdRoot := filepath.Join(root, "md")
|
||||
glob := filepath.Join(root, "telegram", "*", "messages.jsonl")
|
||||
|
||||
@@ -0,0 +1,669 @@
|
||||
package chats
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type LinkedInMCPSource struct {
|
||||
userDataDir string
|
||||
limit int
|
||||
}
|
||||
|
||||
type lnInboxItem struct {
|
||||
ThreadID string `json:"thread_id"`
|
||||
Participants string `json:"participants"`
|
||||
LastMessage string `json:"last_message"`
|
||||
LastMessageDate string `json:"last_message_date"`
|
||||
Unread bool `json:"unread"`
|
||||
}
|
||||
|
||||
// mcp-server-linkedin v4.22 returns get_inbox / get_conversation as
|
||||
// {url, sections:{inbox|conversation: textblob}, references:{...}}.
|
||||
// The conversation list lives in references (kind=conversation); messages live
|
||||
// in the sections text blob, delimited by "<From> sent the following message
|
||||
// at <time>" markers. See testdata/linkedin_*.json for the wire shape.
|
||||
|
||||
type lnEnvelope struct {
|
||||
URL string `json:"url"`
|
||||
Sections map[string]any `json:"sections"`
|
||||
References map[string]any `json:"references"`
|
||||
}
|
||||
|
||||
type lnReference struct {
|
||||
Kind string `json:"kind"`
|
||||
URL string `json:"url"`
|
||||
Text string `json:"text"`
|
||||
Context string `json:"context"`
|
||||
}
|
||||
|
||||
type lnMessage struct {
|
||||
From string `json:"from"`
|
||||
Date string `json:"date"`
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
var (
|
||||
lnWeekdays = map[string]time.Weekday{
|
||||
"SUNDAY": time.Sunday, "MONDAY": time.Monday, "TUESDAY": time.Tuesday,
|
||||
"WEDNESDAY": time.Wednesday, "THURSDAY": time.Thursday,
|
||||
"FRIDAY": time.Friday, "SATURDAY": time.Saturday,
|
||||
}
|
||||
lnMsgStartRe = regexp.MustCompile(`^(.+?) sent the following messages? at (.+)$`)
|
||||
lnTimeRe = regexp.MustCompile(`\d{1,2}:\d{2}\s*[AP]M`)
|
||||
)
|
||||
|
||||
func isWeekdayLine(s string) bool {
|
||||
if _, ok := lnWeekdays[s]; ok {
|
||||
return true
|
||||
}
|
||||
switch s {
|
||||
case "TODAY", "YESTERDAY", "THIS WEEK", "LAST WEEK":
|
||||
return true
|
||||
}
|
||||
return lnMonthDayRe.MatchString(s)
|
||||
}
|
||||
|
||||
var lnMonthDayRe = regexp.MustCompile(`^[A-Z]{3}\s+\d{1,2}$`)
|
||||
|
||||
// parseLinkedInInbox extracts conversations from a get_inbox response.
|
||||
func parseLinkedInInbox(text string) []lnInboxItem {
|
||||
var env lnEnvelope
|
||||
if err := json.Unmarshal([]byte(text), &env); err != nil {
|
||||
return nil
|
||||
}
|
||||
refs, _ := env.References["inbox"].([]any)
|
||||
var items []lnInboxItem
|
||||
for _, r := range refs {
|
||||
rr, ok := r.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if rr["kind"] != "conversation" {
|
||||
continue
|
||||
}
|
||||
u, _ := rr["url"].(string)
|
||||
tid := threadIDFromURL(u)
|
||||
if !validThreadID(tid) {
|
||||
continue
|
||||
}
|
||||
name, _ := rr["text"].(string)
|
||||
items = append(items, lnInboxItem{
|
||||
ThreadID: tid,
|
||||
Participants: name,
|
||||
})
|
||||
}
|
||||
return items
|
||||
}
|
||||
|
||||
// parseLinkedInConversation parses the sections.conversation text blob into
|
||||
// messages. Messages are delimited by "<From> sent the following message(s) at
|
||||
// <time>" lines; each message body runs until the next marker. Day headers
|
||||
// (all-caps weekdays) provide date context; times are mapped to the most
|
||||
// recent matching weekday.
|
||||
func parseLinkedInConversation(text string) []lnMessage {
|
||||
var env lnEnvelope
|
||||
if err := json.Unmarshal([]byte(text), &env); err != nil {
|
||||
return nil
|
||||
}
|
||||
blob, _ := env.Sections["conversation"].(string)
|
||||
if blob == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
var msgs []lnMessage
|
||||
var cur *lnMessage
|
||||
var body []string
|
||||
day := ""
|
||||
|
||||
flush := func() {
|
||||
if cur == nil {
|
||||
return
|
||||
}
|
||||
cur.Text = strings.TrimSpace(strings.Join(body, "\n"))
|
||||
if ts := linkedInTimestamp(day, cur.Date); ts != "" {
|
||||
cur.Date = ts
|
||||
}
|
||||
if cur.Text != "" {
|
||||
msgs = append(msgs, *cur)
|
||||
}
|
||||
cur = nil
|
||||
body = nil
|
||||
}
|
||||
|
||||
for _, raw := range strings.Split(blob, "\n") {
|
||||
line := strings.TrimSpace(raw)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if isWeekdayLine(line) {
|
||||
if line != day {
|
||||
// A new day header terminates the previous message,
|
||||
// which must keep the earlier date context.
|
||||
flush()
|
||||
}
|
||||
day = line
|
||||
continue
|
||||
}
|
||||
if m := lnMsgStartRe.FindStringSubmatch(line); m != nil {
|
||||
flush()
|
||||
cur = &lnMessage{From: strings.TrimSpace(m[1]), Date: strings.TrimSpace(m[2])}
|
||||
continue
|
||||
}
|
||||
if cur == nil {
|
||||
continue
|
||||
}
|
||||
// Skip "View X's profile" and the "<From> (pronouns) <time>" header.
|
||||
if strings.HasPrefix(line, "View ") && strings.HasSuffix(line, "'s profile") {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, cur.From) && lnTimeRe.MatchString(line) {
|
||||
continue
|
||||
}
|
||||
body = append(body, line)
|
||||
}
|
||||
flush()
|
||||
return msgs
|
||||
}
|
||||
|
||||
// linkedInTimestamp maps a weekday, relative, or MON DD date header + clock
|
||||
// string to a timestamp, or returns "" when the clock cannot be parsed.
|
||||
func linkedInTimestamp(day, clock string) string {
|
||||
t, err := time.Parse("3:04 PM", clock)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
now := time.Now()
|
||||
var d time.Time
|
||||
if wd, ok := lnWeekdays[day]; ok {
|
||||
diff := (int(now.Weekday()) - int(wd) + 7) % 7
|
||||
d = now.AddDate(0, 0, -diff)
|
||||
} else {
|
||||
switch day {
|
||||
case "TODAY":
|
||||
d = now
|
||||
case "YESTERDAY":
|
||||
d = now.AddDate(0, 0, -1)
|
||||
case "THIS WEEK":
|
||||
diff := int(now.Weekday())
|
||||
d = now.AddDate(0, 0, -diff)
|
||||
case "LAST WEEK":
|
||||
diff := int(now.Weekday()) + 7
|
||||
d = now.AddDate(0, 0, -diff)
|
||||
default:
|
||||
if m := lnMonthDayRe.FindStringSubmatch(day); m != nil {
|
||||
// MON DD without a year: resolve to the most recent
|
||||
// occurrence that is not in the future.
|
||||
d = monthDayDate(day, now)
|
||||
if d.IsZero() {
|
||||
return t.Format("15:04")
|
||||
}
|
||||
} else {
|
||||
// No date context; keep bare clock time.
|
||||
return t.Format("15:04")
|
||||
}
|
||||
}
|
||||
}
|
||||
res := time.Date(d.Year(), d.Month(), d.Day(), t.Hour(), t.Minute(), 0, 0, time.UTC)
|
||||
return res.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
var lnMonths = map[string]time.Month{
|
||||
"JAN": time.January, "FEB": time.February, "MAR": time.March,
|
||||
"APR": time.April, "MAY": time.May, "JUN": time.June,
|
||||
"JUL": time.July, "AUG": time.August, "SEP": time.September,
|
||||
"OCT": time.October, "NOV": time.November, "DEC": time.December,
|
||||
}
|
||||
|
||||
// monthDayDate resolves "MON DD" to the most recent occurrence of that date,
|
||||
// preferring the current year and falling back to the previous year when the
|
||||
// date is in the future. Returns zero time when unresolvable.
|
||||
func monthDayDate(day string, now time.Time) time.Time {
|
||||
parts := strings.Fields(day)
|
||||
if len(parts) != 2 {
|
||||
return time.Time{}
|
||||
}
|
||||
mo, ok := lnMonths[parts[0]]
|
||||
if !ok {
|
||||
return time.Time{}
|
||||
}
|
||||
var dd int
|
||||
if _, err := fmt.Sscanf(parts[1], "%d", &dd); err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
if dd < 1 || dd > 31 {
|
||||
return time.Time{}
|
||||
}
|
||||
d := time.Date(now.Year(), mo, dd, 0, 0, 0, 0, time.UTC)
|
||||
if d.After(now) {
|
||||
d = d.AddDate(-1, 0, 0)
|
||||
}
|
||||
if d.After(now) {
|
||||
return time.Time{}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func threadIDFromURL(u string) string {
|
||||
u = strings.TrimSuffix(u, "/")
|
||||
idx := strings.LastIndex(u, "/")
|
||||
if idx < 0 {
|
||||
return ""
|
||||
}
|
||||
return u[idx+1:]
|
||||
}
|
||||
|
||||
// validThreadID rejects path segments that are not real thread ids (e.g. the
|
||||
// literal "thread" or an empty trailing segment).
|
||||
func validThreadID(id string) bool {
|
||||
if id == "" || id == "thread" {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func NewLinkedInMCPSource(userDataDir string) *LinkedInMCPSource {
|
||||
return &LinkedInMCPSource{userDataDir: userDataDir}
|
||||
}
|
||||
|
||||
func (s *LinkedInMCPSource) Name() string { return "linkedin" }
|
||||
|
||||
func (s *LinkedInMCPSource) Sync(ctx context.Context, outDir string, limit int) error {
|
||||
if limit > 0 {
|
||||
s.limit = limit
|
||||
}
|
||||
|
||||
// getConversation fetches one thread, recreating the MCP server when it
|
||||
// wedges. A single 429 makes mcp-server-linkedin close its browser and
|
||||
// refuse every later call ("still has a browser open"), so a broken server
|
||||
// must be restarted rather than hammered.
|
||||
getConversation := func(threadID string) ([]lnMessage, error) {
|
||||
client, err := newLinkedInMCP(ctx, s.userDataDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("linkedin mcp: %w", err)
|
||||
}
|
||||
defer client.Close()
|
||||
msgs, err := client.GetConversation(ctx, "", threadID, msgLimitFor(s.limit))
|
||||
if err != nil && wedged(err) {
|
||||
fmt.Fprintf(os.Stderr, "chats: %s: server wedged, restarting broker\n", threadID)
|
||||
time.Sleep(5 * time.Second)
|
||||
client2, cerr := newLinkedInMCP(ctx, s.userDataDir)
|
||||
if cerr == nil {
|
||||
defer client2.Close()
|
||||
msgs, err = client2.GetConversation(ctx, "", threadID, msgLimitFor(s.limit))
|
||||
}
|
||||
}
|
||||
return msgs, err
|
||||
}
|
||||
|
||||
client, err := newLinkedInMCP(ctx, s.userDataDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("linkedin mcp: %w", err)
|
||||
}
|
||||
inbox, err := client.GetInbox(ctx, 50)
|
||||
if err != nil {
|
||||
client.Close()
|
||||
return fmt.Errorf("get_inbox: %w", err)
|
||||
}
|
||||
client.Close()
|
||||
if len(inbox) == 0 {
|
||||
fmt.Println("chats: no LinkedIn conversations found")
|
||||
return nil
|
||||
}
|
||||
|
||||
fmt.Printf("chats: found %d LinkedIn conversations\n", len(inbox))
|
||||
|
||||
for _, conv := range inbox {
|
||||
convID := sanitizeDir(conv.ThreadID)
|
||||
if convID == "" {
|
||||
convID = fmt.Sprintf("conv_%d", time.Now().UnixNano())
|
||||
}
|
||||
|
||||
parts := strings.SplitN(conv.Participants, ",", 2)
|
||||
chatName := strings.TrimSpace(parts[0])
|
||||
if chatName == "" {
|
||||
chatName = convID
|
||||
}
|
||||
|
||||
chatDir := filepath.Join(outDir, "linkedin", convID)
|
||||
if err := os.MkdirAll(chatDir, 0755); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: mkdir %s: %v\n", chatDir, err)
|
||||
continue
|
||||
}
|
||||
|
||||
msgs, err := getConversation(conv.ThreadID)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: get_conversation %s: %v\n", convID, err)
|
||||
continue
|
||||
}
|
||||
|
||||
jsonlPath := filepath.Join(chatDir, "messages.jsonl")
|
||||
|
||||
// A rate-limited response can parse to zero messages. Never clobber
|
||||
// previously synced data with an empty file.
|
||||
if len(msgs) == 0 {
|
||||
fmt.Fprintf(os.Stderr, "chats: %s (%s): 0 messages parsed, keeping existing file\n", chatName, convID)
|
||||
continue
|
||||
}
|
||||
|
||||
f, err := os.Create(jsonlPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: create %s: %v\n", jsonlPath, err)
|
||||
continue
|
||||
}
|
||||
|
||||
enc := json.NewEncoder(f)
|
||||
written := 0
|
||||
for i, m := range msgs {
|
||||
text := m.Text
|
||||
if text == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
ts := m.Date
|
||||
if t, err := time.Parse("2006-01-02T15:04:05Z07:00", m.Date); err == nil {
|
||||
ts = t.UTC().Format(time.RFC3339)
|
||||
} else if t, err := time.Parse(time.RFC3339, m.Date); err == nil {
|
||||
ts = t.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
chatMsg := Message{
|
||||
ID: fmt.Sprintf("li_%s_%d", convID, i),
|
||||
Timestamp: ts,
|
||||
From: m.From,
|
||||
Text: text,
|
||||
Platform: "linkedin",
|
||||
}
|
||||
if err := enc.Encode(chatMsg); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: encode: %v\n", err)
|
||||
continue
|
||||
}
|
||||
written++
|
||||
}
|
||||
f.Close()
|
||||
|
||||
fmt.Printf("chats: synced %s (%s) — %d messages\n", chatName, convID, written)
|
||||
|
||||
// Pause between conversations to reduce LinkedIn rate limiting.
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
type linkedInMCPClient struct {
|
||||
cmd *exec.Cmd
|
||||
stdin *bufio.Writer
|
||||
stdout *bufio.Scanner
|
||||
msgID int
|
||||
}
|
||||
|
||||
func newLinkedInMCP(ctx context.Context, userDataDir string) (*linkedInMCPClient, error) {
|
||||
args := []string{
|
||||
"mcp-server-linkedin@latest",
|
||||
"--user-data-dir", userDataDir,
|
||||
"--no-auto-import",
|
||||
"--no-daemon",
|
||||
"--transport", "stdio",
|
||||
"--login-timeout", "10",
|
||||
"--browser-wait", "1",
|
||||
"--browser-idle-timeout", "10",
|
||||
"--log-level", "ERROR",
|
||||
}
|
||||
|
||||
cmd := exec.CommandContext(ctx, "uvx", args...)
|
||||
cmd.Env = os.Environ()
|
||||
|
||||
stdin, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stdin pipe: %w", err)
|
||||
}
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stdout pipe: %w", err)
|
||||
}
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, fmt.Errorf("start: %w", err)
|
||||
}
|
||||
|
||||
c := &linkedInMCPClient{
|
||||
cmd: cmd,
|
||||
stdin: bufio.NewWriter(stdin),
|
||||
stdout: bufio.NewScanner(stdout),
|
||||
msgID: 0,
|
||||
}
|
||||
c.stdout.Buffer(make([]byte, 1<<20), 1<<20)
|
||||
|
||||
if err := c.initialize(ctx); err != nil {
|
||||
c.Close()
|
||||
return nil, fmt.Errorf("init: %w", err)
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
func (c *linkedInMCPClient) nextID() int {
|
||||
c.msgID++
|
||||
return c.msgID
|
||||
}
|
||||
|
||||
func (c *linkedInMCPClient) initialize(ctx context.Context) error {
|
||||
params := map[string]interface{}{
|
||||
"protocolVersion": "2024-11-05",
|
||||
"capabilities": map[string]interface{}{},
|
||||
"clientInfo": map[string]string{
|
||||
"name": "chats-sync",
|
||||
"version": "0.1.0",
|
||||
},
|
||||
}
|
||||
_, err := c.send(ctx, "initialize", params)
|
||||
return err
|
||||
}
|
||||
|
||||
func (c *linkedInMCPClient) send(ctx context.Context, method string, params interface{}) (json.RawMessage, error) {
|
||||
id := c.nextID()
|
||||
req := map[string]interface{}{
|
||||
"jsonrpc": "2.0",
|
||||
"id": id,
|
||||
"method": method,
|
||||
}
|
||||
if params != nil {
|
||||
req["params"] = params
|
||||
}
|
||||
body, err := json.Marshal(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal: %w", err)
|
||||
}
|
||||
|
||||
if _, err := c.stdin.Write(body); err != nil {
|
||||
return nil, fmt.Errorf("write: %w", err)
|
||||
}
|
||||
if err := c.stdin.WriteByte('\n'); err != nil {
|
||||
return nil, fmt.Errorf("newline: %w", err)
|
||||
}
|
||||
if err := c.stdin.Flush(); err != nil {
|
||||
return nil, fmt.Errorf("flush: %w", err)
|
||||
}
|
||||
|
||||
for c.stdout.Scan() {
|
||||
line := c.stdout.Text()
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
var resp struct {
|
||||
JSONRPC string `json:"jsonrpc"`
|
||||
ID int `json:"id"`
|
||||
Result json.RawMessage `json:"result,omitempty"`
|
||||
Error *struct {
|
||||
Code int `json:"code"`
|
||||
Message string `json:"message"`
|
||||
} `json:"error,omitempty"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(line), &resp); err != nil {
|
||||
return nil, fmt.Errorf("unmarshal: %w\nline: %s", err, line[:min(len(line), 500)])
|
||||
}
|
||||
if resp.Error != nil {
|
||||
return nil, fmt.Errorf("rpc error %d: %s", resp.Error.Code, resp.Error.Message)
|
||||
}
|
||||
return resp.Result, nil
|
||||
}
|
||||
return nil, fmt.Errorf("no response: %w", c.stdout.Err())
|
||||
}
|
||||
|
||||
// msgLimitFor returns the per-conversation message cap for a sync.
|
||||
func msgLimitFor(limit int) int {
|
||||
if limit > 0 {
|
||||
return limit
|
||||
}
|
||||
return 100
|
||||
}
|
||||
|
||||
// wedged reports whether a conversation fetch failure means the MCP server
|
||||
// closed its browser and will refuse every later call.
|
||||
func wedged(err error) bool {
|
||||
return strings.Contains(err.Error(), "still has a browser open")
|
||||
}
|
||||
|
||||
// callTool invokes an MCP tool, retrying transient (rate-limit) failures.
|
||||
func (c *linkedInMCPClient) callTool(ctx context.Context, name string, params map[string]interface{}) (json.RawMessage, error) {
|
||||
var lastErr error
|
||||
for attempt := 0; attempt < 3; attempt++ {
|
||||
if attempt > 0 {
|
||||
delay := time.Duration(1<<uint(attempt)) * 5 * time.Second
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(delay):
|
||||
}
|
||||
}
|
||||
result, err := c.send(ctx, "tools/call", map[string]interface{}{
|
||||
"name": name,
|
||||
"arguments": params,
|
||||
})
|
||||
if err == nil {
|
||||
// Tool-level errors surface as a successful RPC with an
|
||||
// isError=true content entry.
|
||||
if hint := toolErrorHint(result); hint != "" {
|
||||
lastErr = fmt.Errorf("%s error: %s", name, hint)
|
||||
if !isTransientLinkedInError(lastErr.Error()) {
|
||||
return nil, lastErr
|
||||
}
|
||||
continue
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
lastErr = err
|
||||
if !isTransientLinkedInError(err.Error()) {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("%s: %w", name, lastErr)
|
||||
}
|
||||
|
||||
// toolErrorHint returns the tool's error text when the result has isError set.
|
||||
func toolErrorHint(result json.RawMessage) string {
|
||||
var toolRes struct {
|
||||
Content []struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
IsError bool `json:"isError"`
|
||||
}
|
||||
if err := json.Unmarshal(result, &toolRes); err != nil || !toolRes.IsError {
|
||||
return ""
|
||||
}
|
||||
if len(toolRes.Content) > 0 {
|
||||
return toolRes.Content[0].Text
|
||||
}
|
||||
return "unknown tool error"
|
||||
}
|
||||
|
||||
// isTransientLinkedInError reports whether a fetch failed due to rate limiting
|
||||
// or a transient server error, which may succeed on retry.
|
||||
func isTransientLinkedInError(msg string) bool {
|
||||
return strings.Contains(msg, "503") || strings.Contains(msg, "429") ||
|
||||
strings.Contains(msg, "ERR_HTTP_RESPONSE_CODE_FAILURE") ||
|
||||
strings.Contains(msg, "ERR_ABORTED") ||
|
||||
strings.Contains(msg, "Error calling tool") ||
|
||||
strings.Contains(msg, "Unexpected error") ||
|
||||
strings.Contains(msg, "still has a browser open")
|
||||
}
|
||||
|
||||
func (c *linkedInMCPClient) GetInbox(ctx context.Context, limit int) ([]lnInboxItem, error) {
|
||||
params := map[string]interface{}{
|
||||
"limit": limit,
|
||||
}
|
||||
result, err := c.callTool(ctx, "get_inbox", params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var toolRes struct {
|
||||
Content []struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
IsError bool `json:"isError"`
|
||||
}
|
||||
if err := json.Unmarshal(result, &toolRes); err != nil {
|
||||
return nil, fmt.Errorf("unmarshal tool: %w", err)
|
||||
}
|
||||
if len(toolRes.Content) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
text := toolRes.Content[0].Text
|
||||
return parseLinkedInInbox(text), nil
|
||||
}
|
||||
|
||||
func (c *linkedInMCPClient) GetConversation(ctx context.Context, username, threadID string, limit int) ([]lnMessage, error) {
|
||||
params := map[string]interface{}{
|
||||
"linkedin_username": username,
|
||||
"thread_id": threadID,
|
||||
"index": limit,
|
||||
}
|
||||
result, err := c.callTool(ctx, "get_conversation", params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var toolRes struct {
|
||||
Content []struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
IsError bool `json:"isError"`
|
||||
}
|
||||
if err := json.Unmarshal(result, &toolRes); err != nil {
|
||||
return nil, fmt.Errorf("unmarshal tool: %w", err)
|
||||
}
|
||||
if len(toolRes.Content) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
text := toolRes.Content[0].Text
|
||||
return parseLinkedInConversation(text), nil
|
||||
}
|
||||
|
||||
func (c *linkedInMCPClient) Close() error {
|
||||
if c.stdin != nil {
|
||||
c.stdin.Flush()
|
||||
}
|
||||
if c.cmd != nil && c.cmd.Process != nil {
|
||||
c.cmd.Process.Kill()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,215 @@
|
||||
package chats
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func readFixture(t *testing.T, name string) string {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(filepath.Join("testdata", name))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(data)
|
||||
}
|
||||
|
||||
// TestParseLinkedInInbox verifies get_inbox parsing against the v4.22 wire
|
||||
// format (testdata/linkedin_inbox.json — synthetic Alice/Bob/Charlie).
|
||||
func TestParseLinkedInInbox(t *testing.T) {
|
||||
text := readFixture(t, "linkedin_inbox.json")
|
||||
items := parseLinkedInInbox(text)
|
||||
if len(items) != 2 {
|
||||
t.Fatalf("expected 2 conversations (empty thread url skipped), got %d", len(items))
|
||||
}
|
||||
|
||||
first := items[0]
|
||||
if first.ThreadID == "" {
|
||||
t.Error("expected thread id extracted from reference url")
|
||||
}
|
||||
if first.Participants != "Alice Example" {
|
||||
t.Errorf("participants=%q, want Alice Example", first.Participants)
|
||||
}
|
||||
if !strings.HasPrefix(first.ThreadID, "2-") {
|
||||
t.Errorf("unexpected thread id format %q", first.ThreadID)
|
||||
}
|
||||
if items[1].Participants != "Charlie Example" {
|
||||
t.Errorf("second participant=%q, want Charlie Example", items[1].Participants)
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseLinkedInInboxBadJSON verifies a non-JSON response yields no items
|
||||
// rather than a panic or error.
|
||||
func TestParseLinkedInInboxBadJSON(t *testing.T) {
|
||||
if got := parseLinkedInInbox("Session expired"); len(got) != 0 {
|
||||
t.Fatalf("expected no items for non-JSON, got %d", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseLinkedInConversation verifies message extraction from the sections
|
||||
// blob (testdata/linkedin_conversation.json — synthetic Alice/Bob).
|
||||
func TestParseLinkedInConversation(t *testing.T) {
|
||||
text := readFixture(t, "linkedin_conversation.json")
|
||||
msgs := parseLinkedInConversation(text)
|
||||
if len(msgs) != 2 {
|
||||
t.Fatalf("expected 2 messages, got %d", len(msgs))
|
||||
}
|
||||
|
||||
if msgs[0].From != "Alice Example" {
|
||||
t.Errorf("from=%q, want Alice Example", msgs[0].From)
|
||||
}
|
||||
if !strings.Contains(msgs[0].Text, "Senior Software Engineer") {
|
||||
t.Errorf("alice text missing role, got %q", msgs[0].Text)
|
||||
}
|
||||
if msgs[0].Date == "" {
|
||||
t.Error("expected message date")
|
||||
}
|
||||
if msgs[1].From != "Bob Example" {
|
||||
t.Errorf("from=%q, want Bob Example", msgs[1].From)
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseLinkedInConversationEmpty verifies empty/non-JSON blobs parse to
|
||||
// zero messages.
|
||||
func TestParseLinkedInConversationEmpty(t *testing.T) {
|
||||
if got := parseLinkedInConversation("no data here"); len(got) != 0 {
|
||||
t.Fatalf("expected 0 messages, got %d", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// TestLinkedInTimestamp verifies weekday+clock resolution to a recent UTC date.
|
||||
func TestLinkedInTimestamp(t *testing.T) {
|
||||
// The most recent Wednesday before/equal to "now".
|
||||
ts := linkedInTimestamp("WEDNESDAY", "10:02 AM")
|
||||
parsed, err := time.Parse(time.RFC3339, ts)
|
||||
if err != nil {
|
||||
t.Fatalf("unparseable timestamp %q: %v", ts, err)
|
||||
}
|
||||
if parsed.Weekday() != time.Wednesday {
|
||||
t.Errorf("expected Wednesday, got %s", parsed.Weekday())
|
||||
}
|
||||
if parsed.Hour() != 10 || parsed.Minute() != 2 {
|
||||
t.Errorf("expected 10:02, got %02d:%02d", parsed.Hour(), parsed.Minute())
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
diff := now.Sub(parsed)
|
||||
if diff < 0 || diff > 7*24*time.Hour {
|
||||
t.Errorf("timestamp %s is not within the last week of %s", parsed, now)
|
||||
}
|
||||
|
||||
if got := linkedInTimestamp("MONDAY", "garbage"); got != "" {
|
||||
t.Errorf("expected empty for bad clock, got %q", got)
|
||||
}
|
||||
if got := linkedInTimestamp("", "1:22 PM"); got != "13:22" {
|
||||
t.Errorf("expected bare 13:22 for missing weekday, got %q", got)
|
||||
}
|
||||
|
||||
// Relative day headers must resolve to full dates, not bare clocks.
|
||||
today := linkedInTimestamp("TODAY", "9:42 AM")
|
||||
yp, err := time.Parse(time.RFC3339, today)
|
||||
if err != nil {
|
||||
t.Fatalf("TODAY unparseable %q: %v", today, err)
|
||||
}
|
||||
if yp.Year() != now.Year() || yp.Month() != now.Month() || yp.Day() != now.Day() {
|
||||
t.Errorf("TODAY expected %v, got %v", now, yp)
|
||||
}
|
||||
yest := linkedInTimestamp("YESTERDAY", "3:00 PM")
|
||||
yp, err = time.Parse(time.RFC3339, yest)
|
||||
if err != nil {
|
||||
t.Fatalf("YESTERDAY unparseable %q: %v", yest, err)
|
||||
}
|
||||
if yp.Day() != now.AddDate(0, 0, -1).Day() {
|
||||
t.Errorf("YESTERDAY expected day %d, got %d", now.AddDate(0, 0, -1).Day(), yp.Day())
|
||||
}
|
||||
|
||||
// MON DD header (e.g. "JUN 25"): must resolve to a full date. The
|
||||
// timestamp should fall within the current year (falling back to the
|
||||
// prior year if the date would be in the future).
|
||||
md := linkedInTimestamp("JUN 25", "10:48 AM")
|
||||
mp, err := time.Parse(time.RFC3339, md)
|
||||
if err != nil {
|
||||
t.Fatalf("MON DD unparseable %q: %v", md, err)
|
||||
}
|
||||
if mp.Year() != now.Year() && mp.Year() != now.Year()-1 {
|
||||
t.Errorf("JUN 25 expected year %d or %d, got %d", now.Year(), now.Year()-1, mp.Year())
|
||||
}
|
||||
if mp.Month() != time.June || mp.Day() != 25 {
|
||||
t.Errorf("JUN 25 expected Jun 25, got %s %d", mp.Month(), mp.Day())
|
||||
}
|
||||
if mp.After(now) {
|
||||
t.Errorf("JUN 25 resolved to the future: %s > %s", mp, now)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTransientLinkedInError verifies rate-limit errors are retryable but
|
||||
// genuine failures are not.
|
||||
func TestTransientLinkedInError(t *testing.T) {
|
||||
retryable := []string{
|
||||
"get_conversation error: Error calling tool 'get_conversation'",
|
||||
"get_conversation error: Unexpected error in get_conversation: net::ERR_HTTP_RESPONSE_CODE_FAILURE",
|
||||
"rpc error 503: rate limited",
|
||||
"rpc error 429: too many requests",
|
||||
"get_conversation: get_conversation error: This server still has a browser open on the profile.",
|
||||
}
|
||||
for _, msg := range retryable {
|
||||
if !isTransientLinkedInError(msg) {
|
||||
t.Errorf("expected %q to be transient", msg)
|
||||
}
|
||||
}
|
||||
permanent := []string{
|
||||
"get_inbox error: bad credentials",
|
||||
"rpc error -32602: Invalid request parameters",
|
||||
"unmarshal: unexpected end of JSON input",
|
||||
}
|
||||
for _, msg := range permanent {
|
||||
if isTransientLinkedInError(msg) {
|
||||
t.Errorf("expected %q to be permanent", msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestMsgLimitFor verifies the per-conversation message cap resolution.
|
||||
func TestMsgLimitFor(t *testing.T) {
|
||||
if got := msgLimitFor(0); got != 100 {
|
||||
t.Errorf("expected default 100, got %d", got)
|
||||
}
|
||||
if got := msgLimitFor(5); got != 5 {
|
||||
t.Errorf("expected 5, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWedged verifies the browser-open failure is recognized as a wedge.
|
||||
func TestWedged(t *testing.T) {
|
||||
if !wedged(errors.New("get_conversation error: This server still has a browser open on the profile")) {
|
||||
t.Error("expected wedged error to be recognized")
|
||||
}
|
||||
if wedged(errors.New("get_conversation error: bad thing")) {
|
||||
t.Error("unexpected wedge detection")
|
||||
}
|
||||
}
|
||||
|
||||
// TestThreadIDFromURL verifies thread id extraction.
|
||||
func TestThreadIDFromURL(t *testing.T) {
|
||||
cases := []struct {
|
||||
url, want string
|
||||
}{
|
||||
{"/messaging/thread/2-abc123/", "2-abc123"},
|
||||
{"/messaging/thread/2-abc123", "2-abc123"},
|
||||
{"", ""},
|
||||
{"/messaging/thread/", ""},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got := threadIDFromURL(c.url)
|
||||
if c.want == "" && validThreadID(got) {
|
||||
t.Errorf("threadIDFromURL(%q) = %q, want empty", c.url, got)
|
||||
}
|
||||
if c.want != "" && got != c.want {
|
||||
t.Errorf("threadIDFromURL(%q) = %q, want %q", c.url, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
@@ -0,0 +1,39 @@
|
||||
package chats
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Root locates the 2dph project root (KB_ROOT, or walk up for var/ or .git).
|
||||
func Root() string {
|
||||
if v := os.Getenv("KB_ROOT"); v != "" {
|
||||
return v
|
||||
}
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
return "."
|
||||
}
|
||||
for i := 0; i < 10; i++ {
|
||||
if _, err := os.Stat(wd + "/var"); err == nil {
|
||||
return wd
|
||||
}
|
||||
if _, err := os.Stat(wd + "/.git"); err == nil {
|
||||
return wd
|
||||
}
|
||||
parent := wd
|
||||
if idx := strings.LastIndex(wd, "/"); idx >= 0 {
|
||||
parent = wd[:idx]
|
||||
}
|
||||
if parent == wd {
|
||||
break
|
||||
}
|
||||
wd = parent
|
||||
}
|
||||
return "."
|
||||
}
|
||||
|
||||
// Dir is var/chats under the project root.
|
||||
func Dir() string {
|
||||
return Root() + "/var/chats"
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -0,0 +1,105 @@
|
||||
package chats
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
func checkLinkedInSession(userDataDir string) (bool, error) {
|
||||
// Validate the source-session files without launching a browser. A full
|
||||
// `--status` run spawns Chromium and loads /feed/, doubling the automation
|
||||
// exposed to LinkedIn (429 rate limits) before the sync even starts.
|
||||
root := filepath.Dir(userDataDir)
|
||||
sessionFiles := []string{
|
||||
filepath.Join(root, "source-state.json"),
|
||||
filepath.Join(root, "cookies.json"),
|
||||
filepath.Join(userDataDir, "Default", "Cookies"),
|
||||
}
|
||||
for _, f := range sessionFiles {
|
||||
if _, err := os.Stat(f); err != nil {
|
||||
return true, fmt.Errorf("missing session file %s", f)
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
func RunSyncLinkedIn(args []string) int {
|
||||
fs := flag.NewFlagSet("chats sync linkedin", flag.ContinueOnError)
|
||||
limit := fs.Int("limit", 0, "max messages per conversation (0 = all)")
|
||||
refresh := fs.Bool("refresh", false, "refresh session from live webtop browser before sync")
|
||||
help := fs.Bool("help", false, "")
|
||||
fs.SetOutput(os.Stderr)
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return 2
|
||||
}
|
||||
if *help {
|
||||
fmt.Fprintln(os.Stderr, "usage: chats sync linkedin [--limit N] [--refresh]")
|
||||
return 0
|
||||
}
|
||||
|
||||
userDataDir := envVar("LINKEDIN_USER_DATA_DIR", "")
|
||||
if userDataDir == "" {
|
||||
home, _ := os.UserHomeDir()
|
||||
userDataDir = home + "/.linkedin-mcp/profile"
|
||||
}
|
||||
|
||||
if *refresh {
|
||||
if code := refreshLinkedInSession(userDataDir); code != 0 {
|
||||
return code
|
||||
}
|
||||
}
|
||||
|
||||
// Check session files first (no browser launch).
|
||||
loginNeeded, err := checkLinkedInSession(userDataDir)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: linkedin status check: %v\n", err)
|
||||
}
|
||||
if loginNeeded {
|
||||
fmt.Fprintf(os.Stderr, "chats: LinkedIn session missing. Run:\n")
|
||||
fmt.Fprintf(os.Stderr, " chats sync linkedin --refresh\n")
|
||||
fmt.Fprintf(os.Stderr, "or point LINKEDIN_USER_DATA_DIR at a valid session\n")
|
||||
return 1
|
||||
}
|
||||
|
||||
src := NewLinkedInMCPSource(userDataDir)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
start := time.Now()
|
||||
if err := src.Sync(ctx, Dir(), *limit); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats sync linkedin: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
fmt.Printf("chats sync linkedin: completed in %s\n", time.Since(start).Round(time.Millisecond))
|
||||
return 0
|
||||
}
|
||||
|
||||
// refreshLinkedInSession re-syncs the LinkedIn source session from the live
|
||||
// webtop browser via the vendored refresh-linkedin-session helper.
|
||||
func refreshLinkedInSession(userDataDir string) int {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: resolve executable: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
helper := filepath.Join(filepath.Dir(exe), "refresh-linkedin-session")
|
||||
if _, err := os.Stat(helper); err != nil {
|
||||
// Fall back to the source tree helper next to this command file.
|
||||
helper = "bin/chats/refresh-linkedin-session"
|
||||
}
|
||||
root := filepath.Dir(userDataDir)
|
||||
cmd := exec.Command(helper, "--root", root)
|
||||
cmd.Stdout = os.Stderr
|
||||
cmd.Stderr = os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats: linkedin session refresh: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package chats
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -11,7 +11,7 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
func runSyncTelegram(args []string) int {
|
||||
func RunSyncTelegram(args []string) int {
|
||||
fs := flag.NewFlagSet("chats sync telegram", flag.ContinueOnError)
|
||||
limit := fs.Int("limit", 0, "max messages per chat (0 = all)")
|
||||
phone := fs.String("phone", "", "phone number (default env TELEGRAM_PHONE)")
|
||||
@@ -78,7 +78,7 @@ func runSyncTelegram(args []string) int {
|
||||
defer cancel()
|
||||
|
||||
start := time.Now()
|
||||
if err := src.Sync(ctx, chatsDir(), *limit); err != nil {
|
||||
if err := src.Sync(ctx, Dir(), *limit); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "chats sync telegram: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"url": "https://www.linkedin.com/messaging/thread/2-YWxpY2UtYm9iLXRocmVhZC0xMjM=/",
|
||||
"sections": {
|
||||
"conversation": "WEDNESDAY\nAlice Example sent the following message at 10:02 AM\nView Alice Example's profile\nAlice Example (She/Her) 10:02 AM\nHi Bob, we have a Senior Software Engineer role that matches your Go and Python background. Happy to share more if you are open to a chat.\n\nBob Example sent the following messages at 1:22 PM\nView Bob Example's profile\nBob Example 1:22 PM\nHi Alice, thanks for reaching out — yes, I am open to exploring a Senior Software Engineer role. Happy to do a short video call.\n"
|
||||
},
|
||||
"references": {
|
||||
"conversation": [
|
||||
{"kind": "person", "url": "/in/alice-example/", "text": "Alice Example"},
|
||||
{"kind": "person", "url": "/in/bob-example/", "text": "Bob Example"}
|
||||
]
|
||||
}
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"url": "https://www.linkedin.com/messaging/",
|
||||
"sections": {
|
||||
"inbox": "Messaging\nInbox\nConversation List\nAlice Example\nExciting opportunity for a senior software engineer\n"
|
||||
},
|
||||
"references": {
|
||||
"inbox": [
|
||||
{
|
||||
"kind": "conversation",
|
||||
"url": "/messaging/thread/2-YWxpY2UtYm9iLXRocmVhZC0xMjM=/",
|
||||
"context": "inbox",
|
||||
"text": "Alice Example"
|
||||
},
|
||||
{
|
||||
"kind": "person",
|
||||
"url": "/in/alice-example/",
|
||||
"text": "Alice Example",
|
||||
"context": "inbox"
|
||||
},
|
||||
{
|
||||
"kind": "conversation",
|
||||
"url": "/messaging/thread/2-Y2hhcmxpZS1ib2ItdGhyZWFkLTQ1Ng==/",
|
||||
"context": "inbox",
|
||||
"text": "Charlie Example"
|
||||
},
|
||||
{
|
||||
"kind": "conversation",
|
||||
"url": "/messaging/thread/",
|
||||
"context": "inbox",
|
||||
"text": "should-be-skipped"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -2,15 +2,10 @@
|
||||
//
|
||||
// Async by design: every request runs on its own goroutine, and CPU-heavy
|
||||
// searches are serialized through a bounded worker pool (a counting
|
||||
// semaphore) so N requests can't spawn N Python interpreters at once.
|
||||
// semaphore) so N requests can't spawn N search processes at once.
|
||||
//
|
||||
// Used by bin/serve.go which is a self-executing shebang script:
|
||||
//
|
||||
// ///usr/bin/env go run "$0" "$@"; exit
|
||||
// package main
|
||||
// import "github.com/eSlider/2dph/bin/server"
|
||||
// func main() { server.Run() }
|
||||
package server
|
||||
// Used by bin/brain/serve.go.
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -100,8 +95,8 @@ func writeRaw(w http.ResponseWriter, code int, body []byte) {
|
||||
w.Write(body)
|
||||
}
|
||||
|
||||
// brainSearcher shells out to bin/kb/search --json. A single python search
|
||||
// is bounded and short-lived; the worker pool keeps at most N live.
|
||||
// brainSearcher shells out to the Go brain-search binary (not Python).
|
||||
// A single search is bounded and short-lived; the worker pool keeps at most N live.
|
||||
type brainSearcher struct {
|
||||
cmdPath string
|
||||
timeout time.Duration
|
||||
@@ -122,15 +117,18 @@ func (b *brainSearcher) Search(ctx context.Context, query string, limit int) ([]
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Run starts the HTTP server. Reads env: KB_SEARCH_CMD (default bin/kb/search,
|
||||
// relative to the repo root given by KB_ROOT), KB_WORKERS (default 4), KB_PORT
|
||||
// (default 8630).
|
||||
func defaultSearchCmd(root string) string {
|
||||
if env := os.Getenv("KB_SEARCH_CMD"); env != "" {
|
||||
return env
|
||||
}
|
||||
return filepath.Join(root, "var", "bin", "brain-search")
|
||||
}
|
||||
|
||||
// Run starts the HTTP server. Reads env: KB_SEARCH_CMD (default
|
||||
// $KB_ROOT/var/bin/brain-search), KB_WORKERS (default 4), KB_PORT (default 8630).
|
||||
func Run() {
|
||||
root := os.Getenv("KB_ROOT")
|
||||
searchPath := os.Getenv("KB_SEARCH_CMD")
|
||||
if searchPath == "" {
|
||||
searchPath = filepath.Join(root, "bin", "kb", "search")
|
||||
}
|
||||
searchPath := defaultSearchCmd(root)
|
||||
workers := 4
|
||||
if raw := os.Getenv("KB_WORKERS"); raw != "" {
|
||||
if n, err := strconv.Atoi(raw); err == nil && n > 0 {
|
||||
@@ -147,7 +145,7 @@ func Run() {
|
||||
searcher := &brainSearcher{cmdPath: searchPath, timeout: 60 * time.Second}
|
||||
handler := NewServer(searcher, workers)
|
||||
addr := "127.0.0.1:" + strconv.Itoa(port)
|
||||
log.Printf("serve: %s (workers=%d)", addr, workers)
|
||||
log.Printf("serve: %s (workers=%d cmd=%s)", addr, workers, searchPath)
|
||||
if err := http.ListenAndServe(addr, handler); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
@@ -1,10 +1,11 @@
|
||||
package server
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
@@ -141,6 +142,17 @@ func TestSearchRejectsBadLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultSearchCmdIsBrainNotPython(t *testing.T) {
|
||||
t.Setenv("KB_SEARCH_CMD", "")
|
||||
cmd := defaultSearchCmd("/repo")
|
||||
if strings.Contains(strings.ToLower(cmd), "python") {
|
||||
t.Fatalf("search path still python: %s", cmd)
|
||||
}
|
||||
if !strings.Contains(cmd, "brain") {
|
||||
t.Fatalf("search path must be the Go brain binary, got %s", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSearchTimeout(t *testing.T) {
|
||||
fs := &fakeSearcher{delay: time.Second}
|
||||
h := NewServer(fs, 1)
|
||||
@@ -30,12 +30,11 @@ related:
|
||||
---
|
||||
```
|
||||
|
||||
`bin/kb/index` reads this. `type` becomes a searchable column and `related`
|
||||
becomes a graph edge:
|
||||
`bin/kb/index` reads this. `type` becomes a searchable column. `related:` is
|
||||
frontmatter for humans; graph hops from it are not implemented yet.
|
||||
|
||||
```bash
|
||||
bin/kb/search "deploy" --type howto
|
||||
bin/kb/search "Stecktafel" --hop 1 # follow links and related
|
||||
bin/brain/search.go "deploy"
|
||||
```
|
||||
|
||||
## Audit checklist
|
||||
@@ -46,8 +45,7 @@ bin/kb/search "Stecktafel" --hop 1 # follow links and related
|
||||
The indexer chunks on H2, so split files also search better.
|
||||
3. Is `status: archive` set on anything superseded? Archived files stay indexed
|
||||
but stop competing with current ones for a reader's attention.
|
||||
4. Does every explanation link the reference it explains, and vice versa? That
|
||||
link is what `--hop 1` walks.
|
||||
4. Does every explanation link the reference it explains, and vice versa?
|
||||
|
||||
## Rule
|
||||
|
||||
|
||||
+13
-12
@@ -2,9 +2,10 @@
|
||||
name: kb-search
|
||||
description: >-
|
||||
Deduction search over the 2dph brain (Ladybug graph: ops corpus, portfolio,
|
||||
ssh hosts) with bin/kb/search instead of reading files or grepping repos.
|
||||
Use whenever a question starts with "where is", "what runs on", "which file
|
||||
describes", "who is", "how is X done", before opening any documentation.
|
||||
ssh hosts) with bin/brain/search.go instead of reading files or grepping
|
||||
repos. Use whenever a question starts with "where is", "what runs on",
|
||||
"which file describes", "who is", "how is X done", before opening any
|
||||
documentation.
|
||||
---
|
||||
|
||||
# kb-search — deduction over facts and info
|
||||
@@ -22,15 +23,17 @@ second independent source when local roots cannot confirm. An answer is
|
||||
`(not confirmed)`.
|
||||
|
||||
```bash
|
||||
bin/kb/search "Matrix federation" # pointers + snippets, YAML
|
||||
bin/kb/search "what runs on arc-2" --hop 1 # follow graph edges
|
||||
bin/kb/search "onlyoffice postgres" --root facts # restrict to confirmed
|
||||
bin/kb/search "where is cs-lexicon" --json | yq '.[].ref'
|
||||
bin/kb/get <id> --body # full chunk only when needed
|
||||
bin/kb/stats # index health
|
||||
bin/kb/eval # recall@5 >= 0.95 gate
|
||||
bin/brain/search.go "Matrix federation" # pointers + snippets, YAML
|
||||
bin/brain/search.go "onlyoffice postgres" --root facts # restrict to confirmed
|
||||
bin/brain/search.go "where is cs-lexicon" --json | yq '.[].ref'
|
||||
bin/kb/get <id> --body # full chunk only when needed
|
||||
bin/kb/stats # index health
|
||||
bin/kb/eval # recall@5 >= 0.95 gate
|
||||
```
|
||||
|
||||
`bin/kb/search` is a deprecated wrapper. `--hop` errors (File/FROM_FILE edges
|
||||
are not wired yet); do not treat it as a graph walk.
|
||||
|
||||
## Rules
|
||||
|
||||
- Search before you read. Never grep a repo for a concept the graph covers.
|
||||
@@ -38,8 +41,6 @@ bin/kb/eval # recall@5 >= 0.95 gate
|
||||
facts first, then info leafs clearly marked `(not confirmed)`.
|
||||
- If recall looks wrong, run `bin/kb/eval`; it gates control questions and
|
||||
should stay at or above 95% recall@5.
|
||||
- `--hop N` follows sibling leaves, owning files, `related:` links and
|
||||
vector-neighbours — that is the deduction walk, not random expansion.
|
||||
- Escalate to `web-search` (the `web-search` skill) as the independent second
|
||||
source when both local roots cannot confirm; never report an unconfirmed
|
||||
single-source local answer as fact.
|
||||
@@ -18,9 +18,10 @@ bin/web/search "postgres partial index" --lang en --fresh year
|
||||
|
||||
## Web or knowledge base
|
||||
|
||||
`bin/kb/search` holds our own facts: the ops stack, portfolio, ssh hosts, the
|
||||
lexicon. Go there first. Reach for `bin/web/search` when the answer is outside
|
||||
our repos: upstream library behaviour, vendor documentation, public standards.
|
||||
`bin/brain/search.go` holds our own facts: the ops stack, portfolio, ssh hosts,
|
||||
the lexicon. Go there first. Reach for `bin/web/search` when the answer is
|
||||
outside our repos: upstream library behaviour, vendor documentation, public
|
||||
standards.
|
||||
|
||||
Keep the two apart. A finding is stronger when the reader can see that one
|
||||
source was ours and one was not.
|
||||
|
||||
Reference in New Issue
Block a user