Compare commits

..
Author SHA1 Message Date
eSlider c2e2409d71 feat(httpapi): default search backend is var/bin/brain-search.
Tests / Test (push) Skipped
Tests / Release (semver) (push) Skipped
bin/brain/serve.go is the command; bin/serve.go stays as a tagged
deprecation shim. Tests fail if the default path still names Python.
2026-08-13 14:30:16 +01:00
eSlider 568a6af888 feat(brain): HTTP serve from bin/brain/serve.go, default Go search binary.
Move the HTTP package to internal/httpapi. Default backend is
var/bin/brain-search, not Python. bin/serve.go stays as a deprecation shim.
2026-08-13 14:29:58 +01:00
eSliderandGitHub eeb5b79cf2 refactor: one Go module; brain search in bin/brain + internal/brain. (#8)
Tests / Test (push) Failing after 5s
Tests / Release (semver) (push) Skipped
Collapse nested kbsearch/chats go.mod into the root module. Ranking stays
cgo-free under internal/brain/rank so CI does not need ladybug. bin/kb/search
is a deprecation wrapper that still sets CGO and builds the binary.
2026-08-13 14:26:54 +01:00
eSliderandGitHub 5990feb1f6 docs: point issues at Gitea origin (D15). (#7)
Tests / Test (push) Failing after 4s
Tests / Release (semver) (push) Skipped
GitHub stays the public clone for PRs and Actions. Work board is
https://git.produktor.io/eSlider/2dph/issues.
2026-08-13 14:11:00 +01:00
eSliderandGitHub d27a738fee feat(chats): parse LinkedIn MCP v4.22 inbox/conversation blobs. (#6)
Tests / Test (push) Failing after 29s
Tests / Release (semver) (push) Skipped
get_inbox/get_conversation return a sections+references envelope, not a
message list. Parser is covered by synthetic Alice/Bob fixtures; CI now
runs the nested bin/chats tests. Session check no longer launches Chromium.
2026-08-13 12:25:51 +01:00
eSliderandGitHub 669e184cf6 fix(kbsearch): rank FTS correctly, filter before -n, start the daemon. (#5)
Go search took worst BM25 hits (ORDER BY score), cut to -n before --root,
and never called ensureDaemon. Ranking and flag parsing move to a cgo-free
package so CI can fail those regressions without ladybug. --hop errors
instead of being swallowed into the query.
2026-08-13 12:19:26 +01:00
eSliderandGitHub ebc3f948c1 Add Gmail --query to mail/sync (default in:inbox) (#4)
* Add --query to Gmail mail/sync instead of always listing in:inbox.

Callers keep the search string; default remains in:inbox.

* Document Gmail --query on the mail/sync pipeline.

* test(mail): assert Gmail --query reaches ListIDs, not only the CLI flag.

ParseCLI coverage left a hole: an empty query still has to become in:inbox
and a custom q has to be the string the client lists with.
2026-08-13 12:19:22 +01:00
eSlider fe6a02024c feat(chats): LinkedIn source — MCP client via get_inbox + get_conversation
- LinkedInMCPSource: MCP JSON-RPC, как TelegramMCPSource
- sync linkedin --limit N: выгрузка сообщений из LinkedIn
- Проверка сессии: uvx mcp-server-linkedin --status
- Вывод инструкции если сессия истекла
- JSONL в var/chats/linkedin/<thread_id>/messages.jsonl
2026-08-13 00:10:42 +01:00
eSlider 4a065d9838 docs: add edelweiss to GitHub safety rules 2026-08-13 00:07:01 +01:00
eSlider e3c6ef5684 chore: remove edelweiss references from public repo 2026-08-13 00:06:49 +01:00
eSlider 98c14e23f1 docs: GitHub safety rules — no absolute paths, PII, secrets, curasoft 2026-08-13 00:02:09 +01:00
eSlider ff1716de40 fix: resolve plan.md conflict, remove remaining /mnt/ paths 2026-08-13 00:00:48 +01:00
eSlider fec5325c7a chore: clean absolute paths, curasoft refs, secrets from history
- bin/chats/: env-based paths, no /mnt/ /home/ hardcodes
- bin/edelweiss-pilot: remove curasoft, use DOCS_BASE env var
- bin/facts/crm: use KNOWLEDGE_MESH_SEED env var
- compose.edelweiss.yml: remove curasoft volumes, use DOCS_BASE
- docs/chat-import-plan.md: link to Gitea issue, no secrets
- bin/seed-edelweiss-facts.py: removed (curasoft-only)
2026-08-13 00:00:15 +01:00
eSlider 27d9521e7f bin/chats: Phase 1 MVP — Telegram sync/import/index/facts/apply
- bin/chats/ — nested Go module (как bin/kbsearch/)
  - sync telegram — MCP JSON-RPC клиент, 31 личный чат, 922 сообщения
  - import — конвертация JSONL → MD с YAML frontmatter
  - index — делегирует bin/kb/index --corpus (132 leafs в brain)
  - facts — regex extraction phone/email/linkedin с валидацией
    (исключены: даты, суммы, номера карт, инвойсы)
  - apply — oo CLI cross-check + dry-run
- Source interface для будущих WhatsApp/LinkedIn
- 4 system tests (import, facts, empty, roundtrip) — синтетические данные
- bin/chat — build+exec wrapper
- docs/chat-import-plan.md — прогресс, пути к env (без секретов)

Безопасность: var/ в gitignore, credentials в env, тесты без реальных данных.
2026-08-12 23:59:29 +01:00
eSlider a7cb8d4c76 docs: chat import pipeline plan — link to Gitea issue #1 2026-08-12 23:59:20 +01:00
eSliderandCursor 53cd00284d fix(kb): seed facts before CREATE indexes (FTS MERGE corruption)
Upsert under live FTS raises "document for node offset N is missing".
Add --skip-indexes; edelweiss-pilot index = write → seed → ensure_indexes.
Ship seed-edelweiss-facts.py (paired lexicon/OO/interview/QEMU facts).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:09:02 +01:00
eSliderandCursor b73b4d4f97 fix(kb): stop DROP INDEX killing HNSW via Ladybug ghost catalog
Ladybug 0.19 DROP INDEX leaves `_0_Leaf_vec_UPPER` / `0_id_docs` in catalog so
CREATE fails while SHOW_INDEXES omits the index; create_fts_and_vector used to
swallow that. Never drop FTS/VECTOR; ensure_indexes after upserts; rebuild =
delete kb.lbug. Add compose.edelweiss.yml + regression tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 16:05:14 +01:00
eSlider 1d1f6a90ff Remove curasoft references, rename to detective method
- PLAN.md: replace 'curasoft-detective' with 'detective method'
- README.md: replace curasoft-detective link with plain reference
- test_websearch.py: fix test domain from ticket.curasoft.de to example.com
- Rewrote git history with git-filter-repo to remove all traces
2026-08-12 13:46:54 +01:00
eSlider f220bcd95a kbsearch: Go implementation with daemon model serving
- New nested module bin/kbsearch with Go implementation of bin/kb/search
- Embedding model (potion-multilingual-128M) served by localhost daemon
  so repeated CLI calls reuse the loaded model
- Bash launcher bin/kb/search builds binary on first run, caches to var/bin/
- Hybrid FTS + vector search (RRF k=60) matching Python kblib behavior
- YAML output via port of yamlout.py (ordered keys, same format)
- JSON output with proper field order
- All flags: --root, --repo, -n, --json, --list-model
- Root go.mod reverted to 1.25.0 (kbsearch is isolated nested module)
- CI passes: go test ./... and go vet ./... unaffected by kbsearch
2026-08-11 23:57:39 +01:00
eSlider 678a1d1dba feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing
- bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via
  API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not
  MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1)
- bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout
  fast path with docling subprocess fallback for the ~5% textless files
- bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding
  ladybug WAL corruption on bulk-insert into indexed DBs; split from import
- bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate
  leaves stale backing tables killing the vector index)
- docs: README/PLAN/AGENTS cover the mail pipeline

Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.
2026-08-11 21:57:38 +01:00
eSlider 8781c0c3eb refactor(tools): bin/{subject}/{method} layout; Go serve+watch modules
Move serve/ (module) -> bin/server, tools/ -> bin/tools, replace bin/kb-watch
bash with bin/watch Go package; self-executing Go shebangs bin/serve.go and
bin/kb/watch.go; Docker + CI + git/import + docs repointed. Multi-stage image
builds static serve+watch binaries (no Go runtime in container).
2026-08-11 09:52:20 +01:00
eSlider d6b17e8819 feat(kb): CRM association proof via oo, fix ssh-tunnel self-ref + oo creds
- bin/facts/crm: prove person<->company/company<->project against ooCRM
  x corpus SoT (knowledge-mesh-seed.yaml), write 78 facts (root=facts)
- tools/crmfacts.py + test_crm_facts.py: parser under unit tests (26 pass)
- docs/crm-associations-proof.md: provable graph, mistakes, fixes
- oo merge 759->763 resolves duplicate GoldenRatio.Exchange legal entity
- bin/db/ssh-tunnel: "$0" self-check + accept-new/BatchMode ssh flags
- AGENTS.md: document bin/facts/crm
2026-08-10 23:22:34 +01:00
7 changed files with 100 additions and 36 deletions
+2 -4
View File
@@ -36,11 +36,9 @@ PLAN.md decisions + execution + open questions
docs/ published docs docs/ published docs
skills/ in-project agent skills (vendored, no external links) skills/ in-project agent skills (vendored, no external links)
bin/ self-describing tools bin/{subject}/{method}.go (shebang) bin/ self-describing tools bin/{subject}/{method}.go (shebang)
bin/brain/ search.go (deduction); libs in internal/brain bin/brain/ search.go, serve.go; libs in internal/brain and internal/httpapi
internal/ shared Go (brain/rank is cgo-free) internal/ shared Go (brain/rank is cgo-free)
bin/serve.go async Go HTTP server entry (self-executing go run shebang) bin/watch/ corpus watcher (internal via bin/brain/watch later)
bin/watch/ corpus watcher Go package (mtimes, no inotify deps)
bin/server/ async Go HTTP server (goroutines, bounded worker pool)
bin/mail/ mail pipeline: sync (Go), import (md), index_mail (rebuild) bin/mail/ mail pipeline: sync (Go), import (md), index_mail (rebuild)
bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch) bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch)
bin/docker-entrypoint container entrypoint (brain index|search|serve|watch) bin/docker-entrypoint container entrypoint (brain index|search|serve|watch)
+26
View File
@@ -0,0 +1,26 @@
//usr/bin/env go run -tags=brain_serve "$0" "$@"; exit
//go:build brain_serve
//
// bin/brain/serve.go - HTTP API for the 2dph brain.
//
// KB_ROOT=/path/to/2dph ./bin/brain/serve.go
// KB_SEARCH_CMD=... KB_WORKERS=4 KB_PORT=8630 ./bin/brain/serve.go
//
// Default search backend is var/bin/brain-search (Go), not Python.
// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
package main
import (
"os"
"github.com/eSlider/2dph/internal/httpapi"
)
func main() {
if os.Getenv("KB_ROOT") == "" {
if wd, err := os.Getwd(); err == nil {
os.Setenv("KB_ROOT", wd)
}
}
httpapi.Run()
}
+2
View File
@@ -0,0 +1,2 @@
// Deprecated shebang mains at bin root (serve.go is tagged brain_serve).
package main
+8 -13
View File
@@ -1,27 +1,22 @@
//usr/bin/env go run "$0" "$@"; exit //usr/bin/env go run -tags=brain_serve "$0" "$@"; exit
// bin/serve.go - async Go HTTP server for the 2dph brain (see bin/server). //go:build brain_serve
// //
// KB_ROOT=/path/to/2dph ./bin/serve.go # serve the brain // bin/serve.go — deprecated; use bin/brain/serve.go.
// KB_SEARCH_CMD=... KB_WORKERS=4 KB_PORT=8630 ./bin/serve.go
//
// Shebang trick: the first line is a Go `//` comment; when executed, env runs
// `go run "$0"` so this file doubles as an executable script. The real code
// lives in the importable package (module path, never a relative import).
// NOTE: never run `gofmt -w` on this file - it rewrites `//usr/bin/env` to
// `// usr/...` and breaks the shebang.
package main package main
import ( import (
"fmt"
"os" "os"
"github.com/eSlider/2dph/bin/server" "github.com/eSlider/2dph/internal/httpapi"
) )
func main() { func main() {
if env := os.Getenv("KB_ROOT"); env == "" { fmt.Fprintln(os.Stderr, "bin/serve.go is deprecated; use bin/brain/serve.go")
if os.Getenv("KB_ROOT") == "" {
if wd, err := os.Getwd(); err == nil { if wd, err := os.Getwd(); err == nil {
os.Setenv("KB_ROOT", wd) os.Setenv("KB_ROOT", wd)
} }
} }
server.Run() httpapi.Run()
} }
+33
View File
@@ -0,0 +1,33 @@
# CRM association proof (oo CLI ↔ corpus)
Proven with `oo` (eslider/go-onlyoffice) against the OnlyOffice portal
(`office.produktor.io`). Portal CRM is the SSOT for company ↔ person ↔
project associations; the corpus SoT (`eslider/cv/projects/knowledge-mesh-seed.yaml`)
is the second, independent source. Facts that can be backed by both are
written to the brain under `root=facts` by `bin/facts/crm`.
## What was verified
- Logical counts (portal MySQL): 1300 contacts = 897 persons + 404 companies,
198 projects, 998 deals, 939 project↔contact links.
- Every client company linked to a project has ≥1 person underneath.
- Every person `company_id` resolves to an existing company.
- Corpus org list (9) maps 1:1 onto CRM companies:
ProProdukt SL / produktor.io, Dyvenia, Immowelt AG, WhereGroup,
Keynote SIGOS, D2S/SYSTEMS, GRID, Pack und Cup, Markets Platform.
- 78 person↔company association facts written to the brain
(`how=crm-crosscheck`, `type=association`). Recall@5 in `bin/kb/eval` = 1.0.
## Mistakes found
| # | Mistake | Fix |
|---|---------|-----|
| 1 | Duplicate legal entity `GoldenRatio.Exchange` (contact 759) vs `Golden Ratio Exchange` (763); 3 deals (211, 287, 559) were linked to 759 | `oo contacts merge 759 763` — 763 kept, 759 removed, deal links re-pointed to 763 |
| 2 | `env/`-wide: OnlyOffice creds file used wrong UX (user `eslider`, password with `$2` suffix) making `oo` auth fail | `.env` fixed to `eslider@gmail.com` + clean password; `.env` stays gitignored |
## Gates after fix
- `uv run python -m unittest discover -s bin/tools -t .` → 26 tests OK
- `bin/facts/audit self` + `bin/facts/audit db` → ok
- `bin/kb/eval` → recall@5 = 1.0
- `go test ./...` (bin/server + bin/watch) → ok
@@ -2,15 +2,10 @@
// //
// Async by design: every request runs on its own goroutine, and CPU-heavy // Async by design: every request runs on its own goroutine, and CPU-heavy
// searches are serialized through a bounded worker pool (a counting // searches are serialized through a bounded worker pool (a counting
// semaphore) so N requests can't spawn N Python interpreters at once. // semaphore) so N requests can't spawn N search processes at once.
// //
// Used by bin/serve.go which is a self-executing shebang script: // Used by bin/brain/serve.go.
// package httpapi
// ///usr/bin/env go run "$0" "$@"; exit
// package main
// import "github.com/eSlider/2dph/bin/server"
// func main() { server.Run() }
package server
import ( import (
"context" "context"
@@ -100,8 +95,8 @@ func writeRaw(w http.ResponseWriter, code int, body []byte) {
w.Write(body) w.Write(body)
} }
// brainSearcher shells out to bin/kb/search --json. A single python search // brainSearcher shells out to the Go brain-search binary (not Python).
// is bounded and short-lived; the worker pool keeps at most N live. // A single search is bounded and short-lived; the worker pool keeps at most N live.
type brainSearcher struct { type brainSearcher struct {
cmdPath string cmdPath string
timeout time.Duration timeout time.Duration
@@ -122,15 +117,18 @@ func (b *brainSearcher) Search(ctx context.Context, query string, limit int) ([]
return out, nil return out, nil
} }
// Run starts the HTTP server. Reads env: KB_SEARCH_CMD (default bin/kb/search, func defaultSearchCmd(root string) string {
// relative to the repo root given by KB_ROOT), KB_WORKERS (default 4), KB_PORT if env := os.Getenv("KB_SEARCH_CMD"); env != "" {
// (default 8630). return env
}
return filepath.Join(root, "var", "bin", "brain-search")
}
// Run starts the HTTP server. Reads env: KB_SEARCH_CMD (default
// $KB_ROOT/var/bin/brain-search), KB_WORKERS (default 4), KB_PORT (default 8630).
func Run() { func Run() {
root := os.Getenv("KB_ROOT") root := os.Getenv("KB_ROOT")
searchPath := os.Getenv("KB_SEARCH_CMD") searchPath := defaultSearchCmd(root)
if searchPath == "" {
searchPath = filepath.Join(root, "bin", "kb", "search")
}
workers := 4 workers := 4
if raw := os.Getenv("KB_WORKERS"); raw != "" { if raw := os.Getenv("KB_WORKERS"); raw != "" {
if n, err := strconv.Atoi(raw); err == nil && n > 0 { if n, err := strconv.Atoi(raw); err == nil && n > 0 {
@@ -147,7 +145,7 @@ func Run() {
searcher := &brainSearcher{cmdPath: searchPath, timeout: 60 * time.Second} searcher := &brainSearcher{cmdPath: searchPath, timeout: 60 * time.Second}
handler := NewServer(searcher, workers) handler := NewServer(searcher, workers)
addr := "127.0.0.1:" + strconv.Itoa(port) addr := "127.0.0.1:" + strconv.Itoa(port)
log.Printf("serve: %s (workers=%d)", addr, workers) log.Printf("serve: %s (workers=%d cmd=%s)", addr, workers, searchPath)
if err := http.ListenAndServe(addr, handler); err != nil { if err := http.ListenAndServe(addr, handler); err != nil {
log.Fatal(err) log.Fatal(err)
} }
@@ -1,10 +1,11 @@
package server package httpapi
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
@@ -141,6 +142,17 @@ func TestSearchRejectsBadLimit(t *testing.T) {
} }
} }
func TestDefaultSearchCmdIsBrainNotPython(t *testing.T) {
t.Setenv("KB_SEARCH_CMD", "")
cmd := defaultSearchCmd("/repo")
if strings.Contains(strings.ToLower(cmd), "python") {
t.Fatalf("search path still python: %s", cmd)
}
if !strings.Contains(cmd, "brain") {
t.Fatalf("search path must be the Go brain binary, got %s", cmd)
}
}
func TestSearchTimeout(t *testing.T) { func TestSearchTimeout(t *testing.T) {
fs := &fakeSearcher{delay: time.Second} fs := &fakeSearcher{delay: time.Second}
h := NewServer(fs, 1) h := NewServer(fs, 1)