Files

T

eSlider 678a1d1dba feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing

- bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via
  API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not
  MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1)
- bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout
  fast path with docling subprocess fallback for the ~5% textless files
- bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding
  ladybug WAL corruption on bulk-insert into indexed DBs; split from import
- bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate
  leaves stale backing tables killing the vector index)
- docs: README/PLAN/AGENTS cover the mail pipeline

Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.

2026-08-11 21:57:38 +01:00

4.0 KiB

Raw Blame History

AGENTS.md — 2dph (deductionphile)

Evidence-first brain over the ops/eSlider stack. Facts need proof or they are (not confirmed).

Read first: PLAN → docs.

Method (detective, no fork)

≥2 independent sources of evidence, or the finding is (not confirmed). Link the lexicon yaml path that backs each claim.

facts root = assertions backed by ≥2 independent sources (docker ps × compose × ssh-config × docs).
info root = descriptive/narrative leafs, searchable, never asserted as fact.
Search is deduction: facts → info → web-search (second independent source). An answer is confirmed only if it comes off the facts root.

Hard rules

Secrets. ~/.config/brain/, .env, .secrets/ never read into context, printed, or committed. The OnlyOffice password is obtained via the tunnel, never written to this repo.
Read-only data sources. Ladybug var/kb.lbug and Postgres are opened read-only for queries. Index rebuilds write to var/ (gitignored).
PII. brain-test, cs_brain client data is never read or quoted.
No main pushes. Feature branches + PR via gh; CI must be green.
TDD. Failing test before tool code. Unit tests run offline against fixtures; network/db calls are wrapped.
docs reflect behaviour. Any change updates docs/ + PLAN.md status.

Layout

PLAN.md       decisions + execution + open questions
docs/         published docs
skills/       in-project agent skills (vendored, no external links)
bin/          self-describing tools bin/{subject}/{method} (shebang)
bin/serve.go  async Go HTTP server entry (self-executing go run shebang)
bin/watch/    corpus watcher Go package (mtimes, no inotify deps)
bin/server/   async Go HTTP server (goroutines, bounded worker pool)
bin/mail/     mail pipeline: sync (Go), import (md), index_mail (rebuild)
bin/tools/    vendored python libs behind bin/* (kblib, yamlout, websearch)
bin/docker-entrypoint  container entrypoint (brain index|search|serve|watch)
compose.yaml  docker composition (root level, not docker/)
Dockerfile    multi-stage: python deps + static Go binaries
var/          kb.lbug, var/mail/*, caches (gitignored)
.venv/        ladybug + model2vec + mistune

Mail pipeline

bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail   # raw message.json + attachments
bin/mail/import --from-raw var/mail                                     # message.json → message.md (convert only)
bin/mail/index_mail                                                     # rebuild brain incl. all mail (fresh DB)

sync (Go) downloads messages + attachments; Gmail uses paginated list + body.attachmentId (not partId) for attachments.
import converts body + attachments to markdown. PDFs use poppler pdftotext -layout fast path (~15ms); textless/scanned PDFs fall back to docling (isolated subprocess — its native onnx can segfault the parent). Conversion never touches the brain DB (crash safety).
index_mail always rebuilds from scratch (repo corpus + mail). Ladybug corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector indexes exist; a fresh DB with indexes created last is the only safe path. Keep conversion + indexing separate so a conversion crash can't leave the DB mid-transaction.

Tools

bin/facts/audit ["self"|"facts"|"info"|"stale"]   # 2-source + staleness gate
bin/facts/crm [--dry-run]                         # proof person↔company/company↔project (ooCRM × corpus SoT)
bin/kb/search "query" [--hop N] [--repo X]        # deduction search → YAML
bin/md/tables                                     # what the graph holds → YAML
bin/brain/deduce "question"                       # thinking wrapper

Never start a shell command with cd — use the tool working-directory parameter. Search before reading whole files.

Communication

Same tone as the corpus: plain, lists, no hype. Sign-off Andriy Oblivantsev. German C1 where useful. Caveman only for agent chat, never in committed docs.

4.0 KiB Raw Blame History Unescape Escape