feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing

- bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via
  API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not
  MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1)
- bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout
  fast path with docling subprocess fallback for the ~5% textless files
- bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding
  ladybug WAL corruption on bulk-insert into indexed DBs; split from import
- bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate
  leaves stale backing tables killing the vector index)
- docs: README/PLAN/AGENTS cover the mail pipeline

Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.
This commit is contained in:
2026-08-11 21:57:38 +01:00
parent 9f22380e82
commit 60c20ed98d
20 changed files with 5000 additions and 26 deletions
+9
View File
@@ -93,6 +93,15 @@ bin/kb/stats # index health
bin/kb/eval # recall@5 gate
```
Mail is a first-class corpus (retrievable through the same search):
```bash
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw sync (Go)
bin/mail/import --from-raw var/mail # JSON → markdown
bin/mail/index_mail # rebuild brain incl. mail
bin/kb/search "Mietwagen Nürnberg invoice" # now answers from mail
```
## Storage
- **LadybugDB** — single `var/kb.lbug`, Cypher property graph, HNSW + BM25