feat(mail): M365 Graph sync + bot ETL loop (OO+Gmail) (#38)
* feat(mail): add Microsoft 365 Graph sync source with delta state Adds a GET-only Microsoft Graph source to bin/mail/sync (client-credentials, delta query). Wires it into the CLI (--source m365), the worker pool and the compose stack: - m365.go: M365Client (token cache, delta pagination, message normalization, attachment download) + m365Source adapter (per-mailbox folders + delta link) - sync.go: M365Credentials in SyncConfig; Committer interface so delta links only advance after a fully successful run (no skips on failure) - cli.go: --source m365 with M365_/MS_ env passthrough - Dockerfile: mail-build stage produces /mail-sync into the index image - docker-entrypoint: mail-sync loop (sync -> import -> index, default 10s) - compose.yaml: mail-sync service (index image, kb-var volume, secrets ro) * feat(mail): adapt M365 ETL loop for OO+Gmail bots. Default compose mail-sync to onlyoffice,gmail every 300s with import on new mail; gate full --rebuild behind MAIL_SYNC_INDEX. Fix mail-build stage order and add bin/stack/start-mail-sync.
This commit is contained in:
@@ -86,7 +86,7 @@ detective method: **a fact needs ≥2 independent sources or it is
|
||||
mail/ocr.go tesseract eng+deu (pdftoppm scans)
|
||||
md/import (deprecated; bin/markdown/import.go)
|
||||
brain/extract brain/audit brain/deduce (thinking wrapper)
|
||||
stack/start start-assistant stop status
|
||||
stack/start start-assistant start-mail-sync stop status
|
||||
web/search (deprecated shim → web/search.go)
|
||||
db/psql-yq (vendored)
|
||||
ssh-tunnel onlyoffice pg tunnel 5433
|
||||
@@ -141,8 +141,9 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
|
||||
|
||||
## Mail pipeline (done)
|
||||
|
||||
1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail/OnlyOffice download.
|
||||
Gmail attachments key off `body.attachmentId`, not MIME `partId`.
|
||||
1. `bin/mail/sync.go` (Go, 8 workers) — paginated Gmail / OnlyOffice / M365
|
||||
Graph download. Gmail attachments key off `body.attachmentId`, not MIME
|
||||
`partId`. M365 uses client-credentials + delta link (commit after success).
|
||||
2. `bin/mail/import.go --from-raw` — message.json → message.md; PDFs via
|
||||
`pdftotext -layout` (~15ms); textless/scanned PDFs `pdftoppm` + tesseract
|
||||
`eng+deu`. ICS sidecars
|
||||
@@ -151,7 +152,11 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
|
||||
corrupts its WAL on bulk-insert into an already-indexed DB. Conversion and
|
||||
indexing stay separate for crash safety. `bin/mail/index_mail` is a
|
||||
deprecation shim.
|
||||
4. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable
|
||||
4. Compose `mail-sync` / `bin/stack/start-mail-sync` — ETL loop (default
|
||||
`onlyoffice,gmail`, 300s): sync → import on `new>0`; full rebuild only if
|
||||
`MAIL_SYNC_INDEX=1`. Bot digests (ai-bot) and case wrappers reuse sync/OAuth;
|
||||
they do not replace the corpus path.
|
||||
5. Result: 17,835 messages → 28,918 info leafs, FTS + HNSW healthy, searchable
|
||||
via `bin/brain/search.go`.
|
||||
|
||||
## CI/CD pipeline (D15)
|
||||
|
||||
Reference in New Issue
Block a user