feat(mail): M365 Graph sync + bot ETL loop (OO+Gmail) (#38)
Tests / Test (push) Failing after 5s
Tests / OCR (tesseract fixture) (push) Failing after 4s
Tests / Release (semver) (push) Skipped

* feat(mail): add Microsoft 365 Graph sync source with delta state

Adds a GET-only Microsoft Graph source to bin/mail/sync (client-credentials,
delta query). Wires it into the CLI (--source m365), the worker pool and the
compose stack:

- m365.go: M365Client (token cache, delta pagination, message normalization,
  attachment download) + m365Source adapter (per-mailbox folders + delta link)
- sync.go: M365Credentials in SyncConfig; Committer interface so delta links
  only advance after a fully successful run (no skips on failure)
- cli.go: --source m365 with M365_/MS_ env passthrough
- Dockerfile: mail-build stage produces /mail-sync into the index image
- docker-entrypoint: mail-sync loop (sync -> import -> index, default 10s)
- compose.yaml: mail-sync service (index image, kb-var volume, secrets ro)

* feat(mail): adapt M365 ETL loop for OO+Gmail bots.

Default compose mail-sync to onlyoffice,gmail every 300s with import on new
mail; gate full --rebuild behind MAIL_SYNC_INDEX. Fix mail-build stage order
and add bin/stack/start-mail-sync.
This commit is contained in:
2026-08-14 18:02:13 +01:00
committed by GitHub
co-authored by GitHub
parent fc2723c39f
commit e28fb9f428
17 changed files with 788 additions and 19 deletions
+9 -1
View File
@@ -66,12 +66,20 @@ var/ kb.lbug, var/mail/*, caches (gitignored)
```bash
bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail # raw message.json + attachments
bin/mail/sync.go --source gmail --query 'from:example.com' --out var/mail # Gmail search (default in:inbox)
bin/mail/sync.go --source m365 --env ~/.config/brain/mail.env --out var/mail # Microsoft Graph (delta)
bin/mail/import.go --from-raw var/mail # message.json → message.md (convert only)
bin/brain/index.go --rebuild --with-facts --with-chats
bin/stack/start-mail-sync # compose ETL: sync→import every 300s
```
- `sync` (Go) downloads messages + attachments; Gmail uses paginated list +
`body.attachmentId` (not partId) for attachments.
`body.attachmentId` (not partId) for attachments. Sources: `onlyoffice`,
`gmail`, `m365` (client-credentials + delta link; commit after success).
- Compose `mail-sync` / `bin/stack/start-mail-sync`: ETL loop (default
`onlyoffice,gmail`, 300s). On `new>0` runs import; full `--rebuild` only if
`MAIL_SYNC_INDEX=1`. Secrets: `~/.config/brain/mail.env` + `~/.gmail-mcp`.
Case wrappers (e.g. family `gmail-sync-la-quinta.sh`) and ai-bot
`gmail-reauth.sh` reuse this sync/OAuth — do not fork corpus download.
- `import` converts body + attachments to markdown. PDFs use poppler
`pdftotext -layout` fast path (~15ms); textless/scanned PDFs use
`pdftoppm` + tesseract `eng+deu` (`bin/mail/ocr.go`). Optional