eSlider/2dph - 2dph

58 Commits 24 Branches 29 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
eSliderandGitHub	e28fb9f428	feat(mail): M365 Graph sync + bot ETL loop (OO+Gmail) (#38 ) Tests / Test (push) Failing after 5s Details Tests / OCR (tesseract fixture) (push) Failing after 4s Details Tests / Release (semver) (push) Skipped Details * feat(mail): add Microsoft 365 Graph sync source with delta state Adds a GET-only Microsoft Graph source to bin/mail/sync (client-credentials, delta query). Wires it into the CLI (--source m365), the worker pool and the compose stack: - m365.go: M365Client (token cache, delta pagination, message normalization, attachment download) + m365Source adapter (per-mailbox folders + delta link) - sync.go: M365Credentials in SyncConfig; Committer interface so delta links only advance after a fully successful run (no skips on failure) - cli.go: --source m365 with M365_/MS_ env passthrough - Dockerfile: mail-build stage produces /mail-sync into the index image - docker-entrypoint: mail-sync loop (sync -> import -> index, default 10s) - compose.yaml: mail-sync service (index image, kb-var volume, secrets ro) * feat(mail): adapt M365 ETL loop for OO+Gmail bots. Default compose mail-sync to onlyoffice,gmail every 300s with import on new mail; gate full --rebuild behind MAIL_SYNC_INDEX. Fix mail-build stage order and add bin/stack/start-mail-sync.	2026-08-14 18:02:13 +01:00
eSliderandGitHub	c1ee920b0a	feat: brain/index.go shebang; mail import is not a brain write (D14). (#12 ) Tests / Test (push) Skipped Tests / Release (semver) (push) Skipped Commands live at bin/brain/{index,get,stats,eval,watch}.go and bin/mail/import.go, bin/markdown/import.go, bin/postgres/query.go. Python remains the Ladybug write worker. index_mail is a deprecation shim that rebuilds via --with-mail.	2026-08-13 17:46:25 +01:00
eSlider	60c20ed98d	feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing - bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1) - bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout fast path with docling subprocess fallback for the ~5% textless files - bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding ladybug WAL corruption on bulk-insert into indexed DBs; split from import - bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate leaves stale backing tables killing the vector index) - docs: README/PLAN/AGENTS cover the mail pipeline Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.	2026-08-11 21:57:38 +01:00

Author

SHA1

Message

Date

eSliderandGitHub

e28fb9f428

feat(mail): M365 Graph sync + bot ETL loop (OO+Gmail) (#38 )

Tests / Test (push) Failing after 5s

Details

Tests / OCR (tesseract fixture) (push) Failing after 4s

Details

Tests / Release (semver) (push) Skipped

Details

* feat(mail): add Microsoft 365 Graph sync source with delta state

Adds a GET-only Microsoft Graph source to bin/mail/sync (client-credentials,
delta query). Wires it into the CLI (--source m365), the worker pool and the
compose stack:

- m365.go: M365Client (token cache, delta pagination, message normalization,
  attachment download) + m365Source adapter (per-mailbox folders + delta link)
- sync.go: M365Credentials in SyncConfig; Committer interface so delta links
  only advance after a fully successful run (no skips on failure)
- cli.go: --source m365 with M365_/MS_ env passthrough
- Dockerfile: mail-build stage produces /mail-sync into the index image
- docker-entrypoint: mail-sync loop (sync -> import -> index, default 10s)
- compose.yaml: mail-sync service (index image, kb-var volume, secrets ro)

* feat(mail): adapt M365 ETL loop for OO+Gmail bots.

Default compose mail-sync to onlyoffice,gmail every 300s with import on new
mail; gate full --rebuild behind MAIL_SYNC_INDEX. Fix mail-build stage order
and add bin/stack/start-mail-sync.

2026-08-14 18:02:13 +01:00

eSliderandGitHub

c1ee920b0a

feat: brain/index.go shebang; mail import is not a brain write (D14). (#12 )

Tests / Test (push) Skipped

Tests / Release (semver) (push) Skipped

Commands live at bin/brain/{index,get,stats,eval,watch}.go and
bin/mail/import.go, bin/markdown/import.go, bin/postgres/query.go.
Python remains the Ladybug write worker. index_mail is a deprecation
shim that rebuilds via --with-mail.

2026-08-13 17:46:25 +01:00

eSlider

60c20ed98d

feat(mail): full Gmail+OnlyOffice sync, import, and brain indexing

- bin/mail/sync.go: async Go sync engine (8 workers, paginated Gmail via
  API + OnlyOffice IMAP); Gmail attachments key off body.attachmentId, not
  MIME partId; ICS sidecars Latin-1->UTF-8 normalized (TestICSToMarkdownNormalizesLatin1)
- bin/mail/import: message.json -> markdown; PDFs via pdftotext -layout
  fast path with docling subprocess fallback for the ~5% textless files
- bin/mail/index_mail: fresh-rebuild indexer (repo corpus + mail) avoiding
  ladybug WAL corruption on bulk-insert into indexed DBs; split from import
- bin/kb/index: keep FTS/VECTOR indexes across incremental runs (drop+recreate
  leaves stale backing tables killing the vector index)
- docs: README/PLAN/AGENTS cover the mail pipeline

Result: 17,835 messages -> 28,918 info leafs, FTS+HNSW healthy.

2026-08-11 21:57:38 +01:00