Compare commits

..
1 Commits
Author SHA1 Message Date
mdx-1 ae776aa396 feat(mailsync): FetchMailFolder — integration-layer walk for ETL consumers
Release Please / Release Please (push) Skipped
Release / GoReleaser (push) Skipped
Tests / Test (Go 1.25) (push) Skipped
Tests / Test (Go stable) (push) Skipped
Tests / Test (Go 1.25) (pull_request) Successful in 41s
Tests / Test (Go stable) (pull_request) Successful in 1m32s
Adds the high-level mail folder walk that sync pipelines need on top of
the raw mail API (list -> get -> download-attachment), so consumers stop
re-implementing it against private client copies.

  type MailSyncMessage struct { ID, Folder, Subject, From, Date, IsNew,
                                HasAttachments, Attachments }
  type MailSyncAttachment struct { ID, Name, Size, Body }
  func (c *Client) FetchMailFolder(ctx, folderID, MailSyncOptions)
                                   ([]MailSyncMessage, error)

Options: Limit / StartIndex for checkpointed walks, FetchBodies to
eagerly download attachment bytes via download.ashx (session-cookie path).

Hydration details:
- list items may omit the attachment array; when hasAttachments is set
  the full record is fetched and its attachments merged
- attachment ids accepted from id/fileId/attachmentId variants
- timestamps parsed from RFC3339 (any fractional digits) and
  second-precision forms

This is the first step of the 2dph integration layer (#1): the brain's
mail-ingest pipeline can now drop its private OOClient copy and consume
this canonical walk directly.

Tests: httptest-backed coverage for pagination, hydration with
full-record fallback, body download incl. auth-cookie requirement,
Limit/StartIndex windows, timestamp parsing.
2026-08-22 14:35:57 +00:00
31 changed files with 332 additions and 3159 deletions
-45
View File
@@ -10,51 +10,6 @@ permissions:
contents: read
jobs:
secret-scan:
name: Secret scan (gitleaks)
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Compute scan range (diff of new commits only)
id: range
run: |
if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
RANGE="${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}"
else
BEFORE="${{ github.event.before }}"
if [ "$BEFORE" = "0000000000000000000000000000000000000000" ]; then
RANGE="$(git rev-list --max-parents=0 HEAD | tail -1)..$GITHUB_SHA"
else
RANGE="$BEFORE..$GITHUB_SHA"
fi
fi
echo "RANGE=$RANGE" >> "$GITHUB_ENV"
echo "Scanning range: $RANGE"
# Install the gitleaks binary instead of a docker action: the
# docker://zricethezav/gitleaks action hardcodes /github/workspace,
# which does not exist on the Gitea (act) runner. $GITHUB_WORKSPACE is
# the checkout dir on BOTH runners (GitHub and Gitea act). Mirrors the
# fix applied to 2dph (issue #142).
- name: Gitleaks (diff-only, fail on leak)
env:
GITLEAKS_RANGE: ${{ env.RANGE }}
run: |
set -euo pipefail
curl -fsSLo /tmp/gitleaks.tar.gz \
https://github.com/gitleaks/gitleaks/releases/download/v8.30.1/gitleaks_8.30.1_linux_x64.tar.gz
tar -xzf /tmp/gitleaks.tar.gz -C /tmp gitleaks
chmod +x /tmp/gitleaks
/tmp/gitleaks detect \
--source "$GITHUB_WORKSPACE" \
--log-opts="$GITLEAKS_RANGE" \
--redact \
--verbose
test:
name: Test (Go ${{ matrix.go }})
runs-on: ubuntu-latest
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "0.15.0"
".": "0.11.0"
}
-1
View File
@@ -28,7 +28,6 @@ Canonical Go client for OnlyOffice Workspace (Projects + Calendar + CRM) and the
- Prefer `ResponseObject` / `postFormObject` / `putFormObject` / `deleteObject` over hand-rolled `json.Unmarshal(responseField(...))` blocks — they exist for DRY, use them.
- Domain split is by file, **not** by subpackage. Don't introduce `internal/` or `pkg/*` subpackages inside the library — it flattens the `*Client` call surface for a reason.
- CLI commands follow **subject → verb** structure (`oo <subject> <verb>`), never `oo <verb>-<subject>`. Add new commands to the existing subject file if one fits; create a new `cmd/oo/<subject>.go` for a genuinely new domain.
- **Documents for agents:** prefer Markdown in git; OnlyOffice UI is weak for `.md`/`.txt`. Use `oo docs put-md` (md→docx) and `oo docs put-txt` (txt→docx, preserves line breaks). `oo projects files dedupe PROJECT_ID` reports/removes duplicate stem|ext copies (`--apply`, `--cross`).
- Every table output goes through `printTable(headers, rows)`; every single-object through `printObject(v)`. Do not `fmt.Println` rows ad-hoc or the `--output json` flag breaks for that command.
- No secrets in the repo; use `.env` (gitignored). Commit `.env.example` only.
- Follow SemVer on tags; this repo is tagged at GitHub under `git@github.com:eSlider/go-onlyoffice.git`.
-95
View File
@@ -16,101 +16,6 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
* Document that invoice→deal must be set at create (`update --opportunity` often HTTP 400)
## [0.15.0](https://github.com/eSlider/go-onlyoffice/compare/v0.14.0...v0.15.0) (2026-08-29)
### Features
* **files:** `ListFolder`, `CreateFolder`, `MoveFiles`, `UploadToFolder` — Documents folder helpers for OO Documents ingestion ([6ea2fba](https://github.com/eSlider/go-onlyoffice/commit/6ea2fba))
* **files:** dedupe by stem|ext (`files_stem.go`) + `oo projects files dedupe` ([ac06d86](https://github.com/eSlider/go-onlyoffice/commit/ac06d86))
* **docs:** `internal/docpipe` — md↔docx convert, OCR→PDF, hOCR→Markdown via go-hocr, as-md/put-md for agents ([6ea2fba](https://github.com/eSlider/go-onlyoffice/commit/6ea2fba), [a264cbd](https://github.com/eSlider/go-onlyoffice/commit/a264cbd))
* **docs:** optimize PDF via Ghostscript pdfwrite ([6b3f40e](https://github.com/eSlider/go-onlyoffice/commit/6b3f40e))
* **security:** gitleaks secret-scan in CI + pre-push/pre-commit hooks ([9ead554](https://github.com/eSlider/go-onlyoffice/commit/9ead554))
### Fixes
* **files:** `DeleteFiles` via per-file DELETE API; `DeleteDavItems` with `Immediately` true ([622dcdb](https://github.com/eSlider/go-onlyoffice/commit/622dcdb), [d9a7adc](https://github.com/eSlider/go-onlyoffice/commit/d9a7adc))
* **docs:** put-txt preserves line breaks in DOCX; fixed-width extracts in code block; put-md upsert by stem ([309ae44](https://github.com/eSlider/go-onlyoffice/commit/309ae44), [f1739dc](https://github.com/eSlider/go-onlyoffice/commit/f1739dc), [50bd475](https://github.com/eSlider/go-onlyoffice/commit/50bd475))
* **ci:** point gitleaks at `/github/workspace` in docker action ([2c0df0d](https://github.com/eSlider/go-onlyoffice/commit/2c0df0d))
## [0.14.0](https://github.com/eSlider/go-onlyoffice/compare/v0.13.0...v0.14.0) (2026-08-27)
### Features
* **crm:** contact email & person-opportunity indexes, history entity whitelist ([9c450a0](https://github.com/eSlider/go-onlyoffice/commit/9c450a0352c25f12ac16d91062744ab026ffe660))
* **docs:** hOCR → Markdown via go-hocr ([e531de2](https://github.com/eSlider/go-onlyoffice/commit/e531de280f2c521a7411f75212803649b177db58))
* **docs:** md↔docx convert, OCR→PDF, as-md/put-md for agents ([6ea2fba](https://github.com/eSlider/go-onlyoffice/commit/6ea2fbadf768d7d85f6c7a49a9bc1050e642bc56))
* **docs:** md↔docx, OCR→PDF, as-md/put-md ([db9ef12](https://github.com/eSlider/go-onlyoffice/commit/db9ef12ff4096f8aadb557ca128b6edcb503028c))
* **docs:** oo docs hocr — tesseract hOCR → go-hocr Markdown/YAML ([a264cbd](https://github.com/eSlider/go-onlyoffice/commit/a264cbd61c12be9098c8d16e7c1c1253eb460816))
* **mail:** oo mails send — SendMail + guard empty-by-id send ([dda2bd3](https://github.com/eSlider/go-onlyoffice/commit/dda2bd3ca5cea6ebb89e1ac02b785b9838727bda))
* **mail:** oo mails send — SendMail client + guard empty-by-id ([ab7dfbd](https://github.com/eSlider/go-onlyoffice/commit/ab7dfbd8594de5724286e2b460e789f4acb114bc))
* **security:** secret-scan via gitleaks in CI + pre-push/pre-commit hooks ([#142](https://github.com/eSlider/go-onlyoffice/issues/142)) ([9ead554](https://github.com/eSlider/go-onlyoffice/commit/9ead554f5cc229687d3a27934a023fb1dddaef15))
### Bug Fixes
* **ci:** point gitleaks at /github/workspace in docker action ([2c0df0d](https://github.com/eSlider/go-onlyoffice/commit/2c0df0d55fb03555b60481695554d75dceffce48))
* **docs:** put-md upsert — no duplicate folder files ([2b267d3](https://github.com/eSlider/go-onlyoffice/commit/2b267d36a8086b14a94706468f7e92e9179e62d4))
* **docs:** put-md upsert by stem to avoid duplicate folder files ([50bd475](https://github.com/eSlider/go-onlyoffice/commit/50bd47570b6430cebde43641a7b1d2f8d54d32f2))
* **files:** DeleteFiles actually removes files on produktor OO ([a8cb4e9](https://github.com/eSlider/go-onlyoffice/commit/a8cb4e97805226e8af694ec6a64ee3ac6a5e9fdd))
* **files:** DeleteFiles via per-file DELETE API ([622dcdb](https://github.com/eSlider/go-onlyoffice/commit/622dcdb7bffd49acbea5ef07f5d5d009f44b1358))
### Documentation
* **readme:** document oo docs hocr ([70e605b](https://github.com/eSlider/go-onlyoffice/commit/70e605b4ca8ee005e4351b2fc2d2eec081604998))
* **readme:** document oo docs md↔docx / OCR agent workflow ([239c5ea](https://github.com/eSlider/go-onlyoffice/commit/239c5ea67201e62de6ba067f0e7963c7c9bda188))
## [0.13.0](https://github.com/eSlider/go-onlyoffice/compare/v0.12.0...v0.13.0) (2026-08-27)
### Features
* **crm:** contact email & person-opportunity indexes, history entity whitelist ([9c450a0](https://github.com/eSlider/go-onlyoffice/commit/9c450a0352c25f12ac16d91062744ab026ffe660))
* **docs:** hOCR → Markdown via go-hocr ([e531de2](https://github.com/eSlider/go-onlyoffice/commit/e531de280f2c521a7411f75212803649b177db58))
* **docs:** md↔docx convert, OCR→PDF, as-md/put-md for agents ([6ea2fba](https://github.com/eSlider/go-onlyoffice/commit/6ea2fbadf768d7d85f6c7a49a9bc1050e642bc56))
* **docs:** md↔docx, OCR→PDF, as-md/put-md ([db9ef12](https://github.com/eSlider/go-onlyoffice/commit/db9ef12ff4096f8aadb557ca128b6edcb503028c))
* **docs:** oo docs hocr — tesseract hOCR → go-hocr Markdown/YAML ([a264cbd](https://github.com/eSlider/go-onlyoffice/commit/a264cbd61c12be9098c8d16e7c1c1253eb460816))
* **mail:** oo mails send — SendMail + guard empty-by-id send ([dda2bd3](https://github.com/eSlider/go-onlyoffice/commit/dda2bd3ca5cea6ebb89e1ac02b785b9838727bda))
* **mail:** oo mails send — SendMail client + guard empty-by-id ([ab7dfbd](https://github.com/eSlider/go-onlyoffice/commit/ab7dfbd8594de5724286e2b460e789f4acb114bc))
* **security:** secret-scan via gitleaks in CI + pre-push/pre-commit hooks ([#142](https://github.com/eSlider/go-onlyoffice/issues/142)) ([9ead554](https://github.com/eSlider/go-onlyoffice/commit/9ead554f5cc229687d3a27934a023fb1dddaef15))
### Bug Fixes
* **ci:** point gitleaks at /github/workspace in docker action ([2c0df0d](https://github.com/eSlider/go-onlyoffice/commit/2c0df0d55fb03555b60481695554d75dceffce48))
* **files:** rewrite viewUrl host to API base on download ([ecf34ba](https://github.com/eSlider/go-onlyoffice/commit/ecf34ba51aae77f1626a60795f7329f25d9344e5))
### Documentation
* **readme:** document oo docs hocr ([70e605b](https://github.com/eSlider/go-onlyoffice/commit/70e605b4ca8ee005e4351b2fc2d2eec081604998))
* **readme:** document oo docs md↔docx / OCR agent workflow ([239c5ea](https://github.com/eSlider/go-onlyoffice/commit/239c5ea67201e62de6ba067f0e7963c7c9bda188))
## [0.12.0](https://github.com/eSlider/go-onlyoffice/compare/v0.11.0...v0.12.0) (2026-08-27)
### Features
* **crm:** contact email & person-opportunity indexes, history entity whitelist ([9c450a0](https://github.com/eSlider/go-onlyoffice/commit/9c450a0352c25f12ac16d91062744ab026ffe660))
* **docs:** md↔docx convert, OCR→PDF, as-md/put-md for agents ([6ea2fba](https://github.com/eSlider/go-onlyoffice/commit/6ea2fbadf768d7d85f6c7a49a9bc1050e642bc56))
* **docs:** md↔docx, OCR→PDF, as-md/put-md ([db9ef12](https://github.com/eSlider/go-onlyoffice/commit/db9ef12ff4096f8aadb557ca128b6edcb503028c))
* **files:** add WebDAV-oriented Files operations ([ebbd5d5](https://github.com/eSlider/go-onlyoffice/commit/ebbd5d5373abfeca5f160312f201cbd683f55e42))
* **mail:** oo mails send — SendMail + guard empty-by-id send ([dda2bd3](https://github.com/eSlider/go-onlyoffice/commit/dda2bd3ca5cea6ebb89e1ac02b785b9838727bda))
* **mail:** oo mails send — SendMail client + guard empty-by-id ([ab7dfbd](https://github.com/eSlider/go-onlyoffice/commit/ab7dfbd8594de5724286e2b460e789f4acb114bc))
* **security:** secret-scan via gitleaks in CI + pre-push/pre-commit hooks ([#142](https://github.com/eSlider/go-onlyoffice/issues/142)) ([9ead554](https://github.com/eSlider/go-onlyoffice/commit/9ead554f5cc229687d3a27934a023fb1dddaef15))
### Bug Fixes
* **ci:** point gitleaks at /github/workspace in docker action ([2c0df0d](https://github.com/eSlider/go-onlyoffice/commit/2c0df0d55fb03555b60481695554d75dceffce48))
* **files:** rewrite viewUrl host to API base on download ([ecf34ba](https://github.com/eSlider/go-onlyoffice/commit/ecf34ba51aae77f1626a60795f7329f25d9344e5))
### Documentation
* **readme:** document oo docs md↔docx / OCR agent workflow ([239c5ea](https://github.com/eSlider/go-onlyoffice/commit/239c5ea67201e62de6ba067f0e7963c7c9bda188))
## [0.11.0](https://github.com/eSlider/go-onlyoffice/compare/v0.10.0...v0.11.0) (2026-08-18)
+3 -18
View File
@@ -607,25 +607,10 @@ go test -tags=integration ./cmd/office/fetch/... ./cmd/office/preview/...
```bash
# Project Documents (files module)
oo projects files list 33
oo projects files upload 33 ./notes.docx
oo projects files download 12345 --to ./copy.docx
oo projects files rename 12345 notes-v2.docx
oo projects files upload 33 ./notes.md
oo projects files download 12345 --to ./copy.md
oo projects files rename 12345 notes-v2.md
oo projects files delete 12345
oo projects files dedupe 7 # dry-run duplicate report
oo projects files dedupe 7 --apply # remove older stem|ext copies per folder
oo projects files dedupe 7 --cross --apply # cross-folder; keep non-_trash
# Agent document pipeline (md in git ↔ docx in OO; OCR scans)
oo docs tools
oo docs convert ./note.md # → note.docx
oo docs convert ./note.docx # → note.md
oo docs ocr ./scan.jpg --md ./scan.md # searchable PDF + markdown
oo docs hocr ./scan.jpg --lang spa --md ./scan.hocr.md --yaml ./scan.yml
oo docs put-md 7 ./OO-HONDA-7-INDEX.md --folder 490
oo docs as-md 2815 --to ./parte.md # download OO file as MD (OCR if needed)
oo docs as-md 307 --hocr --lang spa # OO download via go-hocr structure
oo projects files put-md 7 ./note.md # alias
oo projects files as-md 2815 # alias
oo tasks files list 208
oo tasks files upload 208 ./notes.pdf
oo tasks files detach 208 12345
-542
View File
@@ -1,542 +0,0 @@
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
onlyoffice "github.com/eslider/go-onlyoffice"
"github.com/eslider/go-onlyoffice/internal/docpipe"
"github.com/spf13/cobra"
)
func init() {
rootCmd.AddCommand(docsCmd())
}
func docsCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "docs",
Short: "Local document pipeline: md↔docx, OCR→PDF, extract Markdown",
Long: `Agent-friendly conversions (requires pandoc / ocrmypdf / pdftotext on PATH).
OnlyOffice Documents UI is poor for .md/.txt — keep sources in git, store .docx in OO.
Upload Markdown as DOCX: oo docs put-md PROJECT_ID file.md
Upload plain text: oo docs put-txt PROJECT_ID file.txt (preserves line breaks)
Read an OO file as MD: oo docs as-md FILE_ID
OCR a scan locally: oo docs ocr scan.pdf --md out.md
Structured OCR (hOCR→MD): oo docs hocr scan.jpg --md out.md --yaml out.yml`,
}
cmd.AddCommand(docsConvertCmd())
cmd.AddCommand(docsOptimizeCmd())
cmd.AddCommand(docsOCRCmd())
cmd.AddCommand(docsHOCRCmd())
cmd.AddCommand(docsAsMDCmd())
cmd.AddCommand(docsPutMDCmd())
cmd.AddCommand(docsPutTxtCmd())
cmd.AddCommand(docsToolsCmd())
return cmd
}
func docsToolsCmd() *cobra.Command {
return &cobra.Command{
Use: "tools",
Short: "Show which converter binaries are on PATH",
RunE: func(cmd *cobra.Command, args []string) error {
t := docpipe.LookPath()
printObject(map[string]any{
"pandoc": strOrNil(t.Pandoc),
"ocrmypdf": strOrNil(t.OCRMyPDF),
"pdftotext": strOrNil(t.PDFToText),
"tesseract": strOrNil(t.Tesseract),
"ghostscript": strOrNil(t.Ghostscript),
})
return nil
},
}
}
func strOrNil(s string) any {
if s == "" {
return nil
}
return s
}
func docsConvertCmd() *cobra.Command {
var to string
cmd := &cobra.Command{
Use: "convert PATH",
Short: "Convert a local file with pandoc (md↔docx by default)",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
in := args[0]
t := docpipe.LookPath()
out := to
if out == "" {
switch docpipe.Ext(in) {
case ".md", ".markdown":
out = docpipe.SiblingDOCX(in)
case ".docx":
out = strings.TrimSuffix(in, docpipe.Ext(in)) + ".md"
default:
return fmt.Errorf("--to required for input type %s", docpipe.Ext(in))
}
}
if err := docpipe.EnsureDir(out); err != nil {
return err
}
if err := t.ConvertFile(in, out); err != nil {
return err
}
printObject(map[string]any{"in": in, "out": out})
return nil
},
}
cmd.Flags().StringVar(&to, "to", "", "output path (default: sibling .docx or .md)")
return cmd
}
func docsOptimizeCmd() *cobra.Command {
var out string
cmd := &cobra.Command{
Use: "optimize PDF_PATH",
Short: "Rewrite PDF via Ghostscript (PostScript pdfwrite) for OO-friendly size/text",
Long: `Use for InDesign/iText PDFs with a good text layer — avoids ocrmypdf invisible
text overlays that break OnlyOffice DocEditor. Skips OCR; rewrites via gs pdfwrite.`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
in := args[0]
t := docpipe.LookPath()
if out == "" {
base := strings.TrimSuffix(filepath.Base(in), filepath.Ext(in))
out = filepath.Join(filepath.Dir(in), base+".optimized.pdf")
}
if err := docpipe.EnsureDir(out); err != nil {
return err
}
chars, _ := t.PDFTextLayerChars(in)
if err := t.OptimizePDF(in, out); err != nil {
return err
}
outChars, _ := t.PDFTextLayerChars(out)
printObject(map[string]any{
"in": in,
"pdf": out,
"text_chars_in": chars,
"text_chars_out": outChars,
"note": "native text layer preserved; no OCR overlay",
})
return nil
},
}
cmd.Flags().StringVar(&out, "out", "", "output PDF (default: <name>.optimized.pdf)")
return cmd
}
func docsOCRCmd() *cobra.Command {
var out, mdOut, lang string
var force bool
var writeMD bool
cmd := &cobra.Command{
Use: "ocr PATH",
Short: "OCR image/PDF → searchable PDF (and optional Markdown)",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
in := args[0]
t := docpipe.LookPath()
if out == "" {
base := strings.TrimSuffix(filepath.Base(in), filepath.Ext(in))
out = filepath.Join(filepath.Dir(in), base+".ocr.pdf")
}
if err := docpipe.EnsureDir(out); err != nil {
return err
}
if err := t.OCRToPDF(in, out, force, lang); err != nil {
return err
}
res := map[string]any{"in": in, "pdf": out}
if writeMD || mdOut != "" {
if mdOut == "" {
mdOut = strings.TrimSuffix(out, filepath.Ext(out)) + ".md"
}
text, err := t.ExtractPDFText(out)
if err != nil {
return err
}
body := "# " + filepath.Base(in) + "\n\n" + strings.TrimSpace(text) + "\n"
if err := os.WriteFile(mdOut, []byte(body), 0o644); err != nil {
return err
}
res["md"] = mdOut
}
printObject(res)
return nil
},
}
cmd.Flags().StringVar(&out, "out", "", "output searchable PDF (default: <name>.ocr.pdf)")
cmd.Flags().StringVar(&mdOut, "md", "", "write Markdown extraction to this path")
cmd.Flags().BoolVar(&writeMD, "markdown", false, "also write sibling .md next to OCR PDF")
cmd.Flags().StringVar(&lang, "lang", "eng", "OCR language(s) for tesseract/ocrmypdf")
cmd.Flags().BoolVar(&force, "force", false, "force OCR even if a text layer exists (default: skip when pdftotext finds enough text)")
return cmd
}
func docsHOCRCmd() *cobra.Command {
var mdOut, yamlOut, hocrOut, lang string
var dpi int
var minConf float32
cmd := &cobra.Command{
Use: "hocr PATH",
Short: "Tesseract hOCR → structured Markdown/YAML (via go-hocr)",
Long: `Runs tesseract with hOCR output, parses with go-hocr, writes Markdown
(and optional YAML). Better reading order / confidence than plain pdftotext.
For Spanish scans: --lang spa or spa+eng (needs tesseract-ocr-spa / TESSDATA_PREFIX).
Phone photos: --dpi 200..300.`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
in := args[0]
t := docpipe.LookPath()
dir, err := os.MkdirTemp("", "oo-docs-hocr-*")
if err != nil {
return err
}
defer os.RemoveAll(dir)
res, err := t.ToHOCRMarkdown(in, dir, lang, dpi, minConf)
if err != nil {
return err
}
if mdOut == "" {
base := strings.TrimSuffix(filepath.Base(in), filepath.Ext(in))
mdOut = filepath.Join(filepath.Dir(in), base+".hocr.md")
}
if err := docpipe.EnsureDir(mdOut); err != nil {
return err
}
if err := os.WriteFile(mdOut, []byte(res.Markdown), 0o644); err != nil {
return err
}
obj := map[string]any{
"in": in,
"md": mdOut,
"did_ocr": res.DidOCR,
}
if hocrOut != "" {
if err := docpipe.EnsureDir(hocrOut); err != nil {
return err
}
b, err := os.ReadFile(res.HOCRPath)
if err != nil {
return err
}
if err := os.WriteFile(hocrOut, b, 0o644); err != nil {
return err
}
obj["hocr"] = hocrOut
} else {
obj["hocr_tmp"] = res.HOCRPath
}
if yamlOut != "" {
if err := docpipe.EnsureDir(yamlOut); err != nil {
return err
}
if err := os.WriteFile(yamlOut, []byte(res.YAML), 0o644); err != nil {
return err
}
obj["yaml"] = yamlOut
}
printObject(obj)
return nil
},
}
cmd.Flags().StringVar(&mdOut, "md", "", "Markdown output (default: <name>.hocr.md)")
cmd.Flags().StringVar(&yamlOut, "yaml", "", "also write structured YAML from go-hocr")
cmd.Flags().StringVar(&hocrOut, "hocr", "", "also keep raw .hocr file at this path")
cmd.Flags().StringVar(&lang, "lang", "eng", "tesseract language(s), e.g. spa+eng")
cmd.Flags().IntVar(&dpi, "dpi", 220, "hint DPI for phone photos / scans (0 = tesseract default)")
cmd.Flags().Float32Var(&minConf, "min-conf", 0, "drop words with OCR confidence below this (0 = keep all)")
return cmd
}
func docsAsMDCmd() *cobra.Command {
var to, lang string
var minChars int
var uploadOCR bool
var useHOCR bool
var dpi int
var minConf float32
cmd := &cobra.Command{
Use: "as-md FILE_ID",
Short: "Download an OO Documents file and emit Markdown (OCR PDF/image if needed)",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
c, err := newOO(cmd)
if err != nil {
return err
}
ctx := cmd.Context()
meta, err := c.GetFile(ctx, args[0])
if err != nil {
return err
}
title := onlyoffice.FileEntryTitle(meta)
dir, err := os.MkdirTemp("", "oo-docs-as-md-*")
if err != nil {
return err
}
defer os.RemoveAll(dir)
local := filepath.Join(dir, onlyoffice.SafeLocalFileName(title))
f, err := os.Create(local)
if err != nil {
return err
}
if _, err := c.DownloadFile(ctx, args[0], f); err != nil {
_ = f.Close()
return err
}
_ = f.Close()
tools := docpipe.LookPath()
var res docpipe.Result
if useHOCR {
hr, err := tools.ToHOCRMarkdown(local, dir, lang, dpi, minConf)
if err != nil {
return err
}
res = docpipe.Result{Markdown: hr.Markdown, DidOCR: hr.DidOCR, Source: hr.Source}
} else {
res, err = tools.ToMarkdown(local, dir, lang, minChars)
if err != nil {
return err
}
}
outPath := to
if outPath == "" {
base := strings.TrimSuffix(onlyoffice.SafeLocalFileName(title), filepath.Ext(onlyoffice.SafeLocalFileName(title)))
if base == "" || base == "download" {
base = "file-" + args[0]
}
outPath = base + ".md"
}
if err := docpipe.EnsureDir(outPath); err != nil {
return err
}
if err := os.WriteFile(outPath, []byte(res.Markdown), 0o644); err != nil {
return err
}
obj := map[string]any{
"file_id": args[0],
"title": title,
"md": outPath,
"did_ocr": res.DidOCR,
}
if res.OCRPDFPath != "" && uploadOCR {
folderID := onlyoffice.FileFolderID(meta)
upName := strings.TrimSuffix(onlyoffice.SafeLocalFileName(title), filepath.Ext(onlyoffice.SafeLocalFileName(title))) + ".ocr.pdf"
tmpUp := filepath.Join(dir, upName)
data, err := os.ReadFile(res.OCRPDFPath)
if err != nil {
return err
}
if err := os.WriteFile(tmpUp, data, 0o644); err != nil {
return err
}
if folderID == "" {
obj["ocr_pdf_local"] = res.OCRPDFPath
obj["note"] = "file has no folderId; OCR PDF left local — pass after moving into a folder"
} else {
ent, err := c.UploadToFolder(ctx, folderID, tmpUp)
if err != nil {
return err
}
obj["ocr_pdf_file_id"] = fileIDStr(ent)
obj["ocr_pdf_title"] = onlyoffice.FileEntryTitle(ent)
}
} else if res.OCRPDFPath != "" {
// Keep OCR PDF outside temp by copying beside md if requested via env-less default:
kept := strings.TrimSuffix(outPath, filepath.Ext(outPath)) + ".ocr.pdf"
if b, err := os.ReadFile(res.OCRPDFPath); err == nil {
_ = os.WriteFile(kept, b, 0o644)
obj["ocr_pdf_local"] = kept
} else {
obj["ocr_pdf_local"] = res.OCRPDFPath
}
}
printObject(obj)
return nil
},
}
cmd.Flags().StringVar(&to, "to", "", "write Markdown to this path (default: ./<title>.md)")
cmd.Flags().StringVar(&lang, "lang", "eng", "OCR language")
cmd.Flags().IntVar(&minChars, "min-chars", docpipe.DefaultMinTextChars, "OCR PDF if text layer shorter than this")
cmd.Flags().BoolVar(&uploadOCR, "upload-ocr", false, "upload searchable OCR PDF back into the same OO folder")
cmd.Flags().BoolVar(&useHOCR, "hocr", false, "use tesseract hOCR + go-hocr instead of ocrmypdf/pdftotext")
cmd.Flags().IntVar(&dpi, "dpi", 220, "DPI hint when --hocr (phone photos)")
cmd.Flags().Float32Var(&minConf, "min-conf", 0, "drop low-confidence words when --hocr")
return cmd
}
func docsPutMDCmd() *cobra.Command {
var folderID string
var keepLocalDOCX string
var replace bool
cmd := &cobra.Command{
Use: "put-md PROJECT_ID MARKDOWN_PATH",
Short: "Convert Markdown→DOCX and upload DOCX into a project (OO-friendly)",
Long: `Agents edit .md locally; this uploads .docx so OnlyOffice can open/version it.`,
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
pid, mdPath := args[0], args[1]
c, err := newOO(cmd)
if err != nil {
return err
}
tools := docpipe.LookPath()
dir, err := os.MkdirTemp("", "oo-docs-put-md-*")
if err != nil {
return err
}
defer os.RemoveAll(dir)
docxName := strings.TrimSuffix(filepath.Base(mdPath), filepath.Ext(mdPath)) + ".docx"
docxPath := filepath.Join(dir, docxName)
if err := tools.MDToDOCX(mdPath, docxPath); err != nil {
return err
}
if keepLocalDOCX != "" {
if err := docpipe.EnsureDir(keepLocalDOCX); err != nil {
return err
}
b, err := os.ReadFile(docxPath)
if err != nil {
return err
}
if err := os.WriteFile(keepLocalDOCX, b, 0o644); err != nil {
return err
}
}
ctx := cmd.Context()
var ent *onlyoffice.FileEntry
var deleted []int
if folderID != "" {
if replace {
ent, deleted, err = c.UploadToFolderReplacing(ctx, folderID, docxPath)
} else {
ent, err = c.UploadToFolder(ctx, folderID, docxPath)
}
if err != nil {
return err
}
obj := map[string]any{
"project_id": pid,
"folder_id": folderID,
"md": mdPath,
"uploaded": fileEntryToMap(ent),
}
if len(deleted) > 0 {
obj["replaced_file_ids"] = deleted
}
printObject(obj)
return nil
}
ent, err = c.UploadProjectFile(ctx, pid, docxPath)
if err != nil {
return err
}
printObject(map[string]any{
"project_id": pid,
"md": mdPath,
"uploaded": fileEntryToMap(ent),
})
return nil
},
}
cmd.Flags().StringVar(&folderID, "folder", "", "Documents folder id (default: project root)")
cmd.Flags().StringVar(&keepLocalDOCX, "keep-docx", "", "also write the generated DOCX to this local path")
cmd.Flags().BoolVar(&replace, "replace", true, "delete same-stem files in folder before upload (put-md upsert)")
return cmd
}
func docsPutTxtCmd() *cobra.Command {
var folderID string
var keepLocalDOCX string
var replace bool
cmd := &cobra.Command{
Use: "put-txt PROJECT_ID TEXT_PATH",
Short: "Convert plain text→DOCX (preserve line breaks) and upload into a project",
Long: `OnlyOffice cannot render .txt well. This keeps each source line on its own DOCX line.`,
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
pid, txtPath := args[0], args[1]
c, err := newOO(cmd)
if err != nil {
return err
}
tools := docpipe.LookPath()
dir, err := os.MkdirTemp("", "oo-docs-put-txt-*")
if err != nil {
return err
}
defer os.RemoveAll(dir)
docxName := strings.TrimSuffix(filepath.Base(txtPath), filepath.Ext(txtPath)) + ".docx"
docxPath := filepath.Join(dir, docxName)
if err := tools.TXTToDOCX(txtPath, docxPath); err != nil {
return err
}
if keepLocalDOCX != "" {
if err := docpipe.EnsureDir(keepLocalDOCX); err != nil {
return err
}
b, err := os.ReadFile(docxPath)
if err != nil {
return err
}
if err := os.WriteFile(keepLocalDOCX, b, 0o644); err != nil {
return err
}
}
ctx := cmd.Context()
var ent *onlyoffice.FileEntry
var deleted []int
if folderID != "" {
if replace {
ent, deleted, err = c.UploadToFolderReplacing(ctx, folderID, docxPath)
} else {
ent, err = c.UploadToFolder(ctx, folderID, docxPath)
}
if err != nil {
return err
}
obj := map[string]any{
"project_id": pid,
"txt": txtPath,
"folder_id": folderID,
"uploaded": fileEntryToMap(ent),
}
if len(deleted) > 0 {
obj["replaced_file_ids"] = deleted
}
printObject(obj)
return nil
}
ent, err = c.UploadProjectFile(ctx, pid, docxPath)
if err != nil {
return err
}
printObject(map[string]any{
"project_id": pid,
"txt": txtPath,
"uploaded": fileEntryToMap(ent),
})
return nil
},
}
cmd.Flags().StringVar(&folderID, "folder", "", "Documents folder id (default: project root)")
cmd.Flags().StringVar(&keepLocalDOCX, "keep-docx", "", "also write the generated DOCX to this local path")
cmd.Flags().BoolVar(&replace, "replace", true, "delete same-stem files in folder before upload (put-txt upsert)")
return cmd
}
-66
View File
@@ -26,7 +26,6 @@ func init() {
mailsCmd.AddCommand(mailsDraftCmd())
mailsCmd.AddCommand(mailsAttachCmd())
mailsCmd.AddCommand(mailsDraftInvoiceCmd())
mailsCmd.AddCommand(mailsSendCmd())
mailsCmd.AddCommand(mailsDeleteCmd())
}
@@ -348,71 +347,6 @@ func writeMailAttachment(path string, body []byte) error {
return os.WriteFile(path, body, 0o644)
}
func mailsSendCmd() *cobra.Command {
var from, to, cc, bcc, subject, body, html string
var id int64
cmd := &cobra.Command{
Use: "send",
Short: "Send a mail message (OnlyOffice Mail)",
Long: `Send via PUT /api/2.0/mail/messages/send.json.
oo mails send --id 7803 --body "…" # send referencing a draft id
oo mails send --to a@b.com --subject "…" --body "…"
oo mails send --id 7803 --to a@b.com --subject "…" --body "…" --cc x@y.com
IMPORTANT: send.json does NOT copy subject/body from the referenced draft — the
content must be in this request (--subject/--body). Cc/Bcc are omitted when empty
(the API 400s on empty strings). The API send does not append the UI signature —
put the chat line in --body if needed.
`,
RunE: func(cmd *cobra.Command, args []string) error {
if to == "" && id == 0 {
return fmt.Errorf("--to is required (or --id of an existing draft)")
}
htmlBody := body
if html != "" {
htmlBody = html
}
if htmlBody == "" && id != 0 {
// The send.json endpoint does NOT copy subject/body from the
// referenced draft — an empty body here sends an empty message.
// Warn instead of silently mailing an empty email.
return fmt.Errorf("--body/--html is required when sending by --id (send.json needs the content in the request)")
}
if htmlBody == "" && to == "" {
return fmt.Errorf("--body is required for a fresh message")
}
c, err := newOO(cmd)
if err != nil {
return err
}
raw, err := c.SendMail(cmd.Context(), onlyoffice.SendMailParams{
ID: id,
From: from,
To: to,
Cc: cc,
Bcc: bcc,
Subject: subject,
Body: htmlBody,
})
if err != nil {
return err
}
fmt.Println(string(raw))
return nil
},
}
cmd.Flags().Int64Var(&id, "id", 0, "existing draft id to send (0 = fresh message)")
cmd.Flags().StringVar(&from, "from", "", "from address (default: first enabled mailbox)")
cmd.Flags().StringVar(&to, "to", "", "recipient (required unless --id)")
cmd.Flags().StringVar(&cc, "cc", "", "cc")
cmd.Flags().StringVar(&bcc, "bcc", "", "bcc")
cmd.Flags().StringVar(&subject, "subject", "", "subject")
cmd.Flags().StringVar(&body, "body", "", "plain text or HTML body")
cmd.Flags().StringVar(&html, "html", "", "HTML body (alias of --body when set)")
return cmd
}
func mailsDeleteCmd() *cobra.Command {
return &cobra.Command{
Use: "delete ID [ID...]",
+1 -2
View File
@@ -3,7 +3,7 @@
// Command tree is subject-based (mirrors the library split and the `tea` CLI):
//
// oo calendar list | events | add | delete
// oo projects list | get | milestones | create | update | delete | files (list|upload|download|rename|delete|as-md|put-md)
// oo projects list | get | milestones | create | update | delete | files (list|upload|download|rename|delete)
// oo tasks list | get | create | update | delete | subtask add | files (list|upload|detach)
// oo users list | self (alias: oo whoami)
// oo contacts list | get | delete | info-add | merge | dedupe-info
@@ -15,7 +15,6 @@
// oo crm cleanup
// oo mails accounts | folders | list | get | download-attachment | draft | attach | draft-invoice | delete
// oo invoices list | get | create | update | pdf | pdf-cleanup | status | delete | items …
// oo docs tools | convert | ocr | as-md | put-md
//
// CRM association rules: docs/crm-associations.md
//
-82
View File
@@ -24,35 +24,9 @@ func projectFilesCmd() *cobra.Command {
cmd.AddCommand(prjFilesDownloadCmd())
cmd.AddCommand(prjFilesRenameCmd())
cmd.AddCommand(prjFilesDeleteCmd())
cmd.AddCommand(prjFilesDedupeCmd())
// Convenience aliases into oo docs (md↔docx / OCR pipeline).
cmd.AddCommand(aliasDocsAsMD())
cmd.AddCommand(aliasDocsPutMD())
cmd.AddCommand(aliasDocsPutTxt())
return cmd
}
func aliasDocsAsMD() *cobra.Command {
c := docsAsMDCmd()
c.Use = "as-md FILE_ID"
c.Short = "Alias of `oo docs as-md` — download OO file as Markdown (OCR if needed)"
return c
}
func aliasDocsPutMD() *cobra.Command {
c := docsPutMDCmd()
c.Use = "put-md PROJECT_ID MARKDOWN_PATH"
c.Short = "Alias of `oo docs put-md` — Markdown→DOCX upload into project"
return c
}
func aliasDocsPutTxt() *cobra.Command {
c := docsPutTxtCmd()
c.Use = "put-txt PROJECT_ID TEXT_PATH"
c.Short = "Alias of `oo docs put-txt` — plain text→DOCX upload into project"
return c
}
func prjFilesListCmd() *cobra.Command {
var showFolders bool
cmd := &cobra.Command{
@@ -210,62 +184,6 @@ func prjFilesDeleteCmd() *cobra.Command {
}
}
func prjFilesDedupeCmd() *cobra.Command {
var apply, cross bool
cmd := &cobra.Command{
Use: "dedupe PROJECT_ID",
Short: "Find (and optionally remove) duplicate files in project Documents folders",
Long: `Duplicates share the same logical name: stem|ext (OO title+fileExst).
Default: dry-run report. Pass --apply to delete older copies (keeps newest; --cross prefers non-trash folders).`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
c, err := newOO(cmd)
if err != nil {
return err
}
groups, deleted, err := c.DedupeProject(cmd.Context(), args[0], onlyoffice.DedupOptions{
CrossFolder: cross,
}, apply)
if err != nil {
return err
}
rows := make([]map[string]any, 0, len(groups))
for _, g := range groups {
row := map[string]any{
"key": g.Key,
"folder_id": g.FolderID,
"folder_title": g.FolderTitle,
"keep_id": fileIDStr(g.Keep),
"keep_title": onlyoffice.FileEntryTitle(g.Keep),
"remove_count": len(g.Remove),
}
removeIDs := make([]string, 0, len(g.Remove))
for _, f := range g.Remove {
removeIDs = append(removeIDs, fileIDStr(f))
}
row["remove_ids"] = removeIDs
rows = append(rows, row)
}
out := map[string]any{
"project_id": args[0],
"dry_run": !apply,
"cross": cross,
"groups": len(groups),
"duplicates": rows,
}
if apply {
out["deleted_ids"] = deleted
}
printObject(out)
return nil
},
}
cmd.Flags().BoolVar(&apply, "apply", false, "delete duplicate files (default: report only)")
cmd.Flags().BoolVar(&cross, "cross", false, "also dedupe same stem|ext across folders (prefers non-_trash)")
return cmd
}
func fileEntryRows(files []*onlyoffice.FileEntry) []map[string]any {
rows := make([]map[string]any, 0, len(files))
for _, f := range files {
-96
View File
@@ -1,96 +0,0 @@
package onlyoffice
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
)
// History entities that OnlyOffice CRM actually accepts for history notes.
// There is NO person/contact history in this API version: POST /api/2.0/crm/history.json
// returns 400 "Value does not fall within the expected range." for entityType
// contact/person/people/client/member. Verified against a live instance (#74).
const (
HistoryEntityOpportunity = "opportunity"
HistoryEntityCase = "case"
)
// IsCompany reports whether a CRM contact row is a company (vs a person).
// The field arrives as JSON bool; be liberal about what we accept.
func IsCompany(person map[string]any) bool {
b, _ := person["isCompany"].(bool)
return b
}
// ContactID returns the CRM id of a contact row as a plain string.
func ContactID(row map[string]any) string {
return fmt.Sprint(row["id"])
}
// BuildContactEmailIndex scans all persons once and maps lowercase email →
// contact id. Use this instead of calling FindPersonByEmail per address:
// the index is O(N) over the whole CRM, the per-address lookup is O(N×M).
func (c *Client) BuildContactEmailIndex(ctx context.Context) (map[string]string, error) {
all, err := c.ListAllContacts(ctx)
if err != nil {
return nil, err
}
index := make(map[string]string, len(all)*2)
for _, person := range all {
if IsCompany(person) {
continue
}
id := ContactID(person)
for _, row := range ContactInfoRows(person) {
if NormalizeContactInfoType(fmt.Sprint(row["infoType"])) != "email" {
continue
}
email := strings.ToLower(strings.TrimSpace(fmt.Sprint(row["data"])))
if email != "" && email != "<nil>" {
index[email] = id
}
}
}
return index, nil
}
// BuildPersonOpportunityIndex maps every opportunity member's contact id to a
// deterministic representative opportunity: the one with the lowest numeric id.
// OnlyOffice has no person-level history, so notes for a person go on their
// deal — this index answers "which deal" in one pass.
func (c *Client) BuildPersonOpportunityIndex(ctx context.Context) (map[string]string, error) {
opps, err := c.ListAllOpportunities(ctx)
if err != nil {
return nil, err
}
index := map[string]string{}
for _, opp := range opps {
oppID := ContactID(opp)
for _, member := range OpportunityMembers(opp) {
pid := ContactID(member)
if cur, ok := index[pid]; !ok || NumericIDLess(oppID, cur) {
index[pid] = oppID
}
}
}
return index, nil
}
// NumericIDLess compares two string ids numerically when possible, falling
// back to lexicographic order so results stay deterministic either way.
func NumericIDLess(a, b string) bool {
na, errA := strconv.Atoi(strings.TrimSpace(a))
nb, errB := strconv.Atoi(strings.TrimSpace(b))
if errA == nil && errB == nil && na != nb {
return na < nb
}
return a < b
}
// SortIDs orders id strings deterministically (numeric first, then lexical).
func SortIDs(ids []string) {
sort.Strings(ids)
sort.SliceStable(ids, func(i, j int) bool { return NumericIDLess(ids[i], ids[j]) })
}
-65
View File
@@ -1,65 +0,0 @@
package onlyoffice
import "testing"
func TestIsCompany(t *testing.T) {
if IsCompany(map[string]any{"isCompany": true}) != true {
t.Fatal("true row not detected")
}
if IsCompany(map[string]any{"isCompany": false}) {
t.Fatal("false row detected as company")
}
if IsCompany(map[string]any{}) {
t.Fatal("missing field detected as company")
}
if IsCompany(nil) {
t.Fatal("nil row detected as company")
}
}
func TestNumericIDLess(t *testing.T) {
cases := []struct {
a, b string
want bool
}{
{"9", "10", true},
{"1747", "1748", true},
{"abc", "abd", true},
{"10", "9", false},
{" 12 ", "13", true},
{"x1", "2", false}, // non-numeric falls back lexical: "x1" > "2"
}
for _, c := range cases {
if got := NumericIDLess(c.a, c.b); got != c.want {
t.Errorf("NumericIDLess(%q,%q)=%v want %v", c.a, c.b, got, c.want)
}
}
}
func TestSortIDs(t *testing.T) {
ids := []string{"20", "3", "100", "1"}
SortIDs(ids)
want := "1 3 20 100"
got := ""
for i, id := range ids {
if i > 0 {
got += " "
}
got += id
}
if got != want {
t.Fatalf("SortIDs=%q want %q", got, want)
}
}
func TestContactID(t *testing.T) {
if ContactID(map[string]any{"id": float64(42)}) != "42" {
t.Fatal("numeric id formatting broken")
}
}
func TestHistoryEntityConstants(t *testing.T) {
if HistoryEntityOpportunity != "opportunity" || HistoryEntityCase != "case" {
t.Fatal("history entity whitelist drifted from live-verified values")
}
}
+11 -85
View File
@@ -263,86 +263,22 @@ func (c *Client) RenameFile(ctx context.Context, fileID, newTitle string) (*File
return decodeResponseFileEntry(raw)
}
type deleteFilesBody struct {
FileIDs []int `json:"fileIds"`
FolderIDs []int `json:"folderIds"`
}
// DeleteFiles permanently deletes files by numeric id (Documents module).
// Uses per-file DELETE (DeleteDavItems); fileops/delete returns 200 on some
// portals (e.g. produktor.io) without removing the file.
func (c *Client) DeleteFiles(ctx context.Context, fileIDs []int) error {
if len(fileIDs) == 0 {
return fmt.Errorf("no file ids to delete")
}
strIDs := make([]string, len(fileIDs))
for i, id := range fileIDs {
strIDs[i] = strconv.Itoa(id)
}
return c.DeleteDavItems(ctx, nil, strIDs)
}
// ListFolder returns the Documents module listing for a folder id
// (GET /api/2.0/files/{folderId}).
func (c *Client) ListFolder(ctx context.Context, folderID string) (map[string]any, error) {
if folderID == "" {
return nil, fmt.Errorf("folder id is required")
}
out, err := c.ResponseObject(ctx, "/api/2.0/files/"+url.PathEscape(folderID)+".json")
body := deleteFilesBody{FileIDs: fileIDs, FolderIDs: nil}
_, err := c.putJSON(ctx, "/api/2.0/files/fileops/delete.json", body)
if err != nil {
out, err = c.ResponseObject(ctx, "/api/2.0/files/"+url.PathEscape(folderID))
_, err = c.putJSON(ctx, "/api/2.0/files/fileops/delete", body)
}
return out, err
}
// CreateFolder creates a subfolder under parentFolderID.
func (c *Client) CreateFolder(ctx context.Context, parentFolderID, title string) (map[string]any, error) {
if parentFolderID == "" || title == "" {
return nil, fmt.Errorf("parent folder id and title are required")
}
body := map[string]any{"title": title}
out, err := c.postJSONObject(ctx, "/api/2.0/files/folder/"+url.PathEscape(parentFolderID)+".json", body)
if err != nil {
out, err = c.postJSONObject(ctx, "/api/2.0/files/folder/"+url.PathEscape(parentFolderID), body)
}
return out, err
}
// MoveFiles moves file ids into destFolderID (Documents fileops/move).
func (c *Client) MoveFiles(ctx context.Context, destFolderID int, fileIDs []int) (map[string]any, error) {
if destFolderID == 0 || len(fileIDs) == 0 {
return nil, fmt.Errorf("dest folder and file ids are required")
}
body := map[string]any{
"folderIds": []int{},
"fileIds": fileIDs,
"destFolderId": destFolderID,
}
out, err := c.putJSONObject(ctx, "/api/2.0/files/fileops/move.json", body)
if err != nil {
out, err = c.putJSONObject(ctx, "/api/2.0/files/fileops/move", body)
}
return out, err
}
// UploadToFolder uploads a local file into an arbitrary Documents folder id.
func (c *Client) UploadToFolder(ctx context.Context, folderID, localPath string) (*FileEntry, error) {
if folderID == "" || localPath == "" {
return nil, fmt.Errorf("folder id and local path are required")
}
uploadPath := fmt.Sprintf("/api/2.0/files/%s/upload.json", url.PathEscape(folderID))
raw, err := c.uploadMultipart(ctx, uploadPath, "file", localPath)
if err != nil {
uploadPath = fmt.Sprintf("/api/2.0/files/%s/upload", url.PathEscape(folderID))
raw, err = c.uploadMultipart(ctx, uploadPath, "file", localPath)
if err != nil {
return nil, err
}
}
return decodeResponseFileEntry(raw)
}
// FileFolderID returns the parent folder id string for a file entry, if known.
func FileFolderID(f *FileEntry) string {
if f == nil || f.FolderID == nil {
return ""
}
return f.FolderID.String()
return err
}
// DownloadFile streams file bytes from the file's viewUrl using the same auth
@@ -383,18 +319,8 @@ func (c *Client) resolveAPIURL(ref string) string {
if ref == "" {
return ref
}
// Rewrite any host to the configured API base so downloads stay on the
// internal network and keep the Authorization header (no cross-host
// redirect that would strip it). Scheme-relative URLs are handled too.
if strings.HasPrefix(ref, "//") {
ref = "http:" + ref
}
if u, err := url.Parse(ref); err == nil && u.IsAbs() {
if base, err2 := url.Parse(c.baseURL()); err2 == nil {
u.Scheme = base.Scheme
u.Host = base.Host
return u.String()
}
if strings.HasPrefix(ref, "http://") || strings.HasPrefix(ref, "https://") {
return ref
}
base := c.baseURL()
if strings.HasPrefix(ref, "/") {
-337
View File
@@ -1,337 +0,0 @@
package onlyoffice
import (
"context"
"path/filepath"
"sort"
"strings"
)
// FileEntryExt returns a normalized extension (lowercase, with leading dot).
func FileEntryExt(f *FileEntry) string {
if f == nil {
return ""
}
exst := ""
if f.FileExst != nil {
exst = strings.TrimSpace(*f.FileExst)
}
if exst != "" {
if !strings.HasPrefix(exst, ".") {
exst = "." + exst
}
return strings.ToLower(exst)
}
if f.Title != nil {
if ext := filepath.Ext(*f.Title); ext != "" {
return strings.ToLower(ext)
}
}
return ""
}
// FileDedupKey is stem|ext — two files with the same key are duplicates.
func FileDedupKey(f *FileEntry) string {
st := FileEntryStem(f)
ext := FileEntryExt(f)
if st == "" {
return ""
}
if ext == "" {
return st
}
return st + "|" + strings.TrimPrefix(ext, ".")
}
// FindFilesByDedupKey returns folder files matching stem and extension.
func FindFilesByDedupKey(files []*FileEntry, stem, ext string) []*FileEntry {
key := dedupKeyFromParts(stem, ext)
if key == "" {
return nil
}
var out []*FileEntry
for _, f := range files {
if FileDedupKey(f) == key {
out = append(out, f)
}
}
return out
}
func dedupKeyFromParts(stem, ext string) string {
stem = strings.TrimSpace(stem)
if stem == "" {
return ""
}
ext = strings.ToLower(strings.TrimSpace(ext))
if ext != "" && !strings.HasPrefix(ext, ".") {
ext = "." + ext
}
if ext == "" {
return stem
}
return stem + "|" + strings.TrimPrefix(ext, ".")
}
// UploadExtFromLocal returns the lowercase extension from a local path.
func UploadExtFromLocal(localPath string) string {
ext := filepath.Ext(localPath)
if ext == "" {
return ""
}
return strings.ToLower(ext)
}
// IsTrashFolderTitle reports staging/trash folders (e.g. _trash-md).
func IsTrashFolderTitle(title string) bool {
t := strings.ToLower(strings.TrimSpace(title))
return strings.HasPrefix(t, "_") || strings.Contains(t, "trash")
}
// ProjectFolderFile ties a file to its project Documents subfolder.
type ProjectFolderFile struct {
FolderID string
FolderTitle string
File *FileEntry
}
// DedupGroup is one duplicate set: keep the newest (or non-trash) file.
type DedupGroup struct {
Key string
FolderID string
FolderTitle string
Keep *FileEntry
Remove []*FileEntry
}
// DedupOptions controls project-wide duplicate scans.
type DedupOptions struct {
CrossFolder bool
}
// FindProjectDuplicates scans project folders for duplicate files.
func FindProjectDuplicates(folders []*FolderEntry, filesByFolder map[string][]*FileEntry, opts DedupOptions) []DedupGroup {
var indexed []ProjectFolderFile
for _, folder := range folders {
if folder == nil || folder.ID == nil {
continue
}
fid := folder.ID.String()
title := ""
if folder.Title != nil {
title = *folder.Title
}
for _, f := range filesByFolder[fid] {
if f == nil {
continue
}
indexed = append(indexed, ProjectFolderFile{
FolderID: fid, FolderTitle: title, File: f,
})
}
}
if opts.CrossFolder {
return findCrossFolderDuplicates(indexed)
}
return findWithinFolderDuplicates(indexed)
}
func findWithinFolderDuplicates(indexed []ProjectFolderFile) []DedupGroup {
byFolder := map[string][]ProjectFolderFile{}
for _, it := range indexed {
byFolder[it.FolderID] = append(byFolder[it.FolderID], it)
}
var out []DedupGroup
for fid, items := range byFolder {
title := ""
if len(items) > 0 {
title = items[0].FolderTitle
}
byKey := map[string][]*FileEntry{}
for _, it := range items {
k := FileDedupKey(it.File)
byKey[k] = append(byKey[k], it.File)
}
for k, group := range byKey {
if len(group) < 2 {
continue
}
keep, remove := pickDuplicateKeeper(group, false)
if keep == nil || len(remove) == 0 {
continue
}
out = append(out, DedupGroup{
Key: k, FolderID: fid, FolderTitle: title, Keep: keep, Remove: remove,
})
}
}
sortDedupGroups(out)
return out
}
func findCrossFolderDuplicates(indexed []ProjectFolderFile) []DedupGroup {
byKey := map[string][]ProjectFolderFile{}
for _, it := range indexed {
k := FileDedupKey(it.File)
byKey[k] = append(byKey[k], it)
}
var out []DedupGroup
for k, items := range byKey {
if len(items) < 2 {
continue
}
files := make([]*FileEntry, len(items))
folders := make([]string, len(items))
folderTitles := make([]string, len(items))
for i, it := range items {
files[i] = it.File
folders[i] = it.FolderID
folderTitles[i] = it.FolderTitle
}
keep, remove := pickDuplicateKeeperWithFolders(files, folders, folderTitles)
if keep == nil || len(remove) == 0 {
continue
}
fid, ftitle := "", ""
for _, it := range items {
if it.File == keep {
fid, ftitle = it.FolderID, it.FolderTitle
break
}
}
out = append(out, DedupGroup{
Key: k, FolderID: fid, FolderTitle: ftitle, Keep: keep, Remove: remove,
})
}
sortDedupGroups(out)
return out
}
func pickDuplicateKeeper(files []*FileEntry, _ bool) (*FileEntry, []*FileEntry) {
return pickDuplicateKeeperWithFolders(files, nil, nil)
}
func pickDuplicateKeeperWithFolders(files []*FileEntry, folderIDs, folderTitles []string) (*FileEntry, []*FileEntry) {
if len(files) == 0 {
return nil, nil
}
type ranked struct {
file *FileEntry
trash bool
}
rankedFiles := make([]ranked, len(files))
for i, f := range files {
trash := false
if folderTitles != nil && i < len(folderTitles) {
trash = IsTrashFolderTitle(folderTitles[i])
}
rankedFiles[i] = ranked{file: f, trash: trash}
}
sort.SliceStable(rankedFiles, func(i, j int) bool {
ri, rj := rankedFiles[i], rankedFiles[j]
if ri.trash != rj.trash {
return !ri.trash // non-trash first
}
ti, tj := rankedFiles[i].file.Updated, rankedFiles[j].file.Updated
if ti == nil {
return false
}
if tj == nil {
return true
}
return ti.After(*tj) // newest first
})
keep := rankedFiles[0].file
var remove []*FileEntry
for _, r := range rankedFiles[1:] {
remove = append(remove, r.file)
}
return keep, remove
}
func sortDedupGroups(groups []DedupGroup) {
sort.Slice(groups, func(i, j int) bool {
if groups[i].FolderTitle != groups[j].FolderTitle {
return groups[i].FolderTitle < groups[j].FolderTitle
}
return groups[i].Key < groups[j].Key
})
}
// ApplyDedupGroups deletes Remove files from each group.
func (c *Client) ApplyDedupGroups(ctx context.Context, groups []DedupGroup) ([]int, error) {
seen := map[int]struct{}{}
var ids []int
for _, g := range groups {
for _, f := range g.Remove {
n := int(FileEntryNumericID(f))
if n == 0 {
continue
}
if _, ok := seen[n]; ok {
continue
}
seen[n] = struct{}{}
ids = append(ids, n)
}
}
if len(ids) == 0 {
return nil, nil
}
if err := c.DeleteFiles(ctx, ids); err != nil {
return ids, err
}
return ids, nil
}
// DeleteFilesByDedupKey removes all files in folderID matching stem+ext.
func (c *Client) DeleteFilesByDedupKey(ctx context.Context, folderID, stem, ext string) ([]int, error) {
files, err := c.FolderFiles(ctx, folderID)
if err != nil {
return nil, err
}
matches := FindFilesByDedupKey(files, stem, ext)
if len(matches) == 0 {
return nil, nil
}
ids := make([]int, 0, len(matches))
for _, f := range matches {
n := int(FileEntryNumericID(f))
if n != 0 {
ids = append(ids, n)
}
}
if len(ids) == 0 {
return nil, nil
}
if err := c.DeleteFiles(ctx, ids); err != nil {
return nil, err
}
return ids, nil
}
// DedupeProject scans project folders and optionally deletes duplicates.
func (c *Client) DedupeProject(ctx context.Context, projectID string, opts DedupOptions, apply bool) ([]DedupGroup, []int, error) {
pf, err := c.GetProjectFiles(ctx, projectID)
if err != nil {
return nil, nil, err
}
filesByFolder := make(map[string][]*FileEntry, len(pf.Folders))
for _, folder := range pf.Folders {
if folder == nil || folder.ID == nil {
continue
}
fid := folder.ID.String()
files, err := c.FolderFiles(ctx, fid)
if err != nil {
return nil, nil, err
}
filesByFolder[fid] = files
}
groups := FindProjectDuplicates(pf.Folders, filesByFolder, opts)
if !apply || len(groups) == 0 {
return groups, nil, nil
}
deleted, err := c.ApplyDedupGroups(ctx, groups)
return groups, deleted, err
}
-81
View File
@@ -1,81 +0,0 @@
package onlyoffice
import (
"encoding/json"
"testing"
"time"
)
func TestFileDedupKey(t *testing.T) {
title := "OO-HONDA-7-INDEX.docx"
exst := ".docx"
f := &FileEntry{Title: &title, FileExst: &exst}
if got := FileDedupKey(f); got != "OO-HONDA-7-INDEX|docx" {
t.Fatalf("got %q", got)
}
}
func TestFindFilesByDedupKey(t *testing.T) {
a := &FileEntry{Title: strPtr("foo.docx"), FileExst: strPtr(".docx")}
b := &FileEntry{Title: strPtr("foo.md"), FileExst: strPtr(".md")}
files := []*FileEntry{a, b}
got := FindFilesByDedupKey(files, "foo", ".docx")
if len(got) != 1 || got[0] != a {
t.Fatalf("got %+v", got)
}
}
func TestFindWithinFolderDuplicates(t *testing.T) {
t1 := time.Date(2026, 8, 27, 16, 0, 0, 0, time.UTC)
t2 := t1.Add(time.Hour)
old := &FileEntry{ID: jsonNum("1"), Title: strPtr("idx.docx"), FileExst: strPtr(".docx"), Updated: &t1}
new := &FileEntry{ID: jsonNum("2"), Title: strPtr("idx.docx"), FileExst: strPtr(".docx"), Updated: &t2}
indexed := []ProjectFolderFile{
{FolderID: "490", FolderTitle: "00-Index", File: old},
{FolderID: "490", FolderTitle: "00-Index", File: new},
}
groups := findWithinFolderDuplicates(indexed)
if len(groups) != 1 {
t.Fatalf("groups=%d", len(groups))
}
if FileEntryNumericID(groups[0].Keep) != 2 {
t.Fatalf("keep id=%d", FileEntryNumericID(groups[0].Keep))
}
if len(groups[0].Remove) != 1 || FileEntryNumericID(groups[0].Remove[0]) != 1 {
t.Fatalf("remove=%v", groups[0].Remove)
}
}
func TestCrossFolderPrefersNonTrash(t *testing.T) {
t1 := time.Date(2026, 8, 27, 18, 0, 0, 0, time.UTC)
t2 := t1.Add(-time.Hour)
trash := &FileEntry{ID: jsonNum("10"), Title: strPtr("INDEX.md"), FileExst: strPtr(".md"), Updated: &t1}
good := &FileEntry{ID: jsonNum("20"), Title: strPtr("INDEX.md"), FileExst: strPtr(".md"), Updated: &t2}
indexed := []ProjectFolderFile{
{FolderID: "493", FolderTitle: "_trash-md", File: trash},
{FolderID: "492", FolderTitle: "OCR", File: good},
}
groups := findCrossFolderDuplicates(indexed)
if len(groups) != 1 {
t.Fatalf("groups=%d", len(groups))
}
if FileEntryNumericID(groups[0].Keep) != 20 {
t.Fatalf("keep id=%d", FileEntryNumericID(groups[0].Keep))
}
}
func TestIsTrashFolderTitle(t *testing.T) {
if !IsTrashFolderTitle("_trash-md") {
t.Fatal("expected trash")
}
if IsTrashFolderTitle("00-Index") {
t.Fatal("expected not trash")
}
}
func strPtr(s string) *string { return &s }
func jsonNum(s string) *json.Number {
n := json.Number(s)
return &n
}
-129
View File
@@ -1,129 +0,0 @@
package onlyoffice
import (
"context"
"encoding/json"
"path/filepath"
"strings"
)
// FileEntryStem returns the logical basename without duplicated extensions.
// OO often stores title="foo.docx" and fileExst=".docx" (UI shows foo.docx.docx).
func FileEntryStem(f *FileEntry) string {
if f == nil || f.Title == nil {
return ""
}
exst := ""
if f.FileExst != nil {
exst = *f.FileExst
}
return NormalizeUploadStem(*f.Title, exst)
}
// NormalizeUploadStem derives a stable stem for matching uploads.
func NormalizeUploadStem(title, exst string) string {
t := strings.TrimSpace(title)
t = strings.TrimSuffix(t, ".")
if exst != "" && strings.HasSuffix(t, exst) {
t = strings.TrimSuffix(t, exst)
}
if ext := filepath.Ext(t); ext != "" {
t = strings.TrimSuffix(t, ext)
}
return strings.TrimSpace(t)
}
// UploadStemFromLocal returns the stem used to match/replace folder files.
func UploadStemFromLocal(localPath string) string {
base := filepath.Base(localPath)
ext := filepath.Ext(base)
if ext != "" {
base = strings.TrimSuffix(base, ext)
}
return base
}
// FolderFiles returns file entries in a Documents folder.
func (c *Client) FolderFiles(ctx context.Context, folderID string) ([]*FileEntry, error) {
raw, err := c.ListFolder(ctx, folderID)
if err != nil {
return nil, err
}
return ParseFolderFileEntries(raw), nil
}
// ParseFolderFileEntries extracts []*FileEntry from ListFolder JSON.
func ParseFolderFileEntries(raw map[string]any) []*FileEntry {
items, _ := raw["files"].([]any)
out := make([]*FileEntry, 0, len(items))
for _, it := range items {
m, ok := it.(map[string]any)
if !ok {
continue
}
b, err := json.Marshal(m)
if err != nil {
continue
}
var f FileEntry
if err := json.Unmarshal(b, &f); err != nil {
continue
}
out = append(out, &f)
}
return out
}
// FindFilesByStem returns folder files whose logical stem matches.
func FindFilesByStem(files []*FileEntry, stem string) []*FileEntry {
stem = strings.TrimSpace(stem)
if stem == "" {
return nil
}
var out []*FileEntry
for _, f := range files {
if FileEntryStem(f) == stem {
out = append(out, f)
}
}
return out
}
// DeleteFilesByStem removes all files in folderID matching stem (any extension).
// Prefer DeleteFilesByDedupKey when the upload extension is known.
func (c *Client) DeleteFilesByStem(ctx context.Context, folderID, stem string) ([]int, error) {
files, err := c.FolderFiles(ctx, folderID)
if err != nil {
return nil, err
}
matches := FindFilesByStem(files, stem)
if len(matches) == 0 {
return nil, nil
}
ids := make([]int, 0, len(matches))
for _, f := range matches {
n := int(FileEntryNumericID(f))
if n != 0 {
ids = append(ids, n)
}
}
if len(ids) == 0 {
return nil, nil
}
if err := c.DeleteFiles(ctx, ids); err != nil {
return nil, err
}
return ids, nil
}
// UploadToFolderReplacing deletes same stem+ext files then uploads localPath.
func (c *Client) UploadToFolderReplacing(ctx context.Context, folderID, localPath string) (*FileEntry, []int, error) {
stem := UploadStemFromLocal(localPath)
ext := UploadExtFromLocal(localPath)
deleted, err := c.DeleteFilesByDedupKey(ctx, folderID, stem, ext)
if err != nil {
return nil, deleted, err
}
ent, err := c.UploadToFolder(ctx, folderID, localPath)
return ent, deleted, err
}
-35
View File
@@ -1,35 +0,0 @@
package onlyoffice
import "testing"
func TestNormalizeUploadStem(t *testing.T) {
tests := []struct {
title, exst, want string
}{
{"OO-HONDA-7-INDEX.docx", ".docx", "OO-HONDA-7-INDEX"},
{"README.txt", ".txt", "README"},
{"car-docs-print.docx", ".docx", "car-docs-print"},
{"plain.", "", "plain"},
{"foo", ".docx", "foo"},
}
for _, tc := range tests {
if got := NormalizeUploadStem(tc.title, tc.exst); got != tc.want {
t.Fatalf("NormalizeUploadStem(%q,%q)=%q want %q", tc.title, tc.exst, got, tc.want)
}
}
}
func TestFileEntryStem(t *testing.T) {
title := "00-INDEX.docx"
exst := ".docx"
f := &FileEntry{Title: &title, FileExst: &exst}
if got := FileEntryStem(f); got != "00-INDEX" {
t.Fatalf("got %q", got)
}
}
func TestUploadStemFromLocal(t *testing.T) {
if got := UploadStemFromLocal("/tmp/OO-HONDA-7-INDEX.docx"); got != "OO-HONDA-7-INDEX" {
t.Fatalf("got %q", got)
}
}
-421
View File
@@ -1,421 +0,0 @@
package onlyoffice
// WebDAV-oriented Files operations. These expose the Documents module through
// value types and cover everything needed to back a filesystem mapping:
// listing (including the virtual @root sections), folder/file CRUD, move/copy,
// and streaming upload/download. They are intentionally small and dependency
// free (only net/http), so callers are not forced to import heavier parts of
// the library.
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/url"
"strconv"
"strings"
"time"
)
// DavFolder is a folder row from the Files module.
type DavFolder struct {
ID string
Title string
ParentID string
RootType int // 1=Common, 3=Trash, 5=My, 6=Share, 8=Projects, ...
FilesCount int
FoldersCount int
Access int
Shared bool
Updated string
}
// DavFile is a file row from the Files module.
type DavFile struct {
ID string
Title string
Size int64
Updated string
ViewURL string
}
// DavListing is the contents of one folder.
type DavListing struct {
Current DavFolder
Files []DavFile
Folders []DavFolder
}
// ListDavFolder returns the contents of a folder by id, which may be a
// symbolic root such as "@my". For "@root" use ListDavSections.
func (c *Client) ListDavFolder(ctx context.Context, id string) (*DavListing, error) {
raw, err := c.getJSON(ctx, "/api/2.0/files/"+url.PathEscape(id))
if err != nil {
return nil, err
}
resp, err := responseField(raw, "response")
if err != nil {
return nil, err
}
// @root returns an array with a single blob; a normal folder returns an
// object. Normalize both.
if len(resp) > 0 && resp[0] == '[' {
var arr []*DavListing
if err := json.Unmarshal(resp, &arr); err != nil {
return nil, err
}
if len(arr) == 0 {
return &DavListing{}, nil
}
return arr[0], nil
}
var l DavListing
if err := json.Unmarshal(resp, &l); err != nil {
return nil, err
}
return &l, nil
}
// ListDavSections returns the virtual top-level sections shown by @root
// ("In projects", "My documents", "Shared with me", "Common", "Favorites",
// "Recent", "Trash"). Each is the `current` folder of one @root element.
func (c *Client) ListDavSections(ctx context.Context) ([]DavFolder, error) {
raw, err := c.getJSON(ctx, "/api/2.0/files/@root")
if err != nil {
return nil, err
}
resp, err := responseField(raw, "response")
if err != nil {
return nil, err
}
var arr []struct {
Current DavFolder `json:"current"`
}
if err := json.Unmarshal(resp, &arr); err != nil {
// Tolerate a non-array (single listing) response.
var single DavListing
if err2 := json.Unmarshal(resp, &single); err2 != nil {
return nil, err
}
return []DavFolder{single.Current}, nil
}
sections := make([]DavFolder, 0, len(arr))
for i := range arr {
sections = append(sections, arr[i].Current)
}
return sections, nil
}
// CreateDavFolder creates a folder titled title inside parentID.
func (c *Client) CreateDavFolder(ctx context.Context, parentID, title string) (*DavFolder, error) {
raw, err := c.postJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(parentID),
map[string]string{"title": title})
if err != nil {
return nil, err
}
var env struct {
Response *DavFolder `json:"response"`
}
if err := json.Unmarshal(raw, &env); err != nil {
return nil, err
}
if env.Response == nil {
return nil, fmt.Errorf("onlyoffice: empty create-folder response")
}
return env.Response, nil
}
// RenameDavFolder renames a folder.
func (c *Client) RenameDavFolder(ctx context.Context, id, title string) error {
_, err := c.putJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(id),
map[string]string{"title": title})
return err
}
// RenameDavFile renames a file (title includes the extension).
func (c *Client) RenameDavFile(ctx context.Context, id, title string) error {
_, err := c.putJSON(ctx, "/api/2.0/files/file/"+url.PathEscape(id),
map[string]string{"title": title})
return err
}
// MoveDavItems moves the given folders and/or files into destFolderID.
func (c *Client) MoveDavItems(ctx context.Context, folderIDs, fileIDs []string, destFolderID string) error {
_, err := c.putJSON(ctx, "/api/2.0/files/fileops/move", map[string]any{
"folderIds": nums(folderIDs),
"fileIds": nums(fileIDs),
"destFolderId": num(destFolderID),
"resolveType": "Skip",
"holdResult": true,
})
return err
}
// CopyDavItems copies the given folders and/or files into destFolderID.
func (c *Client) CopyDavItems(ctx context.Context, folderIDs, fileIDs []string, destFolderID string) error {
_, err := c.putJSON(ctx, "/api/2.0/files/fileops/copy", map[string]any{
"folderIds": nums(folderIDs),
"fileIds": nums(fileIDs),
"destFolderId": num(destFolderID),
"conflictResolveType": "Skip",
"deleteAfter": true,
})
return err
}
// DeleteDavItems deletes the given folders and/or files.
func (c *Client) DeleteDavItems(ctx context.Context, folderIDs, fileIDs []string) error {
body := map[string]any{"DeleteAfter": true, "Immediately": true}
for _, id := range folderIDs {
if _, err := c.deleteJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(id), body); err != nil {
return err
}
}
for _, id := range fileIDs {
if _, err := c.deleteJSON(ctx, "/api/2.0/files/file/"+url.PathEscape(id), body); err != nil {
return err
}
}
return nil
}
// UploadDavFile uploads src (fileName) into folderID, streaming from src.
func (c *Client) UploadDavFile(ctx context.Context, folderID, fileName string, src io.Reader) (*DavFile, error) {
raw, err := c.uploadReader(ctx, "/api/2.0/files/"+url.PathEscape(folderID)+"/upload", "file", fileName, src)
if err != nil {
return nil, err
}
var env struct {
Response *DavFile `json:"response"`
}
if err := json.Unmarshal(raw, &env); err != nil {
return nil, err
}
if env.Response == nil {
return nil, fmt.Errorf("onlyoffice: empty upload response")
}
return env.Response, nil
}
// DownloadDavFile streams the file identified by id to w, returning bytes copied.
func (c *Client) DownloadDavFile(ctx context.Context, id string, w io.Writer) (int64, error) {
file, err := c.GetFile(ctx, id)
if err != nil {
return 0, err
}
if file.ViewURL == nil || *file.ViewURL == "" {
return 0, fmt.Errorf("onlyoffice: file %s has no viewUrl", id)
}
u := c.resolveAPIURL(*file.ViewURL)
auth, err := c.authHeader()
if err != nil {
return 0, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return 0, err
}
req.Header.Set("Authorization", auth)
resp, err := c.client.Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return 0, fmt.Errorf("onlyoffice: download: %d", resp.StatusCode)
}
return io.Copy(w, resp.Body)
}
// --- internal helpers -------------------------------------------------------
// deleteJSON performs an authenticated DELETE with an optional JSON body.
func (c *Client) deleteJSON(ctx context.Context, path string, body any) (json.RawMessage, error) {
var reader io.Reader
if body != nil {
buf, err := json.Marshal(body)
if err != nil {
return nil, err
}
reader = bytes.NewReader(buf)
}
auth, err := c.authHeader()
if err != nil {
return nil, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL()+path, reader)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", auth)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
resp, err := c.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
raw, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
if resp.StatusCode >= 400 {
return nil, fmt.Errorf("DELETE %s: %d %s", path, resp.StatusCode, truncate(string(raw), 400))
}
return raw, nil
}
// uploadReader uploads a stream to path under the given form field name.
func (c *Client) uploadReader(ctx context.Context, path, fieldName, fileName string, src io.Reader) (json.RawMessage, error) {
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
part, err := mw.CreateFormFile(fieldName, fileName)
if err != nil {
return nil, err
}
if _, err := io.Copy(part, src); err != nil {
return nil, err
}
if err := mw.Close(); err != nil {
return nil, err
}
auth, err := c.authHeader()
if err != nil {
return nil, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL()+path, &buf)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", auth)
req.Header.Set("Content-Type", mw.FormDataContentType())
req.Header.Set("Accept", "application/json")
resp, err := c.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
raw, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
if resp.StatusCode >= 400 {
return nil, fmt.Errorf("upload %s: %d %s", path, resp.StatusCode, truncate(string(raw), 400))
}
return raw, nil
}
func nums(ids []string) []json.Number {
out := make([]json.Number, 0, len(ids))
for _, id := range ids {
if _, err := strconv.Atoi(id); err == nil {
out = append(out, json.Number(id))
}
}
return out
}
func num(id string) any {
if _, err := strconv.Atoi(id); err == nil {
return json.Number(id)
}
return id
}
// UnmarshalJSON decodes a folder from the portal envelope, including fields
// that the base FolderEntry omits (parentId, rootFolderType, access, ...).
func (f *DavFolder) UnmarshalJSON(b []byte) error {
var raw struct {
ID *json.Number `json:"id"`
Title *string `json:"title"`
ParentID *json.Number `json:"parentId"`
RootType *int `json:"rootFolderType"`
FilesCount *int `json:"filesCount"`
FoldersCount *int `json:"foldersCount"`
Access *int `json:"access"`
Shared *bool `json:"shared"`
Updated *string `json:"updated"`
}
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
if raw.ID != nil {
f.ID = raw.ID.String()
}
if raw.Title != nil {
f.Title = *raw.Title
}
if raw.ParentID != nil {
f.ParentID = raw.ParentID.String()
}
if raw.RootType != nil {
f.RootType = *raw.RootType
}
if raw.FilesCount != nil {
f.FilesCount = *raw.FilesCount
}
if raw.FoldersCount != nil {
f.FoldersCount = *raw.FoldersCount
}
if raw.Access != nil {
f.Access = *raw.Access
}
if raw.Shared != nil {
f.Shared = *raw.Shared
}
if raw.Updated != nil {
f.Updated = *raw.Updated
}
return nil
}
// UnmarshalJSON decodes a file row, capturing size and timestamps.
func (f *DavFile) UnmarshalJSON(b []byte) error {
var raw struct {
ID *json.Number `json:"id"`
Title *string `json:"title"`
PureSize *int64 `json:"pureContentLength"`
SizeStr *string `json:"contentLength"`
Updated *string `json:"updated"`
ViewURL *string `json:"viewUrl"`
}
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
if raw.ID != nil {
f.ID = raw.ID.String()
}
if raw.Title != nil {
f.Title = *raw.Title
}
if raw.PureSize != nil {
f.Size = *raw.PureSize
} else if raw.SizeStr != nil {
if n, err := strconv.ParseInt(strings.Fields(*raw.SizeStr)[0], 10, 64); err == nil {
f.Size = n
}
}
if raw.Updated != nil {
f.Updated = *raw.Updated
}
if raw.ViewURL != nil {
f.ViewURL = *raw.ViewURL
}
return nil
}
// ModTime parses the folder's updated timestamp.
func (f *DavFolder) ModTime() time.Time {
t, _ := time.Parse("2006-01-02T15:04:05.0000000-07:00", f.Updated)
return t
}
// ModTime parses the file's updated timestamp.
func (f *DavFile) ModTime() time.Time {
t, _ := time.Parse("2006-01-02T15:04:05.0000000-07:00", f.Updated)
return t
}
-1
View File
@@ -10,7 +10,6 @@ require (
github.com/charmbracelet/lipgloss v0.12.1
github.com/charmbracelet/x/ansi v0.1.4
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002
github.com/eslider/go-xls/v2 v2.1.0
github.com/google/go-querystring v1.2.0
github.com/joho/godotenv v1.5.1
-2
View File
@@ -37,8 +37,6 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3 h1:B9YK+Tck5mTccyDhtxBzWyqGYcFxLyB6+noMNW4/VgI=
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3/go.mod h1:HMJKR5wlh/ziNp+sHEDV2ltblO4JD2+IdDOWtGcQBTM=
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002 h1:LOFxQG4mxvlH7+lffbO2SIn2ThlClJlrHHfs1OqMNXs=
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002/go.mod h1:fIgfH/E1j3rU8du4X4+7mxTD0GPtPQibTzytgitdJWU=
github.com/eslider/go-xls/v2 v2.1.0 h1:HszWKqYQbXxACmAXXWdMsfNl1NDBfGVBnJUPtyUHQ7A=
github.com/eslider/go-xls/v2 v2.1.0/go.mod h1:xgxO6JrfuBr9jGUB+0z5l/yDmFFZ5diGk0ATGihxlMU=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
-419
View File
@@ -1,419 +0,0 @@
// Package docpipe converts documents for OnlyOffice agent workflows:
// Markdown ↔ DOCX (pandoc) and image/PDF OCR → searchable PDF + Markdown text.
//
// External tools (optional at runtime; helpers skip/error clearly when missing):
// - pandoc — md↔docx
// - ocrmypdf — OCR into a searchable PDF
// - pdftotext — extract text layer
// - tesseract — OCR single images when ocrmypdf is unsuitable
// - ghostscript (gs) — PDF rewrite/optimize via PostScript (pdfwrite)
package docpipe
import (
"bytes"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
)
// DefaultMinTextChars: below this, a PDF is treated as needing OCR.
const DefaultMinTextChars = 200
// Tools reports which converters are available on PATH.
type Tools struct {
Pandoc string
OCRMyPDF string
PDFToText string
Tesseract string
Ghostscript string
}
// LookPath resolves converter binaries (empty string if missing).
func LookPath() Tools {
find := func(names ...string) string {
for _, n := range names {
if p, err := exec.LookPath(n); err == nil {
return p
}
}
return ""
}
return Tools{
Pandoc: find("pandoc"),
OCRMyPDF: find("ocrmypdf"),
PDFToText: find("pdftotext"),
Tesseract: find("tesseract"),
Ghostscript: find("gs", "ghostscript"),
}
}
func (t Tools) requirePandoc() error {
if t.Pandoc == "" {
return fmt.Errorf("pandoc not found on PATH (needed for md↔docx)")
}
return nil
}
// Ext returns lower-case extension including dot (".pdf").
func Ext(path string) string {
return strings.ToLower(filepath.Ext(path))
}
// ConvertFile converts between md and docx (and other pandoc formats) via pandoc.
// outExt may be ".md", ".docx", or a full output path.
func (t Tools) ConvertFile(inPath, outPath string) error {
if err := t.requirePandoc(); err != nil {
return err
}
if strings.TrimSpace(outPath) == "" {
return fmt.Errorf("output path required")
}
cmd := exec.Command(t.Pandoc, inPath, "-o", outPath)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return fmt.Errorf("pandoc %s → %s: %w (%s)", inPath, outPath, err, strings.TrimSpace(stderr.String()))
}
return nil
}
// TXTToDOCX converts plain text to DOCX preserving line breaks (via markdown hard breaks).
func (t Tools) TXTToDOCX(txtPath, docxPath string) error {
if Ext(txtPath) != ".txt" {
return fmt.Errorf("expected .txt input, got %q", txtPath)
}
if docxPath == "" {
docxPath = strings.TrimSuffix(txtPath, Ext(txtPath)) + ".docx"
}
b, err := os.ReadFile(txtPath)
if err != nil {
return err
}
dir := filepath.Dir(docxPath)
if dir == "" || dir == "." {
dir = os.TempDir()
}
tmpMD := filepath.Join(dir, trimExt(filepath.Base(txtPath))+".txt2docx.md")
md := TxtToMarkdown(string(b))
if err := os.WriteFile(tmpMD, []byte(md), 0o644); err != nil {
return err
}
defer os.Remove(tmpMD)
return t.MDToDOCX(tmpMD, docxPath)
}
// TxtToMarkdown converts plain text to Markdown for DOCX output.
// Prose text: each line is a hard break. Fixed-width extracts (INE, pdftotext -layout):
// wrapped in a fenced code block (monospace, columns preserved).
func TxtToMarkdown(content string) string {
content = normalizeTxtNewlines(content)
if isFixedWidthTxt(content) {
return "```\n" + content + "\n```\n"
}
var b strings.Builder
for _, line := range strings.Split(content, "\n") {
if strings.TrimSpace(line) == "" {
b.WriteByte('\n')
continue
}
b.WriteString(line)
b.WriteString(" \n")
}
return b.String()
}
func normalizeTxtNewlines(content string) string {
content = strings.ReplaceAll(content, "\r\n", "\n")
return strings.ReplaceAll(content, "\r", "\n")
}
// isFixedWidthTxt detects pdftotext -layout style extracts (many indented/spaced columns).
func isFixedWidthTxt(content string) bool {
lines := strings.Split(content, "\n")
if len(lines) < 8 {
return false
}
indented, long := 0, 0
for _, line := range lines {
if strings.TrimSpace(line) == "" {
continue
}
if len(line) >= 72 {
long++
}
if len(line) > 0 && (line[0] == ' ' || line[0] == '\t') {
indented++
}
}
n := len(lines)
return indented*100/n >= 20 || (long >= 5 && indented*100/n >= 10)
}
// MDToDOCX writes a DOCX next to or at outPath from a Markdown file.
func (t Tools) MDToDOCX(mdPath, docxPath string) error {
if Ext(mdPath) != ".md" && Ext(mdPath) != ".markdown" {
return fmt.Errorf("expected markdown input, got %q", mdPath)
}
if docxPath == "" {
docxPath = strings.TrimSuffix(mdPath, Ext(mdPath)) + ".docx"
}
return t.ConvertFile(mdPath, docxPath)
}
// DOCXToMD writes Markdown from a DOCX file.
func (t Tools) DOCXToMD(docxPath, mdPath string) error {
if Ext(docxPath) != ".docx" {
return fmt.Errorf("expected .docx input, got %q", docxPath)
}
if mdPath == "" {
mdPath = strings.TrimSuffix(docxPath, Ext(docxPath)) + ".md"
}
return t.ConvertFile(docxPath, mdPath)
}
// OptimizePDF rewrites a PDF through Ghostscript (PostScript pdfwrite).
// Preserves native text layers; strips broken OCR overlays; shrinks for OO preview.
// Use instead of ocrmypdf when pdftotext already extracts enough text.
func (t Tools) OptimizePDF(inPath, outPath string) error {
if t.Ghostscript == "" {
return fmt.Errorf("ghostscript (gs) not found on PATH")
}
if outPath == "" {
return fmt.Errorf("output PDF path required")
}
args := []string{
"-sDEVICE=pdfwrite",
"-dCompatibilityLevel=1.5",
"-dNOPAUSE", "-dQUIET", "-dBATCH",
"-dPDFSETTINGS=/ebook",
"-dEmbedAllFonts=true",
"-dSubsetFonts=true",
"-dCompressFonts=true",
"-dCompressPages=true",
"-dDetectDuplicateImages=true",
"-dAutoRotatePages=/None",
"-sOutputFile=" + outPath,
inPath,
}
cmd := exec.Command(t.Ghostscript, args...)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return fmt.Errorf("ghostscript pdfwrite: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
return nil
}
// PDFTextLayerChars returns approximate extracted character count (0 if unavailable).
func (t Tools) PDFTextLayerChars(pdfPath string) (int, error) {
if t.PDFToText == "" {
return 0, fmt.Errorf("pdftotext not found on PATH")
}
cmd := exec.Command(t.PDFToText, "-layout", pdfPath, "-")
out, err := cmd.Output()
if err != nil {
return 0, err
}
return len(bytes.TrimSpace(out)), nil
}
// NeedsOCR reports whether path likely needs OCR before text extraction.
func (t Tools) NeedsOCR(path string, minChars int) (bool, error) {
if minChars <= 0 {
minChars = DefaultMinTextChars
}
switch Ext(path) {
case ".jpg", ".jpeg", ".png", ".tif", ".tiff", ".webp", ".gif", ".bmp":
return true, nil
case ".pdf":
n, err := t.PDFTextLayerChars(path)
if err != nil {
// If we cannot measure, prefer OCR.
return true, nil
}
return n < minChars, nil
default:
return false, nil
}
}
// OCRToPDF runs ocrmypdf into outPDF (searchable). Forces OCR when force is true.
func (t Tools) OCRToPDF(inPath, outPDF string, force bool, lang string) error {
if t.OCRMyPDF == "" {
return fmt.Errorf("ocrmypdf not found on PATH")
}
if outPDF == "" {
return fmt.Errorf("output PDF path required")
}
if lang == "" {
lang = "eng"
}
args := []string{"-l", lang, "--skip-big", "100"}
if force {
args = append(args, "--force-ocr")
} else {
args = append(args, "--skip-text")
}
args = append(args, inPath, outPDF)
cmd := exec.Command(t.OCRMyPDF, args...)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
// Retry with force if skip-text refused.
if !force && strings.Contains(stderr.String(), "PriorOcrFoundError") == false {
args2 := []string{"-l", lang, "--force-ocr", inPath, outPDF}
cmd2 := exec.Command(t.OCRMyPDF, args2...)
var stderr2 bytes.Buffer
cmd2.Stderr = &stderr2
if err2 := cmd2.Run(); err2 == nil {
return nil
}
}
return fmt.Errorf("ocrmypdf: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
return nil
}
// ImageToText OCRs a raster image with tesseract (stdout text).
func (t Tools) ImageToText(imgPath, lang string) (string, error) {
if t.Tesseract == "" {
return "", fmt.Errorf("tesseract not found on PATH")
}
if lang == "" {
lang = "eng"
}
cmd := exec.Command(t.Tesseract, imgPath, "stdout", "-l", lang)
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.Output()
if err != nil {
return "", fmt.Errorf("tesseract: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
return string(out), nil
}
// ExtractPDFText returns layout text from a PDF via pdftotext.
func (t Tools) ExtractPDFText(pdfPath string) (string, error) {
if t.PDFToText == "" {
return "", fmt.Errorf("pdftotext not found on PATH")
}
cmd := exec.Command(t.PDFToText, "-layout", pdfPath, "-")
out, err := cmd.Output()
if err != nil {
return "", err
}
return string(out), nil
}
// Result of ToMarkdown.
type Result struct {
Markdown string
OCRPDFPath string // set when a searchable PDF was produced
DidOCR bool
Source string
}
// ToMarkdown turns a local file into Markdown text.
// PDFs/images with weak/no text layer are OCR'd to a searchable PDF first (when tools exist).
func (t Tools) ToMarkdown(path string, workDir string, lang string, minChars int) (Result, error) {
res := Result{Source: path}
ext := Ext(path)
switch ext {
case ".md", ".markdown", ".txt":
b, err := os.ReadFile(path)
if err != nil {
return res, err
}
res.Markdown = string(b)
return res, nil
case ".docx", ".odt", ".rtf", ".html", ".htm":
if err := t.requirePandoc(); err != nil {
return res, err
}
tmp := filepath.Join(workDir, "out.md")
if err := t.ConvertFile(path, tmp); err != nil {
return res, err
}
b, err := os.ReadFile(tmp)
if err != nil {
return res, err
}
res.Markdown = string(b)
return res, nil
case ".pdf":
need, _ := t.NeedsOCR(path, minChars)
pdf := path
if need {
if workDir == "" {
workDir = os.TempDir()
}
outPDF := filepath.Join(workDir, trimExt(filepath.Base(path))+".ocr.pdf")
if err := t.OCRToPDF(path, outPDF, true, lang); err != nil {
return res, err
}
res.DidOCR = true
res.OCRPDFPath = outPDF
pdf = outPDF
}
text, err := t.ExtractPDFText(pdf)
if err != nil {
return res, err
}
res.Markdown = wrapMD(filepath.Base(path), text)
return res, nil
case ".jpg", ".jpeg", ".png", ".tif", ".tiff", ".webp", ".gif", ".bmp":
if workDir == "" {
workDir = os.TempDir()
}
outPDF := filepath.Join(workDir, trimExt(filepath.Base(path))+".ocr.pdf")
if t.OCRMyPDF != "" {
if err := t.OCRToPDF(path, outPDF, true, lang); err == nil {
res.DidOCR = true
res.OCRPDFPath = outPDF
text, err := t.ExtractPDFText(outPDF)
if err != nil {
return res, err
}
res.Markdown = wrapMD(filepath.Base(path), text)
return res, nil
}
}
text, err := t.ImageToText(path, lang)
if err != nil {
return res, err
}
res.DidOCR = true
res.Markdown = wrapMD(filepath.Base(path), text)
return res, nil
default:
return res, fmt.Errorf("unsupported type %q for markdown extraction", ext)
}
}
func wrapMD(title, body string) string {
body = strings.TrimSpace(body)
if body == "" {
return "# " + title + "\n\n_(empty text layer)_\n"
}
return "# " + title + "\n\n" + body + "\n"
}
func trimExt(name string) string {
return strings.TrimSuffix(name, filepath.Ext(name))
}
// SiblingDOCX returns path with .docx extension replacing the original ext.
func SiblingDOCX(mdPath string) string {
return strings.TrimSuffix(mdPath, Ext(mdPath)) + ".docx"
}
// EnsureDir creates parent directories for path.
func EnsureDir(path string) error {
dir := filepath.Dir(path)
if dir == "" || dir == "." {
return nil
}
return os.MkdirAll(dir, 0o755)
}
-167
View File
@@ -1,167 +0,0 @@
package docpipe
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestExt(t *testing.T) {
if Ext("Foo.PDF") != ".pdf" {
t.Fatalf("Ext: %q", Ext("Foo.PDF"))
}
}
func TestSiblingDOCX(t *testing.T) {
if got := SiblingDOCX("notes.md"); got != "notes.docx" {
t.Fatalf("got %q", got)
}
}
func TestWrapMD(t *testing.T) {
s := wrapMD("a.pdf", " hello ")
if !strings.HasPrefix(s, "# a.pdf\n") || !strings.Contains(s, "hello") {
t.Fatalf("wrap: %q", s)
}
}
func TestNeedsOCR_Image(t *testing.T) {
tools := LookPath()
need, err := tools.NeedsOCR("x.jpg", 0)
if err != nil || !need {
t.Fatalf("jpg should need OCR: need=%v err=%v", need, err)
}
}
func TestTxtToMarkdown_FixedWidthUsesCodeBlock(t *testing.T) {
var lines []string
for i := 0; i < 12; i++ {
lines = append(lines, " column layout line "+strings.Repeat("x", 40))
}
in := strings.Join(lines, "\n")
md := TxtToMarkdown(in)
if !strings.HasPrefix(md, "```\n") || !strings.Contains(md, "```") {
t.Fatalf("expected code block: %q", md[:min(80, len(md))])
}
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
func TestTxtToMarkdownPreservesLines(t *testing.T) {
in := "line1\nline2\n\nline4"
md := TxtToMarkdown(in)
if !strings.Contains(md, "line1 \n") || !strings.Contains(md, "line2 \n") {
t.Fatalf("hard breaks missing: %q", md)
}
if !strings.Contains(md, "line4 \n") {
t.Fatalf("last line: %q", md)
}
}
func TestTXTToDOCXPreservesLines(t *testing.T) {
tools := LookPath()
if tools.Pandoc == "" {
t.Skip("pandoc not installed")
}
dir := t.TempDir()
txt := filepath.Join(dir, "sample.txt")
docx := filepath.Join(dir, "sample.docx")
body := "MyBox Auto — resumen\nNº contrato: 123\n\nEstado: Vigente\n"
if err := os.WriteFile(txt, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
if err := tools.TXTToDOCX(txt, docx); err != nil {
t.Fatal(err)
}
mdOut := filepath.Join(dir, "out.md")
if err := tools.DOCXToMD(docx, mdOut); err != nil {
t.Fatal(err)
}
got, err := os.ReadFile(mdOut)
if err != nil {
t.Fatal(err)
}
s := string(got)
for _, want := range []string{"MyBox Auto", "Nº contrato", "Estado: Vigente"} {
if !strings.Contains(s, want) {
t.Fatalf("missing %q in %q", want, s)
}
}
if strings.Contains(s, "MyBox Auto — resumen Nº") {
t.Fatalf("lines collapsed: %q", s)
}
}
func TestMDDocxRoundTrip(t *testing.T) {
tools := LookPath()
if tools.Pandoc == "" {
t.Skip("pandoc not installed")
}
dir := t.TempDir()
md := filepath.Join(dir, "n.md")
docx := filepath.Join(dir, "n.docx")
md2 := filepath.Join(dir, "n2.md")
if err := os.WriteFile(md, []byte("# Title\n\nHello **world**.\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := tools.MDToDOCX(md, docx); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(docx); err != nil {
t.Fatal(err)
}
if err := tools.DOCXToMD(docx, md2); err != nil {
t.Fatal(err)
}
b, err := os.ReadFile(md2)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), "Hello") {
t.Fatalf("round-trip missing Hello: %s", b)
}
}
func TestOptimizePDF(t *testing.T) {
tools := LookPath()
if tools.Ghostscript == "" || tools.PDFToText == "" {
t.Skip("ghostscript/pdftotext not installed")
}
in := "/tmp/ccgg-original.pdf"
if _, err := os.Stat(in); err != nil {
t.Skip("local fixture not present")
}
dir := t.TempDir()
out := filepath.Join(dir, "out.pdf")
charsIn, err := tools.PDFTextLayerChars(in)
if err != nil || charsIn < 1000 {
t.Skip("fixture has no text layer")
}
if err := tools.OptimizePDF(in, out); err != nil {
t.Fatal(err)
}
charsOut, err := tools.PDFTextLayerChars(out)
if err != nil {
t.Fatal(err)
}
if charsOut < charsIn/2 {
t.Fatalf("text layer lost: in=%d out=%d", charsIn, charsOut)
}
}
func TestToMarkdown_PlainMD(t *testing.T) {
tools := LookPath()
dir := t.TempDir()
p := filepath.Join(dir, "a.md")
_ = os.WriteFile(p, []byte("hi"), 0o644)
res, err := tools.ToMarkdown(p, dir, "eng", 0)
if err != nil || res.Markdown != "hi" {
t.Fatalf("got %+v err=%v", res, err)
}
}
-153
View File
@@ -1,153 +0,0 @@
package docpipe
import (
"bytes"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
hocr "github.com/eslider/go-hocr"
)
// HOCRResult is structured OCR output for agents.
type HOCRResult struct {
HOCRPath string
Markdown string
YAML string
DidOCR bool
Source string
}
// ImageToHOCR runs tesseract hOCR into outBase+".hocr" (tesseract adds the extension).
// outBase must not include ".hocr". dpi 0 uses tesseract default; phone photos often need 200–300.
func (t Tools) ImageToHOCR(imgPath, outBase, lang string, dpi int) (string, error) {
if t.Tesseract == "" {
return "", fmt.Errorf("tesseract not found on PATH")
}
if lang == "" {
lang = "eng"
}
if outBase == "" {
return "", fmt.Errorf("hOCR output base path required")
}
args := []string{imgPath, outBase, "-l", lang}
if dpi > 0 {
args = append(args, "--dpi", fmt.Sprintf("%d", dpi))
}
args = append(args, resolveHOCRConfig())
cmd := exec.Command(t.Tesseract, args...)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return "", fmt.Errorf("tesseract hocr: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
out := outBase + ".hocr"
if _, err := os.Stat(out); err != nil {
// Some builds write .html
alt := outBase + ".html"
if _, err2 := os.Stat(alt); err2 == nil {
return alt, nil
}
return "", fmt.Errorf("tesseract hocr: missing output %s (%s)", out, strings.TrimSpace(stderr.String()))
}
return out, nil
}
// resolveHOCRConfig returns a tesseract config path that works when TESSDATA_PREFIX
// points at a custom traineddata dir without relative "hocr" configs.
func resolveHOCRConfig() string {
var candidates []string
if p := strings.TrimSpace(os.Getenv("TESSDATA_PREFIX")); p != "" {
candidates = append(candidates,
filepath.Join(p, "configs", "hocr"),
filepath.Join(p, "tessdata", "configs", "hocr"),
)
}
candidates = append(candidates,
"/usr/share/tesseract-ocr/5/tessdata/configs/hocr",
"/usr/share/tesseract-ocr/4.00/tessdata/configs/hocr",
"/usr/share/tessdata/configs/hocr",
)
for _, c := range candidates {
if _, err := os.Stat(c); err == nil {
return c
}
}
return "hocr"
}
// HOCRToMarkdown parses an hOCR file via go-hocr and returns Markdown (+ optional YAML).
// Words with confidence in (0, minConf) are dropped; minConf 0 keeps all.
func HOCRToMarkdown(hocrPath string, minConf float32) (md string, yml string, err error) {
doc, err := hocr.ReadFile(hocrPath)
if err != nil {
return "", "", fmt.Errorf("go-hocr: %w", err)
}
md = doc.ToMarkdown(minConf)
yml, err = doc.ToYaml()
if err != nil {
return md, "", err
}
return md, yml, nil
}
// ToHOCRMarkdown OCRs an image (or rasterizes first PDF page) to hOCR → Markdown/YAML.
func (t Tools) ToHOCRMarkdown(path, workDir, lang string, dpi int, minConf float32) (HOCRResult, error) {
res := HOCRResult{Source: path}
if workDir == "" {
workDir = os.TempDir()
}
ext := Ext(path)
img := path
switch ext {
case ".jpg", ".jpeg", ".png", ".tif", ".tiff", ".webp", ".gif", ".bmp":
// ok
case ".pdf":
raster, err := t.pdfFirstPagePNG(path, workDir)
if err != nil {
return res, err
}
img = raster
if dpi == 0 {
dpi = 300
}
default:
return res, fmt.Errorf("hOCR path expects image or PDF, got %q", ext)
}
base := filepath.Join(workDir, trimExt(filepath.Base(path))+".hocr-out")
hocrPath, err := t.ImageToHOCR(img, base, lang, dpi)
if err != nil {
return res, err
}
res.DidOCR = true
res.HOCRPath = hocrPath
md, yml, err := HOCRToMarkdown(hocrPath, minConf)
if err != nil {
return res, err
}
res.Markdown = wrapMD(filepath.Base(path), strings.TrimSpace(md))
res.YAML = yml
return res, nil
}
// pdfFirstPagePNG uses pdftoppm when available.
func (t Tools) pdfFirstPagePNG(pdfPath, workDir string) (string, error) {
pdftoppm, err := exec.LookPath("pdftoppm")
if err != nil {
return "", fmt.Errorf("pdftoppm not found (needed to rasterize PDF for hOCR)")
}
outBase := filepath.Join(workDir, trimExt(filepath.Base(pdfPath))+".page")
cmd := exec.Command(pdftoppm, "-png", "-f", "1", "-singlefile", "-r", "200", pdfPath, outBase)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return "", fmt.Errorf("pdftoppm: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
png := outBase + ".png"
if _, err := os.Stat(png); err != nil {
return "", fmt.Errorf("pdftoppm: missing %s", png)
}
return png, nil
}
-49
View File
@@ -1,49 +0,0 @@
package docpipe
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestHOCRToMarkdown_Fixture(t *testing.T) {
// Minimal hOCR 1.2 snippet
hocrXML := `<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>tesseract</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<meta name="ocr-system" content="tesseract 5"/>
<meta name="ocr-capabilities" content="ocr_page ocr_carea ocr_par ocr_line ocrx_word"/>
</head>
<body>
<div class="ocr_page" id="page_1" title="image &quot;x.png&quot;; bbox 0 0 100 50; ppageno 0">
<div class="ocr_carea" id="block_1_1" title="bbox 0 0 100 50">
<p class="ocr_par" id="par_1_1" lang="eng" title="bbox 0 0 100 50">
<span class="ocr_line" id="line_1_1" title="bbox 0 0 100 20; baseline 0 0; x_size 20">
<span class="ocrx_word" id="word_1_1" title="bbox 0 0 40 20; x_wconf 96">Hello</span>
<span class="ocrx_word" id="word_1_2" title="bbox 45 0 100 20; x_wconf 92">world</span>
</span>
</p>
</div>
</div>
</body>
</html>`
dir := t.TempDir()
p := filepath.Join(dir, "sample.hocr")
if err := os.WriteFile(p, []byte(hocrXML), 0o644); err != nil {
t.Fatal(err)
}
md, yml, err := HOCRToMarkdown(p, 0)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(md, "Hello") || !strings.Contains(md, "world") {
t.Fatalf("md=%q", md)
}
if !strings.Contains(yml, "Hello") {
t.Fatalf("yaml missing word: %s", yml)
}
}
+1 -44
View File
@@ -8,8 +8,8 @@ import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/mail"
"net/http"
"net/url"
"strconv"
"strings"
@@ -193,49 +193,6 @@ func (c *Client) SaveMailDraft(ctx context.Context, p SaveMailDraftParams) (map[
return c.putJSONObject(ctx, "/api/2.0/mail/drafts/save", body)
}
// SendMailParams describes a message to send via PUT /api/2.0/mail/messages/send.
// ID refers to an existing draft/message id; From falls back to the first enabled
// mailbox. Cc/Bcc are omitted when empty (the API 400s on empty strings). Chat
// line goes into Body (API send does not append the UI signature).
type SendMailParams struct {
ID int64
From string
To string
Cc string
Bcc string
Subject string
Body string // HTML
}
// SendMail sends an existing draft (or a fresh message) via the OnlyOffice Mail
// send endpoint. Returns the raw send response.
func (c *Client) SendMail(ctx context.Context, p SendMailParams) (json.RawMessage, error) {
if strings.TrimSpace(p.To) == "" {
return nil, fmt.Errorf("SendMail: to is required")
}
if strings.TrimSpace(p.From) == "" {
from, err := c.defaultMailFrom(ctx)
if err != nil {
return nil, err
}
p.From = from
}
body := map[string]any{
"id": p.ID,
"from": p.From,
"to": p.To,
"subject": p.Subject,
"body": p.Body,
}
if strings.TrimSpace(p.Cc) != "" {
body["cc"] = p.Cc
}
if strings.TrimSpace(p.Bcc) != "" {
body["bcc"] = p.Bcc
}
return c.putJSON(ctx, "/api/2.0/mail/messages/send.json", body)
}
func (c *Client) defaultMailFrom(ctx context.Context) (string, error) {
accounts, err := c.ListMailAccounts(ctx)
if err != nil {
-53
View File
@@ -2,7 +2,6 @@ package onlyoffice
import (
"context"
"encoding/json"
"net/http"
"net/http/cookiejar"
"net/http/httptest"
@@ -160,55 +159,3 @@ func TestDownloadMailAttachmentUsesAuthCookie(t *testing.T) {
t.Fatalf("path = %q", gotPath)
}
}
func TestSendMailOmitsEmptyCcBcc(t *testing.T) {
var gotBody map[string]any
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/2.0/authentication.json":
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"response":{"token":"tok","expires":"2099-01-01T00:00:00.0000000+00:00"}}`))
case "/api/2.0/mail/messages/send.json":
gotPath = r.URL.Path
dec := json.NewDecoder(r.Body)
_ = dec.Decode(&gotBody)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"response":{"id":1}}`))
default:
http.NotFound(w, r)
}
}))
defer srv.Close()
c := NewClient(Credentials{Url: srv.URL, User: "u", Password: "p"})
ctx := context.Background()
raw, err := c.SendMail(ctx, SendMailParams{
ID: 99,
From: "me@x.com",
To: "a@b.com",
Subject: "hi",
Body: "<p>hello</p>",
})
if err != nil {
t.Fatalf("SendMail: %v", err)
}
if gotPath != "/api/2.0/mail/messages/send.json" {
t.Fatalf("path = %q", gotPath)
}
if _, hasCC := gotBody["cc"]; hasCC {
t.Fatalf("empty cc should be omitted: %v", gotBody)
}
if _, hasBcc := gotBody["bcc"]; hasBcc {
t.Fatalf("empty bcc should be omitted: %v", gotBody)
}
if gotBody["to"] != "a@b.com" {
t.Fatalf("to = %v", gotBody["to"])
}
if gotBody["id"] != float64(99) {
t.Fatalf("id = %v", gotBody["id"])
}
if !strings.Contains(string(raw), `"id"`) {
t.Fatalf("raw = %s", raw)
}
}
+175
View File
@@ -0,0 +1,175 @@
package onlyoffice
// High-level mail folder walk for ETL consumers (2dph brain mail-ingest,
// cv tools). This is the "integration layer" half of reusing the canonical
// client instead of private per-project OOClient copies: the caller gets a
// single hydrated stream instead of hand-rolling list → get → download
// against the raw API.
import (
"context"
"fmt"
"strconv"
"time"
)
// MailSyncAttachment is one attachment of a hydrated mail message.
type MailSyncAttachment struct {
ID string // id accepted by Client.DownloadMailAttachment
Name string
Size int64
Body []byte // non-nil only when MailSyncOptions.FetchBodies is set
}
// MailSyncMessage is a hydrated mail message for sync pipelines.
type MailSyncMessage struct {
ID int64
Folder int
Subject string
From string // raw RFC 5322 header value ("Name" <addr>)
Date time.Time
IsNew bool
HasAttachments bool
Attachments []MailSyncAttachment
}
// MailSyncOptions controls FetchMailFolder.
type MailSyncOptions struct {
Limit int // max messages to hydrate; 0 = whole folder
StartIndex int // skip this many messages before collecting
FetchBodies bool // eagerly download attachment bytes
}
// FetchMailFolder walks a mail folder page by page and hydrates every
// message: list → get → (optionally) download attachments. It is the single
// entry point sync pipelines need on top of the mail API.
//
// Messages are returned in API order (newest first). The folder walk stops
// at the first empty or short page.
func (c *Client) FetchMailFolder(ctx context.Context, folderID int, opts MailSyncOptions) ([]MailSyncMessage, error) {
if folderID <= 0 {
folderID = MailFolderInbox
}
var out []MailSyncMessage
skipped := 0
for page := 1; ; page++ {
batch, err := c.ResponseArray(ctx,
mailMessagesPath(MailMessagesFilter{Folder: folderID}, page, mailMessagesPageSize))
if err != nil {
return nil, fmt.Errorf("FetchMailFolder: %w", err)
}
if len(batch) == 0 {
break
}
for _, raw := range batch {
if skipped < opts.StartIndex {
skipped++
continue
}
msg, err := c.hydrateMailMessage(ctx, raw, opts)
if err != nil {
return nil, err
}
out = append(out, *msg)
if opts.Limit > 0 && len(out) >= opts.Limit {
return out, nil
}
}
if len(batch) < mailMessagesPageSize {
break
}
}
return out, nil
}
// hydrateMailMessage converts one raw API message into a MailSyncMessage,
// fetching the full record when the list item does not carry the attachment
// metadata, and downloading bodies when requested.
func (c *Client) hydrateMailMessage(ctx context.Context, m map[string]any, opts MailSyncOptions) (*MailSyncMessage, error) {
msg := &MailSyncMessage{
ID: Int64FromMap(m, "id"),
Folder: int(Int64FromMap(m, "folder")),
Subject: stringFromMap(m, "subject"),
From: stringFromMap(m, "from"),
IsNew: boolFromMap(m, "isNew") == "true",
}
msg.Date = parseMailTime(stringFromMap(m, "date"))
atts, _ := m["attachments"].([]any)
hasFlag := boolFromMap(m, "hasAttachments") == "true"
if hasFlag && len(atts) == 0 {
// List items may omit the attachment array; pull the full record.
full, err := c.GetMailMessage(ctx, strconv.FormatInt(msg.ID, 10))
if err != nil {
return nil, fmt.Errorf("FetchMailFolder: hydrate message %d: %w", msg.ID, err)
}
atts, _ = full["attachments"].([]any)
}
for _, a := range atts {
am, ok := a.(map[string]any)
if !ok {
continue
}
att := MailSyncAttachment{
ID: mailAttachmentID(am),
Name: stringFromMap(am, "fileName"),
Size: Int64FromMap(am, "size"),
}
if att.Name == "" {
att.Name = stringFromMap(am, "name")
}
if att.ID != "" {
msg.Attachments = append(msg.Attachments, att)
}
}
msg.HasAttachments = hasFlag || len(msg.Attachments) > 0
if opts.FetchBodies {
for i := range msg.Attachments {
body, err := c.DownloadMailAttachment(ctx, msg.Attachments[i].ID)
if err != nil {
return nil, fmt.Errorf("FetchMailFolder: message %d attachment %q: %w",
msg.ID, msg.Attachments[i].Name, err)
}
msg.Attachments[i].Body = body
}
}
return msg, nil
}
// mailAttachmentID extracts the download id from an attachment object.
// OnlyOffice variants use "id", "fileId" or "attachmentId".
func mailAttachmentID(am map[string]any) string {
for _, key := range []string{"id", "fileId", "attachmentId"} {
switch v := am[key].(type) {
case string:
if s := v; s != "" {
return s
}
case float64:
if n := int64(v); n != 0 {
return strconv.FormatInt(n, 10)
}
case int64:
if v != 0 {
return strconv.FormatInt(v, 10)
}
}
}
return ""
}
// parseMailTime accepts the OnlyOffice timestamp shapes seen in the wild:
// RFC3339 (with any fractional digits) and second-precision local form.
func parseMailTime(s string) time.Time {
if s == "" {
return time.Time{}
}
if t, err := time.Parse(time.RFC3339, s); err == nil {
return t
}
if t, err := time.Parse("2006-01-02T15:04:05", s); err == nil {
return t
}
return time.Time{}
}
+137
View File
@@ -0,0 +1,137 @@
package onlyoffice
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// mailsSyncMock serves a two-page inbox: page 1 has two list items (one
// reporting hasAttachments but omitting the attachment array, forcing the
// full-record fetch), page 2 is empty. The full record for message 102
// carries one attachment whose body is served by download.ashx.
func newMailSyncTestServer(t *testing.T, msgsPage1 string) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/api/2.0/authentication.json":
http.SetCookie(w, &http.Cookie{Name: "sessionid", Value: "abc", Path: "/"})
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"response":{"token":"tok","expires":"2099-01-01T00:00:00.0000000+00:00"}}`))
case r.URL.Path == "/api/2.0/mail/messages":
w.Header().Set("Content-Type", "application/json")
if r.URL.Query().Get("page") > "1" {
_, _ = w.Write([]byte(`{"response":[]}`))
return
}
_, _ = w.Write([]byte(`{"response":[` + msgsPage1 + `]}`))
case r.URL.Path == "/api/2.0/mail/messages/102":
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"response":{
"id":102,"subject":"Full record","from":"\"A\" <a@b.com>",
"date":"2026-08-22T10:15:00+02:00","folder":1,"isNew":false,
"hasAttachments":true,
"attachments":[{"id":77,"fileName":"report.pdf","size":3}]}}`))
case r.URL.Path == "/addons/mail/httphandlers/download.ashx":
if r.Header.Get("Cookie") == "" {
http.Error(w, "missing cookie", http.StatusUnauthorized)
return
}
_, _ = w.Write([]byte("PDF!"))
default:
http.NotFound(w, r)
}
}))
}
func TestFetchMailFolderHydratesAndDownloads(t *testing.T) {
page1 := `
{"id":101,"subject":"Plain","from":"x@y.z","date":"2026-08-21T09:00:00Z",
"folder":1,"isNew":true,"hasAttachments":false},
{"id":102,"subject":"With attachment (list item)","from":"a@b.com",
"date":"2026-08-22T10:15:00+02:00","folder":1,"isNew":false,
"hasAttachments":true}
`
srv := newMailSyncTestServer(t, page1)
defer srv.Close()
c := NewClient(Credentials{Url: srv.URL, User: "u", Password: "p"})
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
msgs, err := c.FetchMailFolder(ctx, MailFolderInbox, MailSyncOptions{FetchBodies: true})
if err != nil {
t.Fatalf("FetchMailFolder: %v", err)
}
if len(msgs) != 2 {
t.Fatalf("got %d messages, want 2", len(msgs))
}
first := msgs[0]
if first.ID != 101 || first.Subject != "Plain" || !first.IsNew {
t.Fatalf("first = %+v", first)
}
if first.Date.IsZero() || first.Date.Year() != 2026 {
t.Fatalf("first date = %v", first.Date)
}
if first.HasAttachments {
t.Fatalf("first should have no attachments")
}
second := msgs[1]
if !second.HasAttachments || len(second.Attachments) != 1 {
t.Fatalf("second attachments = %+v", second.Attachments)
}
att := second.Attachments[0]
if att.ID != "77" || att.Name != "report.pdf" || att.Size != 3 || string(att.Body) != "PDF!" {
t.Fatalf("attachment = %+v", att)
}
if second.Date.Location() == time.UTC && second.Date.Hour() != 8 {
t.Fatalf("second date = %v (want +02:00 offset preserved)", second.Date)
}
}
func TestFetchMailFolderLimitAndStartIndex(t *testing.T) {
var items []string
for i := 1; i <= 5; i++ {
items = append(items, `{"id":`+string(rune('0'+i))+`,"subject":"m`+string(rune('0'+i))+`",
"from":"x@y.z","date":"2026-08-20T00:00:00Z","folder":1}`)
}
srv := newMailSyncTestServer(t, strings.Join(items, ","))
defer srv.Close()
c := NewClient(Credentials{Url: srv.URL, User: "u", Password: "p"})
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
got, err := c.FetchMailFolder(ctx, MailFolderInbox, MailSyncOptions{StartIndex: 1, Limit: 2})
if err != nil {
t.Fatalf("FetchMailFolder: %v", err)
}
if len(got) != 2 {
t.Fatalf("got %d messages, want 2", len(got))
}
if got[0].ID != 2 || got[1].ID != 3 {
t.Fatalf("ids = %d,%d want 2,3", got[0].ID, got[1].ID)
}
}
func TestParseMailTime(t *testing.T) {
fractions := "2026-08-22T10:15:00.1234567+02:00"
if parseMailTime(fractions).IsZero() {
t.Fatalf("RFC3339 with 7-digit fraction failed: %q", fractions)
}
if parseMailTime("2026-08-22T10:15:00").IsZero() {
t.Fatal("second-precision form failed")
}
if !parseMailTime("").IsZero() || !parseMailTime("garbage").IsZero() {
t.Fatal("unparseable input must yield zero time")
}
}
-26
View File
@@ -1,26 +0,0 @@
#!/usr/bin/env bash
#
# install.sh — symlinks the shared githooks (pre-push, pre-commit) into .git/hooks
# for this repository. Safe to run repeatedly.
#
# Usage:
# ./scripts/githooks/install.sh
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
SRC="$ROOT/scripts/githooks"
HOOKS="$ROOT/.git/hooks"
mkdir -p "$HOOKS"
chmod +x "$SRC"/secret-scan.sh "$SRC"/pre-push "$SRC"/pre-commit
for h in pre-push pre-commit; do
if [[ -e "$HOOKS/$h" ]] && [[ ! -L "$HOOKS/$h" ]]; then
echo "error: $HOOKS/$h already exists and is not a symlink; remove it first" >&2
exit 1
fi
ln -sfn "$SRC/$h" "$HOOKS/$h"
echo "installed $h -> $SRC/$h"
done
echo "githooks installed for $(basename "$ROOT")"
-20
View File
@@ -1,20 +0,0 @@
#!/usr/bin/env bash
#
# pre-commit git hook — blocks a commit if staged changes contain a secret.
# Scans only the staged (index) diff with gitleaks (via secret-scan.sh).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
ROOT="$(git rev-parse --show-toplevel)"
if [[ "$SCRIPT_DIR" == "$ROOT/scripts/githooks" ]]; then
SCAN="$SCRIPT_DIR/secret-scan.sh"
else
SCAN="$ROOT/scripts/githooks/secret-scan.sh"
fi
if ! "$SCAN" --staged; then
echo "pre-commit: LEAK FOUND in staged changes; commit BLOCKED. Remove the secret first." >&2
exit 1
fi
exit 0
-66
View File
@@ -1,66 +0,0 @@
#!/usr/bin/env bash
#
# pre-push git hook — blocks a push if any NEW commit leaks a secret.
#
# Reads the refs being pushed from stdin (format:
# <local ref> <local sha> <remote ref> <remote sha>
# per ref). Scans only the new commits with gitleaks (via secret-scan.sh) and
# aborts (exit 1) if anything is found.
#
# Install: ln -s ../../scripts/githooks/pre-push .git/hooks/pre-push
# (or run scripts/githooks/install.sh)
set -euo pipefail
# Resolve symlinks so the hook works whether copied into .git/hooks or
# symlinked from scripts/githooks (and in worktrees sharing the main repo hooks).
SCRIPT_DIR="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
ROOT="$(git rev-parse --show-toplevel)"
if [[ "$SCRIPT_DIR" == "$ROOT/scripts/githooks" ]]; then
SCAN="$SCRIPT_DIR/secret-scan.sh"
else
SCAN="$ROOT/scripts/githooks/secret-scan.sh"
fi
zero=0000000000000000000000000000000000000000
failed=0
while read -r local_ref local_sha remote_ref remote_sha; do
[[ -n "$local_sha" ]] || continue
# Deletion push — nothing to scan.
if [[ "$local_sha" == "$zero" ]]; then
continue
fi
# New branch (no remote ref yet): scan only the commits this branch ADDS over
# its merge-base with the integration branch (PR target), NOT all history.
# This keeps pre-existing historical leaks (see #140) from blocking new work.
if [[ "$remote_sha" == "$zero" ]]; then
base=""
for base_ref in origin/release/v1 origin/release/v2 origin/master origin/main; do
if git rev-parse --verify "$base_ref" >/dev/null 2>&1; then
base="$(git merge-base "$base_ref" "$local_sha" 2>/dev/null)"
break
fi
done
if [[ -z "$base" ]] && git rev-parse --verify origin/HEAD >/dev/null 2>&1; then
base="$(git merge-base origin/HEAD "$local_sha" 2>/dev/null)"
fi
[[ -z "$base" ]] && base="$(git rev-list --max-parents=0 "$local_sha" 2>/dev/null | tail -1)"
range="${base}..${local_sha}"
else
range="${remote_sha}..${local_sha}"
fi
echo "secret-scan: scanning new commits ${range} (ref ${local_ref})"
if ! "$SCAN" --range "$range"; then
echo "secret-scan: LEAK FOUND in ${local_ref}; push BLOCKED. Remove the secret before pushing." >&2
failed=1
fi
done
if [[ "$failed" -ne 0 ]]; then
exit 1
fi
exit 0
-55
View File
@@ -1,55 +0,0 @@
#!/usr/bin/env bash
#
# secret-scan.sh — shared secret scanner used by git hooks (pre-push, pre-commit)
# and by SE agents before any push.
#
# Scans ONLY the diff of new commits (or staged changes) with gitleaks, never the
# full history. Fails (exit non-zero) on any finding, so a leaking push is blocked.
#
# Uses the gitleaks Docker image (gitleaks/gitleaks) if docker is available,
# otherwise a locally installed `gitleaks` binary. No secret VALUES are ever
# printed: findings are emitted redacted.
#
# Usage:
# secret-scan.sh <range> scan a git log range, e.g. origin/release/v1..HEAD
# secret-scan.sh --staged scan staged (index) changes
# secret-scan.sh --all scan full history (warning: not for normal use)
#
# Exit codes: 0 = clean, 1 = leaks found (caller should abort), 2 = scan failed.
set -euo pipefail
GITLEAKS_IMAGE="zricethezav/gitleaks:latest"
run_gitleaks() {
# $@ = gitleaks args; runs in current dir (a git repo).
if command -v gitleaks >/dev/null 2>&1; then
gitleaks "$@"
elif command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then
docker run --rm -v "$PWD:/repo" -w /repo "$GITLEAKS_IMAGE" "$@"
else
echo "error: secret-scan: neither 'gitleaks' binary nor docker image available" >&2
exit 2
fi
}
mode="${1:---range}"
shift || true
case "$mode" in
--range)
range="${1:?usage: secret-scan.sh <range>}"
run_gitleaks detect --source "$PWD" --no-banner --redact --log-opts="$range" >&2
;;
--staged)
# Scan only staged (index) content: pipe `git diff --cached` through gitleaks --pipe.
git diff --cached --binary | run_gitleaks detect --pipe --no-banner --redact >&2
;;
--all)
run_gitleaks detect --source "$PWD" --no-banner --redact >&2
;;
*)
echo "error: secret-scan: unknown mode '$mode'" >&2
exit 2
;;
esac