Compare commits
40
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3cc288d281 | ||
|
|
ce4778bdf1 | ||
|
|
3fe43ee82c | ||
|
|
6d93ab5b1f | ||
|
|
95d79925ea | ||
|
|
bf3ef025aa | ||
|
|
ba29738482 | ||
|
|
72c7cd5173 | ||
|
|
b3dfff1a4c | ||
|
|
b5a61ad420 | ||
|
|
1aba545e1d | ||
|
|
94951fc6c5 | ||
|
|
d95985e6d0 | ||
|
|
57b383d579 | ||
|
|
a73f6c172d | ||
|
|
e49693ad2c | ||
|
|
b86d71211f | ||
|
|
a2b919478e | ||
|
|
ccabc11153 | ||
|
|
9a6da1a4f7 | ||
|
|
504d13ed08 | ||
|
|
efc0864d42 | ||
|
|
59caf5e560 | ||
|
|
e7803c5269 | ||
|
|
a08e7c49ad | ||
|
|
4310aa7002 | ||
|
|
1a7a962183 | ||
|
|
af8e3b053a | ||
|
|
8ac777c031 | ||
|
|
c576bfe2ee | ||
|
|
47a2c256f3 | ||
|
|
c1881e1e61 | ||
|
|
7f56332285 | ||
|
|
a5807b9c31 | ||
|
|
d650a16a36 | ||
|
|
3da11586f9 | ||
|
|
e9c969a89c | ||
|
|
4d91726179 | ||
|
|
4d8af7a2fe | ||
|
|
34745e349e |
@@ -25,3 +25,38 @@ ONLYOFFICE_PROJECT_ID=33
|
|||||||
# cmd/office TUI — optional Document Server for DOCX→HTML preview:
|
# cmd/office TUI — optional Document Server for DOCX→HTML preview:
|
||||||
# ONLYOFFICE_DOCS_URL=https://docs.example.com
|
# ONLYOFFICE_DOCS_URL=https://docs.example.com
|
||||||
# ONLYOFFICE_DOCS_SECRET=
|
# ONLYOFFICE_DOCS_SECRET=
|
||||||
|
|
||||||
|
# MinIO download fallback for the portal's stale AWS S3 consumer (older
|
||||||
|
# Documents folders). When the portal redirects to amazonaws.com with access
|
||||||
|
# key "minio" (403 InvalidAccessKeyId), files are fetched from the local MinIO
|
||||||
|
# store instead. Without a key/secret the fallback is disabled.
|
||||||
|
# MINIO_ENDPOINT=http://192.168.188.10:9000
|
||||||
|
# MINIO_BUCKET=office
|
||||||
|
# MINIO_ACCESS_KEY=
|
||||||
|
# MINIO_SECRET_KEY=
|
||||||
|
|
||||||
|
# oo search — direct Elasticsearch access for name + content search. ES lives
|
||||||
|
# inside the OnlyOffice VM on localhost:9200; expose it with an SSH tunnel
|
||||||
|
# (see docs/elasticsearch.md). ONLYOFFICE_ES_INDEX defaults to files_file.
|
||||||
|
# ONLYOFFICE_ES_URL=http://127.0.0.1:9200
|
||||||
|
# ONLYOFFICE_ES_INDEX=files_file
|
||||||
|
# ONLYOFFICE_TENANT=
|
||||||
|
|
||||||
|
# Read-only SQL file store over the Community Server database (see
|
||||||
|
# docs/community-server-db.md). The live portal runs MySQL; ONLYOFFICE_DSN is
|
||||||
|
# `user:pass@tcp(host:port)/onlyoffice?parseTime=true`, or a `postgres://` URL.
|
||||||
|
# ONLYOFFICE_DSN=
|
||||||
|
# ONLYOFFICE_PG_DRIVER= # postgres | mysql (auto-detected from DSN)
|
||||||
|
# ONLYOFFICE_PG_TENANT=1 # falls back to ONLYOFFICE_TENANT
|
||||||
|
# Alternatively build a PostgreSQL DSN from parts:
|
||||||
|
# ONLYOFFICE_PG_HOST=
|
||||||
|
# ONLYOFFICE_PG_PORT=5432
|
||||||
|
# ONLYOFFICE_PG_USER=
|
||||||
|
# ONLYOFFICE_PG_PASSWORD=
|
||||||
|
# ONLYOFFICE_PG_DBNAME=onlyoffice
|
||||||
|
# ONLYOFFICE_PG_SSLMODE=disable
|
||||||
|
|
||||||
|
# oo index / oo search --backend own — own full-text index for PDF/scans,
|
||||||
|
# filled by `oo index` from internal/docpipe (pdftotext + OCR). Defaults to
|
||||||
|
# oo_docs_text. Uses the same ONLYOFFICE_ES_URL.
|
||||||
|
# ONLYOFFICE_ES_TEXT_INDEX=oo_docs_text
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
github: eSlider
|
||||||
|
ko_fi: eslider
|
||||||
|
liberapay: eslider
|
||||||
|
patreon: eslider
|
||||||
|
custom:
|
||||||
|
- https://polar.sh/eslider
|
||||||
@@ -1,3 +1,3 @@
|
|||||||
{
|
{
|
||||||
".": "0.17.0"
|
".": "0.18.0"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,16 +9,17 @@ Canonical Go client for OnlyOffice Workspace (Projects + Calendar + CRM) and the
|
|||||||
- `request.go` — `Request`, `Query`, `Time`, `Token`, `MetaResponse`, `Permissions`.
|
- `request.go` — `Request`, `Query`, `Time`, `Token`, `MetaResponse`, `Permissions`.
|
||||||
- `auth.go` — `Authenticate`, `AuthenticateContext`, `InvalidateToken`, `Auth`, token lifecycle.
|
- `auth.go` — `Authenticate`, `AuthenticateContext`, `InvalidateToken`, `Auth`, token lifecycle.
|
||||||
- `http.go` — transport + DRY response decoders (`ResponseArray`/`ResponseObject`/`postFormObject`/`putFormObject`/`deleteObject`).
|
- `http.go` — transport + DRY response decoders (`ResponseArray`/`ResponseObject`/`postFormObject`/`putFormObject`/`deleteObject`).
|
||||||
- `projects.go`, `tasks.go`, `users.go`, `calendar.go`, `crm.go`, `files.go`, `mails.go`, `invoices.go` — typed / untyped domain methods. **`files.go`** — CRM opportunity upload plus **project/task Documents**. **`mails.go`** — OnlyOffice Workspace Mail. **`invoices.go`** — CRM invoices, PDF regen/cleanup, status. Association rules: [`docs/crm-associations.md`](docs/crm-associations.md).
|
- `projects.go`, `tasks.go`, `users.go`, `calendar.go`, `crm.go`, `files.go`, `files_webdav.go`, `files_stem.go`, `retry.go`, `mails.go`, `invoices.go` — typed / untyped domain methods. **`files.go`** — CRM opportunity upload plus **project/task Documents** (`UpdateFile`, `UploadToFolderReplacing`). **`files_webdav.go`** — Documents module by id (`ListDavFolder`, `MoveDavItems`/`CopyDavItems` with per-operation error surfacing, `ListFileOps`). **`retry.go`** — `DoRetry`: deterministic linear backoff (no jitter) on 429/502/503/504; every bulk tool routes API calls through it. **`mails.go`** — OnlyOffice Workspace Mail. **`invoices.go`** — CRM invoices, PDF regen/cleanup, status. Association rules: [`docs/crm-associations.md`](docs/crm-associations.md).
|
||||||
- Pure stdlib + `google/go-querystring`; no UI, no dotenv.
|
- Pure stdlib + `google/go-querystring`; no UI, no dotenv.
|
||||||
- **CLI — `cmd/oo/` as `package main`.** Cobra wrapper that loads `.env` via `godotenv` at startup. **Subject-based command tree** mirroring [`tea`](https://gitea.com/gitea/tea):
|
- **CLI — `cmd/oo/` as `package main`.** Cobra wrapper that loads `.env` via `godotenv` at startup. **Subject-based command tree** mirroring [`tea`](https://gitea.com/gitea/tea):
|
||||||
- `main.go` — entry point (docstring lists the command tree).
|
- `main.go` — entry point (docstring lists the command tree).
|
||||||
- `common.go` — `rootCmd`, `newOO`, `printTable`/`printObject`, `--output table|json` flag.
|
- `common.go` — `rootCmd`, `newOO`, `printTable`/`printObject`, `--output table|json` flag.
|
||||||
- `calendar.go`, `projects.go`, `projects_files.go`, `tasks.go`, `tasks_files.go`, `users.go`, `contacts.go`, `opportunities.go`, `cases.go`, `crm_tasks.go`, `mails.go`, `invoices.go` — one file per subject (or per subject facet), each registers in `init()`.
|
- `calendar.go`, `projects.go`, `projects_files.go`, `tasks.go`, `tasks_files.go`, `users.go`, `contacts.go`, `opportunities.go`, `cases.go`, `crm.go`, `crm_tasks.go`, `catalog.go`, `docs.go`, `dav.go`, `mails.go`, `invoices.go` — one file per subject (or per subject facet), each registers in `init()`. `dav.go` exposes the Documents module by id (`oo dav ls|move|copy|mkdir|rename-file|rename-folder|download|fileops`).
|
||||||
- CLI-only deps (`spf13/cobra`, `joho/godotenv`) stay out of the library.
|
- CLI-only deps (`spf13/cobra`, `joho/godotenv`) stay out of the library.
|
||||||
- **TUI — `cmd/office/` as `package main`.** Bubble Tea three-pane browser (module tree, selectable list, markdown preview). Reuses `cmd/internal/bootstrap` for env/auth and the root `onlyoffice` library for all API calls. UI logic in `cmd/office/ui/`; preview/formatting in `cmd/office/preview/`; list loaders in `cmd/office/fetch/`.
|
- **TUI — `cmd/office/` as `package main`.** Bubble Tea three-pane browser (module tree, selectable list, markdown preview). Reuses `cmd/internal/bootstrap` for env/auth and the root `onlyoffice` library for all API calls. UI logic in `cmd/office/ui/`; preview/formatting in `cmd/office/preview/`; list loaders in `cmd/office/fetch/`.
|
||||||
- **List table (`DataTable`)** — `cmd/office/ui/table*.go`. Column layout policies live in `cmd/office/model/table_layout.go` (`TableFlexLayoutFor`); cell rendering uses the bubbles/table inline pattern in `table_render.go` (`renderTableCell`, `padANSIWidth`). See `.cursor/skills/office-tui-table/SKILL.md` before changing center-pane tables.
|
- **List table (`DataTable`)** — `cmd/office/ui/table*.go`. Column layout policies live in `cmd/office/model/table_layout.go` (`TableFlexLayoutFor`); cell rendering uses the bubbles/table inline pattern in `table_render.go` (`renderTableCell`, `padANSIWidth`). See `.cursor/skills/office-tui-table/SKILL.md` before changing center-pane tables.
|
||||||
- **Shared bootstrap — `cmd/internal/bootstrap/`.** `LoadEnv()` + `NewClient(ctx)` extracted from `oo`; both binaries import it.
|
- **Shared bootstrap — `cmd/internal/bootstrap/`.** `LoadEnv()` + `NewClient(ctx)` extracted from `oo`; both binaries import it.
|
||||||
|
- **Bulk Documents tools — `cmd/ooscan/`, `cmd/pdfamount/`, `cmd/kontoblatt/`, `cmd/kontolink/`.** Single-purpose binaries (folder index, PDF amounts, Kontoblatt summary/linking). Pace requests, route API calls through `DoRetry`; usage in README.
|
||||||
- **Personal ops tooling** (disk inventory, dossier→CRM sync, SearXNG) lives in private [`eSlider/oo-workspace`](https://git.produktor.io/eSlider/oo-workspace) (`oow`), not in this public tree.
|
- **Personal ops tooling** (disk inventory, dossier→CRM sync, SearXNG) lives in private [`eSlider/oo-workspace`](https://git.produktor.io/eSlider/oo-workspace) (`oow`), not in this public tree.
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
@@ -27,7 +28,7 @@ Canonical Go client for OnlyOffice Workspace (Projects + Calendar + CRM) and the
|
|||||||
- New endpoints go into the library first; CLI commands are thin wrappers.
|
- New endpoints go into the library first; CLI commands are thin wrappers.
|
||||||
- Prefer `ResponseObject` / `postFormObject` / `putFormObject` / `deleteObject` over hand-rolled `json.Unmarshal(responseField(...))` blocks — they exist for DRY, use them.
|
- Prefer `ResponseObject` / `postFormObject` / `putFormObject` / `deleteObject` over hand-rolled `json.Unmarshal(responseField(...))` blocks — they exist for DRY, use them.
|
||||||
- Domain split is by file, **not** by subpackage. Don't introduce `internal/` or `pkg/*` subpackages inside the library — it flattens the `*Client` call surface for a reason.
|
- Domain split is by file, **not** by subpackage. Don't introduce `internal/` or `pkg/*` subpackages inside the library — it flattens the `*Client` call surface for a reason.
|
||||||
- CLI commands follow **subject → verb** structure (`oo <subject> <verb>`), never `oo <verb>-<subject>`. Add new commands to the existing subject file if one fits; create a new `cmd/oo/<subject>.go` for a genuinely new domain.
|
- CLI commands follow **subject → verb** structure (`oo <subject> <verb>`), never `oo <verb>-<subject>`. Add new commands to the existing subject file if one fits; create a new `cmd/oo/<subject>.go` for a genuinely new domain. The subject→verb tree in `cmd/oo/main.go` and the README table are documentation — update them with the code.
|
||||||
- **Documents for agents:** prefer Markdown in git; OnlyOffice UI is weak for `.md`/`.txt`. Use `oo docs put-md` (md→docx) and `oo docs put-txt` (txt→docx, preserves line breaks). All upload paths default to **upsert** by `stem|ext` (`--replace`, default true); `--no-replace` fails on conflict; `--allow-duplicate` opts into raw OO append. `oo projects files dedupe PROJECT_ID` reports/removes duplicate stem|ext copies (`--apply`, `--cross`; includes project root folder).
|
- **Documents for agents:** prefer Markdown in git; OnlyOffice UI is weak for `.md`/`.txt`. Use `oo docs put-md` (md→docx) and `oo docs put-txt` (txt→docx, preserves line breaks). All upload paths default to **upsert** by `stem|ext` (`--replace`, default true); `--no-replace` fails on conflict; `--allow-duplicate` opts into raw OO append. `oo projects files dedupe PROJECT_ID` reports/removes duplicate stem|ext copies (`--apply`, `--cross`; includes project root folder).
|
||||||
- Every table output goes through `printTable(headers, rows)`; every single-object through `printObject(v)`. Do not `fmt.Println` rows ad-hoc or the `--output json` flag breaks for that command.
|
- Every table output goes through `printTable(headers, rows)`; every single-object through `printObject(v)`. Do not `fmt.Println` rows ad-hoc or the `--output json` flag breaks for that command.
|
||||||
- No secrets in the repo; use `.env` (gitignored). Commit `.env.example` only.
|
- No secrets in the repo; use `.env` (gitignored). Commit `.env.example` only.
|
||||||
|
|||||||
@@ -6,6 +6,23 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
## [0.18.0](https://github.com/eSlider/go-onlyoffice/compare/v0.17.0...v0.18.0) (2026-09-04)
|
||||||
|
|
||||||
|
|
||||||
|
### Features
|
||||||
|
|
||||||
|
* **crm:** add UpdateContactName and CloseCRMTask helpers ([4d8af7a](https://github.com/eSlider/go-onlyoffice/commit/4d8af7a2fef1fcfd96ad913cefc6497e3389963b))
|
||||||
|
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
* **crm:** deterministic sortBy=id in contact paged lists ([4d91726](https://github.com/eSlider/go-onlyoffice/commit/4d917261792203732ac739644c9c82124b2166eb))
|
||||||
|
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
* **funding:** eSlider support links (reverse-import GitHub e9c969a) ([d650a16](https://github.com/eSlider/go-onlyoffice/commit/d650a16a36037949505eb017c92bd3283e6e2f0f))
|
||||||
|
|
||||||
## [0.17.0](https://github.com/eSlider/go-onlyoffice/compare/v0.16.0...v0.17.0) (2026-08-31)
|
## [0.17.0](https://github.com/eSlider/go-onlyoffice/compare/v0.16.0...v0.17.0) (2026-08-31)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -503,6 +503,29 @@ type Task struct {
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `GetUsers()` | List all users with profiles |
|
| `GetUsers()` | List all users with profiles |
|
||||||
|
|
||||||
|
### Documents Files
|
||||||
|
|
||||||
|
| Method | Description |
|
||||||
|
|---|---|
|
||||||
|
| `ListDavFolder(ctx, id)` | List a Documents folder (`@root` for virtual sections) |
|
||||||
|
| `ListDavSections(ctx)` | Virtual sections (Documents, Projects, …) |
|
||||||
|
| `CreateDavFolder(ctx, parentID, title)` | Create a subfolder |
|
||||||
|
| `RenameDavFolder(ctx, id, title)` / `RenameDavFile(ctx, id, title)` | Rename folder / file |
|
||||||
|
| `DownloadFile(ctx, id, dst)` / `DownloadDavFile(ctx, id, w)` | Download file bytes |
|
||||||
|
| `UploadDavFile(ctx, folderID, fileName, src)` | Upload from a reader |
|
||||||
|
| `UploadToFolder(ctx, folderID, localPath)` | Upload a local file into a folder |
|
||||||
|
| `UploadToFolderReplacing(ctx, folderID, localPath)` | Upsert by `stem\|ext`; returns replaced ids |
|
||||||
|
| `UpdateFile(ctx, fileID, localPath)` | New version of an existing file (same id, no copy) |
|
||||||
|
| `MoveDavItems(ctx, folderIDs, fileIDs, dest)` | Move (`resolveType=Skip`); per-operation errors surfaced, not silent nil |
|
||||||
|
| `CopyDavItems(ctx, folderIDs, fileIDs, dest)` | Copy (`conflictResolveType=Skip`); errors surfaced |
|
||||||
|
| `MoveFiles(ctx, destFolderID, fileIDs)` | Move with `resolveType=Skip` + `holdResult`; errors surfaced |
|
||||||
|
| `ListFileOps(ctx)` | Active file operations (move/copy status polling) |
|
||||||
|
| `FolderFiles(ctx, folderID)` | Flat file list of a folder (stem helpers) |
|
||||||
|
| `DeleteFilesByStem(ctx, folderID, stem)` | Remove `stem\|ext` copies |
|
||||||
|
| `DoRetry(ctx, policy, fn)` | Deterministic linear backoff (N·Base, no jitter) on 429/502/503/504 |
|
||||||
|
| `DefaultRetryPolicy()` | 5 attempts, 1s·2s·3s·4s waits, 30s cap |
|
||||||
|
| `Transient(err)` | True for retriable OnlyOffice answers |
|
||||||
|
|
||||||
### Helper Types
|
### Helper Types
|
||||||
|
|
||||||
| Type | Description |
|
| Type | Description |
|
||||||
@@ -622,6 +645,8 @@ oo docs convert ./note.docx # → note.md
|
|||||||
oo docs ocr ./scan.jpg --md ./scan.md # searchable PDF + markdown
|
oo docs ocr ./scan.jpg --md ./scan.md # searchable PDF + markdown
|
||||||
oo docs hocr ./scan.jpg --lang spa --md ./scan.hocr.md --yaml ./scan.yml
|
oo docs hocr ./scan.jpg --lang spa --md ./scan.hocr.md --yaml ./scan.yml
|
||||||
oo docs put-md 7 ./OO-HONDA-7-INDEX.md --folder 490
|
oo docs put-md 7 ./OO-HONDA-7-INDEX.md --folder 490
|
||||||
|
oo docs put-txt 7 ./notes.txt --folder 490
|
||||||
|
oo docs put-xlsx 7 ./table.xlsx --folder 490
|
||||||
oo docs as-md 2815 --to ./parte.md # download OO file as MD (OCR if needed)
|
oo docs as-md 2815 --to ./parte.md # download OO file as MD (OCR if needed)
|
||||||
oo docs as-md 307 --hocr --lang spa # OO download via go-hocr structure
|
oo docs as-md 307 --hocr --lang spa # OO download via go-hocr structure
|
||||||
oo projects files put-md 7 ./note.md # alias
|
oo projects files put-md 7 ./note.md # alias
|
||||||
@@ -631,21 +656,106 @@ oo tasks files upload 208 ./notes.pdf
|
|||||||
oo tasks files detach 208 12345
|
oo tasks files detach 208 12345
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Documents module (`oo dav`)
|
||||||
|
|
||||||
|
Direct access to the Documents module by folder/file id — the same calls that
|
||||||
|
back `oo-webdav` and the project/task file commands. `move` sends
|
||||||
|
`resolveType=Skip` + `holdResult=true`: without those params the legacy
|
||||||
|
`fileops/move` endpoint answers 200 without moving anything, and the library
|
||||||
|
surfaces such per-operation errors instead of a silent nil
|
||||||
|
(`MoveDavItems` / `CopyDavItems` / `MoveFiles`).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
oo dav ls 659
|
||||||
|
oo dav ls @root # virtual sections (Documents, Projects, …)
|
||||||
|
oo dav mkdir 659 "2026 inbox"
|
||||||
|
oo dav move 659 22881 22882 # DEST_FOLDER_ID FILE_ID…
|
||||||
|
oo dav move 659 22881 --folders 670 # move folders along with files
|
||||||
|
oo dav copy 659 22881
|
||||||
|
oo dav rename-file 22881 invoice-v2.pdf
|
||||||
|
oo dav rename-folder 671 o2-archive
|
||||||
|
oo dav download 22881 --to ./copy.pdf # default path: ./<server title>
|
||||||
|
oo dav fileops # active move/copy operations (status polling)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Search (`oo search`)
|
||||||
|
|
||||||
|
Full-text search over the Documents index. The REST endpoint
|
||||||
|
`/api/2.0/files/@search/{query}` only searches file names in the database, so
|
||||||
|
`oo search` talks to the OnlyOffice **Elasticsearch** directly (index
|
||||||
|
`files_file`). Name search is default; `--content` also matches extracted
|
||||||
|
document text (`document.attachment.content`, Office formats only).
|
||||||
|
See [`docs/elasticsearch.md`](docs/elasticsearch.md) for the tunnel setup.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
oo search "Rechnung" # names only
|
||||||
|
oo search "Mahngebühr" --content # names + document text
|
||||||
|
oo search "Rechnung" --folder 649 --limit 50
|
||||||
|
oo search "Rechnung" --json # shorthand for -o json
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires `ONLYOFFICE_ES_URL` (plus optional `ONLYOFFICE_ES_INDEX`,
|
||||||
|
`ONLYOFFICE_TENANT`).
|
||||||
|
|
||||||
|
#### PDF/scans: own index (`oo index` + `--backend own`)
|
||||||
|
|
||||||
|
The OnlyOffice index covers Office formats only, so PDFs (`S1019`-style invoice
|
||||||
|
numbers) are not searchable by content. `oo index` extracts PDF text with
|
||||||
|
`internal/docpipe` (pdftotext, OCR for scans) — including the text of embedded
|
||||||
|
PDF attachments (`pdfdetach`: `<doc>.md`, `.xml`, covers the original/scan and
|
||||||
|
ZUGFeRD e-invoice XML) — into a separate index (`ONLYOFFICE_ES_TEXT_INDEX`,
|
||||||
|
default `oo_docs_text`); the OnlyOffice server and its index are **not**
|
||||||
|
modified. Then search it with `--backend own`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
oo index folder 634 --recursive --exts pdf # populate (idempotent upsert)
|
||||||
|
oo index files 3576 3578 # specific files
|
||||||
|
oo index folder 634 --dry-run # plan only
|
||||||
|
oo search "S1021" --content --backend own # finds the PDF
|
||||||
|
oo search "Rechnung" --backend own --folder 634 --json
|
||||||
|
```
|
||||||
|
|
||||||
|
See [`docs/elasticsearch.md`](docs/elasticsearch.md) for the decision and
|
||||||
|
trade-offs.
|
||||||
|
|
||||||
|
### Bulk tools (`cmd/`)
|
||||||
|
|
||||||
|
Small single-purpose binaries for bulk Documents work. All of them pace
|
||||||
|
requests and retry transient OnlyOffice answers (429/502/503/504) with a
|
||||||
|
deterministic linear backoff — no jitter, same waits on every run
|
||||||
|
(see `DoRetry` below). Build with `go build ./cmd/<tool>`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ooscan 659 # recursive index → TSV: file_id, folder_id, path, title
|
||||||
|
ooscan 659 666 > oo-index.tsv # several roots into one index
|
||||||
|
pdfamount 671 # "Zu zahlender Betrag" per PDF → TSV: file_id, title, amount
|
||||||
|
kontoblatt 3906 ./kontoblatt.xlsx # summary (Gegenkonto/Monat) uploaded next to source
|
||||||
|
kontolink IN.xlsx oo-index.tsv OUT.xlsx [FILE_ID] [AMOUNTS_TSV]
|
||||||
|
# kontolink writes DocEditor links into the Link column: Beleg → supplier+month
|
||||||
|
# → amount+date (5th arg = pdfamount output); with FILE_ID it updates the
|
||||||
|
# source file in place, else uploads an "(links)" copy next to it.
|
||||||
|
```
|
||||||
|
|
||||||
| Subject | Verbs |
|
| Subject | Verbs |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `calendar` | `list`, `events`, `add`, `delete` |
|
| `calendar` | `list`, `events`, `add`, `delete` |
|
||||||
| `projects` | `list`, `get`, `milestones`, `create`, `update`, `delete`, **`files`** (`list`, `upload`, `download`, `rename`, `delete`) |
|
| `projects` | `list`, `get`, `milestones`, `milestone-create`, `create`, `update`, `delete`, `contacts` (`add`, `remove`), `link-authors`, `link-git`, **`files`** (`list`, `upload`, `download`, `rename`, `delete`, `dedupe`, `as-md`, `put-md`, `put-txt`, `put-xlsx`) |
|
||||||
| `tasks` | `list`, `get`, `create`, `update`, `delete`, `subtask add`, **`files`** (`list`, `upload`, `detach`) |
|
| `tasks` | `list`, `get`, `create`, `update`, `delete`, `subtask add`, **`files`** (`list`, `upload`, `detach`) |
|
||||||
| `users` | `list`, `self` (alias: `oo whoami`) |
|
| `users` | `list`, `self` (alias: `oo whoami`) |
|
||||||
| `contacts` | `list`, `get`, `delete`, `info-add`, `merge`, `dedupe-info` |
|
| `contacts` | `list`, `get`, `delete`, `info-add`, `merge`, `dedupe-info`, `tags`, `tag-add`, `tag-create`, `tag-remove` |
|
||||||
| `persons` | `list`, `create`, `delete`, `dedupe` |
|
| `persons` | `list`, `create`, `delete`, `dedupe` |
|
||||||
| `companies` | `list`, `create`, `delete`, `dedupe`, `dedupe-persons` |
|
| `companies` | `list`, `create`, `delete`, `dedupe`, `dedupe-persons` |
|
||||||
| `opportunities` | `list`, `get`, `create`, `delete`, `stages`, `member-add`, `dedupe`, `dedupe-members`, `fix-titles` |
|
| `opportunities` | `list`, `get`, `create`, `update`, `delete`, `stages`, `member-add`, `dedupe`, `dedupe-members`, `fix-titles` |
|
||||||
| `invoices` | `list`, `get`, `create`, `update`, `pdf`, `pdf-cleanup`, `status`, `delete`, `items …` |
|
| `invoices` | `list`, `get`, `create`, `update`, `pdf`, `pdf-cleanup`, `status`, `delete`, `items …` |
|
||||||
| `crm` | `cleanup` |
|
| `crm` | `cleanup` |
|
||||||
| `mails` | `accounts`, `folders`, `list`, `get`, `draft`, `attach`, `draft-invoice`, `delete` |
|
| `mails` | `accounts`, `folders`, `list`, `get`, `download-attachment`, `draft`, `attach`, `draft-invoice`, `send`, `delete` |
|
||||||
| `cases` | `list`, `create`, `delete`, `member-add` |
|
| `cases` | `list`, `create`, `delete`, `member-add` |
|
||||||
| `crm-tasks` | `list`, `create`, `delete`, `categories` |
|
| `crm-tasks` | `list`, `create`, `delete`, `categories`, `reassign-self` |
|
||||||
|
| `docs` | `tools`, `convert`, `optimize`, `ocr`, `hocr`, `as-md`, `put-md`, `put-txt`, `put-xlsx` |
|
||||||
|
| `catalog` | `match`, `merge`, `apply`, `scan-contacts`, `scan-projects`, `scan-thunderbird` |
|
||||||
|
| `dav` | `ls`, `move`, `copy`, `mkdir`, `rename-file`, `rename-folder`, `download`, `fileops` |
|
||||||
|
| `search` | `QUERY` (`--content`, `--folder ID`, `--limit N`, `--backend oo\|own`, `--json`) |
|
||||||
|
| `index` | `folder FOLDER_ID`, `files FILE_ID...` (`--recursive`, `--exts pdf`, `--limit N`, `--dry-run`) |
|
||||||
|
|
||||||
The CLI reads only `.env` from the current working directory (godotenv is a
|
The CLI reads only `.env` from the current working directory (godotenv is a
|
||||||
CLI-only concern — the library itself never loads dotfiles).
|
CLI-only concern — the library itself never loads dotfiles).
|
||||||
|
|||||||
@@ -0,0 +1,220 @@
|
|||||||
|
// Command kontoblatt builds a summary ("сводная таблица") of a Kontoblatt XLSX
|
||||||
|
// (Datum, Gegenkonto, Buchungstext, Beleg, Soll, Haben, Bemerkung) and uploads
|
||||||
|
// it back to the same OnlyOffice folder as the source file.
|
||||||
|
//
|
||||||
|
// Usage: kontoblatt <FILE_ID> <LOCAL_XLSX>
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
"github.com/xuri/excelize/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
type agg struct {
|
||||||
|
count int
|
||||||
|
soll float64
|
||||||
|
haben float64
|
||||||
|
reFehlt int
|
||||||
|
}
|
||||||
|
|
||||||
|
type rec struct {
|
||||||
|
date, month, konto, text string
|
||||||
|
soll, haben float64
|
||||||
|
reFehlt bool
|
||||||
|
}
|
||||||
|
|
||||||
|
var dateRe = regexp.MustCompile(`^\d{2}\.\d{2}\.\d{4}$`)
|
||||||
|
|
||||||
|
func parseAmount(s string) float64 {
|
||||||
|
s = strings.TrimSpace(s)
|
||||||
|
s = strings.ReplaceAll(s, "€", "")
|
||||||
|
s = strings.ReplaceAll(s, " ", "")
|
||||||
|
s = strings.ReplaceAll(s, ",", "") // German thousands separator
|
||||||
|
s = strings.TrimSpace(s)
|
||||||
|
if s == "" {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
v, err := strconv.ParseFloat(s, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
func cell(row []string, i int) string {
|
||||||
|
if i < len(row) {
|
||||||
|
return strings.TrimSpace(row[i])
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
if len(os.Args) < 3 {
|
||||||
|
fmt.Fprintln(os.Stderr, "usage: kontoblatt <FILE_ID> <LOCAL_XLSX>")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
fileID, path := os.Args[1], os.Args[2]
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
f, err := excelize.OpenFile(path)
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
|
||||||
|
var recs []rec
|
||||||
|
for _, sh := range f.GetSheetList() {
|
||||||
|
rows, err := f.GetRows(sh)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, r := range rows {
|
||||||
|
d := cell(r, 0)
|
||||||
|
if !dateRe.MatchString(d) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
text := cell(r, 2)
|
||||||
|
recs = append(recs, rec{
|
||||||
|
date: d,
|
||||||
|
month: d[3:10],
|
||||||
|
konto: cell(r, 1),
|
||||||
|
text: text,
|
||||||
|
soll: parseAmount(cell(r, 4)),
|
||||||
|
haben: parseAmount(cell(r, 5)),
|
||||||
|
reFehlt: strings.Contains(strings.ToUpper(text), "FEHLT"),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
byKonto := map[string]*agg{}
|
||||||
|
byMonth := map[string]*agg{}
|
||||||
|
getK := func(k string) *agg {
|
||||||
|
if byKonto[k] == nil {
|
||||||
|
byKonto[k] = &agg{}
|
||||||
|
}
|
||||||
|
return byKonto[k]
|
||||||
|
}
|
||||||
|
getM := func(k string) *agg {
|
||||||
|
if byMonth[k] == nil {
|
||||||
|
byMonth[k] = &agg{}
|
||||||
|
}
|
||||||
|
return byMonth[k]
|
||||||
|
}
|
||||||
|
var tot agg
|
||||||
|
for _, r := range recs {
|
||||||
|
k := getK(r.konto)
|
||||||
|
k.count++
|
||||||
|
k.soll += r.soll
|
||||||
|
k.haben += r.haben
|
||||||
|
if r.reFehlt {
|
||||||
|
k.reFehlt++
|
||||||
|
}
|
||||||
|
m := getM(r.month)
|
||||||
|
m.count++
|
||||||
|
m.soll += r.soll
|
||||||
|
m.haben += r.haben
|
||||||
|
if r.reFehlt {
|
||||||
|
m.reFehlt++
|
||||||
|
}
|
||||||
|
tot.count++
|
||||||
|
tot.soll += r.soll
|
||||||
|
tot.haben += r.haben
|
||||||
|
if r.reFehlt {
|
||||||
|
tot.reFehlt++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out := excelize.NewFile()
|
||||||
|
defer out.Close()
|
||||||
|
writeSheet(out, "Nach Gegenkonto", "Gegenkonto", byKonto, tot)
|
||||||
|
writeSheet(out, "Nach Monat", "Monat", byMonth, tot)
|
||||||
|
outPath := "/tmp/opencode/kontoblatt-zusammenfassung.xlsx"
|
||||||
|
if err := out.SaveAs(outPath); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// upload next to the source file
|
||||||
|
creds := onlyoffice.GetEnvironmentCredentials()
|
||||||
|
c := onlyoffice.NewClient(creds)
|
||||||
|
var src *onlyoffice.FileEntry
|
||||||
|
if derr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
var err error
|
||||||
|
src, err = c.GetFile(ctx, fileID)
|
||||||
|
return err
|
||||||
|
}); derr != nil {
|
||||||
|
panic(derr)
|
||||||
|
}
|
||||||
|
folder := ""
|
||||||
|
if src.FolderID != nil {
|
||||||
|
folder = src.FolderID.String()
|
||||||
|
}
|
||||||
|
title := ""
|
||||||
|
if src.Title != nil {
|
||||||
|
title = *src.Title
|
||||||
|
}
|
||||||
|
fmt.Printf("source: id=%s title=%q folder=%s\n", fileID, title, folder)
|
||||||
|
|
||||||
|
name := "Kontoblatt-1591-2025-Zusammenfassung.xlsx"
|
||||||
|
tmp := "/tmp/opencode/" + name
|
||||||
|
data, _ := os.ReadFile(outPath)
|
||||||
|
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
var entry *onlyoffice.FileEntry
|
||||||
|
if derr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
var err error
|
||||||
|
entry, _, err = c.UploadToFolderReplacing(ctx, folder, tmp)
|
||||||
|
return err
|
||||||
|
}); derr != nil {
|
||||||
|
panic(derr)
|
||||||
|
}
|
||||||
|
fmt.Printf("uploaded: %s -> folder %s (id %v)\n", name, folder, entry.ID)
|
||||||
|
|
||||||
|
// print the summary
|
||||||
|
printAgg("Nach Gegenkonto", byKonto, tot)
|
||||||
|
printAgg("Nach Monat", byMonth, tot)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeSheet(f *excelize.File, sheet, key string, m map[string]*agg, tot agg) {
|
||||||
|
f.NewSheet(sheet)
|
||||||
|
rows := [][]any{{key, "Anzahl", "Soll", "Haben", "Saldo", `davon "fehlt"`}}
|
||||||
|
keys := make([]string, 0, len(m))
|
||||||
|
for k := range m {
|
||||||
|
keys = append(keys, k)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
for _, k := range keys {
|
||||||
|
a := m[k]
|
||||||
|
rows = append(rows, []any{k, a.count, a.soll, a.haben, a.soll - a.haben, a.reFehlt})
|
||||||
|
}
|
||||||
|
rows = append(rows, []any{"GESAMT", tot.count, tot.soll, tot.haben, tot.soll - tot.haben, tot.reFehlt})
|
||||||
|
for i, row := range rows {
|
||||||
|
for j, v := range row {
|
||||||
|
cellRef, _ := excelize.CoordinatesToCellName(j+1, i+1)
|
||||||
|
_ = f.SetCellValue(sheet, cellRef, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func printAgg(title string, m map[string]*agg, tot agg) {
|
||||||
|
fmt.Printf("\n== %s ==\n", title)
|
||||||
|
keys := make([]string, 0, len(m))
|
||||||
|
for k := range m {
|
||||||
|
keys = append(keys, k)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
fmt.Printf("%-12s %6s %12s %12s %12s %7s\n", "key", "count", "soll", "haben", "saldo", "fehlt")
|
||||||
|
for _, k := range keys {
|
||||||
|
a := m[k]
|
||||||
|
fmt.Printf("%-12s %6d %12.2f %12.2f %12.2f %7d\n", k, a.count, a.soll, a.haben, a.soll-a.haben, a.reFehlt)
|
||||||
|
}
|
||||||
|
fmt.Printf("%-12s %6d %12.2f %12.2f %12.2f %7d\n", "GESAMT", tot.count, tot.soll, tot.haben, tot.soll-tot.haben, tot.reFehlt)
|
||||||
|
}
|
||||||
@@ -0,0 +1,478 @@
|
|||||||
|
// Command kontolink fills the "Link" column of a Kontoblatt ("ungeklärte
|
||||||
|
// Posten") XLSX by matching each row to an OnlyOffice document.
|
||||||
|
//
|
||||||
|
// Strategy (deterministic, conservative — no LLM):
|
||||||
|
// 1. Beleg token (letters/digits from the "Beleg" column) appears in the file
|
||||||
|
// title; among candidates prefer (a) the row's month, (b) real invoices over
|
||||||
|
// copies/dupes, and require the result to be unique;
|
||||||
|
// 2. else supplier + row month + "rechnung", again unique.
|
||||||
|
//
|
||||||
|
// A file is linked at most once (rows already carrying a link are kept and their
|
||||||
|
// file counts as used). Ambiguous rows are left UNLINKED for manual review.
|
||||||
|
//
|
||||||
|
// Usage: kontolink <IN_XLSX> <INDEX_TSV> <OUT_XLSX>
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
"github.com/xuri/excelize/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
dateRe = regexp.MustCompile(`^\d{2}\.\d{2}\.\d{4}$`)
|
||||||
|
nonAln = regexp.MustCompile(`[^0-9a-z]+`)
|
||||||
|
fileID = regexp.MustCompile(`fileid=(\d+)`)
|
||||||
|
)
|
||||||
|
|
||||||
|
func parseDay(s string) (time.Time, bool) {
|
||||||
|
t, err := time.Parse("02.01.2006", strings.TrimSpace(s))
|
||||||
|
return t, err == nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func titleDay(title string) (time.Time, bool) {
|
||||||
|
if len(title) >= 10 {
|
||||||
|
if t, err := time.Parse("2006-01-02", title[:10]); err == nil {
|
||||||
|
return t, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return time.Time{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
// nearest picks the candidate whose title date is closest to rd. Ties and
|
||||||
|
// undated candidates (when >1) are rejected.
|
||||||
|
func nearest(cands []entry, rd time.Time) (entry, bool) {
|
||||||
|
if len(cands) == 1 {
|
||||||
|
return cands[0], true
|
||||||
|
}
|
||||||
|
best, bestD, tie := -1, 0.0, false
|
||||||
|
for i, e := range cands {
|
||||||
|
td, ok := titleDay(e.title)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
d := td.Sub(rd).Hours() / 24
|
||||||
|
if d < 0 {
|
||||||
|
d = -d
|
||||||
|
}
|
||||||
|
if best < 0 || d < bestD {
|
||||||
|
best, bestD, tie = i, d, false
|
||||||
|
} else if d == bestD {
|
||||||
|
tie = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if best < 0 || tie {
|
||||||
|
return entry{}, false
|
||||||
|
}
|
||||||
|
return cands[best], true
|
||||||
|
}
|
||||||
|
|
||||||
|
const linkPrefix = "https://office.pro-dukt.de/Products/Files/DocEditor.aspx?fileid="
|
||||||
|
|
||||||
|
type entry struct {
|
||||||
|
id, path, title, norm string
|
||||||
|
}
|
||||||
|
|
||||||
|
func norm(s string) string { return nonAln.ReplaceAllString(strings.ToLower(s), "") }
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
if len(os.Args) < 4 {
|
||||||
|
fmt.Fprintln(os.Stderr, "usage: kontolink <IN_XLSX> <INDEX_TSV> <OUT_XLSX>")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
in, idxPath, out := os.Args[1], os.Args[2], os.Args[3]
|
||||||
|
|
||||||
|
idxRaw, err := os.ReadFile(idxPath)
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
var entries []entry
|
||||||
|
for _, line := range strings.Split(string(idxRaw), "\n") {
|
||||||
|
parts := strings.Split(line, "\t")
|
||||||
|
if len(parts) < 4 || parts[0] == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
entries = append(entries, entry{id: parts[0], path: parts[2], title: parts[3], norm: norm(parts[3])})
|
||||||
|
}
|
||||||
|
|
||||||
|
f, err := excelize.OpenFile(in)
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
sheet := f.GetSheetList()[0]
|
||||||
|
rows, err := f.GetRows(sheet)
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// optional 5th arg: amounts TSV "file_id\ttitle\tamount" (see cmd/pdfamount)
|
||||||
|
var amts []amtEntry
|
||||||
|
if len(os.Args) >= 6 && os.Args[5] != "" {
|
||||||
|
amts = loadAmounts(os.Args[5])
|
||||||
|
}
|
||||||
|
|
||||||
|
used := map[string]bool{}
|
||||||
|
for _, r := range rows {
|
||||||
|
if m := fileID.FindStringSubmatch(cell(r, 7)); m != nil {
|
||||||
|
used[m[1]] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var linked, byBeleg, bySupplier, byAmount, unmatched, ambiguous int
|
||||||
|
for i, r := range rows {
|
||||||
|
if i == 0 || !dateRe.MatchString(cell(r, 0)) || strings.TrimSpace(cell(r, 7)) != "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
beleg := norm(cell(r, 3))
|
||||||
|
supplier := supplierNorm(cell(r, 2))
|
||||||
|
month := monthYear(cell(r, 0))
|
||||||
|
rd, _ := parseDay(cell(r, 0))
|
||||||
|
|
||||||
|
e, kind, ok := pick(entries, used, beleg, supplier, month, rd)
|
||||||
|
if !ok {
|
||||||
|
if ae, aok := amountPick(amts, used, supplier, rowAmount(r), rd); aok {
|
||||||
|
e, kind, ok = entry{id: ae.id, title: ae.title}, "amount", true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
if beleg != "" {
|
||||||
|
ambiguous++
|
||||||
|
} else {
|
||||||
|
unmatched++
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ref, _ := excelize.CoordinatesToCellName(8, i+1)
|
||||||
|
if err := f.SetCellValue(sheet, ref, linkPrefix+e.id); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
used[e.id] = true
|
||||||
|
linked++
|
||||||
|
switch kind {
|
||||||
|
case "beleg":
|
||||||
|
byBeleg++
|
||||||
|
case "supplier":
|
||||||
|
bySupplier++
|
||||||
|
case "amount":
|
||||||
|
byAmount++
|
||||||
|
}
|
||||||
|
fmt.Printf("row %3d %-30s -> %s [%s]\n", i+1, cell(r, 2), e.title, kind)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := f.SaveAs(out); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
fmt.Printf("\nlinked=%d (beleg=%d, supplier=%d, amount=%d), ambiguous=%d, no-candidate=%d\n",
|
||||||
|
linked, byBeleg, bySupplier, byAmount, ambiguous, unmatched)
|
||||||
|
|
||||||
|
// Optional 4th arg: source OnlyOffice file id. Try to update it in place;
|
||||||
|
// if it is locked (OnlyOffice 500), upload a "(links)" copy next to it.
|
||||||
|
if len(os.Args) >= 5 && os.Args[4] != "" {
|
||||||
|
c := onlyoffice.NewClient(onlyoffice.GetEnvironmentCredentials())
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
var src *onlyoffice.FileEntry
|
||||||
|
if derr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
var err error
|
||||||
|
src, err = c.GetFile(ctx, os.Args[4])
|
||||||
|
return err
|
||||||
|
}); derr != nil {
|
||||||
|
panic(derr)
|
||||||
|
}
|
||||||
|
folder, title := "", ""
|
||||||
|
if src.FolderID != nil {
|
||||||
|
folder = src.FolderID.String()
|
||||||
|
}
|
||||||
|
if src.Title != nil {
|
||||||
|
title = *src.Title
|
||||||
|
}
|
||||||
|
uderr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
_, err := c.UpdateFile(ctx, os.Args[4], out)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
if uderr == nil {
|
||||||
|
fmt.Printf("updated file %s in place\n", os.Args[4])
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Printf("in-place update failed (locked?); uploading a copy to folder %s\n", folder)
|
||||||
|
ext := filepath.Ext(title)
|
||||||
|
name := strings.TrimSuffix(title, ext) + " (links)" + ext
|
||||||
|
tmp := filepath.Join(os.TempDir(), name)
|
||||||
|
data, _ := os.ReadFile(out)
|
||||||
|
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
if derr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
_, _, err := c.UploadToFolderReplacing(ctx, folder, tmp)
|
||||||
|
return err
|
||||||
|
}); derr != nil {
|
||||||
|
panic(derr)
|
||||||
|
}
|
||||||
|
fmt.Printf("uploaded copy: %s -> folder %s\n", name, folder)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cell(r []string, i int) string {
|
||||||
|
if i < len(r) {
|
||||||
|
return strings.TrimSpace(r[i])
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func supplierNorm(s string) string {
|
||||||
|
s = strings.ToUpper(s)
|
||||||
|
if i := strings.Index(s, ","); i >= 0 {
|
||||||
|
s = s[:i]
|
||||||
|
}
|
||||||
|
for _, w := range []string{"RE FEHLT", "GS FEHLT", "WOFR", "WOFÜR"} {
|
||||||
|
s = strings.ReplaceAll(s, w, "")
|
||||||
|
}
|
||||||
|
return norm(s)
|
||||||
|
}
|
||||||
|
|
||||||
|
func monthYear(date string) string {
|
||||||
|
if len(date) == 10 {
|
||||||
|
return date[6:10] + "-" + date[3:5]
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// pick returns an unused candidate. Beleg match wins; supplier+month is a
|
||||||
|
// fallback. When several candidates qualify, the one closest in time to the row
|
||||||
|
// date wins; a tie is rejected (ambiguous) rather than guessed.
|
||||||
|
func pick(entries []entry, used map[string]bool, beleg, supplier, month string, rd time.Time) (entry, string, bool) {
|
||||||
|
free := func(e entry) bool { return !used[e.id] }
|
||||||
|
|
||||||
|
if len(beleg) >= 5 {
|
||||||
|
var inMonth []entry
|
||||||
|
for _, e := range entries {
|
||||||
|
if free(e) && belegMatches(e.norm, beleg) &&
|
||||||
|
(month == "" || strings.Contains(e.title, month)) {
|
||||||
|
inMonth = append(inMonth, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if supplier != "" {
|
||||||
|
var s []entry
|
||||||
|
for _, e := range inMonth {
|
||||||
|
if strings.Contains(e.norm, supplier) {
|
||||||
|
s = append(s, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(s) > 0 {
|
||||||
|
inMonth = s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inMonth = topRank(inMonth)
|
||||||
|
if e, ok := nearest(inMonth, rd); ok {
|
||||||
|
return e, "beleg", true
|
||||||
|
}
|
||||||
|
// A Beleg is present but no file carries it: do NOT fall back to a
|
||||||
|
// supplier guess (that links the wrong invoice).
|
||||||
|
return entry{}, "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
if supplier != "" && month != "" {
|
||||||
|
var c []entry
|
||||||
|
for _, e := range entries {
|
||||||
|
if free(e) && strings.Contains(e.norm, supplier) &&
|
||||||
|
strings.Contains(e.title, month) && strings.Contains(e.norm, "rechnung") {
|
||||||
|
c = append(c, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
c = topRank(c)
|
||||||
|
if e, ok := nearest(c, rd); ok {
|
||||||
|
return e, "supplier", true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return entry{}, "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
// topRank keeps only the highest-ranked candidates (real invoice over copy /
|
||||||
|
// dupe / op), so a tie with a duplicate does not mask the real file.
|
||||||
|
func topRank(cands []entry) []entry {
|
||||||
|
if len(cands) < 2 {
|
||||||
|
return cands
|
||||||
|
}
|
||||||
|
best := 0
|
||||||
|
for _, e := range cands {
|
||||||
|
if rank(e) > best {
|
||||||
|
best = rank(e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out := cands[:0]
|
||||||
|
for _, e := range cands {
|
||||||
|
if rank(e) == best {
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func rank(e entry) int {
|
||||||
|
s := 0
|
||||||
|
if strings.Contains(e.path, "/2025") || strings.Contains(e.path, "/2024") {
|
||||||
|
s += 4
|
||||||
|
}
|
||||||
|
if strings.Contains(e.norm, "rechnung") {
|
||||||
|
s += 2
|
||||||
|
}
|
||||||
|
if strings.Contains(e.norm, "dupe") || strings.Contains(e.norm, "copy") ||
|
||||||
|
strings.Contains(e.norm, "op") {
|
||||||
|
s--
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// belegMatches reports whether a Beleg identifies the file: the whole normalized
|
||||||
|
// Beleg appears, or (for long numeric Belege, e.g. "24/641393110") an 8-digit
|
||||||
|
// window of its longest digit run appears.
|
||||||
|
func belegMatches(titleNorm, beleg string) bool {
|
||||||
|
if strings.Contains(titleNorm, beleg) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
run := longestDigitRun(beleg)
|
||||||
|
for i := 0; i+8 <= len(run); i++ {
|
||||||
|
if strings.Contains(titleNorm, run[i:i+8]) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func longestDigitRun(s string) string {
|
||||||
|
var best, cur strings.Builder
|
||||||
|
for _, r := range s {
|
||||||
|
if r >= '0' && r <= '9' {
|
||||||
|
cur.WriteRune(r)
|
||||||
|
if cur.Len() > best.Len() {
|
||||||
|
best.Reset()
|
||||||
|
best.WriteString(cur.String())
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
cur.Reset()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
type amtEntry struct {
|
||||||
|
id string
|
||||||
|
title string
|
||||||
|
norm string
|
||||||
|
amount float64
|
||||||
|
date time.Time
|
||||||
|
hasDate bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadAmounts(path string) []amtEntry {
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var out []amtEntry
|
||||||
|
for _, line := range strings.Split(string(raw), "\n") {
|
||||||
|
p := strings.Split(line, "\t")
|
||||||
|
if len(p) < 3 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
v, err := strconv.ParseFloat(strings.TrimSpace(p[2]), 64)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
e := amtEntry{id: p[0], title: p[1], norm: norm(p[1]), amount: v}
|
||||||
|
if len(p[1]) >= 10 {
|
||||||
|
if t, err := time.Parse("2006-01-02", p[1][:10]); err == nil {
|
||||||
|
e.date, e.hasDate = t, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func rowAmount(r []string) float64 {
|
||||||
|
if v := parseAmount(cell(r, 4)); v != 0 {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return parseAmount(cell(r, 5))
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseAmount(s string) float64 {
|
||||||
|
s = strings.ReplaceAll(s, "€", "")
|
||||||
|
s = strings.ReplaceAll(s, " ", "")
|
||||||
|
s = strings.ReplaceAll(s, ",", ".")
|
||||||
|
if s == "" {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
v, err := strconv.ParseFloat(s, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// amountPick matches a row to an O2 invoice by amount + nearest date. Scoped to
|
||||||
|
// Telefonica/O2 rows and O2 files, so it cannot cross-link other suppliers.
|
||||||
|
func amountPick(amts []amtEntry, used map[string]bool, supplier string, amt float64, rd time.Time) (amtEntry, bool) {
|
||||||
|
if amt <= 0 || len(amts) == 0 {
|
||||||
|
return amtEntry{}, false
|
||||||
|
}
|
||||||
|
if !strings.Contains(supplier, "telefonica") && !strings.Contains(supplier, "o2") {
|
||||||
|
return amtEntry{}, false
|
||||||
|
}
|
||||||
|
var cands []amtEntry
|
||||||
|
for _, a := range amts {
|
||||||
|
if used[a.id] || !strings.Contains(a.norm, "o2") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
d := a.amount - amt
|
||||||
|
if d < 0 {
|
||||||
|
d = -d
|
||||||
|
}
|
||||||
|
if d > 0.005 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if a.hasDate && !rd.IsZero() {
|
||||||
|
days := a.date.Sub(rd).Hours() / 24
|
||||||
|
if days < 0 {
|
||||||
|
days = -days
|
||||||
|
}
|
||||||
|
if days > 75 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cands = append(cands, a)
|
||||||
|
}
|
||||||
|
if len(cands) == 1 {
|
||||||
|
return cands[0], true
|
||||||
|
}
|
||||||
|
best, bestD, tie := -1, 0.0, false
|
||||||
|
for i, a := range cands {
|
||||||
|
if !a.hasDate {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
d := a.date.Sub(rd).Hours() / 24
|
||||||
|
if d < 0 {
|
||||||
|
d = -d
|
||||||
|
}
|
||||||
|
if best < 0 || d < bestD {
|
||||||
|
best, bestD, tie = i, d, false
|
||||||
|
} else if d == bestD {
|
||||||
|
tie = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if best < 0 || tie {
|
||||||
|
return amtEntry{}, false
|
||||||
|
}
|
||||||
|
return cands[best], true
|
||||||
|
}
|
||||||
@@ -17,6 +17,18 @@ const MailListPageSize = 25
|
|||||||
// Loader fetches list items for a menu subject using the OnlyOffice client.
|
// Loader fetches list items for a menu subject using the OnlyOffice client.
|
||||||
type Loader struct {
|
type Loader struct {
|
||||||
Client *onlyoffice.Client
|
Client *onlyoffice.Client
|
||||||
|
|
||||||
|
// Files is the backend-agnostic file store used for file download, preview
|
||||||
|
// and delete. When nil it falls back to Client.FileStore(ProviderREST).
|
||||||
|
Files onlyoffice.FileStore
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileStore returns the configured file store, defaulting to REST.
|
||||||
|
func (l *Loader) fileStore() onlyoffice.FileStore {
|
||||||
|
if l.Files != nil {
|
||||||
|
return l.Files
|
||||||
|
}
|
||||||
|
return l.Client.FileStore(onlyoffice.ProviderREST)
|
||||||
}
|
}
|
||||||
|
|
||||||
// List returns items for the given list spec (nav leaf).
|
// List returns items for the given list spec (nav leaf).
|
||||||
@@ -171,11 +183,7 @@ func (l *Loader) executeDelete(ctx context.Context, item model.Item) (string, er
|
|||||||
}
|
}
|
||||||
return fmt.Sprintf("Deleted message %s", item.Title), nil
|
return fmt.Sprintf("Deleted message %s", item.Title), nil
|
||||||
case model.KindFile:
|
case model.KindFile:
|
||||||
id, err := strconv.Atoi(item.ID)
|
if err := l.fileStore().Delete(ctx, []string{item.ID}); err != nil {
|
||||||
if err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
if err := l.Client.DeleteFiles(ctx, []int{id}); err != nil {
|
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
return fmt.Sprintf("Deleted file %s", item.Title), nil
|
return fmt.Sprintf("Deleted file %s", item.Title), nil
|
||||||
@@ -199,7 +207,7 @@ func (l *Loader) executeDownload(ctx context.Context, item model.Item, destPath
|
|||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
defer f.Close()
|
defer f.Close()
|
||||||
if _, err := l.Client.DownloadFile(ctx, item.ID, f); err != nil {
|
if _, err := l.fileStore().Download(ctx, item.ID, f); err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
return fmt.Sprintf("Downloaded to %s", destPath), nil
|
return fmt.Sprintf("Downloaded to %s", destPath), nil
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
onlyoffice "github.com/eslider/go-onlyoffice"
|
|
||||||
"github.com/eslider/go-onlyoffice/cmd/office/model"
|
"github.com/eslider/go-onlyoffice/cmd/office/model"
|
||||||
"github.com/eslider/go-onlyoffice/cmd/office/preview"
|
"github.com/eslider/go-onlyoffice/cmd/office/preview"
|
||||||
)
|
)
|
||||||
@@ -30,13 +29,11 @@ func (l *Loader) filePreviewMarkdown(ctx context.Context, item model.Item) (stri
|
|||||||
return "", fmt.Errorf("file id missing")
|
return "", fmt.Errorf("file id missing")
|
||||||
}
|
}
|
||||||
name := item.Title
|
name := item.Title
|
||||||
if meta, err := l.Client.GetFile(ctx, item.ID); err == nil && meta != nil {
|
if e, err := l.fileStore().Stat(ctx, item.ID); err == nil && e.Title != "" {
|
||||||
if t := onlyoffice.FileEntryTitle(meta); t != "" {
|
name = e.Title
|
||||||
name = t
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
var buf bytes.Buffer
|
var buf bytes.Buffer
|
||||||
if _, err := l.Client.DownloadFile(ctx, item.ID, &buf); err != nil {
|
if _, err := l.fileStore().Download(ctx, item.ID, &buf); err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
return preview.FileBytesToMarkdown(name, buf.Bytes())
|
return preview.FileBytesToMarkdown(name, buf.Bytes())
|
||||||
|
|||||||
+322
@@ -0,0 +1,322 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
rootCmd.AddCommand(davCmd())
|
||||||
|
}
|
||||||
|
|
||||||
|
// davCmd exposes the Documents module through the backend-agnostic FileStore
|
||||||
|
// (DAV backend). The underlying Dav calls are the oo-webdav proven path:
|
||||||
|
// MoveDavItems sends resolveType=Skip + holdResult=true, which the legacy
|
||||||
|
// fileops/move call without those params silently ignores (200 without move).
|
||||||
|
func davCmd() *cobra.Command {
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "dav",
|
||||||
|
Short: "Documents module by folder/file id (oo-webdav proven path)",
|
||||||
|
}
|
||||||
|
cmd.AddCommand(davLsCmd())
|
||||||
|
cmd.AddCommand(davMoveCmd())
|
||||||
|
cmd.AddCommand(davCopyCmd())
|
||||||
|
cmd.AddCommand(davMkdirCmd())
|
||||||
|
cmd.AddCommand(davRemoveCmd())
|
||||||
|
cmd.AddCommand(davRenameFileCmd())
|
||||||
|
cmd.AddCommand(davRenameFolderCmd())
|
||||||
|
cmd.AddCommand(davDownloadCmd())
|
||||||
|
cmd.AddCommand(davFileOpsCmd())
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func davLsCmd() *cobra.Command {
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "ls FOLDER_ID",
|
||||||
|
Short: "List a Documents folder (@root for virtual sections)",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ctx := cmd.Context()
|
||||||
|
if args[0] == "@root" {
|
||||||
|
sections, err := c.ListDavSections(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rows := make([]map[string]any, 0, len(sections))
|
||||||
|
for _, s := range sections {
|
||||||
|
rows = append(rows, map[string]any{
|
||||||
|
"id": s.ID,
|
||||||
|
"title": s.Title,
|
||||||
|
"filesCount": s.FilesCount,
|
||||||
|
"foldersCount": s.FoldersCount,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
printTable([]string{"id", "title", "filesCount", "foldersCount"}, rows)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
entries, err := c.FileStore(onlyoffice.ProviderDAV).List(ctx, args[0])
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
folders := make([]onlyoffice.Entry, 0, len(entries))
|
||||||
|
files := make([]onlyoffice.Entry, 0, len(entries))
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.Kind == onlyoffice.Folder {
|
||||||
|
folders = append(folders, e)
|
||||||
|
} else {
|
||||||
|
files = append(files, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
frows := make([]map[string]any, 0, len(folders))
|
||||||
|
for _, f := range folders {
|
||||||
|
frows = append(frows, map[string]any{
|
||||||
|
"id": f.ID,
|
||||||
|
"title": f.Title,
|
||||||
|
"filesCount": f.FilesCount,
|
||||||
|
"foldersCount": f.FoldersCount,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
rows := make([]map[string]any, 0, len(files))
|
||||||
|
for _, f := range files {
|
||||||
|
rows = append(rows, map[string]any{
|
||||||
|
"id": f.ID,
|
||||||
|
"title": f.Title,
|
||||||
|
"size": f.Size,
|
||||||
|
"updated": entryUpdated(f),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if outputFormat == "json" {
|
||||||
|
printObject(map[string]any{"folders": frows, "files": rows})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if len(frows) > 0 {
|
||||||
|
fmt.Println("folders:")
|
||||||
|
printTable([]string{"id", "title", "filesCount", "foldersCount"}, frows)
|
||||||
|
}
|
||||||
|
fmt.Println("files:")
|
||||||
|
printTable([]string{"id", "title", "size", "updated"}, rows)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
// entryUpdated prefers the backend-native timestamp string so table/JSON output
|
||||||
|
// round-trips what the API returned.
|
||||||
|
func entryUpdated(e onlyoffice.Entry) string {
|
||||||
|
if e.Updated != "" {
|
||||||
|
return e.Updated
|
||||||
|
}
|
||||||
|
if e.Modified.IsZero() {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return e.Modified.Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
|
||||||
|
func davMoveCmd() *cobra.Command {
|
||||||
|
var folderIDs []string
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "move DEST_FOLDER_ID FILE_ID [FILE_ID...]",
|
||||||
|
Short: "Move file(s) into a Documents folder (resolveType=Skip, holdResult)",
|
||||||
|
Args: cobra.MinimumNArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ids := append(append([]string{}, folderIDs...), args[1:]...)
|
||||||
|
if err := c.FileStore(onlyoffice.ProviderDAV).Move(cmd.Context(), ids, args[0]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"moved_files": args[1:], "moved_folders": folderIDs, "dest": args[0]})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cmd.Flags().StringSliceVar(&folderIDs, "folders", nil, "folder ids to move along with the files")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func davCopyCmd() *cobra.Command {
|
||||||
|
var folderIDs []string
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "copy DEST_FOLDER_ID FILE_ID [FILE_ID...]",
|
||||||
|
Short: "Copy file(s) into a Documents folder (conflictResolveType=Skip)",
|
||||||
|
Args: cobra.MinimumNArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ids := append(append([]string{}, folderIDs...), args[1:]...)
|
||||||
|
if err := c.FileStore(onlyoffice.ProviderDAV).Copy(cmd.Context(), ids, args[0]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"copied_files": args[1:], "copied_folders": folderIDs, "dest": args[0]})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cmd.Flags().StringSliceVar(&folderIDs, "folders", nil, "folder ids to copy along with the files")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func davMkdirCmd() *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "mkdir PARENT_FOLDER_ID TITLE",
|
||||||
|
Short: "Create a subfolder in Documents",
|
||||||
|
Args: cobra.ExactArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
f, err := c.FileStore(onlyoffice.ProviderDAV).CreateFolder(cmd.Context(), args[0], args[1])
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"id": f.ID, "title": f.Title, "parent": args[0]})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func davRemoveCmd() *cobra.Command {
|
||||||
|
var folderIDs []string
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "rm [FILE_ID...]",
|
||||||
|
Aliases: []string{"delete"},
|
||||||
|
Short: "Permanently delete file(s) and/or folder(s) from Documents",
|
||||||
|
Args: cobra.ArbitraryArgs,
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
if len(args) == 0 && len(folderIDs) == 0 {
|
||||||
|
return fmt.Errorf("dav rm: give at least one FILE_ID or --folders")
|
||||||
|
}
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ids := append(append([]string{}, args...), folderIDs...)
|
||||||
|
if err := c.FileStore(onlyoffice.ProviderDAV).Delete(cmd.Context(), ids); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"deleted_files": args, "deleted_folders": folderIDs})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cmd.Flags().StringSliceVar(&folderIDs, "folders", nil, "folder ids to delete")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func davRenameFileCmd() *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "rename-file FILE_ID NEW_TITLE",
|
||||||
|
Short: "Rename a Documents file (include extension in NEW_TITLE)",
|
||||||
|
Args: cobra.ExactArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := c.FileStore(onlyoffice.ProviderDAV).Rename(cmd.Context(), args[0], args[1]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"id": args[0], "title": args[1]})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func davRenameFolderCmd() *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "rename-folder FOLDER_ID NEW_TITLE",
|
||||||
|
Short: "Rename a Documents folder",
|
||||||
|
Args: cobra.ExactArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := c.FileStore(onlyoffice.ProviderDAV).Rename(cmd.Context(), args[0], args[1]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"id": args[0], "title": args[1]})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func davDownloadCmd() *cobra.Command {
|
||||||
|
var to string
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "download FILE_ID",
|
||||||
|
Short: "Download Documents file bytes (default path: ./<title>)",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ctx := cmd.Context()
|
||||||
|
store := c.FileStore(onlyoffice.ProviderDAV)
|
||||||
|
e, err := store.Stat(ctx, args[0])
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
path := to
|
||||||
|
if path == "" {
|
||||||
|
path = onlyoffice.SafeLocalFileName(e.Title)
|
||||||
|
}
|
||||||
|
out, err := os.Create(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer out.Close()
|
||||||
|
n, err := store.Download(ctx, args[0], out)
|
||||||
|
if err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{"path": path, "bytes": n})
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cmd.Flags().StringVar(&to, "to", "", "output path (default: ./<server title>)")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func davFileOpsCmd() *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "fileops",
|
||||||
|
Short: "List active file operations (move/copy status polling)",
|
||||||
|
Args: cobra.NoArgs,
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ops, err := c.ListFileOps(cmd.Context())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rows := make([]map[string]any, 0, len(ops))
|
||||||
|
for _, op := range ops {
|
||||||
|
rows = append(rows, map[string]any{
|
||||||
|
"id": fmt.Sprint(op["id"]),
|
||||||
|
"operation": fmt.Sprint(op["operation"]),
|
||||||
|
"progress": fmt.Sprint(op["progress"]),
|
||||||
|
"finished": fmt.Sprint(op["finished"]),
|
||||||
|
"error": fmt.Sprint(op["error"]),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
printTable([]string{"id", "operation", "progress", "finished", "error"}, rows)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
+158
@@ -0,0 +1,158 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
rootCmd.AddCommand(indexCmd())
|
||||||
|
}
|
||||||
|
|
||||||
|
// indexFlags are shared by the `oo index folder` and `oo index files` verbs.
|
||||||
|
type indexFlags struct {
|
||||||
|
recursive bool
|
||||||
|
exts string
|
||||||
|
limit int
|
||||||
|
workers int
|
||||||
|
lang string
|
||||||
|
minChars int
|
||||||
|
workDir string
|
||||||
|
backend string
|
||||||
|
dryRun bool
|
||||||
|
asJSON bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// indexCmd populates the own full-text index (oo_docs_text) that makes PDF and
|
||||||
|
// scanned content searchable. The OnlyOffice index is left untouched.
|
||||||
|
func indexCmd() *cobra.Command {
|
||||||
|
f := &indexFlags{}
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "index",
|
||||||
|
Short: "Populate the own full-text index for PDF/scan content",
|
||||||
|
Long: "Index document text that the OnlyOffice Elasticsearch index does not\n" +
|
||||||
|
"cover (PDFs and scans) into a separate index (ONLYOFFICE_ES_TEXT_INDEX,\n" +
|
||||||
|
"default oo_docs_text). Text is extracted with docpipe (pdftotext, OCR)\n" +
|
||||||
|
"and the OnlyOffice server is never modified.\n\n" +
|
||||||
|
"Requires ONLYOFFICE_URL/USER/PASS (to download files) and ONLYOFFICE_ES_URL\n" +
|
||||||
|
"(to write the index). See docs/elasticsearch.md.",
|
||||||
|
}
|
||||||
|
cmd.PersistentFlags().BoolVar(&f.recursive, "recursive", false, "folder: descend into subfolders")
|
||||||
|
cmd.PersistentFlags().StringVar(&f.exts, "exts", "pdf", "comma-separated extensions to index")
|
||||||
|
cmd.PersistentFlags().IntVar(&f.limit, "limit", 0, "maximum number of files to index (0 = all)")
|
||||||
|
cmd.PersistentFlags().IntVar(&f.workers, "workers", 3, "parallel downloads/extractions")
|
||||||
|
cmd.PersistentFlags().StringVar(&f.lang, "lang", "deu+eng", "OCR language(s)")
|
||||||
|
cmd.PersistentFlags().IntVar(&f.minChars, "min-chars", 0, "text-layer threshold below which OCR runs")
|
||||||
|
cmd.PersistentFlags().StringVar(&f.workDir, "work-dir", "", "temp dir for downloads (default: system temp)")
|
||||||
|
cmd.PersistentFlags().StringVar(&f.backend, "backend", "rest", "file backend: rest|dav")
|
||||||
|
cmd.PersistentFlags().BoolVar(&f.dryRun, "dry-run", false, "list what would be indexed, without changes")
|
||||||
|
cmd.PersistentFlags().BoolVar(&f.asJSON, "json", false, "shorthand for --output json")
|
||||||
|
|
||||||
|
cmd.AddCommand(indexFolderCmd(f), indexFilesCmd(f))
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func indexFolderCmd(f *indexFlags) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "folder FOLDER_ID",
|
||||||
|
Short: "Index every matching file in a Documents folder",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
return runIndex(cmd, f, args[0], nil)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func indexFilesCmd(f *indexFlags) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "files FILE_ID...",
|
||||||
|
Short: "Index specific Documents files",
|
||||||
|
Args: cobra.MinimumNArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
return runIndex(cmd, f, "", args)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func runIndex(cmd *cobra.Command, f *indexFlags, folderID string, ids []string) error {
|
||||||
|
if f.asJSON {
|
||||||
|
outputFormat = "json"
|
||||||
|
}
|
||||||
|
c, err := newOO(cmd)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
idx, err := onlyoffice.NewESTextIndex(onlyoffice.ESTextConfigFromEnv())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ti := onlyoffice.NewTextIndexer(c.FileStore(f.backend), idx)
|
||||||
|
ti.WorkDir = f.workDir
|
||||||
|
opts := onlyoffice.IndexOptions{
|
||||||
|
Recursive: f.recursive,
|
||||||
|
Extensions: splitList(f.exts),
|
||||||
|
Limit: f.limit,
|
||||||
|
Lang: f.lang,
|
||||||
|
MinChars: f.minChars,
|
||||||
|
Workers: f.workers,
|
||||||
|
}
|
||||||
|
ctx := cmd.Context()
|
||||||
|
|
||||||
|
if f.dryRun {
|
||||||
|
var entries []onlyoffice.Entry
|
||||||
|
if folderID != "" {
|
||||||
|
entries, err = ti.PlanFolder(ctx, folderID, opts)
|
||||||
|
} else {
|
||||||
|
entries, err = ti.PlanFiles(ctx, ids, opts)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rows := make([]map[string]any, 0, len(entries))
|
||||||
|
for _, e := range entries {
|
||||||
|
rows = append(rows, map[string]any{
|
||||||
|
"id": e.ID,
|
||||||
|
"title": e.Title,
|
||||||
|
"folder": e.ParentID,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
printTable([]string{"id", "title", "folder"}, rows)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := ti.Ensure(ctx); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var res onlyoffice.IndexResult
|
||||||
|
if folderID != "" {
|
||||||
|
res, err = ti.IndexFolder(ctx, folderID, opts)
|
||||||
|
} else {
|
||||||
|
res, err = ti.IndexFiles(ctx, ids, opts)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
printObject(map[string]any{
|
||||||
|
"index": idx.Index(),
|
||||||
|
"scanned": res.Scanned,
|
||||||
|
"indexed": res.Indexed,
|
||||||
|
"skipped": res.Skipped,
|
||||||
|
"failed": res.Failed,
|
||||||
|
"errors": res.Errors,
|
||||||
|
})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// splitList parses a comma-separated flag value, dropping blanks.
|
||||||
|
func splitList(s string) []string {
|
||||||
|
parts := strings.Split(s, ",")
|
||||||
|
out := make([]string, 0, len(parts))
|
||||||
|
for _, p := range parts {
|
||||||
|
if p = strings.TrimSpace(p); p != "" {
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestIndexCommandRegistered(t *testing.T) {
|
||||||
|
cmd, _, err := rootCmd.Find([]string{"index"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cmd.Name() != "index" {
|
||||||
|
t.Fatalf("index resolved to %q", cmd.Name())
|
||||||
|
}
|
||||||
|
for _, name := range []string{"exts", "limit", "workers", "lang", "min-chars", "work-dir", "backend", "dry-run", "recursive", "json"} {
|
||||||
|
if cmd.PersistentFlags().Lookup(name) == nil {
|
||||||
|
t.Errorf("index: missing --%s flag", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if cmd.PersistentFlags().Lookup("exts").DefValue != "pdf" {
|
||||||
|
t.Errorf("--exts default = %q, want pdf", cmd.PersistentFlags().Lookup("exts").DefValue)
|
||||||
|
}
|
||||||
|
for _, verb := range []string{"index folder", "index files"} {
|
||||||
|
sub, _, err := rootCmd.Find(strings.Fields(verb))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: %v", verb, err)
|
||||||
|
}
|
||||||
|
if sub.Name() != strings.Fields(verb)[1] {
|
||||||
|
t.Errorf("%s resolved to %q", verb, sub.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIndexSearchBackendFlag(t *testing.T) {
|
||||||
|
cmd, _, err := rootCmd.Find([]string{"search"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cmd.Flags().Lookup("backend") == nil {
|
||||||
|
t.Fatal("search: missing --backend flag")
|
||||||
|
}
|
||||||
|
if cmd.Flags().Lookup("backend").DefValue != "oo" {
|
||||||
|
t.Errorf("--backend default = %q, want oo", cmd.Flags().Lookup("backend").DefValue)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSplitList(t *testing.T) {
|
||||||
|
got := splitList(" pdf , .PDF, docx ,, ")
|
||||||
|
want := []string{"pdf", ".PDF", "docx"}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("splitList = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
+10
-6
@@ -3,19 +3,23 @@
|
|||||||
// Command tree is subject-based (mirrors the library split and the `tea` CLI):
|
// Command tree is subject-based (mirrors the library split and the `tea` CLI):
|
||||||
//
|
//
|
||||||
// oo calendar list | events | add | delete
|
// oo calendar list | events | add | delete
|
||||||
// oo projects list | get | milestones | create | update | delete | files (list|upload|download|rename|delete|as-md|put-md)
|
// oo projects list | get | milestones | milestone-create | create | update | delete | contacts (add|remove) | link-authors | link-git | files (list|upload|download|rename|delete|dedupe|as-md|put-md|put-txt|put-xlsx)
|
||||||
// oo tasks list | get | create | update | delete | subtask add | files (list|upload|detach)
|
// oo tasks list | get | create | update | delete | subtask add | files (list|upload|detach)
|
||||||
// oo users list | self (alias: oo whoami)
|
// oo users list | self (alias: oo whoami)
|
||||||
// oo contacts list | get | delete | info-add | merge | dedupe-info
|
// oo contacts list | get | delete | info-add | merge | dedupe-info | tags | tag-add | tag-create | tag-remove
|
||||||
// oo persons list | create | delete | dedupe
|
// oo persons list | create | delete | dedupe
|
||||||
// oo companies list | create | delete | dedupe | dedupe-persons
|
// oo companies list | create | delete | dedupe | dedupe-persons
|
||||||
// oo opportunities list | get | create | delete | stages | member-add | dedupe | dedupe-members | fix-titles
|
// oo opportunities list | get | create | update | delete | stages | member-add | dedupe | dedupe-members | fix-titles
|
||||||
// oo cases list | create | delete | member-add
|
// oo cases list | create | delete | member-add
|
||||||
// oo crm-tasks list | create | delete | categories
|
// oo crm-tasks list | create | delete | categories | reassign-self
|
||||||
// oo crm cleanup
|
// oo crm cleanup
|
||||||
// oo mails accounts | folders | list | get | download-attachment | draft | attach | draft-invoice | delete
|
// oo mails accounts | folders | list | get | download-attachment | draft | attach | draft-invoice | send | delete
|
||||||
// oo invoices list | get | create | update | pdf | pdf-cleanup | status | delete | items …
|
// oo invoices list | get | create | update | pdf | pdf-cleanup | status | delete | items …
|
||||||
// oo docs tools | convert | ocr | as-md | put-md
|
// oo docs tools | convert | optimize | ocr | hocr | as-md | put-md | put-txt | put-xlsx
|
||||||
|
// oo catalog match | merge | apply | scan-contacts | scan-projects | scan-thunderbird
|
||||||
|
// oo dav ls | move | copy | mkdir | rename-file | rename-folder | download | fileops
|
||||||
|
// oo search QUERY [--content] [--folder ID] [--limit N] [--backend oo|own] [--json]
|
||||||
|
// oo index folder FOLDER_ID | files FILE_ID... [--recursive] [--exts pdf] [--dry-run]
|
||||||
//
|
//
|
||||||
// CRM association rules: docs/crm-associations.md
|
// CRM association rules: docs/crm-associations.md
|
||||||
//
|
//
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package main
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -159,20 +160,22 @@ func prjFilesDownloadCmd() *cobra.Command {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
f, err := c.GetFile(cmd.Context(), args[0])
|
ctx := cmd.Context()
|
||||||
|
store := c.Files()
|
||||||
|
e, err := store.Stat(ctx, args[0])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
path := to
|
path := to
|
||||||
if path == "" {
|
if path == "" {
|
||||||
path = onlyoffice.SafeLocalFileName(onlyoffice.FileEntryTitle(f))
|
path = onlyoffice.SafeLocalFileName(e.Title)
|
||||||
}
|
}
|
||||||
out, err := os.Create(path)
|
out, err := os.Create(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
defer out.Close()
|
defer out.Close()
|
||||||
n, err := c.DownloadFile(cmd.Context(), args[0], out)
|
n, err := store.Download(ctx, args[0], out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = os.Remove(path)
|
_ = os.Remove(path)
|
||||||
return err
|
return err
|
||||||
@@ -199,11 +202,16 @@ func prjFilesRenameCmd() *cobra.Command {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
entry, err := c.RenameFile(cmd.Context(), args[0], args[1])
|
store := c.Files()
|
||||||
|
if err := store.Rename(cmd.Context(), args[0], args[1]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
entry, err := store.Stat(cmd.Context(), args[0])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
printObject(fileEntryToMap(entry))
|
entry.Title = args[1]
|
||||||
|
printObject(entryToMap(entry))
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -228,7 +236,7 @@ func prjFilesDeleteCmd() *cobra.Command {
|
|||||||
}
|
}
|
||||||
ids = append(ids, id)
|
ids = append(ids, id)
|
||||||
}
|
}
|
||||||
if err := c.DeleteFiles(cmd.Context(), ids); err != nil {
|
if err := c.Files().Delete(cmd.Context(), args); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
printObject(map[string]any{"deleted": ids})
|
printObject(map[string]any{"deleted": ids})
|
||||||
@@ -317,6 +325,29 @@ func fileEntryToMap(f *onlyoffice.FileEntry) map[string]any {
|
|||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// entryToMap renders a canonical Entry with the same keys as fileEntryToMap.
|
||||||
|
func entryToMap(e onlyoffice.Entry) map[string]any {
|
||||||
|
m := map[string]any{
|
||||||
|
"id": e.ID,
|
||||||
|
"title": e.Title,
|
||||||
|
"fileExst": filepath.Ext(e.Title),
|
||||||
|
"contentLength": contentLengthString(e.Size),
|
||||||
|
}
|
||||||
|
if e.Updated != "" {
|
||||||
|
m["updated"] = e.Updated
|
||||||
|
} else if !e.Modified.IsZero() {
|
||||||
|
m["updated"] = e.Modified.Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
func contentLengthString(n int64) string {
|
||||||
|
if n <= 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strconv.FormatInt(n, 10)
|
||||||
|
}
|
||||||
|
|
||||||
func fileIDStr(f *onlyoffice.FileEntry) string {
|
func fileIDStr(f *onlyoffice.FileEntry) string {
|
||||||
if f == nil || f.ID == nil {
|
if f == nil || f.ID == nil {
|
||||||
return ""
|
return ""
|
||||||
|
|||||||
@@ -0,0 +1,94 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
"github.com/eslider/go-onlyoffice/cmd/internal/bootstrap"
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
rootCmd.AddCommand(searchCmd())
|
||||||
|
}
|
||||||
|
|
||||||
|
// searchCmd queries the OnlyOffice document index through the file facade. The
|
||||||
|
// REST /api/2.0/files/@search endpoint only searches file names in the database;
|
||||||
|
// content search needs Elasticsearch (see docs/elasticsearch.md).
|
||||||
|
func searchCmd() *cobra.Command {
|
||||||
|
var (
|
||||||
|
content bool
|
||||||
|
folder string
|
||||||
|
limit int
|
||||||
|
backend string
|
||||||
|
asJSON bool
|
||||||
|
)
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "search QUERY",
|
||||||
|
Short: "Full-text search over documents by name, optionally by content (Elasticsearch)",
|
||||||
|
Long: "Search the OnlyOffice Documents index.\n\n" +
|
||||||
|
"By default only file names are matched. With --content the query also\n" +
|
||||||
|
"matches extracted document text (document.attachment.content); this covers\n" +
|
||||||
|
"Office formats (docx/xlsx/pptx) and is slower.\n\n" +
|
||||||
|
"--backend own queries the separate index populated by `oo index`\n" +
|
||||||
|
"(ONLYOFFICE_ES_TEXT_INDEX, default oo_docs_text) instead, which also holds\n" +
|
||||||
|
"PDFs and scans (see docs/elasticsearch.md).\n\n" +
|
||||||
|
"Requires ONLYOFFICE_ES_URL (and optionally ONLYOFFICE_ES_INDEX,\n" +
|
||||||
|
"ONLYOFFICE_TENANT). See docs/elasticsearch.md for the tunnel setup.",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
if asJSON {
|
||||||
|
outputFormat = "json"
|
||||||
|
}
|
||||||
|
bootstrap.LoadEnv()
|
||||||
|
c := onlyoffice.NewClient(onlyoffice.GetEnvironmentCredentials())
|
||||||
|
var (
|
||||||
|
searcher onlyoffice.Searcher
|
||||||
|
err error
|
||||||
|
)
|
||||||
|
switch strings.ToLower(strings.TrimSpace(backend)) {
|
||||||
|
case "", "oo", "elasticsearch":
|
||||||
|
searcher, err = c.Files().Search()
|
||||||
|
case "own", "es-text":
|
||||||
|
searcher, err = onlyoffice.NewESTextIndex(onlyoffice.ESTextConfigFromEnv())
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("unknown search backend %q (want oo|own)", backend)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
hits, err := searcher.Search(cmd.Context(), onlyoffice.SearchQuery{
|
||||||
|
Text: args[0],
|
||||||
|
InContent: content,
|
||||||
|
FolderID: folder,
|
||||||
|
Limit: limit,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rows := make([]map[string]any, 0, len(hits))
|
||||||
|
for _, h := range hits {
|
||||||
|
rows = append(rows, map[string]any{
|
||||||
|
"id": h.ID,
|
||||||
|
"title": h.Title,
|
||||||
|
"folder": h.ParentID,
|
||||||
|
"score": h.Score,
|
||||||
|
"highlight": h.Highlight,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if outputFormat == "json" {
|
||||||
|
printJSON(rows)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
printTable([]string{"id", "title", "folder", "score", "highlight"}, rows)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cmd.Flags().BoolVar(&content, "content", false, "also match extracted document content")
|
||||||
|
cmd.Flags().StringVar(&folder, "folder", "", "limit to a Documents folder id")
|
||||||
|
cmd.Flags().IntVar(&limit, "limit", 20, "maximum number of results")
|
||||||
|
cmd.Flags().StringVar(&backend, "backend", "oo", "index to query: oo (OnlyOffice) | own (oo index)")
|
||||||
|
cmd.Flags().BoolVar(&asJSON, "json", false, "shorthand for --output json")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSearchCommandRegisteredWithFlags(t *testing.T) {
|
||||||
|
cmd, _, err := rootCmd.Find([]string{"search"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cmd.Name() != "search" {
|
||||||
|
t.Fatalf("search resolved to %q", cmd.Name())
|
||||||
|
}
|
||||||
|
for _, name := range []string{"content", "folder", "limit", "json"} {
|
||||||
|
if cmd.Flags().Lookup(name) == nil {
|
||||||
|
t.Errorf("search: missing --%s flag", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := cmd.Flags().Lookup("limit").DefValue; got != "20" {
|
||||||
|
t.Errorf("--limit default = %q, want 20", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSearchWithoutESURLIsClearError(t *testing.T) {
|
||||||
|
clearEnv(t, "ONLYOFFICE_ES_URL", "ONLYOFFICE_ES_INDEX", "ONLYOFFICE_TENANT")
|
||||||
|
errBuf := &bytes.Buffer{}
|
||||||
|
rootCmd.SetErr(errBuf)
|
||||||
|
rootCmd.SetOut(&bytes.Buffer{})
|
||||||
|
rootCmd.SetArgs([]string{"search", "Rechnung"})
|
||||||
|
t.Cleanup(func() {
|
||||||
|
rootCmd.SetArgs(nil)
|
||||||
|
rootCmd.SetOut(nil)
|
||||||
|
rootCmd.SetErr(nil)
|
||||||
|
})
|
||||||
|
|
||||||
|
err := rootCmd.Execute()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error without ONLYOFFICE_ES_URL")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "ONLYOFFICE_ES_URL") {
|
||||||
|
t.Fatalf("error %q missing ONLYOFFICE_ES_URL", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
// Command ooscan recursively lists OnlyOffice Documents folders into a TSV
|
||||||
|
// index: file_id, folder_id, path, title.
|
||||||
|
//
|
||||||
|
// Usage: ooscan <FOLDER_ID> [<FOLDER_ID>...]
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
ctx := context.Background()
|
||||||
|
c := onlyoffice.NewClient(onlyoffice.GetEnvironmentCredentials())
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, root := range os.Args[1:] {
|
||||||
|
walk(ctx, c, root, "", 0, seen)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func walk(ctx context.Context, c *onlyoffice.Client, folderID, path string, depth int, seen map[string]bool) {
|
||||||
|
if depth > 8 || seen[folderID] {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
seen[folderID] = true
|
||||||
|
// Throttle: OnlyOffice rate-limits (429) and the host must not be flooded.
|
||||||
|
time.Sleep(350 * time.Millisecond)
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
var l *onlyoffice.DavListing
|
||||||
|
derr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
var err error
|
||||||
|
l, err = c.ListDavFolder(ctx, folderID)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
if derr != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "list %s (%s): %v\n", path, folderID, derr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, f := range l.Files {
|
||||||
|
fmt.Printf("%s\t%s\t%s\t%s\n", f.ID, folderID, path, f.Title)
|
||||||
|
}
|
||||||
|
for _, sub := range l.Folders {
|
||||||
|
walk(ctx, c, sub.ID, path+"/"+sub.Title, depth+1, seen)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,308 @@
|
|||||||
|
// Command pdfamount walks a Documents folder, downloads matching PDFs and
|
||||||
|
// extracts the payable amount, printing "file_id\ttitle\tamount".
|
||||||
|
//
|
||||||
|
// Usage: pdfamount <FOLDER_ID> [TITLE_FILTER_REGEX]
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
onlyoffice "github.com/eslider/go-onlyoffice"
|
||||||
|
)
|
||||||
|
|
||||||
|
// amountPat is the amount capture shared by every amount regex.
|
||||||
|
const amountPat = `([0-9]+(?:[.,][0-9]+)*)`
|
||||||
|
|
||||||
|
// amountRE builds "<label> [optional (comment)] [: -] <number>".
|
||||||
|
func amountRE(label string) *regexp.Regexp {
|
||||||
|
return regexp.MustCompile(
|
||||||
|
`(?i)\b` + regexp.QuoteMeta(label) + `\b\s*(?:\([^)]*\))?\s*[:\-]?\s*` + amountPat)
|
||||||
|
}
|
||||||
|
|
||||||
|
// amountRes lists the payable-amount patterns in strict priority order: the
|
||||||
|
// first pattern with a usable amount wins, and a lower-priority label can never
|
||||||
|
// override a higher-priority one ("zu zahlender betrag" > "rechnungsbetrag" >
|
||||||
|
// "rechnungsendbetrag" > "gesamtbetrag" > "gesamtsumme (inkl. steuern)").
|
||||||
|
//
|
||||||
|
// "gesamtbetrag" and "gesamtsumme" are not in the original set but are the real
|
||||||
|
// labels on Diashop invoices ("Gesamtsumme (inkl. Steuern)"). The inclusive
|
||||||
|
// variant is matched before a plain "gesamtsumme". Everything after those
|
||||||
|
// primary labels is the broader fallback set, consulted only when no primary
|
||||||
|
// label yields an amount. Within one pattern the last usable amount is taken,
|
||||||
|
// because totals usually come last.
|
||||||
|
var amountRes = []*regexp.Regexp{
|
||||||
|
amountRE("zu zahlender betrag"),
|
||||||
|
amountRE("rechnungsbetrag"),
|
||||||
|
amountRE("rechnungsendbetrag"),
|
||||||
|
amountRE("gesamtbetrag"),
|
||||||
|
regexp.MustCompile(`(?i)\bgesamtsumme\b\s*\(\s*inkl\.?\s*steuern\s*\)\s*[:\-]?\s*` + amountPat),
|
||||||
|
amountRE("gesamtsumme"),
|
||||||
|
amountRE("endbetrag"),
|
||||||
|
amountRE("zahlbetrag"),
|
||||||
|
amountRE("bruttobetrag"),
|
||||||
|
amountRE("betrag"),
|
||||||
|
amountRE("total"),
|
||||||
|
amountRE("summe"),
|
||||||
|
}
|
||||||
|
|
||||||
|
// taxLineRe marks a line whose number is a tax rate/percentage: an explicit
|
||||||
|
// percent sign or a VAT/tax keyword. "Steuern" (plural, as in "inkl. Steuern")
|
||||||
|
// is handled separately so the inclusive total stays usable.
|
||||||
|
var taxLineRe = regexp.MustCompile(`(?i)%|\bMwSt\b|\bUSt\b|\bProzent\b`)
|
||||||
|
|
||||||
|
// steuerRe finds "Steuer"/"Umsatzsteuer" etc. RE2 has no lookahead, so the
|
||||||
|
// plural "Steuern" is excluded in isTaxLine.
|
||||||
|
var steuerRe = regexp.MustCompile(`(?i)steuer`)
|
||||||
|
|
||||||
|
// percentAfterRe detects a percent sign directly after a number (spaces ok).
|
||||||
|
var percentAfterRe = regexp.MustCompile(`^\s*%`)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
if len(os.Args) < 2 {
|
||||||
|
fmt.Fprintln(os.Stderr, "usage: pdfamount <FOLDER_ID> [TITLE_FILTER_REGEX]")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
folder := os.Args[1]
|
||||||
|
filter := regexp.MustCompile(`(?i)rechnung`)
|
||||||
|
if len(os.Args) >= 3 {
|
||||||
|
filter = regexp.MustCompile(os.Args[2])
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
c := onlyoffice.NewClient(onlyoffice.GetEnvironmentCredentials())
|
||||||
|
|
||||||
|
files := listAll(ctx, c, folder)
|
||||||
|
for _, f := range files {
|
||||||
|
if !filter.MatchString(f.title) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(strings.ToLower(f.title), ".pdf") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
amount, err := pdfAmount(ctx, c, f.id)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "%s: %v\n", f.title, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if amount == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Printf("%s\t%s\t%s\n", f.id, f.title, amount)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type file struct{ id, title string }
|
||||||
|
|
||||||
|
func listAll(ctx context.Context, c *onlyoffice.Client, folder string) []file {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var out []file
|
||||||
|
var walk func(string)
|
||||||
|
walk = func(id string) {
|
||||||
|
if seen[id] {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
seen[id] = true
|
||||||
|
time.Sleep(300 * time.Millisecond)
|
||||||
|
l, err := c.ListDavFolder(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "list %s: %v\n", id, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, f := range l.Files {
|
||||||
|
out = append(out, file{f.ID, f.Title})
|
||||||
|
}
|
||||||
|
for _, sub := range l.Folders {
|
||||||
|
walk(sub.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
walk(folder)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func pdfAmount(ctx context.Context, c *onlyoffice.Client, id string) (string, error) {
|
||||||
|
time.Sleep(time.Second)
|
||||||
|
tmp, err := os.CreateTemp("", "pdf-*.pdf")
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
defer os.Remove(tmp.Name())
|
||||||
|
derr := onlyoffice.DoRetry(ctx, onlyoffice.DefaultRetryPolicy(), func() error {
|
||||||
|
_ = tmp.Truncate(0)
|
||||||
|
_, _ = tmp.Seek(0, 0)
|
||||||
|
_, err := c.DownloadFile(ctx, id, tmp)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
if derr != nil {
|
||||||
|
tmp.Close()
|
||||||
|
return "", derr
|
||||||
|
}
|
||||||
|
tmp.Close()
|
||||||
|
var buf bytes.Buffer
|
||||||
|
cmd := exec.CommandContext(ctx, "pdftotext", "-layout", tmp.Name(), "-")
|
||||||
|
cmd.Stdout = &buf
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return extractAmount(buf.String()), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// extractAmount returns the normalised ("1234.56") payable amount found in
|
||||||
|
// text, or "" if no usable amount matches.
|
||||||
|
//
|
||||||
|
// DKV invoices are special-cased first: they repeat a per-vehicle "TOTAL:" line
|
||||||
|
// and carry the real total only in the "Gesamtsummenaufstellung" section.
|
||||||
|
func extractAmount(text string) string {
|
||||||
|
if v, ok := dkvGrandTotal(text); ok {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
for _, re := range amountRes {
|
||||||
|
if v, ok := lastUsableAmount(text, re); ok {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// dkvGrandTotal extracts the total of a DKV "Gesamtsummenaufstellung" section.
|
||||||
|
//
|
||||||
|
// Rule: DKV invoices repeat a per-vehicle "TOTAL:" line, so the last TOTAL is
|
||||||
|
// not the invoice total. When a "Gesamtsummenaufstellung" section exists, its
|
||||||
|
// total wins over every "TOTAL:" line: the first amount after the "»" marker,
|
||||||
|
// or, if there is none, the last amount in the section. The section ends at the
|
||||||
|
// page break (form feed) or end of text.
|
||||||
|
func dkvGrandTotal(text string) (string, bool) {
|
||||||
|
idx := strings.Index(strings.ToLower(text), "gesamtsummenaufstellung")
|
||||||
|
if idx < 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
section := text[idx:]
|
||||||
|
if ff := strings.IndexByte(section, '\f'); ff >= 0 {
|
||||||
|
section = section[:ff]
|
||||||
|
}
|
||||||
|
if m := strings.Index(section, "»"); m >= 0 {
|
||||||
|
if v, ok := firstAmount(section[m:]); ok {
|
||||||
|
return v, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return lastAmount(section)
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastUsableAmount returns the last amount matched by re that is not a tax rate
|
||||||
|
// or percentage. Within one label the last usable amount wins.
|
||||||
|
func lastUsableAmount(text string, re *regexp.Regexp) (string, bool) {
|
||||||
|
ms := re.FindAllStringSubmatchIndex(text, -1)
|
||||||
|
for i := len(ms) - 1; i >= 0; i-- {
|
||||||
|
m := ms[i]
|
||||||
|
if isTaxRate(text, m[2], m[3]) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if v, ok := normalizeAmount(text[m[2]:m[3]]); ok {
|
||||||
|
return v, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
// isTaxRate reports whether the number at text[start:end] is a tax rate or a
|
||||||
|
// percentage instead of a payable amount. A candidate is rejected when the
|
||||||
|
// token right after the number is "%" or the number's line carries a percent
|
||||||
|
// sign or a tax keyword. Rejecting is deliberate: office matching treats a
|
||||||
|
// known-but-different amount as a hard disqualifier, so an empty result is
|
||||||
|
// safer than the VAT rate.
|
||||||
|
func isTaxRate(text string, start, end int) bool {
|
||||||
|
if percentAfterRe.MatchString(text[end:]) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
lineStart := strings.LastIndexByte(text[:start], '\n') + 1
|
||||||
|
line := text[lineStart:]
|
||||||
|
if n := strings.IndexByte(text[end:], '\n'); n >= 0 {
|
||||||
|
line = text[lineStart : end+n]
|
||||||
|
}
|
||||||
|
return isTaxLine(line)
|
||||||
|
}
|
||||||
|
|
||||||
|
// isTaxLine reports whether a line looks like a tax rate rather than a payable
|
||||||
|
// amount. "Steuern" is treated as a qualifier ("inkl. Steuern"), not a rate.
|
||||||
|
func isTaxLine(line string) bool {
|
||||||
|
if taxLineRe.MatchString(line) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, loc := range steuerRe.FindAllStringIndex(line, -1) {
|
||||||
|
if loc[1] >= len(line) || (line[loc[1]] != 'n' && line[loc[1]] != 'N') {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// numberRe finds bare numbers (with optional thousands/decimal separators).
|
||||||
|
var numberRe = regexp.MustCompile(`[0-9]+(?:[.,][0-9]+)*`)
|
||||||
|
|
||||||
|
func firstAmount(s string) (string, bool) {
|
||||||
|
for _, m := range numberRe.FindAllString(s, -1) {
|
||||||
|
if v, ok := normalizeAmount(m); ok {
|
||||||
|
return v, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
func lastAmount(s string) (string, bool) {
|
||||||
|
ms := numberRe.FindAllString(s, -1)
|
||||||
|
for i := len(ms) - 1; i >= 0; i-- {
|
||||||
|
if v, ok := normalizeAmount(ms[i]); ok {
|
||||||
|
return v, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
// normalizeAmount turns "1.234,56" (DE), "1,234.56" (EN) or "1234.56" into
|
||||||
|
// "1234.56". The rightmost separator is decimal only when followed by one or
|
||||||
|
// two digits; otherwise every separator is a thousands separator.
|
||||||
|
func normalizeAmount(s string) (string, bool) {
|
||||||
|
last := -1
|
||||||
|
for i := 0; i < len(s); i++ {
|
||||||
|
if s[i] == '.' || s[i] == ',' {
|
||||||
|
last = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var dec byte
|
||||||
|
if last >= 0 {
|
||||||
|
digits := 0
|
||||||
|
for i := last + 1; i < len(s); i++ {
|
||||||
|
if s[i] < '0' || s[i] > '9' {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
digits++
|
||||||
|
}
|
||||||
|
if digits == 1 || digits == 2 {
|
||||||
|
dec = s[last]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var b strings.Builder
|
||||||
|
for i := 0; i < len(s); i++ {
|
||||||
|
switch c := s[i]; {
|
||||||
|
case c >= '0' && c <= '9':
|
||||||
|
b.WriteByte(c)
|
||||||
|
case (c == '.' || c == ',') && c == dec:
|
||||||
|
b.WriteByte('.')
|
||||||
|
case c == '.' || c == ',':
|
||||||
|
// thousands separator
|
||||||
|
default:
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
v, err := strconv.ParseFloat(b.String(), 64)
|
||||||
|
if err != nil {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return strconv.FormatFloat(v, 'f', 2, 64), true
|
||||||
|
}
|
||||||
@@ -0,0 +1,175 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestExtractAmount(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name, text, want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "rechnungsbetrag de format",
|
||||||
|
text: "Rechnungsbetrag: 1.234,56 €",
|
||||||
|
want: "1234.56",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rechnungsbetrag en thousands and dot",
|
||||||
|
text: "Rechnungsbetrag: 1,234.56",
|
||||||
|
want: "1234.56",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rechnungsbetrag plain dot",
|
||||||
|
text: "Rechnungsbetrag: 1234.56",
|
||||||
|
want: "1234.56",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rechnungsbetrag de comma only",
|
||||||
|
text: "Rechnungsbetrag: 1234,56",
|
||||||
|
want: "1234.56",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "currency suffix eur",
|
||||||
|
text: "Rechnungsbetrag: 1.234,56 EUR",
|
||||||
|
want: "1234.56",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "zu zahlender betrag wins over rechnungsbetrag",
|
||||||
|
text: "Zu zahlender Betrag: 10,00\nRechnungsbetrag: 99,00",
|
||||||
|
want: "10.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rechnungsbetrag wins over endbetrag",
|
||||||
|
text: "Endbetrag: 20,00\nRechnungsbetrag: 30,00",
|
||||||
|
want: "30.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "bruttobetrag wins over bare betrag",
|
||||||
|
text: "Bruttobetrag: 50,00\nBetrag: 10,00",
|
||||||
|
want: "50.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "gesamtbetrag wins over bare betrag",
|
||||||
|
text: "Gesamtbetrag: 80,00\nBetrag: 10,00",
|
||||||
|
want: "80.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "last occurrence of same label wins",
|
||||||
|
text: "Rechnungsbetrag: 10,00\nRechnungsbetrag: 20,00",
|
||||||
|
want: "20.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "endbetrag fallback",
|
||||||
|
text: "Endbetrag: 42,00",
|
||||||
|
want: "42.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "zahlbetrag fallback without colon",
|
||||||
|
text: "Zahlbetrag 7,50 €",
|
||||||
|
want: "7.50",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rechnungsendbetrag beats endbetrag",
|
||||||
|
text: "Rechnungsendbetrag: 12,00\nEndbetrag: 13,00",
|
||||||
|
want: "12.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "dkv style total line",
|
||||||
|
text: "Kundenbezogene Daten\n» TOTAL: 123,45 100,00 23,45 123,45\n",
|
||||||
|
want: "123.45",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "dkv gesamtsummenaufstellung grand total after marker",
|
||||||
|
text: "» TOTAL: 111,11 100,00 11,11 111,11\n" +
|
||||||
|
"» TOTAL: 222,22 200,00 22,22 222,22\n" +
|
||||||
|
"Gesamtsummenaufstellung\n" +
|
||||||
|
"Netto 240,00\n" +
|
||||||
|
"MwSt 47,25\n" +
|
||||||
|
"» 287,25\n",
|
||||||
|
want: "287.25",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "dkv gesamtsummenaufstellung total on next line",
|
||||||
|
text: "» TOTAL: 111,11\nGesamtsummenaufstellung\n»\n287,25\n",
|
||||||
|
want: "287.25",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "tax rate with percent sign is not an amount",
|
||||||
|
text: "Betrag: 19,00 % MwSt",
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "mehrwertsteuer rate is not an amount",
|
||||||
|
text: "Gesamtsumme: 19,00% MwSt",
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "steuer word on the number line rejects it",
|
||||||
|
text: "Betrag: 2,83 Steuer",
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "rejected primary falls back to a usable label",
|
||||||
|
text: "Gesamtsumme: 19,00 % MwSt\nEndbetrag: 42,00",
|
||||||
|
want: "42.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "labeled zu zahlender betrag beats unlabeled larger number",
|
||||||
|
text: "unlabeled 999,99\nZu zahlender Betrag: 10,00",
|
||||||
|
want: "10.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "labeled zu zahlender betrag beats lower label larger number",
|
||||||
|
text: "Endbetrag: 999,99\nZu zahlender Betrag: 10,00",
|
||||||
|
want: "10.00",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "diashop style gesamtsumme with comment",
|
||||||
|
text: "Zwischensumme\n12,34 €\nZwischensumme\n12,34 €\nVersand & Bearbeitung\n4,95 €\nGesamtsumme (inkl. Steuern)\n17,29 €\n",
|
||||||
|
want: "17.29",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "diashop picks inclusive total last",
|
||||||
|
text: "Gesamtsumme (exkl. Steuern)\n12,34 €\nGesamtsumme (inkl. Steuern)\n17,29 €",
|
||||||
|
want: "17.29",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "no label",
|
||||||
|
text: "some text without any amount label 12,34",
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := extractAmount(tc.text); got != tc.want {
|
||||||
|
t.Fatalf("extractAmount()=%q want %q", got, tc.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeAmount(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
in string
|
||||||
|
want string
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{"1.234,56", "1234.56", true},
|
||||||
|
{"1,234.56", "1234.56", true},
|
||||||
|
{"1234.56", "1234.56", true},
|
||||||
|
{"1234,56", "1234.56", true},
|
||||||
|
{"1.234.567,89", "1234567.89", true},
|
||||||
|
{"1,234,567.89", "1234567.89", true},
|
||||||
|
{"1.234", "1234.00", true},
|
||||||
|
{"12,5", "12.50", true},
|
||||||
|
{"12", "12.00", true},
|
||||||
|
{"", "0.00", false},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
got, ok := normalizeAmount(tc.in)
|
||||||
|
if ok != tc.ok {
|
||||||
|
t.Fatalf("normalizeAmount(%q) ok=%v want %v", tc.in, ok, tc.ok)
|
||||||
|
}
|
||||||
|
if ok && got != tc.want {
|
||||||
|
t.Fatalf("normalizeAmount(%q)=%q want %q", tc.in, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -15,10 +15,15 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// ListContacts returns a page of CRM contacts and the total count.
|
// ListContacts returns a page of CRM contacts and the total count.
|
||||||
|
// sortBy=id is always set: OnlyOffice filter.json without an explicit sort
|
||||||
|
// order is non-deterministic on large contact sets, so a paged walk
|
||||||
|
// (ListAllContacts, ListContactsByTag, FindCompany, FindPerson) can skip or
|
||||||
|
// duplicate contacts across page boundaries.
|
||||||
func (c *Client) ListContacts(ctx context.Context, count, startIndex int, search string) ([]map[string]any, int, error) {
|
func (c *Client) ListContacts(ctx context.Context, count, startIndex int, search string) ([]map[string]any, int, error) {
|
||||||
q := url.Values{}
|
q := url.Values{}
|
||||||
q.Set("count", strconv.Itoa(count))
|
q.Set("count", strconv.Itoa(count))
|
||||||
q.Set("startIndex", strconv.Itoa(startIndex))
|
q.Set("startIndex", strconv.Itoa(startIndex))
|
||||||
|
q.Set("sortBy", "id")
|
||||||
if search != "" {
|
if search != "" {
|
||||||
q.Set("filterValue", search)
|
q.Set("filterValue", search)
|
||||||
}
|
}
|
||||||
@@ -243,6 +248,24 @@ func (c *Client) DeleteContact(ctx context.Context, contactID string) (map[strin
|
|||||||
return c.deleteObject(ctx, fmt.Sprintf("/api/2.0/crm/contact/%s.json", url.PathEscape(contactID)))
|
return c.deleteObject(ctx, fmt.Sprintf("/api/2.0/crm/contact/%s.json", url.PathEscape(contactID)))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateContactName renames a CRM company contact displayName.
|
||||||
|
// Uses the company endpoint (person names go through /crm/contact/person/{id}).
|
||||||
|
func (c *Client) UpdateContactName(ctx context.Context, contactID, newName string) (map[string]any, error) {
|
||||||
|
body := map[string]any{
|
||||||
|
"displayName": newName,
|
||||||
|
"companyName": newName,
|
||||||
|
"isCompany": true,
|
||||||
|
}
|
||||||
|
out, err := c.putJSONObject(ctx, fmt.Sprintf("/api/2.0/crm/contact/company/%s.json", url.PathEscape(contactID)), body)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
if fresh, gerr := c.GetContact(ctx, contactID); gerr == nil && fresh != nil {
|
||||||
|
out = fresh
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
// ListContactTags returns all CRM contact tags (title + relativeItemsCount).
|
// ListContactTags returns all CRM contact tags (title + relativeItemsCount).
|
||||||
func (c *Client) ListContactTags(ctx context.Context) ([]map[string]any, error) {
|
func (c *Client) ListContactTags(ctx context.Context) ([]map[string]any, error) {
|
||||||
return c.ResponseArray(ctx, "/api/2.0/crm/contact/tag.json")
|
return c.ResponseArray(ctx, "/api/2.0/crm/contact/tag.json")
|
||||||
@@ -288,6 +311,7 @@ func (c *Client) ListContactsByTag(ctx context.Context, tagName string, count, s
|
|||||||
q.Set("count", strconv.Itoa(count))
|
q.Set("count", strconv.Itoa(count))
|
||||||
q.Set("startIndex", strconv.Itoa(startIndex))
|
q.Set("startIndex", strconv.Itoa(startIndex))
|
||||||
q.Set("tags", tagName)
|
q.Set("tags", tagName)
|
||||||
|
q.Set("sortBy", "id")
|
||||||
raw, err := c.getJSON(ctx, "/api/2.0/crm/contact/filter.json?"+q.Encode())
|
raw, err := c.getJSON(ctx, "/api/2.0/crm/contact/filter.json?"+q.Encode())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, 0, err
|
return nil, 0, err
|
||||||
@@ -717,6 +741,11 @@ func (c *Client) DeleteCRMTask(ctx context.Context, id string) (map[string]any,
|
|||||||
return c.deleteObject(ctx, fmt.Sprintf("/api/2.0/crm/task/%s.json", url.PathEscape(id)))
|
return c.deleteObject(ctx, fmt.Sprintf("/api/2.0/crm/task/%s.json", url.PathEscape(id)))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CloseCRMTask closes (completes) a CRM task via the task close endpoint.
|
||||||
|
func (c *Client) CloseCRMTask(ctx context.Context, id string) (map[string]any, error) {
|
||||||
|
return c.putFormObject(ctx, fmt.Sprintf("/api/2.0/crm/task/%s/close.json", url.PathEscape(id)), url.Values{})
|
||||||
|
}
|
||||||
|
|
||||||
// ListTaskCategories returns CRM task categories.
|
// ListTaskCategories returns CRM task categories.
|
||||||
func (c *Client) ListTaskCategories(ctx context.Context) ([]map[string]any, error) {
|
func (c *Client) ListTaskCategories(ctx context.Context) ([]map[string]any, error) {
|
||||||
return c.ResponseArray(ctx, "/api/2.0/crm/task/category.json")
|
return c.ResponseArray(ctx, "/api/2.0/crm/task/category.json")
|
||||||
|
|||||||
@@ -0,0 +1,130 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
status: current
|
||||||
|
related:
|
||||||
|
- README.md
|
||||||
|
- file_pg.go
|
||||||
|
- docs/elasticsearch.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# Community Server DB — прямой SQL-доступ (read-only)
|
||||||
|
|
||||||
|
## Что это
|
||||||
|
|
||||||
|
Бэкенд `pgStore` (`file_pg.go`) читает файлы и папки **напрямую из БД
|
||||||
|
Community Server**, без HTTP-слоя. Реализует `FileStore` (`List`/`Stat`/
|
||||||
|
`Download`) и `Searcher` по имени. Запись запрещена: все write-методы
|
||||||
|
возвращают `ErrReadOnly`.
|
||||||
|
|
||||||
|
## Что за БД (research, live)
|
||||||
|
|
||||||
|
Проверено на VM `onlyoffice-v2` (SSH `127.0.0.1:32`):
|
||||||
|
|
||||||
|
- Community Server работает на **MySQL 8.0**, не на PostgreSQL.
|
||||||
|
- Хост: `127.0.0.1:3306` внутри VM, база `onlyoffice`.
|
||||||
|
- Конфиг: `/etc/onlyoffice/communityserver/appsettings.production.json`,
|
||||||
|
`providerName: MySql.Data.MySqlClient`.
|
||||||
|
- Таблицы: `files_file`, `files_folder`, `files_folder_tree`,
|
||||||
|
`files_security`, тенанты — `tenants_tenants` (не `tenants`).
|
||||||
|
- PostgreSQL 16 в той же VM — **наш** контур (`edw_docs`, роли `edw`/`edw_ro`,
|
||||||
|
office-assistant), к OnlyOffice отношения не имеет. `files_file` в PG нет.
|
||||||
|
- Портал хранит файлы в **S3/MinIO** (DiscStorage только для мелочи).
|
||||||
|
Бакет `office`, объект — по ключу (см. ниже).
|
||||||
|
|
||||||
|
Вывод: бэкенд назван по issue «PostgreSQL», но живой источник — MySQL.
|
||||||
|
`database/sql` + драйвер по DSN: `mysql` для MySQL, `pgx` для PostgreSQL.
|
||||||
|
`Name()` возвращает фактический движок (`mysql` или `postgres`).
|
||||||
|
|
||||||
|
## Схема
|
||||||
|
|
||||||
|
`files_file` — одна строка **на версию** (PK `tenant_id, id, version`):
|
||||||
|
|
||||||
|
| поле | смысл |
|
||||||
|
|------|-------|
|
||||||
|
| `id` | id файла (тот же, что в REST/ES) |
|
||||||
|
| `version` | номер версии этой строки |
|
||||||
|
| `version_group` | номер версии |
|
||||||
|
| `current_version` | `1` = текущая версия, `0` = старая |
|
||||||
|
| `folder_id` | id родительской папки |
|
||||||
|
| `title` | имя файла с расширением |
|
||||||
|
| `content_length` | размер в байтах |
|
||||||
|
| `create_on`, `modified_on` | даты (UTC, без зоны) |
|
||||||
|
| `tenant_id` | тенант (портал) |
|
||||||
|
|
||||||
|
`files_folder`: `id`, `parent_id`, `title`, `create_on`, `modified_on`,
|
||||||
|
`tenant_id`. `files_folder_tree`: `folder_id`, `parent_id`, `level` — готовое
|
||||||
|
дерево, пока не используется.
|
||||||
|
|
||||||
|
Текущую строку файла берём по `current_version = 1`.
|
||||||
|
|
||||||
|
## Доступ (SSH-туннель)
|
||||||
|
|
||||||
|
MySQL слушает только `127.0.0.1:3306` внутри VM. Снаружи — SSH-туннель
|
||||||
|
(SSH в VM открыт как `127.0.0.1:32`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -f -N -o ControlMaster=no -o ControlPath=none \
|
||||||
|
-p 32 -i ~/.ssh/id_ed25519 \
|
||||||
|
-L 13306:127.0.0.1:3306 root@127.0.0.1
|
||||||
|
|
||||||
|
# MySQL DSN затем:
|
||||||
|
# root:<pw>@tcp(127.0.0.1:13306)/onlyoffice?parseTime=true
|
||||||
|
```
|
||||||
|
|
||||||
|
`-o ControlMaster=no -o ControlPath=none` обязательны: иначе forward уходит в
|
||||||
|
persistent master из `~/.ssh/config`.
|
||||||
|
|
||||||
|
## Переменные
|
||||||
|
|
||||||
|
| env | default | смысл |
|
||||||
|
|-----|---------|-------|
|
||||||
|
| `ONLYOFFICE_DSN` | — | DSN драйвера (MySQL `...@tcp(...)/...` или `postgres://...`) |
|
||||||
|
| `ONLYOFFICE_PG_DRIVER` | авто | `postgres` или `mysql`; иначе по форме DSN |
|
||||||
|
| `ONLYOFFICE_PG_TENANT` | `ONLYOFFICE_TENANT` | фильтр `tenant_id` (пусто = все) |
|
||||||
|
| `ONLYOFFICE_PG_HOST/PORT/USER/PASSWORD/DBNAME/SSLMODE` | — | собрать PG DSN, если `ONLYOFFICE_DSN` пуст |
|
||||||
|
|
||||||
|
Имена — в [`.env.example`](../.env.example). Секретов нет.
|
||||||
|
|
||||||
|
## Download (MinIO)
|
||||||
|
|
||||||
|
`Download` не ходит в REST. Ключ объекта собирается из строки `files_file`:
|
||||||
|
|
||||||
|
```
|
||||||
|
00/00/<tenant>/files/folder_<shard>/file_<id>/v<version>/content.<ext>
|
||||||
|
shard = (id/1000 + 1) * 1000
|
||||||
|
```
|
||||||
|
|
||||||
|
`shard` — не `folder_id`, а следующая тысяча над `id` (файл 3727 →
|
||||||
|
`folder_4000`). Проверено live по бакету `office`.
|
||||||
|
|
||||||
|
Стриминг переиспользует `downloadMinioObject` из `storage_fallback.go`
|
||||||
|
(та же подпись SigV4 и `MINIO_*`), без дублирования.
|
||||||
|
|
||||||
|
## Тесты
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./... # unit: rebind, csObjectKey, маппинг
|
||||||
|
go test -race ./...
|
||||||
|
|
||||||
|
# integration (нужен DSN; skip без него)
|
||||||
|
ONLYOFFICE_DSN='root:<pw>@tcp(127.0.0.1:13306)/onlyoffice?parseTime=true' \
|
||||||
|
ONLYOFFICE_PG_TENANT=1 \
|
||||||
|
ONLYOFFICE_PG_TEST_FILE_ID=22484 \
|
||||||
|
ONLYOFFICE_PG_TEST_FOLDER_ID=649 \
|
||||||
|
MINIO_ACCESS_KEY=... MINIO_SECRET_KEY=... \
|
||||||
|
go test -tags=integration -run TestIntegrationPGStore -v ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Integration сверяет `Stat`/`List`/`Download` с REST (`c.Files()`) и проверяет,
|
||||||
|
что write-методы дают `ErrReadOnly`.
|
||||||
|
|
||||||
|
## Грабли
|
||||||
|
|
||||||
|
- MySQL хранит `datetime` без зоны; `parseTime=true` (ставится автоматически)
|
||||||
|
читает их как UTC. REST отдаёт `+02:00` — сравнивать моменты, не строки.
|
||||||
|
- `GetFile` (REST) не отдаёт `contentLength` — размер сверять с `Stat` SQL.
|
||||||
|
- Один файл = много строк `files_file` (по версиям). Без `current_version = 1`
|
||||||
|
получите дубликаты.
|
||||||
|
- `folder_id` не входит в ключ MinIO; ключ считает `shard` от `id`.
|
||||||
|
- Searcher SQL ищет только по имени (`LIKE`). Контент — Elasticsearch
|
||||||
|
([elasticsearch.md](elasticsearch.md)).
|
||||||
@@ -0,0 +1,255 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
status: current
|
||||||
|
related:
|
||||||
|
- README.md
|
||||||
|
- file_es.go
|
||||||
|
---
|
||||||
|
|
||||||
|
# Elasticsearch — полнотекстовый поиск OnlyOffice
|
||||||
|
|
||||||
|
## Что это
|
||||||
|
|
||||||
|
Полнотекстовый поиск OnlyOffice Workspace работает на **Elasticsearch**.
|
||||||
|
Клиент на сервере — NEST. Индекс — имя таблицы.
|
||||||
|
|
||||||
|
Для файлов индекс `files_file`:
|
||||||
|
|
||||||
|
| поле | тип | смысл |
|
||||||
|
|------|-----|-------|
|
||||||
|
| `id` | integer | id файла (тот же, что в REST/Documents) |
|
||||||
|
| `title` | text (`whitespacecustom`) | имя файла |
|
||||||
|
| `tenantId` | integer | тенант (портал) |
|
||||||
|
| `folders` | nested | список папок: `folderId` (строка), `id`, `tenantId` |
|
||||||
|
| `document.attachment.content` | text (`document`) | извлеченный текст (ingest-attachment) |
|
||||||
|
| `document.attachment.content_type` | text | MIME |
|
||||||
|
|
||||||
|
Важно:
|
||||||
|
- Живой сервер — **Elasticsearch 7.16.3**, кластер `elasticsearch`.
|
||||||
|
- REST `GET /api/2.0/files/@search/{query}` ищет **только по имени в БД**
|
||||||
|
(`fileDao.Search`), ES не задействует. Для поиска по содержимому нужен
|
||||||
|
прямой ES — это и делает `oo search`.
|
||||||
|
- `title` analyzer `whitespacecustom` режет по пробелам и lower-case. Полное
|
||||||
|
имя файла — один токен (`Rechnung-4711.pdf`), поэтому поиск по имени ищет
|
||||||
|
слово целиком, а не подстроку.
|
||||||
|
- `document.attachment.content` заполняется **только для Office-форматов**
|
||||||
|
(docx / xlsx / pptx). У PDF/txt, залитых через API, контент не извлекается.
|
||||||
|
- Индексация асинхронная (TeamLabSvc) — файл появляется в ES не мгновенно.
|
||||||
|
|
||||||
|
## Доступ
|
||||||
|
|
||||||
|
ES слушает `127.0.0.1:9200` **внутри** VM OnlyOffice. Снаружи порт закрыт,
|
||||||
|
SSH в VM открыт на хосте как `127.0.0.1:32` (контейнер `onlyoffice-v2`,
|
||||||
|
QEMU). Схема — SSH-туннель.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# из корня go-onlyoffice (ключ и хост — как в infra-доках)
|
||||||
|
ssh -f -N -o ControlMaster=no -o ControlPath=none \
|
||||||
|
-p 32 -i ~/.ssh/id_ed25519 \
|
||||||
|
-L 9200:127.0.0.1:9200 root@127.0.0.1
|
||||||
|
|
||||||
|
curl -s http://127.0.0.1:9200/ | head # tagline + version
|
||||||
|
curl -s 'http://127.0.0.1:9200/_cat/indices?h=index,docs.count'
|
||||||
|
```
|
||||||
|
|
||||||
|
`-o ControlMaster=no -o ControlPath=none` обязательны: иначе forward уходит
|
||||||
|
в persistent master-соединение из `~/.ssh/config` и порт остаётся занят.
|
||||||
|
|
||||||
|
Проверить, что туннель жив:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://127.0.0.1:9200/files_file/_count
|
||||||
|
```
|
||||||
|
|
||||||
|
## Переменные
|
||||||
|
|
||||||
|
| env | default | смысл |
|
||||||
|
|-----|---------|-------|
|
||||||
|
| `ONLYOFFICE_ES_URL` | — (обязателен) | `scheme://host:port` ES |
|
||||||
|
| `ONLYOFFICE_ES_INDEX` | `files_file` | индекс |
|
||||||
|
| `ONLYOFFICE_TENANT` | пусто (все) | фильтр `tenantId` |
|
||||||
|
|
||||||
|
Имена — в [`.env.example`](../.env.example). Секретов нет: ES без пароля.
|
||||||
|
|
||||||
|
## CLI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ONLYOFFICE_ES_URL=http://127.0.0.1:9200 oo search "Rechnung"
|
||||||
|
ONLYOFFICE_ES_URL=http://127.0.0.1:9200 oo search "Mahngebühr" --content
|
||||||
|
oo search "Rechnung" --folder 649 --limit 50 --json
|
||||||
|
```
|
||||||
|
|
||||||
|
Флаги: `--content` (искать и по тексту), `--folder ID` (папка
|
||||||
|
`folders.folderId`), `--limit N` (по умолчанию 20, максимум 200),
|
||||||
|
`--json` = `-o json`.
|
||||||
|
|
||||||
|
## Библиотека
|
||||||
|
|
||||||
|
`file_es.go` — `ESSearcher` (`Name() = "elasticsearch"`), прямой ES REST на
|
||||||
|
stdlib `net/http`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
es, _ := onlyoffice.NewESSearcher(onlyoffice.ESConfigFromEnv())
|
||||||
|
hits, _ := es.Search(ctx, onlyoffice.SearchQuery{
|
||||||
|
Text: "Rechnung", InContent: true, Limit: 20,
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
Запрос: `multi_match` по `title^2` (+ `document.attachment.content` при
|
||||||
|
`InContent`), фильтры `tenantId` и `folders.folderId`, `_source`
|
||||||
|
id/title/folders, `highlight` для фрагмента. Ответ → `[]SearchHit` (модель из
|
||||||
|
эпика #34; пока объявлена в `file_es.go`, переедет в `file_core.go` с F1 #35).
|
||||||
|
|
||||||
|
## Тесты
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# unit — чистые builders/парсеры, без сети
|
||||||
|
go test ./ -run ES
|
||||||
|
|
||||||
|
# integration — нужен ONLYOFFICE_ES_URL (+ креды REST для залива)
|
||||||
|
set -a; . .env; set +a
|
||||||
|
ONLYOFFICE_ES_URL=http://127.0.0.1:9200 ONLYOFFICE_TENANT=1 \
|
||||||
|
go test -tags=integration -run TestIntegrationESSearch -v .
|
||||||
|
```
|
||||||
|
|
||||||
|
Интеграционный тест заливает временный xlsx (в имени и в ячейке — уникальные
|
||||||
|
токены), ждёт индексации, проверяет поиск по имени и по содержимому, затем
|
||||||
|
удаляет проект.
|
||||||
|
|
||||||
|
## Грабли
|
||||||
|
|
||||||
|
- `locale`/версия ES: 7.16.3, `_search` совместим с REST 7.x.
|
||||||
|
- ES без auth и слушает только localhost — туннель обязателен.
|
||||||
|
- Фильтр `tenantId` сузит выдачу; без него видны документы всех тенантов.
|
||||||
|
- Поиск по содержимому PDF в индексе OnlyOffice не работает (для PDF нет
|
||||||
|
`attachment.content`) — только Office-форматы. Решение для PDF — свой индекс
|
||||||
|
`oo_docs_text` (F6 #42), см. ниже.
|
||||||
|
|
||||||
|
# PDF и сканы — свой индекс (F6 #42)
|
||||||
|
|
||||||
|
## Проблема
|
||||||
|
|
||||||
|
`oo search --content "S1019"` не находил номер внутри PDF-счёта: в индексе
|
||||||
|
OnlyOffice PDF лежит только по имени.
|
||||||
|
|
||||||
|
## Почему PDF исключён (исходники CommunityServer)
|
||||||
|
|
||||||
|
Разобрано в `ONLYOFFICE/CommunityServer`:
|
||||||
|
|
||||||
|
- `web/core/ASC.Web.Core/Files/FileUtility.cs` — `CanIndex(fileName)` читает
|
||||||
|
серверную настройку `files.index.formats` (в `web/studio/ASC.Web.Studio/web.appsettings.config`
|
||||||
|
значение по умолчанию `".pptx|.xlsx|.docx"`).
|
||||||
|
- `web/studio/ASC.Web.Studio/Products/Files/Core/Search/FilesWrapper.cs` —
|
||||||
|
`GetDocumentStream*` возвращает `null`, если `!FileUtility.CanIndex(Title)`,
|
||||||
|
файл зашифрован или больше `MaxFileSize`.
|
||||||
|
- `module/ASC.ElasticSearch/Core/WrapperWithDoc.cs` + mapping в `Wrapper.cs` —
|
||||||
|
маппинг `document.attachment.content` и ingest-pipeline `attachments`
|
||||||
|
формат-агностичны: они распарсят любой поток.
|
||||||
|
|
||||||
|
Вывод: PDF исключён **только настройкой** `files.index.formats`; жёсткого
|
||||||
|
ограничения на формат в коде нет.
|
||||||
|
|
||||||
|
## Варианты и решение
|
||||||
|
|
||||||
|
| # | Вариант | Оценка |
|
||||||
|
|---|---------|--------|
|
||||||
|
| a | Включить `.pdf` в `files.index.formats` + reindex | Правка сервера OO; настройка может потеряться при обновлении; полный reindex 39k док-в; Tika **не OCR** — сканы без текстового слоя дадут пустой контент. Отклонён без решения PO. |
|
||||||
|
| b | Server-side ingest/attachment для PDF | По факту то же, что (a): сервер кормит поток только для `CanIndex`. |
|
||||||
|
| c | **Свой индекс** `oo_docs_text`, наполняемый `internal/docpipe` | **Выбран.** Сервер OO не трогаем; детерминированно; работает OCR для сканов; независимо от обновлений OO; любые форматы; фильтры папка/тип. |
|
||||||
|
| d | Локальный поиск без индекса | Отклонён как основной: качаем и извлекаем на каждый запрос, нет выдачи/ранжирования/highlight. |
|
||||||
|
|
||||||
|
Итог: **вариант c**. Индекс OnlyOffice (`files_file`) не изменяется; наш
|
||||||
|
индекс живёт рядом.
|
||||||
|
|
||||||
|
## Устройство
|
||||||
|
|
||||||
|
- `file_es_text.go` — `ESTextIndex` (`Name() = "es-text"`):
|
||||||
|
`Ensure` (создаёт индекс с явным маппингом), `Put` (bulk, `refresh`),
|
||||||
|
`Delete` (по `id`), `Search` (`multi_match` по `title^2` + `content`,
|
||||||
|
фильтры `folder`/`ext`, highlight).
|
||||||
|
- `file_text_index.go` — `TextIndexer`: листает папки (`FileStore.List`),
|
||||||
|
качает файлы (`FileStore.Download`), извлекает текст через
|
||||||
|
`internal/docpipe` (`pdftotext`, для сканов — `ocrmypdf`/`tesseract`),
|
||||||
|
пишет в `TextIndex`. Пул воркеров (по умолчанию 3).
|
||||||
|
- CLI: `oo index folder|files` наполняет индекс; `oo search --backend own`
|
||||||
|
ищет по нему.
|
||||||
|
|
||||||
|
### Встроенные вложения PDF
|
||||||
|
|
||||||
|
Оцифрованные PDF несут вложения (`<doc>.md` — текст/таблицы скана,
|
||||||
|
`<doc>.yaml`/`.json` — метаданные, `.xml` — EN 16931 CII eRechnung,
|
||||||
|
`factur-x.xml` у ZUGFeRD; см. `office-assistant/docs/reference/document-metadata.md`).
|
||||||
|
`TextIndexer` обходит их: `pdfdetach -list` перечисляет, `-save` сохраняет,
|
||||||
|
каждое вложение проходит штатный `docpipe.ToMarkdown` (PDF/картинки → OCR,
|
||||||
|
`.md`/`.txt` — как есть). Форматы, которые docpipe не конвертирует
|
||||||
|
(`.xml`/`.html` — снимаются теги; `.json`/`.csv` — как текст), извлекаются
|
||||||
|
текстом; нечитаемые — пропускаются.
|
||||||
|
|
||||||
|
Текст склеивается: тело, затем по секции на вложение с маркером
|
||||||
|
`[attachment: <имя>]` (функция `docpipe.JoinWithAttachments`). Индекс — тот же
|
||||||
|
`file_id`, upsert идемпотентен. Нет вложений или pdfdetach/формат нечитаем —
|
||||||
|
индексируется тело (без падения).
|
||||||
|
|
||||||
|
Поля `oo_docs_text`:
|
||||||
|
|
||||||
|
| поле | тип | смысл |
|
||||||
|
|------|-----|-------|
|
||||||
|
| `id` | keyword | id файла Documents |
|
||||||
|
| `title` | text (+`.keyword`) | имя файла |
|
||||||
|
| `folder` | keyword | id папки |
|
||||||
|
| `ext` | keyword | расширение |
|
||||||
|
| `content` | text | извлечённый текст (pdftotext/OCR) |
|
||||||
|
|
||||||
|
## CLI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
set -a; . .env; set +a # ONLYOFFICE_URL/USER/PASS + ONLYOFFICE_ES_URL
|
||||||
|
oo index folder 634 # PDF в папке 634
|
||||||
|
oo index folder 634 --recursive --exts pdf,png --limit 100
|
||||||
|
oo index files 3576 3578 # точечно
|
||||||
|
oo index folder 634 --dry-run # показать план, ничего не менять
|
||||||
|
|
||||||
|
oo search "S1021" --content --backend own
|
||||||
|
oo search "S1021" --backend own --folder 634 --json
|
||||||
|
```
|
||||||
|
|
||||||
|
`--backend` у `oo search`: `oo` (по умолчанию, индекс OnlyOffice) или `own`
|
||||||
|
(наш `ONLYOFFICE_ES_TEXT_INDEX`).
|
||||||
|
|
||||||
|
## Переменные (дополнение)
|
||||||
|
|
||||||
|
| env | default | смысл |
|
||||||
|
|-----|---------|-------|
|
||||||
|
| `ONLYOFFICE_ES_TEXT_INDEX` | `oo_docs_text` | индекс своего конвейера |
|
||||||
|
|
||||||
|
`ONLYOFFICE_ES_URL` — общий для обоих индексов.
|
||||||
|
|
||||||
|
## Тесты
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test -run 'ESText|TextIndexer|Index' ./ ./cmd/oo/ # unit, без сети
|
||||||
|
ONLYOFFICE_ES_URL=http://127.0.0.1:9200 \
|
||||||
|
go test -tags=integration -run TestIntegrationESTextIndex -v .
|
||||||
|
```
|
||||||
|
|
||||||
|
Интеграционный тест создаёт временный индекс, наполняет, ищет по контенту,
|
||||||
|
проверяет фильтры и удаление, затем удаляет индекс;
|
||||||
|
`TestIntegrationESTextIndexPDFAttachment` индексирует
|
||||||
|
`testdata/pdf-with-attachment.pdf` реальным конвейером (pdfdetach + pdftotext)
|
||||||
|
и ищет токен, лежащий только во вложении. Unit-тесты используют
|
||||||
|
fake-store/fake-extractor и не требуют pdftotext/OCR (парсер списка, склейка
|
||||||
|
`JoinWithAttachments`, снятие тегов `xmlToText` — чистые).
|
||||||
|
|
||||||
|
## Грабли
|
||||||
|
|
||||||
|
- Наполнение — ручное (`oo index`); после изменения/добавления PDF повтори.
|
||||||
|
Повтор идемпотентен (upsert по id файла).
|
||||||
|
- В индексе ищется только то, что проиндексировано; `oo index` качает каждый
|
||||||
|
файл и (для сканов) гоняет OCR — это медленно, отсюда `--limit`/`--exts`.
|
||||||
|
- `folder` фильтруется как id папки, а не как путь.
|
||||||
|
- Дубликаты (напр. `S1055.pdf` и `2026-08-20-S1055-…`) дадут несколько строк —
|
||||||
|
это ожидаемо, дедуп — на стороне потребителя.
|
||||||
|
- Вложения: нужен `pdfdetach` (poppler); если его нет — индексируется только
|
||||||
|
тело. Вложенный PDF/картинка с плохим текстовым слоем проходит OCR, это
|
||||||
|
медленно. `.json`-метаданные (CuraSoft) индексируются как текст и могут
|
||||||
|
добавить шумовых токенов.
|
||||||
+214
@@ -0,0 +1,214 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// Canonical file model and the backend-agnostic store interface. REST
|
||||||
|
// (files.go), WebDAV (files_webdav.go) and future backends (PostgreSQL,
|
||||||
|
// Elasticsearch) implement FileStore/Searcher so callers stop depending on a
|
||||||
|
// concrete transport. This file holds only types and pure conversions — no IO.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"mime"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Kind distinguishes files from folders in the canonical model.
|
||||||
|
type Kind int
|
||||||
|
|
||||||
|
const (
|
||||||
|
File Kind = iota
|
||||||
|
Folder
|
||||||
|
)
|
||||||
|
|
||||||
|
// String renders the kind for logs and table output.
|
||||||
|
func (k Kind) String() string {
|
||||||
|
switch k {
|
||||||
|
case File:
|
||||||
|
return "file"
|
||||||
|
case Folder:
|
||||||
|
return "folder"
|
||||||
|
default:
|
||||||
|
return "unknown"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Provider names for the FileStore adapters.
|
||||||
|
const (
|
||||||
|
ProviderREST = "rest"
|
||||||
|
ProviderDAV = "dav"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Entry is the backend-independent representation of a document or folder.
|
||||||
|
// Fields that a backend cannot supply stay at their zero value.
|
||||||
|
type Entry struct {
|
||||||
|
ID string
|
||||||
|
ParentID string
|
||||||
|
Title string
|
||||||
|
Kind Kind
|
||||||
|
Size int64
|
||||||
|
MIME string
|
||||||
|
Created time.Time
|
||||||
|
Modified time.Time
|
||||||
|
// Updated is the backend-native timestamp string, when the backend exposes
|
||||||
|
// one. It lets list output round-trip the API value; Modified is the
|
||||||
|
// parsed form for logic.
|
||||||
|
Updated string
|
||||||
|
Version int
|
||||||
|
Provider string
|
||||||
|
|
||||||
|
// Folder-only counters. Zero for files and for backends that do not
|
||||||
|
// report them.
|
||||||
|
FilesCount int
|
||||||
|
FoldersCount int
|
||||||
|
}
|
||||||
|
|
||||||
|
// FileStore is the operation surface every file backend implements.
|
||||||
|
type FileStore interface {
|
||||||
|
Name() string
|
||||||
|
List(ctx context.Context, parentID string) ([]Entry, error)
|
||||||
|
Stat(ctx context.Context, id string) (Entry, error)
|
||||||
|
CreateFolder(ctx context.Context, parentID, title string) (Entry, error)
|
||||||
|
Upload(ctx context.Context, parentID, title string, r io.Reader) (Entry, error)
|
||||||
|
Download(ctx context.Context, id string, w io.Writer) (int64, error)
|
||||||
|
Move(ctx context.Context, ids []string, parentID string) error
|
||||||
|
Copy(ctx context.Context, ids []string, parentID string) error
|
||||||
|
Rename(ctx context.Context, id, title string) error
|
||||||
|
Delete(ctx context.Context, ids []string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// SearchQuery narrows a Searcher request. InContent asks the backend to match
|
||||||
|
// document bodies, not just titles.
|
||||||
|
type SearchQuery struct {
|
||||||
|
Text string
|
||||||
|
InContent bool
|
||||||
|
FolderID string
|
||||||
|
Extensions []string
|
||||||
|
Limit int
|
||||||
|
}
|
||||||
|
|
||||||
|
// SearchHit is one Searcher result: the matching entry plus backend-specific
|
||||||
|
// ranking metadata.
|
||||||
|
type SearchHit struct {
|
||||||
|
Entry
|
||||||
|
Score float64
|
||||||
|
Highlight string
|
||||||
|
Path []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Searcher is the optional content/name search surface. Only some backends
|
||||||
|
// (for example Elasticsearch) provide it.
|
||||||
|
type Searcher interface {
|
||||||
|
Search(ctx context.Context, q SearchQuery) ([]SearchHit, error)
|
||||||
|
Name() string
|
||||||
|
}
|
||||||
|
|
||||||
|
// FileStore returns the adapter for a backend name: ProviderREST (default) or
|
||||||
|
// ProviderDAV. Unknown or empty names select the REST backend. The composed
|
||||||
|
// facade (backend selection/fallback) lives on FileClient in file_facade.go.
|
||||||
|
func (c *Client) FileStore(backend string) FileStore {
|
||||||
|
switch strings.ToLower(strings.TrimSpace(backend)) {
|
||||||
|
case ProviderDAV, "webdav":
|
||||||
|
return &davStore{c: c}
|
||||||
|
default:
|
||||||
|
return &restStore{c: c}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Files returns the composed file facade. The returned *FileClient implements
|
||||||
|
// FileStore, so callers that used Files() as the plain REST store keep working.
|
||||||
|
func (c *Client) Files() *FileClient { return c.newFileClient() }
|
||||||
|
|
||||||
|
// retryStoreOp runs one store operation under the shared deterministic
|
||||||
|
// transient-error policy (429/502/503/504).
|
||||||
|
func retryStoreOp(ctx context.Context, fn func() error) error {
|
||||||
|
return DoRetry(ctx, DefaultRetryPolicy(), fn)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FileEntryToEntry converts a Files-module file row to the canonical model.
|
||||||
|
func FileEntryToEntry(f *FileEntry, provider string) Entry {
|
||||||
|
e := Entry{Kind: File, Provider: provider}
|
||||||
|
if f == nil {
|
||||||
|
return e
|
||||||
|
}
|
||||||
|
if f.ID != nil {
|
||||||
|
e.ID = f.ID.String()
|
||||||
|
}
|
||||||
|
e.ParentID = FileFolderID(f)
|
||||||
|
if f.Title != nil {
|
||||||
|
e.Title = *f.Title
|
||||||
|
}
|
||||||
|
if f.ContentLength != nil {
|
||||||
|
e.Size = parseContentLength(*f.ContentLength)
|
||||||
|
}
|
||||||
|
exst := ""
|
||||||
|
if f.FileExst != nil {
|
||||||
|
exst = *f.FileExst
|
||||||
|
}
|
||||||
|
e.MIME = mimeForTitle(e.Title, exst)
|
||||||
|
if f.Updated != nil {
|
||||||
|
e.Modified = *f.Updated
|
||||||
|
e.Updated = f.Updated.Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
return e
|
||||||
|
}
|
||||||
|
|
||||||
|
// DavFileToEntry converts a WebDAV file row to the canonical model.
|
||||||
|
func DavFileToEntry(f DavFile, provider string) Entry {
|
||||||
|
return Entry{
|
||||||
|
ID: f.ID,
|
||||||
|
Title: f.Title,
|
||||||
|
Kind: File,
|
||||||
|
Size: f.Size,
|
||||||
|
MIME: mimeForTitle(f.Title, ""),
|
||||||
|
Modified: f.ModTime(),
|
||||||
|
Updated: f.Updated,
|
||||||
|
Provider: provider,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DavFolderToEntry converts a WebDAV folder row to the canonical model.
|
||||||
|
func DavFolderToEntry(f DavFolder, provider string) Entry {
|
||||||
|
return Entry{
|
||||||
|
ID: f.ID,
|
||||||
|
ParentID: f.ParentID,
|
||||||
|
Title: f.Title,
|
||||||
|
Kind: Folder,
|
||||||
|
Modified: f.ModTime(),
|
||||||
|
Updated: f.Updated,
|
||||||
|
Provider: provider,
|
||||||
|
FilesCount: f.FilesCount,
|
||||||
|
FoldersCount: f.FoldersCount,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseContentLength reads the leading integer of an OnlyOffice contentLength
|
||||||
|
// string (the API sometimes appends a unit, e.g. "12345 b").
|
||||||
|
func parseContentLength(s string) int64 {
|
||||||
|
fields := strings.Fields(s)
|
||||||
|
if len(fields) == 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
n, err := strconv.ParseInt(fields[0], 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
// mimeForTitle derives a MIME type from an explicit extension or the title.
|
||||||
|
func mimeForTitle(title, exst string) string {
|
||||||
|
ext := strings.TrimSpace(exst)
|
||||||
|
if ext == "" {
|
||||||
|
ext = filepath.Ext(title)
|
||||||
|
}
|
||||||
|
if ext == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(ext, ".") {
|
||||||
|
ext = "." + ext
|
||||||
|
}
|
||||||
|
return mime.TypeByExtension(strings.ToLower(ext))
|
||||||
|
}
|
||||||
@@ -0,0 +1,193 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFileEntryToEntry(t *testing.T) {
|
||||||
|
id := json.Number("42")
|
||||||
|
title := "invoice.pdf"
|
||||||
|
exst := ".pdf"
|
||||||
|
size := "12345"
|
||||||
|
parent := json.Number("7")
|
||||||
|
updated := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
f := &FileEntry{
|
||||||
|
ID: &id,
|
||||||
|
Title: &title,
|
||||||
|
FileExst: &exst,
|
||||||
|
ContentLength: &size,
|
||||||
|
FolderID: &parent,
|
||||||
|
Updated: &updated,
|
||||||
|
}
|
||||||
|
|
||||||
|
e := FileEntryToEntry(f, ProviderREST)
|
||||||
|
if e.ID != "42" {
|
||||||
|
t.Errorf("ID = %q, want 42", e.ID)
|
||||||
|
}
|
||||||
|
if e.ParentID != "7" {
|
||||||
|
t.Errorf("ParentID = %q, want 7", e.ParentID)
|
||||||
|
}
|
||||||
|
if e.Title != title {
|
||||||
|
t.Errorf("Title = %q, want %q", e.Title, title)
|
||||||
|
}
|
||||||
|
if e.Kind != File {
|
||||||
|
t.Errorf("Kind = %v, want file", e.Kind)
|
||||||
|
}
|
||||||
|
if e.Size != 12345 {
|
||||||
|
t.Errorf("Size = %d, want 12345", e.Size)
|
||||||
|
}
|
||||||
|
if e.MIME != "application/pdf" {
|
||||||
|
t.Errorf("MIME = %q, want application/pdf", e.MIME)
|
||||||
|
}
|
||||||
|
if !e.Modified.Equal(updated) {
|
||||||
|
t.Errorf("Modified = %v, want %v", e.Modified, updated)
|
||||||
|
}
|
||||||
|
if e.Provider != ProviderREST {
|
||||||
|
t.Errorf("Provider = %q, want %q", e.Provider, ProviderREST)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileEntryToEntryNil(t *testing.T) {
|
||||||
|
e := FileEntryToEntry(nil, ProviderDAV)
|
||||||
|
if e.Kind != File {
|
||||||
|
t.Errorf("Kind = %v, want file", e.Kind)
|
||||||
|
}
|
||||||
|
if e.ID != "" || e.Title != "" {
|
||||||
|
t.Errorf("nil entry should be empty: %+v", e)
|
||||||
|
}
|
||||||
|
if e.Provider != ProviderDAV {
|
||||||
|
t.Errorf("Provider = %q, want %q", e.Provider, ProviderDAV)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileEntryToEntrySizeFormats(t *testing.T) {
|
||||||
|
cases := map[string]int64{
|
||||||
|
"12345": 12345,
|
||||||
|
"12345 b": 12345,
|
||||||
|
"0": 0,
|
||||||
|
"": 0,
|
||||||
|
"notanum": 0,
|
||||||
|
}
|
||||||
|
for in, want := range cases {
|
||||||
|
got := parseContentLength(in)
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("parseContentLength(%q) = %d, want %d", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDavFileToEntry(t *testing.T) {
|
||||||
|
f := DavFile{
|
||||||
|
ID: "9",
|
||||||
|
Title: "note.txt",
|
||||||
|
Size: 10,
|
||||||
|
Updated: "2026-01-02T03:04:05.0000000+01:00",
|
||||||
|
}
|
||||||
|
e := DavFileToEntry(f, ProviderDAV)
|
||||||
|
if e.ID != "9" || e.Title != "note.txt" {
|
||||||
|
t.Errorf("identity mismatch: %+v", e)
|
||||||
|
}
|
||||||
|
if e.Kind != File {
|
||||||
|
t.Errorf("Kind = %v, want file", e.Kind)
|
||||||
|
}
|
||||||
|
if e.Size != 10 {
|
||||||
|
t.Errorf("Size = %d, want 10", e.Size)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(e.MIME, "text/plain") {
|
||||||
|
t.Errorf("MIME = %q, want text/plain*", e.MIME)
|
||||||
|
}
|
||||||
|
if e.Modified.IsZero() {
|
||||||
|
t.Error("Modified not parsed")
|
||||||
|
}
|
||||||
|
if e.Provider != ProviderDAV {
|
||||||
|
t.Errorf("Provider = %q, want %q", e.Provider, ProviderDAV)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDavFolderToEntry(t *testing.T) {
|
||||||
|
f := DavFolder{
|
||||||
|
ID: "5",
|
||||||
|
Title: "inbox",
|
||||||
|
ParentID: "1",
|
||||||
|
Updated: "2026-01-02T03:04:05.0000000+01:00",
|
||||||
|
}
|
||||||
|
e := DavFolderToEntry(f, ProviderDAV)
|
||||||
|
if e.ID != "5" || e.Title != "inbox" || e.ParentID != "1" {
|
||||||
|
t.Errorf("identity mismatch: %+v", e)
|
||||||
|
}
|
||||||
|
if e.Kind != Folder {
|
||||||
|
t.Errorf("Kind = %v, want folder", e.Kind)
|
||||||
|
}
|
||||||
|
if e.MIME != "" {
|
||||||
|
t.Errorf("folder MIME = %q, want empty", e.MIME)
|
||||||
|
}
|
||||||
|
if e.Modified.IsZero() {
|
||||||
|
t.Error("Modified not parsed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEntriesFromFolderMap(t *testing.T) {
|
||||||
|
m := map[string]any{
|
||||||
|
"files": []any{
|
||||||
|
map[string]any{"id": float64(42), "title": "a.pdf", "pureContentLength": float64(7)},
|
||||||
|
},
|
||||||
|
"folders": []any{
|
||||||
|
map[string]any{"id": float64(7), "title": "sub", "parentId": float64(1)},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
entries, err := entriesFromFolderMap(m, ProviderREST)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("entriesFromFolderMap: %v", err)
|
||||||
|
}
|
||||||
|
if len(entries) != 2 {
|
||||||
|
t.Fatalf("got %d entries, want 2: %+v", len(entries), entries)
|
||||||
|
}
|
||||||
|
byID := map[string]Entry{}
|
||||||
|
for _, e := range entries {
|
||||||
|
byID[e.ID] = e
|
||||||
|
}
|
||||||
|
if got := byID["42"]; got.Kind != File || got.Size != 7 || got.Title != "a.pdf" {
|
||||||
|
t.Errorf("file entry = %+v", got)
|
||||||
|
}
|
||||||
|
if got := byID["7"]; got.Kind != Folder || got.ParentID != "1" || got.Title != "sub" {
|
||||||
|
t.Errorf("folder entry = %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEntriesFromFolderMapNil(t *testing.T) {
|
||||||
|
entries, err := entriesFromFolderMap(nil, ProviderREST)
|
||||||
|
if err != nil || entries != nil {
|
||||||
|
t.Fatalf("got %v, %v; want nil, nil", entries, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKindString(t *testing.T) {
|
||||||
|
if File.String() != "file" || Folder.String() != "folder" {
|
||||||
|
t.Errorf("kind strings: %q %q", File.String(), Folder.String())
|
||||||
|
}
|
||||||
|
if Kind(9).String() != "unknown" {
|
||||||
|
t.Errorf("unknown kind = %q", Kind(9).String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientFileStoreSelection(t *testing.T) {
|
||||||
|
c := NewClient(Credentials{})
|
||||||
|
if got := c.FileStore(ProviderDAV).Name(); got != ProviderDAV {
|
||||||
|
t.Errorf("FileStore(dav).Name() = %q", got)
|
||||||
|
}
|
||||||
|
if got := c.FileStore("webdav").Name(); got != ProviderDAV {
|
||||||
|
t.Errorf("FileStore(webdav).Name() = %q", got)
|
||||||
|
}
|
||||||
|
if got := c.FileStore(ProviderREST).Name(); got != ProviderREST {
|
||||||
|
t.Errorf("FileStore(rest).Name() = %q", got)
|
||||||
|
}
|
||||||
|
if got := c.FileStore("").Name(); got != ProviderREST {
|
||||||
|
t.Errorf("FileStore(\"\").Name() = %q", got)
|
||||||
|
}
|
||||||
|
if got := c.Files().Name(); got != ProviderREST {
|
||||||
|
t.Errorf("Files().Name() = %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
+189
@@ -0,0 +1,189 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// davStore implements FileStore on top of the Documents/WebDAV methods in
|
||||||
|
// files_webdav.go. The Documents fileops calls need folder and file ids
|
||||||
|
// separated, so ids are classified through Stat before move/copy/rename/delete.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
)
|
||||||
|
|
||||||
|
// davStore is a FileStore over the WebDAV-oriented Documents API.
|
||||||
|
type davStore struct{ c *Client }
|
||||||
|
|
||||||
|
// Name reports the backend name.
|
||||||
|
func (s *davStore) Name() string { return ProviderDAV }
|
||||||
|
|
||||||
|
// List returns the files and folders directly below parentID.
|
||||||
|
func (s *davStore) List(ctx context.Context, parentID string) ([]Entry, error) {
|
||||||
|
var out []Entry
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
l, err := s.c.ListDavFolder(ctx, parentID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
entries := make([]Entry, 0, len(l.Folders)+len(l.Files))
|
||||||
|
for _, f := range l.Folders {
|
||||||
|
entries = append(entries, DavFolderToEntry(f, ProviderDAV))
|
||||||
|
}
|
||||||
|
for _, f := range l.Files {
|
||||||
|
entries = append(entries, DavFileToEntry(f, ProviderDAV))
|
||||||
|
}
|
||||||
|
out = entries
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stat resolves a folder or file entry by id. A folder answers ListDavFolder
|
||||||
|
// with its own metadata in Current; otherwise the file metadata API is used.
|
||||||
|
func (s *davStore) Stat(ctx context.Context, id string) (Entry, error) {
|
||||||
|
return s.stat(ctx, id)
|
||||||
|
}
|
||||||
|
|
||||||
|
// CreateFolder creates a subfolder under parentID.
|
||||||
|
func (s *davStore) CreateFolder(ctx context.Context, parentID, title string) (Entry, error) {
|
||||||
|
var out Entry
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
f, err := s.c.CreateDavFolder(ctx, parentID, title)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if f == nil {
|
||||||
|
return fmt.Errorf("onlyoffice: dav store: empty create-folder response")
|
||||||
|
}
|
||||||
|
out = DavFolderToEntry(*f, ProviderDAV)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload streams r into parentID as title. The reader is buffered once so a
|
||||||
|
// retry re-sends the same bytes instead of an exhausted stream.
|
||||||
|
func (s *davStore) Upload(ctx context.Context, parentID, title string, r io.Reader) (Entry, error) {
|
||||||
|
data, err := io.ReadAll(r)
|
||||||
|
if err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
var out Entry
|
||||||
|
err = retryStoreOp(ctx, func() error {
|
||||||
|
f, err := s.c.UploadDavFile(ctx, parentID, title, bytes.NewReader(data))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if f == nil {
|
||||||
|
return fmt.Errorf("onlyoffice: dav store: empty upload response")
|
||||||
|
}
|
||||||
|
out = DavFileToEntry(*f, ProviderDAV)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Download streams the file bytes into w.
|
||||||
|
func (s *davStore) Download(ctx context.Context, id string, w io.Writer) (int64, error) {
|
||||||
|
var n int64
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
var e error
|
||||||
|
n, e = s.c.DownloadDavFile(ctx, id, w)
|
||||||
|
return e
|
||||||
|
})
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move moves ids into parentID, splitting folders from files.
|
||||||
|
func (s *davStore) Move(ctx context.Context, ids []string, parentID string) error {
|
||||||
|
folders, files, err := s.split(ctx, ids)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(folders) == 0 && len(files) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
return s.c.MoveDavItems(ctx, folders, files, parentID)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy copies ids into parentID, splitting folders from files.
|
||||||
|
func (s *davStore) Copy(ctx context.Context, ids []string, parentID string) error {
|
||||||
|
folders, files, err := s.split(ctx, ids)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(folders) == 0 && len(files) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
return s.c.CopyDavItems(ctx, folders, files, parentID)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rename renames a folder or file.
|
||||||
|
func (s *davStore) Rename(ctx context.Context, id, title string) error {
|
||||||
|
e, err := s.stat(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
if e.Kind == Folder {
|
||||||
|
return s.c.RenameDavFolder(ctx, id, title)
|
||||||
|
}
|
||||||
|
return s.c.RenameDavFile(ctx, id, title)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete removes ids, splitting folders from files.
|
||||||
|
func (s *davStore) Delete(ctx context.Context, ids []string) error {
|
||||||
|
folders, files, err := s.split(ctx, ids)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(folders) == 0 && len(files) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
return s.c.DeleteDavItems(ctx, folders, files)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// stat resolves a single id to a folder or file Entry.
|
||||||
|
func (s *davStore) stat(ctx context.Context, id string) (Entry, error) {
|
||||||
|
var out Entry
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
if l, err := s.c.ListDavFolder(ctx, id); err == nil {
|
||||||
|
if l != nil && l.Current.ID != "" && l.Current.ID == id {
|
||||||
|
out = DavFolderToEntry(l.Current, ProviderDAV)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
} else if Transient(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
f, err := s.c.GetFile(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out = FileEntryToEntry(f, ProviderDAV)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// split classifies ids into folder and file id lists.
|
||||||
|
func (s *davStore) split(ctx context.Context, ids []string) (folders, files []string, err error) {
|
||||||
|
for _, id := range ids {
|
||||||
|
e, err := s.stat(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
if e.Kind == Folder {
|
||||||
|
folders = append(folders, id)
|
||||||
|
} else {
|
||||||
|
files = append(files, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return folders, files, nil
|
||||||
|
}
|
||||||
+287
@@ -0,0 +1,287 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// Elasticsearch backend of the unified file client (epic #34, F3 #37).
|
||||||
|
//
|
||||||
|
// OnlyOffice full-text search runs on Elasticsearch (index `files_file`, NEST
|
||||||
|
// client on the server). The REST endpoint GET /api/2.0/files/@search/{query}
|
||||||
|
// only searches file names in the database, so content search needs a direct
|
||||||
|
// ES query. The live server is Elasticsearch 7.16.3; the request shape below
|
||||||
|
// is plain REST and stays stdlib-only, matching the repo's no-extra-deps rule.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The canonical model (Kind, Entry, SearchQuery, SearchHit, Searcher) lives in
|
||||||
|
// file_core.go (F1 #35).
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultESIndex = "files_file"
|
||||||
|
defaultESLimit = 20
|
||||||
|
maxESLimit = 200
|
||||||
|
maxESResponseSize = 8 << 20
|
||||||
|
)
|
||||||
|
|
||||||
|
// ESConfig configures the direct Elasticsearch searcher.
|
||||||
|
type ESConfig struct {
|
||||||
|
URL string // scheme://host:port of the ES HTTP endpoint
|
||||||
|
Index string // index name, default files_file
|
||||||
|
Tenant string // tenantId filter, empty means all tenants
|
||||||
|
}
|
||||||
|
|
||||||
|
// ESConfigFromEnv reads ONLYOFFICE_ES_URL, ONLYOFFICE_ES_INDEX (default
|
||||||
|
// files_file) and ONLYOFFICE_TENANT. The library never loads dotfiles — the
|
||||||
|
// CLI does that.
|
||||||
|
func ESConfigFromEnv() ESConfig {
|
||||||
|
return ESConfig{
|
||||||
|
URL: strings.TrimRight(strings.TrimSpace(os.Getenv("ONLYOFFICE_ES_URL")), "/"),
|
||||||
|
Index: firstNonEmpty(os.Getenv("ONLYOFFICE_ES_INDEX"), defaultESIndex),
|
||||||
|
Tenant: strings.TrimSpace(os.Getenv("ONLYOFFICE_TENANT")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ESSearcher queries OnlyOffice's Elasticsearch index directly for file name
|
||||||
|
// and document content.
|
||||||
|
type ESSearcher struct {
|
||||||
|
cfg ESConfig
|
||||||
|
http *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewESSearcher returns a searcher for the OnlyOffice Elasticsearch index.
|
||||||
|
// The URL is required; an empty index falls back to files_file.
|
||||||
|
func NewESSearcher(cfg ESConfig) (*ESSearcher, error) {
|
||||||
|
if strings.TrimSpace(cfg.URL) == "" {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: elasticsearch URL is empty (set ONLYOFFICE_ES_URL)")
|
||||||
|
}
|
||||||
|
cfg.URL = strings.TrimRight(cfg.URL, "/")
|
||||||
|
if cfg.Index == "" {
|
||||||
|
cfg.Index = defaultESIndex
|
||||||
|
}
|
||||||
|
return &ESSearcher{cfg: cfg, http: &http.Client{Timeout: 30 * time.Second}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Name implements Searcher.
|
||||||
|
func (s *ESSearcher) Name() string { return "elasticsearch" }
|
||||||
|
|
||||||
|
// Search runs a multi_match over title (and, when q.InContent is set,
|
||||||
|
// document.attachment.content), filtered by tenant and optional folder.
|
||||||
|
func (s *ESSearcher) Search(ctx context.Context, q SearchQuery) ([]SearchHit, error) {
|
||||||
|
q.Text = strings.TrimSpace(q.Text)
|
||||||
|
if q.Text == "" {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: empty search query")
|
||||||
|
}
|
||||||
|
body, err := json.Marshal(esSearchRequest(q, s.cfg.Tenant))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: build elasticsearch query: %w", err)
|
||||||
|
}
|
||||||
|
endpoint := s.cfg.URL + "/" + s.cfg.Index + "/_search"
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
req.Header.Set("Accept", "application/json")
|
||||||
|
resp, err := s.http.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: elasticsearch search: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
raw, err := io.ReadAll(io.LimitReader(resp.Body, maxESResponseSize))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if resp.StatusCode >= 400 {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: elasticsearch search: %d %s", resp.StatusCode, truncate(string(raw), 400))
|
||||||
|
}
|
||||||
|
return parseESSearchResponse(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
// esSearchRequest builds the ES query body. Pure, so it is unit-tested.
|
||||||
|
func esSearchRequest(q SearchQuery, tenant string) esRequest {
|
||||||
|
limit := q.Limit
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = defaultESLimit
|
||||||
|
}
|
||||||
|
if limit > maxESLimit {
|
||||||
|
limit = maxESLimit
|
||||||
|
}
|
||||||
|
fields := []string{"title^2"}
|
||||||
|
if q.InContent {
|
||||||
|
fields = append(fields, "document.attachment.content")
|
||||||
|
}
|
||||||
|
must := []esClause{{MultiMatch: &esMultiMatch{Query: q.Text, Fields: fields}}}
|
||||||
|
|
||||||
|
var filter []esClause
|
||||||
|
if t := strings.TrimSpace(tenant); t != "" {
|
||||||
|
filter = append(filter, esClause{Term: map[string]any{"tenantId": numericOrString(t)}})
|
||||||
|
}
|
||||||
|
if f := strings.TrimSpace(q.FolderID); f != "" {
|
||||||
|
filter = append(filter, esClause{Term: map[string]any{"folders.folderId": f}})
|
||||||
|
}
|
||||||
|
for _, ext := range normalizeExtensions(q.Extensions) {
|
||||||
|
filter = append(filter, esClause{Wildcard: map[string]any{"title": "*." + ext}})
|
||||||
|
}
|
||||||
|
|
||||||
|
highlightFields := map[string]struct{}{"title": {}}
|
||||||
|
if q.InContent {
|
||||||
|
highlightFields["document.attachment.content"] = struct{}{}
|
||||||
|
}
|
||||||
|
return esRequest{
|
||||||
|
Size: limit,
|
||||||
|
Source: []string{"id", "title", "folders"},
|
||||||
|
Query: esQuery{Bool: esBool{Must: must, Filter: filter}},
|
||||||
|
Highlight: esHighlight{PreTags: []string{"<em>"}, PostTags: []string{"</em>"}, Fields: highlightFields},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// normalizeExtensions lowercases, trims leading dots and drops empties.
|
||||||
|
func normalizeExtensions(exts []string) []string {
|
||||||
|
out := make([]string, 0, len(exts))
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, e := range exts {
|
||||||
|
e = strings.ToLower(strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(e), ".")))
|
||||||
|
if e == "" || seen[e] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[e] = true
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// numericOrString keeps an integer-looking filter value numeric (tenantId is
|
||||||
|
// a long) and leaves anything else as a string (folderId is a text token).
|
||||||
|
func numericOrString(s string) any {
|
||||||
|
if n, err := strconv.ParseInt(s, 10, 64); err == nil {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// esRequest is the subset of the ES query DSL this client emits.
|
||||||
|
type esRequest struct {
|
||||||
|
Size int `json:"size"`
|
||||||
|
Source []string `json:"_source"`
|
||||||
|
Query esQuery `json:"query"`
|
||||||
|
Highlight esHighlight `json:"highlight"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type esQuery struct {
|
||||||
|
Bool esBool `json:"bool"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type esBool struct {
|
||||||
|
Must []esClause `json:"must,omitempty"`
|
||||||
|
Filter []esClause `json:"filter,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type esClause struct {
|
||||||
|
MultiMatch *esMultiMatch `json:"multi_match,omitempty"`
|
||||||
|
Term map[string]any `json:"term,omitempty"`
|
||||||
|
Terms map[string]any `json:"terms,omitempty"`
|
||||||
|
Wildcard map[string]any `json:"wildcard,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type esMultiMatch struct {
|
||||||
|
Query string `json:"query"`
|
||||||
|
Fields []string `json:"fields"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type esHighlight struct {
|
||||||
|
PreTags []string `json:"pre_tags,omitempty"`
|
||||||
|
PostTags []string `json:"post_tags,omitempty"`
|
||||||
|
Fields map[string]struct{} `json:"fields"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// esResponse is the subset of an ES search response we consume.
|
||||||
|
type esResponse struct {
|
||||||
|
Took int `json:"took"`
|
||||||
|
Hits struct {
|
||||||
|
Total struct {
|
||||||
|
Value int `json:"value"`
|
||||||
|
Relation string `json:"relation"`
|
||||||
|
} `json:"total"`
|
||||||
|
Hits []esResponseHit `json:"hits"`
|
||||||
|
} `json:"hits"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type esResponseHit struct {
|
||||||
|
ID string `json:"_id"`
|
||||||
|
Score float64 `json:"_score"`
|
||||||
|
Source struct {
|
||||||
|
ID int `json:"id"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Folders []struct {
|
||||||
|
FolderID string `json:"folderId"`
|
||||||
|
ID int `json:"id"`
|
||||||
|
} `json:"folders"`
|
||||||
|
} `json:"_source"`
|
||||||
|
Highlight map[string][]string `json:"highlight"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseESSearchResponse converts an ES search response into SearchHit values.
|
||||||
|
// Pure, so it is unit-tested.
|
||||||
|
func parseESSearchResponse(raw []byte) ([]SearchHit, error) {
|
||||||
|
var r esResponse
|
||||||
|
if err := json.Unmarshal(raw, &r); err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: decode elasticsearch response: %w", err)
|
||||||
|
}
|
||||||
|
hits := make([]SearchHit, 0, len(r.Hits.Hits))
|
||||||
|
for _, h := range r.Hits.Hits {
|
||||||
|
id := strconv.Itoa(h.Source.ID)
|
||||||
|
if h.Source.ID == 0 {
|
||||||
|
id = h.ID
|
||||||
|
}
|
||||||
|
var parent string
|
||||||
|
path := make([]string, 0, len(h.Source.Folders))
|
||||||
|
for i, f := range h.Source.Folders {
|
||||||
|
path = append(path, f.FolderID)
|
||||||
|
if i == 0 {
|
||||||
|
parent = f.FolderID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hits = append(hits, SearchHit{
|
||||||
|
Entry: Entry{
|
||||||
|
ID: id,
|
||||||
|
ParentID: parent,
|
||||||
|
Title: h.Source.Title,
|
||||||
|
Kind: File,
|
||||||
|
Provider: "elasticsearch",
|
||||||
|
},
|
||||||
|
Score: h.Score,
|
||||||
|
Highlight: esHighlightText(h.Highlight),
|
||||||
|
Path: path,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return hits, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var esHighlightTag = regexp.MustCompile(`</?em[^>]*>`)
|
||||||
|
|
||||||
|
// esHighlightText flattens a highlight map into one plain-text snippet,
|
||||||
|
// preferring the content fragment over the title. It covers both the
|
||||||
|
// OnlyOffice content field and the own-index "content" field.
|
||||||
|
func esHighlightText(hl map[string][]string) string {
|
||||||
|
for _, key := range []string{"document.attachment.content", "content", "title"} {
|
||||||
|
frags := hl[key]
|
||||||
|
if len(frags) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
clean := make([]string, 0, len(frags))
|
||||||
|
for _, f := range frags {
|
||||||
|
clean = append(clean, esHighlightTag.ReplaceAllString(f, ""))
|
||||||
|
}
|
||||||
|
return strings.Join(clean, " … ")
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
@@ -0,0 +1,135 @@
|
|||||||
|
//go:build integration
|
||||||
|
|
||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/xuri/excelize/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestIntegrationESSearch uploads a throwaway workbook and verifies that the
|
||||||
|
// direct Elasticsearch search finds it by file name and by content.
|
||||||
|
//
|
||||||
|
// The content index (document.attachment.content) is only populated for Office
|
||||||
|
// formats (docx/xlsx/pptx), so the fixture is an xlsx whose cell carries a
|
||||||
|
// unique token. Requires ONLYOFFICE_ES_URL (a reachable ES endpoint — in the
|
||||||
|
// current setup a tunnel to the ES inside the OnlyOffice VM, see
|
||||||
|
// docs/elasticsearch.md) plus the regular REST credentials for the upload.
|
||||||
|
// Skips when either is missing.
|
||||||
|
func TestIntegrationESSearch(t *testing.T) {
|
||||||
|
esURL := strings.TrimSpace(os.Getenv("ONLYOFFICE_ES_URL"))
|
||||||
|
if esURL == "" {
|
||||||
|
t.Skip("ONLYOFFICE_ES_URL not set — skipping Elasticsearch integration test")
|
||||||
|
}
|
||||||
|
c := liveClient(t)
|
||||||
|
t.Cleanup(func() { cleanupTestProjects(t, c) })
|
||||||
|
|
||||||
|
stamp := time.Now().UTC().Format("20060102-150405")
|
||||||
|
nameToken := "goesname" + stamp
|
||||||
|
contentToken := "goescontent" + stamp
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
project, err := c.CreateProject(NewProjectRequest{
|
||||||
|
Title: testProjectPrefix + "es-" + stamp,
|
||||||
|
Description: "go-onlyoffice elasticsearch integration",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateProject: %v", err)
|
||||||
|
}
|
||||||
|
if project.ID == nil {
|
||||||
|
t.Fatal("created project without id")
|
||||||
|
}
|
||||||
|
pid := strconv.Itoa(*project.ID)
|
||||||
|
|
||||||
|
title := nameToken + ".xlsx"
|
||||||
|
localPath := filepath.Join(t.TempDir(), title)
|
||||||
|
book := excelize.NewFile()
|
||||||
|
if err := book.SetCellValue("Sheet1", "A1", "OnlyOffice Elasticsearch content fixture "+contentToken); err != nil {
|
||||||
|
t.Fatalf("SetCellValue: %v", err)
|
||||||
|
}
|
||||||
|
if err := book.SaveAs(localPath); err != nil {
|
||||||
|
t.Fatalf("SaveAs: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
entry, err := c.UploadProjectFile(ctx, pid, localPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UploadProjectFile: %v", err)
|
||||||
|
}
|
||||||
|
fileID := strconv.Itoa(int(FileEntryNumericID(entry)))
|
||||||
|
if fileID == "0" {
|
||||||
|
t.Fatalf("upload returned no file id: %+v", entry)
|
||||||
|
}
|
||||||
|
|
||||||
|
es, err := NewESSearcher(ESConfig{
|
||||||
|
URL: esURL,
|
||||||
|
Index: os.Getenv("ONLYOFFICE_ES_INDEX"),
|
||||||
|
Tenant: os.Getenv("ONLYOFFICE_TENANT"),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewESSearcher: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Indexing is asynchronous on the server; poll until the file shows up.
|
||||||
|
// The server's title analyzer splits on whitespace, so the name query is
|
||||||
|
// the full file name token (including extension), as a user would type it.
|
||||||
|
nameHit := waitForHit(t, ctx, es, SearchQuery{Text: title}, fileID)
|
||||||
|
if nameHit.Title != title {
|
||||||
|
t.Errorf("name hit title = %q, want %q", nameHit.Title, title)
|
||||||
|
}
|
||||||
|
contentHit := waitForHit(t, ctx, es, SearchQuery{Text: contentToken, InContent: true}, fileID)
|
||||||
|
if contentHit.Highlight == "" {
|
||||||
|
t.Error("content hit has no highlight fragment")
|
||||||
|
}
|
||||||
|
if !strings.Contains(contentHit.Title, nameToken) {
|
||||||
|
t.Errorf("content hit title = %q, want the uploaded workbook", contentHit.Title)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The content token is absent from the title, so a name-only search must
|
||||||
|
// not return the file — this proves the content field is really queried.
|
||||||
|
if hits := searchQuiet(t, es, SearchQuery{Text: contentToken}); len(hits) != 0 {
|
||||||
|
t.Errorf("name-only search for content token returned %d hits, want 0", len(hits))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitForHit polls ES until the file with fileID appears and returns that hit.
|
||||||
|
func waitForHit(t *testing.T, ctx context.Context, s *ESSearcher, q SearchQuery, fileID string) SearchHit {
|
||||||
|
t.Helper()
|
||||||
|
var lastErr error
|
||||||
|
for {
|
||||||
|
hits, err := s.Search(ctx, q)
|
||||||
|
if err != nil {
|
||||||
|
lastErr = err
|
||||||
|
} else {
|
||||||
|
for _, h := range hits {
|
||||||
|
if h.ID == fileID {
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
t.Fatalf("search %q: file %s not indexed in time (last err: %v)", q.Text, fileID, lastErr)
|
||||||
|
case <-time.After(3 * time.Second):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func searchQuiet(t *testing.T, s *ESSearcher, q SearchQuery) []SearchHit {
|
||||||
|
t.Helper()
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
hits, err := s.Search(ctx, q)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Search(%q): %v", q.Text, err)
|
||||||
|
}
|
||||||
|
return hits
|
||||||
|
}
|
||||||
+188
@@ -0,0 +1,188 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestESSearchRequestNameOnly(t *testing.T) {
|
||||||
|
got := esSearchRequest(SearchQuery{Text: "Rechnung"}, "1")
|
||||||
|
if got.Size != defaultESLimit {
|
||||||
|
t.Errorf("size = %d, want %d", got.Size, defaultESLimit)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got.Source, []string{"id", "title", "folders"}) {
|
||||||
|
t.Errorf("_source = %v", got.Source)
|
||||||
|
}
|
||||||
|
if len(got.Query.Bool.Must) != 1 || got.Query.Bool.Must[0].MultiMatch == nil {
|
||||||
|
t.Fatalf("must = %+v, want one multi_match", got.Query.Bool.Must)
|
||||||
|
}
|
||||||
|
mm := got.Query.Bool.Must[0].MultiMatch
|
||||||
|
if mm.Query != "Rechnung" {
|
||||||
|
t.Errorf("query = %q", mm.Query)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(mm.Fields, []string{"title^2"}) {
|
||||||
|
t.Errorf("fields = %v, want title only", mm.Fields)
|
||||||
|
}
|
||||||
|
if _, ok := got.Highlight.Fields["document.attachment.content"]; ok {
|
||||||
|
t.Error("content highlight present without InContent")
|
||||||
|
}
|
||||||
|
if _, ok := got.Highlight.Fields["title"]; !ok {
|
||||||
|
t.Error("title highlight missing")
|
||||||
|
}
|
||||||
|
if len(got.Query.Bool.Filter) != 1 || got.Query.Bool.Filter[0].Term["tenantId"] != int64(1) {
|
||||||
|
t.Errorf("tenant filter = %+v, want numeric tenantId=1", got.Query.Bool.Filter)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestESSearchRequestContentFields(t *testing.T) {
|
||||||
|
got := esSearchRequest(SearchQuery{Text: "Mahnung", InContent: true}, "")
|
||||||
|
mm := got.Query.Bool.Must[0].MultiMatch
|
||||||
|
want := []string{"title^2", "document.attachment.content"}
|
||||||
|
if !reflect.DeepEqual(mm.Fields, want) {
|
||||||
|
t.Errorf("fields = %v, want %v", mm.Fields, want)
|
||||||
|
}
|
||||||
|
if _, ok := got.Highlight.Fields["document.attachment.content"]; !ok {
|
||||||
|
t.Error("content highlight missing with InContent")
|
||||||
|
}
|
||||||
|
if len(got.Query.Bool.Filter) != 0 {
|
||||||
|
t.Errorf("filter = %+v, want none without tenant/folder", got.Query.Bool.Filter)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestESSearchRequestFiltersAndLimit(t *testing.T) {
|
||||||
|
got := esSearchRequest(SearchQuery{
|
||||||
|
Text: "Storchen",
|
||||||
|
FolderID: "649",
|
||||||
|
Extensions: []string{".PDF", "pdf", "docx"},
|
||||||
|
Limit: 999,
|
||||||
|
}, "42")
|
||||||
|
if got.Size != maxESLimit {
|
||||||
|
t.Errorf("size = %d, want cap %d", got.Size, maxESLimit)
|
||||||
|
}
|
||||||
|
var tenant, folder, wildcards int
|
||||||
|
for _, f := range got.Query.Bool.Filter {
|
||||||
|
switch {
|
||||||
|
case f.Term != nil && f.Term["tenantId"] != nil:
|
||||||
|
tenant++
|
||||||
|
case f.Term != nil && f.Term["folders.folderId"] != nil:
|
||||||
|
folder++
|
||||||
|
if f.Term["folders.folderId"] != "649" {
|
||||||
|
t.Errorf("folder filter = %+v", f.Term)
|
||||||
|
}
|
||||||
|
case f.Wildcard != nil:
|
||||||
|
wildcards++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if tenant != 1 || folder != 1 {
|
||||||
|
t.Errorf("term filters tenant=%d folder=%d, want 1 each", tenant, folder)
|
||||||
|
}
|
||||||
|
if wildcards != 2 {
|
||||||
|
t.Errorf("wildcard filters = %d, want deduped PDF+docx", wildcards)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestESSearchRequestRejectsEmptyTextAtSearch(t *testing.T) {
|
||||||
|
s, err := NewESSearcher(ESConfig{URL: "http://localhost:9200"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewESSearcher: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.Search(t.Context(), SearchQuery{Text: " "}); err == nil {
|
||||||
|
t.Error("empty query: want error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewESSearcherRequiresURL(t *testing.T) {
|
||||||
|
if _, err := NewESSearcher(ESConfig{}); err == nil {
|
||||||
|
t.Error("empty URL: want error")
|
||||||
|
}
|
||||||
|
s, err := NewESSearcher(ESConfig{URL: "http://es:9200/"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewESSearcher: %v", err)
|
||||||
|
}
|
||||||
|
if s.cfg.Index != defaultESIndex {
|
||||||
|
t.Errorf("index = %q, want %q", s.cfg.Index, defaultESIndex)
|
||||||
|
}
|
||||||
|
if s.cfg.URL != "http://es:9200" {
|
||||||
|
t.Errorf("url = %q, want trimmed", s.cfg.URL)
|
||||||
|
}
|
||||||
|
if s.Name() != "elasticsearch" {
|
||||||
|
t.Errorf("Name() = %q", s.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeExtensions(t *testing.T) {
|
||||||
|
got := normalizeExtensions([]string{" .PDF ", "pdf", "", "xlsx"})
|
||||||
|
want := []string{"pdf", "xlsx"}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("normalizeExtensions = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseESSearchResponse(t *testing.T) {
|
||||||
|
raw := []byte(`{
|
||||||
|
"took": 12,
|
||||||
|
"hits": {
|
||||||
|
"total": {"value": 2, "relation": "eq"},
|
||||||
|
"hits": [
|
||||||
|
{
|
||||||
|
"_id": "2395",
|
||||||
|
"_score": 7.31,
|
||||||
|
"_source": {"id": 2395, "title": "Rechnung-4711.pdf",
|
||||||
|
"folders": [{"folderId": "438", "id": 0}, {"folderId": "11", "id": 0}]},
|
||||||
|
"highlight": {
|
||||||
|
"title": ["<em>Rechnung</em>-4711.pdf"],
|
||||||
|
"document.attachment.content": ["… Zahlung der <em>Rechnung</em> …"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"_id": "2318",
|
||||||
|
"_score": 6.02,
|
||||||
|
"_source": {"id": 2318, "title": "Mahnung.pdf", "folders": []},
|
||||||
|
"highlight": {"title": ["<em>Mahnung</em>.pdf"]}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}`)
|
||||||
|
hits, err := parseESSearchResponse(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parseESSearchResponse: %v", err)
|
||||||
|
}
|
||||||
|
if len(hits) != 2 {
|
||||||
|
t.Fatalf("hits = %d, want 2", len(hits))
|
||||||
|
}
|
||||||
|
h0 := hits[0]
|
||||||
|
if h0.ID != "2395" || h0.Title != "Rechnung-4711.pdf" || h0.Kind != File {
|
||||||
|
t.Errorf("hit0 entry = %+v", h0.Entry)
|
||||||
|
}
|
||||||
|
if h0.ParentID != "438" || !reflect.DeepEqual(h0.Path, []string{"438", "11"}) {
|
||||||
|
t.Errorf("hit0 path = %v parent = %q", h0.Path, h0.ParentID)
|
||||||
|
}
|
||||||
|
if h0.Score != 7.31 {
|
||||||
|
t.Errorf("hit0 score = %v", h0.Score)
|
||||||
|
}
|
||||||
|
if h0.Highlight != "… Zahlung der Rechnung …" {
|
||||||
|
t.Errorf("hit0 highlight = %q, want content fragment", h0.Highlight)
|
||||||
|
}
|
||||||
|
if hits[1].Highlight != "Mahnung.pdf" {
|
||||||
|
t.Errorf("hit1 highlight = %q, want title without tags", hits[1].Highlight)
|
||||||
|
}
|
||||||
|
if hits[1].ParentID != "" || len(hits[1].Path) != 0 {
|
||||||
|
t.Errorf("hit1 path = %v", hits[1].Path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestESSearchRequestJSONShape(t *testing.T) {
|
||||||
|
got := esSearchRequest(SearchQuery{Text: "Rechnung", InContent: true}, "1")
|
||||||
|
b, err := json.Marshal(got)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal: %v", err)
|
||||||
|
}
|
||||||
|
var back map[string]any
|
||||||
|
if err := json.Unmarshal(b, &back); err != nil {
|
||||||
|
t.Fatalf("unmarshal: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := back["query"].(map[string]any)["bool"]; !ok {
|
||||||
|
t.Errorf("query.bool missing: %s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
+336
@@ -0,0 +1,336 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// Own full-text index (epic #34, F6 #42).
|
||||||
|
//
|
||||||
|
// The OnlyOffice Elasticsearch index (files_file) only holds extracted content
|
||||||
|
// for Office formats. FileUtility.CanIndex gates extraction by the server
|
||||||
|
// setting files.index.formats, whose default is ".pptx|.xlsx|.docx", so PDFs
|
||||||
|
// are indexed by name only. Instead of patching the server (risky: lost on
|
||||||
|
// upgrade, forces a full reindex) this file implements a second, independent
|
||||||
|
// index (default oo_docs_text) that our own pipeline fills from
|
||||||
|
// internal/docpipe (pdftotext + OCR). The OnlyOffice index is never touched.
|
||||||
|
//
|
||||||
|
// See docs/elasticsearch.md for the decision and the trade-offs.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const defaultESTextIndex = "oo_docs_text"
|
||||||
|
|
||||||
|
// ESTextConfig configures the own full-text index.
|
||||||
|
type ESTextConfig struct {
|
||||||
|
URL string // scheme://host:port of the ES HTTP endpoint
|
||||||
|
Index string // index name, default oo_docs_text
|
||||||
|
Tenant string // reserved for future multi-tenant data; unused for now
|
||||||
|
}
|
||||||
|
|
||||||
|
// ESTextConfigFromEnv reads ONLYOFFICE_ES_URL and ONLYOFFICE_ES_TEXT_INDEX
|
||||||
|
// (default oo_docs_text). The library never loads dotfiles — the CLI does that.
|
||||||
|
func ESTextConfigFromEnv() ESTextConfig {
|
||||||
|
return ESTextConfig{
|
||||||
|
URL: strings.TrimRight(strings.TrimSpace(os.Getenv("ONLYOFFICE_ES_URL")), "/"),
|
||||||
|
Index: firstNonEmpty(os.Getenv("ONLYOFFICE_ES_TEXT_INDEX"), defaultESTextIndex),
|
||||||
|
Tenant: strings.TrimSpace(os.Getenv("ONLYOFFICE_TENANT")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TextDoc is one document in the own full-text index. It is keyed by the
|
||||||
|
// OnlyOffice file id so hits map straight back to Documents entries.
|
||||||
|
type TextDoc struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
FolderID string `json:"folder,omitempty"`
|
||||||
|
Ext string `json:"ext,omitempty"`
|
||||||
|
Content string `json:"content"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TextIndex is the storage/search surface for locally extracted document text.
|
||||||
|
// It complements Searcher: ESSearcher reads OnlyOffice's index, ESTextIndex
|
||||||
|
// reads ours.
|
||||||
|
type TextIndex interface {
|
||||||
|
Put(ctx context.Context, docs []TextDoc) error
|
||||||
|
Delete(ctx context.Context, ids []string) error
|
||||||
|
Search(ctx context.Context, q SearchQuery) ([]SearchHit, error)
|
||||||
|
Name() string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ESTextIndex is a TextIndex (and Searcher) over a dedicated Elasticsearch
|
||||||
|
// index filled by TextIndexer.
|
||||||
|
type ESTextIndex struct {
|
||||||
|
cfg ESTextConfig
|
||||||
|
http *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ TextIndex = (*ESTextIndex)(nil)
|
||||||
|
_ Searcher = (*ESTextIndex)(nil)
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewESTextIndex returns a searcher/writer for the own full-text index. The URL
|
||||||
|
// is required; an empty index falls back to oo_docs_text.
|
||||||
|
func NewESTextIndex(cfg ESTextConfig) (*ESTextIndex, error) {
|
||||||
|
if strings.TrimSpace(cfg.URL) == "" {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: elasticsearch URL is empty (set ONLYOFFICE_ES_URL)")
|
||||||
|
}
|
||||||
|
cfg.URL = strings.TrimRight(cfg.URL, "/")
|
||||||
|
if cfg.Index == "" {
|
||||||
|
cfg.Index = defaultESTextIndex
|
||||||
|
}
|
||||||
|
return &ESTextIndex{cfg: cfg, http: &http.Client{Timeout: 120 * time.Second}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Name implements Searcher and TextIndex.
|
||||||
|
func (x *ESTextIndex) Name() string { return "es-text" }
|
||||||
|
|
||||||
|
// Index returns the configured index name.
|
||||||
|
func (x *ESTextIndex) Index() string { return x.cfg.Index }
|
||||||
|
|
||||||
|
// esTextMapping pins explicit types: content must stay a plain text field (no
|
||||||
|
// keyword subfield) and folder/ext stay exact keywords for filters.
|
||||||
|
const esTextMapping = `{
|
||||||
|
"mappings": {
|
||||||
|
"properties": {
|
||||||
|
"id": {"type": "keyword"},
|
||||||
|
"title": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 512}}},
|
||||||
|
"folder": {"type": "keyword"},
|
||||||
|
"ext": {"type": "keyword"},
|
||||||
|
"content": {"type": "text"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`
|
||||||
|
|
||||||
|
// Ensure creates the index with the explicit mapping. A missing index is
|
||||||
|
// created; an already existing one is left untouched.
|
||||||
|
func (x *ESTextIndex) Ensure(ctx context.Context) error {
|
||||||
|
status, raw, err := x.do(ctx, http.MethodPut, "/"+x.cfg.Index, []byte(esTextMapping), "application/json")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if status == http.StatusOK {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if status == http.StatusBadRequest && bytes.Contains(raw, []byte("resource_already_exists_exception")) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return fmt.Errorf("onlyoffice: create text index %s: %d %s", x.cfg.Index, status, truncate(string(raw), 300))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put upserts documents via the bulk API and refreshes so they are immediately
|
||||||
|
// searchable.
|
||||||
|
func (x *ESTextIndex) Put(ctx context.Context, docs []TextDoc) error {
|
||||||
|
if len(docs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
status, raw, err := x.do(ctx, http.MethodPost, "/"+x.cfg.Index+"/_bulk?refresh=true", esTextBulkBody(docs), "application/x-ndjson")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if status >= 400 {
|
||||||
|
return fmt.Errorf("onlyoffice: bulk index %s: %d %s", x.cfg.Index, status, truncate(string(raw), 400))
|
||||||
|
}
|
||||||
|
var res esBulkResponse
|
||||||
|
if err := json.Unmarshal(raw, &res); err != nil {
|
||||||
|
return fmt.Errorf("onlyoffice: decode bulk response: %w", err)
|
||||||
|
}
|
||||||
|
if !res.Errors {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return fmt.Errorf("onlyoffice: bulk index %s: %s", x.cfg.Index, res.firstError())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete removes documents by OnlyOffice file id. A missing index means there
|
||||||
|
// is nothing to delete.
|
||||||
|
func (x *ESTextIndex) Delete(ctx context.Context, ids []string) error {
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
body, err := json.Marshal(map[string]any{"query": map[string]any{"terms": map[string]any{"id": ids}}})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
status, raw, err := x.do(ctx, http.MethodPost, "/"+x.cfg.Index+"/_delete_by_query?refresh=true", body, "application/json")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if status == http.StatusNotFound {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if status >= 400 {
|
||||||
|
return fmt.Errorf("onlyoffice: delete from %s: %d %s", x.cfg.Index, status, truncate(string(raw), 400))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Search runs a multi_match over title (boosted) and content, with optional
|
||||||
|
// folder and extension filters. A missing index yields no hits, not an error.
|
||||||
|
func (x *ESTextIndex) Search(ctx context.Context, q SearchQuery) ([]SearchHit, error) {
|
||||||
|
q.Text = strings.TrimSpace(q.Text)
|
||||||
|
if q.Text == "" {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: empty search query")
|
||||||
|
}
|
||||||
|
body, err := json.Marshal(esTextSearchRequest(q))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: build elasticsearch query: %w", err)
|
||||||
|
}
|
||||||
|
status, raw, err := x.do(ctx, http.MethodPost, "/"+x.cfg.Index+"/_search", body, "application/json")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if status == http.StatusNotFound {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if status >= 400 {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: search %s: %d %s", x.cfg.Index, status, truncate(string(raw), 400))
|
||||||
|
}
|
||||||
|
return parseESTextResponse(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
// do sends one request and returns the status and body (bounded). The caller
|
||||||
|
// decides which statuses are errors.
|
||||||
|
func (x *ESTextIndex) do(ctx context.Context, method, path string, body []byte, contentType string) (int, []byte, error) {
|
||||||
|
var r io.Reader
|
||||||
|
if body != nil {
|
||||||
|
r = bytes.NewReader(body)
|
||||||
|
}
|
||||||
|
req, err := http.NewRequestWithContext(ctx, method, x.cfg.URL+path, r)
|
||||||
|
if err != nil {
|
||||||
|
return 0, nil, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Accept", "application/json")
|
||||||
|
if contentType != "" {
|
||||||
|
req.Header.Set("Content-Type", contentType)
|
||||||
|
}
|
||||||
|
resp, err := x.http.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return 0, nil, fmt.Errorf("onlyoffice: elasticsearch %s: %w", method, err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
raw, err := io.ReadAll(io.LimitReader(resp.Body, maxESResponseSize))
|
||||||
|
if err != nil {
|
||||||
|
return resp.StatusCode, nil, err
|
||||||
|
}
|
||||||
|
return resp.StatusCode, raw, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// esTextBulkBody renders the NDJSON bulk payload. Pure, so it is unit-tested.
|
||||||
|
func esTextBulkBody(docs []TextDoc) []byte {
|
||||||
|
var b bytes.Buffer
|
||||||
|
enc := json.NewEncoder(&b)
|
||||||
|
enc.SetEscapeHTML(false)
|
||||||
|
for _, d := range docs {
|
||||||
|
_ = enc.Encode(map[string]any{"index": map[string]any{"_id": d.ID}})
|
||||||
|
_ = enc.Encode(d)
|
||||||
|
}
|
||||||
|
return b.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
// esTextSearchRequest builds the own-index query. Pure, so it is unit-tested.
|
||||||
|
func esTextSearchRequest(q SearchQuery) esRequest {
|
||||||
|
limit := q.Limit
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = defaultESLimit
|
||||||
|
}
|
||||||
|
if limit > maxESLimit {
|
||||||
|
limit = maxESLimit
|
||||||
|
}
|
||||||
|
fields := []string{"title^2", "content"}
|
||||||
|
must := []esClause{{MultiMatch: &esMultiMatch{Query: q.Text, Fields: fields}}}
|
||||||
|
|
||||||
|
var filter []esClause
|
||||||
|
if f := strings.TrimSpace(q.FolderID); f != "" {
|
||||||
|
filter = append(filter, esClause{Term: map[string]any{"folder": f}})
|
||||||
|
}
|
||||||
|
if exts := normalizeExtensions(q.Extensions); len(exts) > 0 {
|
||||||
|
filter = append(filter, esClause{Terms: map[string]any{"ext": exts}})
|
||||||
|
}
|
||||||
|
|
||||||
|
return esRequest{
|
||||||
|
Size: limit,
|
||||||
|
Source: []string{"id", "title", "folder", "ext"},
|
||||||
|
Query: esQuery{Bool: esBool{Must: must, Filter: filter}},
|
||||||
|
Highlight: esHighlight{PreTags: []string{"<em>"}, PostTags: []string{"</em>"}, Fields: map[string]struct{}{"title": {}, "content": {}}},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// esBulkResponse is the subset of an ES bulk response we consume.
|
||||||
|
type esBulkResponse struct {
|
||||||
|
Errors bool `json:"errors"`
|
||||||
|
Items []map[string]struct {
|
||||||
|
ID string `json:"_id"`
|
||||||
|
Status int `json:"status"`
|
||||||
|
Error *struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
} `json:"error"`
|
||||||
|
} `json:"items"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstError returns a compact description of the first failed bulk item.
|
||||||
|
func (r esBulkResponse) firstError() string {
|
||||||
|
for _, item := range r.Items {
|
||||||
|
for op, res := range item {
|
||||||
|
if res.Error != nil {
|
||||||
|
return fmt.Sprintf("%s %s: %s %s", op, res.ID, res.Error.Type, res.Error.Reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "unknown bulk error"
|
||||||
|
}
|
||||||
|
|
||||||
|
// esTextResponse is the subset of an own-index search response we consume.
|
||||||
|
type esTextResponse struct {
|
||||||
|
Hits struct {
|
||||||
|
Total struct {
|
||||||
|
Value int `json:"value"`
|
||||||
|
} `json:"total"`
|
||||||
|
Hits []struct {
|
||||||
|
ID string `json:"_id"`
|
||||||
|
Score float64 `json:"_score"`
|
||||||
|
Source TextDoc `json:"_source"`
|
||||||
|
HL map[string][]string `json:"highlight"`
|
||||||
|
} `json:"hits"`
|
||||||
|
} `json:"hits"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseESTextResponse converts an own-index search response into SearchHit
|
||||||
|
// values. Pure, so it is unit-tested.
|
||||||
|
func parseESTextResponse(raw []byte) ([]SearchHit, error) {
|
||||||
|
var r esTextResponse
|
||||||
|
if err := json.Unmarshal(raw, &r); err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: decode elasticsearch response: %w", err)
|
||||||
|
}
|
||||||
|
hits := make([]SearchHit, 0, len(r.Hits.Hits))
|
||||||
|
for _, h := range r.Hits.Hits {
|
||||||
|
id := h.Source.ID
|
||||||
|
if id == "" {
|
||||||
|
id = h.ID
|
||||||
|
}
|
||||||
|
parent := h.Source.FolderID
|
||||||
|
var path []string
|
||||||
|
if parent != "" {
|
||||||
|
path = []string{parent}
|
||||||
|
}
|
||||||
|
hits = append(hits, SearchHit{
|
||||||
|
Entry: Entry{
|
||||||
|
ID: id,
|
||||||
|
ParentID: parent,
|
||||||
|
Title: h.Source.Title,
|
||||||
|
Kind: File,
|
||||||
|
Provider: "es-text",
|
||||||
|
},
|
||||||
|
Score: h.Score,
|
||||||
|
Highlight: esHighlightText(h.HL),
|
||||||
|
Path: path,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return hits, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
//go:build integration
|
||||||
|
|
||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/eslider/go-onlyoffice/internal/docpipe"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestIntegrationESTextIndex verifies the own full-text index end to end
|
||||||
|
// against a live Elasticsearch: create the index with its mapping, index a
|
||||||
|
// document, find it by content (and reject it via a folder filter and after
|
||||||
|
// deletion), then drop the throwaway index.
|
||||||
|
//
|
||||||
|
// Requires ONLYOFFICE_ES_URL (a reachable ES endpoint — in the current setup a
|
||||||
|
// tunnel to the ES inside the OnlyOffice VM, see docs/elasticsearch.md). It
|
||||||
|
// does not need OnlyOffice credentials because no file is downloaded: the
|
||||||
|
// TextIndexer write path is covered by unit tests with a fake extractor.
|
||||||
|
func TestIntegrationESTextIndex(t *testing.T) {
|
||||||
|
esURL := strings.TrimSpace(os.Getenv("ONLYOFFICE_ES_URL"))
|
||||||
|
if esURL == "" {
|
||||||
|
t.Skip("ONLYOFFICE_ES_URL not set — skipping Elasticsearch integration test")
|
||||||
|
}
|
||||||
|
stamp := time.Now().UTC().Format("20060102150405")
|
||||||
|
idx, err := NewESTextIndex(ESTextConfig{URL: esURL, Index: "oo_docs_text_it_" + stamp})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewESTextIndex: %v", err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
cleanupCtx, done := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer done()
|
||||||
|
_, _, _ = idx.do(cleanupCtx, http.MethodDelete, "/"+idx.Index(), nil, "")
|
||||||
|
})
|
||||||
|
|
||||||
|
if err := idx.Ensure(ctx); err != nil {
|
||||||
|
t.Fatalf("Ensure: %v", err)
|
||||||
|
}
|
||||||
|
// Ensure is idempotent.
|
||||||
|
if err := idx.Ensure(ctx); err != nil {
|
||||||
|
t.Fatalf("Ensure (second): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
token := "gotes" + stamp
|
||||||
|
doc := TextDoc{
|
||||||
|
ID: "3578",
|
||||||
|
Title: "2026-07-28-S1021-Edelweiss-rechnung.pdf",
|
||||||
|
FolderID: "634",
|
||||||
|
Ext: "pdf",
|
||||||
|
Content: "Begleitzettel SGB XI — Rechnung " + token,
|
||||||
|
}
|
||||||
|
if err := idx.Put(ctx, []TextDoc{doc}); err != nil {
|
||||||
|
t.Fatalf("Put: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
hits, err := idx.Search(ctx, SearchQuery{Text: token})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Search: %v", err)
|
||||||
|
}
|
||||||
|
if len(hits) != 1 || hits[0].ID != "3578" {
|
||||||
|
t.Fatalf("content search hits = %+v, want doc 3578", hits)
|
||||||
|
}
|
||||||
|
if !strings.Contains(hits[0].Highlight, token) {
|
||||||
|
t.Errorf("highlight = %q, want token", hits[0].Highlight)
|
||||||
|
}
|
||||||
|
|
||||||
|
if hits, err := idx.Search(ctx, SearchQuery{Text: token, FolderID: "999"}); err != nil {
|
||||||
|
t.Fatalf("Search with folder filter: %v", err)
|
||||||
|
} else if len(hits) != 0 {
|
||||||
|
t.Errorf("folder filter returned %d hits, want 0", len(hits))
|
||||||
|
}
|
||||||
|
if hits, err := idx.Search(ctx, SearchQuery{Text: token, Extensions: []string{"docx"}}); err != nil {
|
||||||
|
t.Fatalf("Search with ext filter: %v", err)
|
||||||
|
} else if len(hits) != 0 {
|
||||||
|
t.Errorf("ext filter returned %d hits, want 0", len(hits))
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := idx.Delete(ctx, []string{"3578"}); err != nil {
|
||||||
|
t.Fatalf("Delete: %v", err)
|
||||||
|
}
|
||||||
|
if hits, err := idx.Search(ctx, SearchQuery{Text: token}); err != nil {
|
||||||
|
t.Fatalf("Search after delete: %v", err)
|
||||||
|
} else if len(hits) != 0 {
|
||||||
|
t.Errorf("after delete search returned %d hits, want 0", len(hits))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestIntegrationESTextIndexPDFAttachment indexes testdata/pdf-with-attachment.pdf
|
||||||
|
// through the real pipeline (TextIndexer + docpipe: pdfdetach + pdftotext) and
|
||||||
|
// verifies that text living only in the embedded attachment is searchable.
|
||||||
|
//
|
||||||
|
// Requires ONLYOFFICE_ES_URL plus poppler (pdfdetach/pdftotext). No OnlyOffice
|
||||||
|
// credentials are needed: a fixture FileStore serves the PDF bytes.
|
||||||
|
func TestIntegrationESTextIndexPDFAttachment(t *testing.T) {
|
||||||
|
esURL := strings.TrimSpace(os.Getenv("ONLYOFFICE_ES_URL"))
|
||||||
|
if esURL == "" {
|
||||||
|
t.Skip("ONLYOFFICE_ES_URL not set — skipping Elasticsearch integration test")
|
||||||
|
}
|
||||||
|
if docpipe.LookPath().PDFDetach == "" {
|
||||||
|
t.Skip("pdfdetach not on PATH — skipping PDF attachment integration test")
|
||||||
|
}
|
||||||
|
pdf, err := os.ReadFile("testdata/pdf-with-attachment.pdf")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read fixture: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
stamp := time.Now().UTC().Format("20060102150405")
|
||||||
|
idx, err := NewESTextIndex(ESTextConfig{URL: esURL, Index: "oo_docs_text_it_att_" + stamp})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewESTextIndex: %v", err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
t.Cleanup(func() {
|
||||||
|
cleanupCtx, done := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer done()
|
||||||
|
_, _, _ = idx.do(cleanupCtx, http.MethodDelete, "/"+idx.Index(), nil, "")
|
||||||
|
})
|
||||||
|
if err := idx.Ensure(ctx); err != nil {
|
||||||
|
t.Fatalf("Ensure: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
store := &textFakeStore{files: map[string][]byte{"9001": pdf}}
|
||||||
|
ix := NewTextIndexer(store, idx)
|
||||||
|
res, err := ix.IndexEntries(ctx, []Entry{{ID: "9001", Title: "scan.pdf", ParentID: "777", Kind: File}}, IndexOptions{MinChars: 1})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("IndexEntries: %v", err)
|
||||||
|
}
|
||||||
|
if res.Indexed != 1 || res.Failed != 0 {
|
||||||
|
t.Fatalf("result = %+v, want one indexed doc", res)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Token appears only inside the embedded goo-note.txt attachment.
|
||||||
|
hits, err := idx.Search(ctx, SearchQuery{Text: "gooattachmenttoken"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Search attachment token: %v", err)
|
||||||
|
}
|
||||||
|
if len(hits) != 1 || hits[0].ID != "9001" {
|
||||||
|
t.Fatalf("attachment-token hits = %+v, want doc 9001", hits)
|
||||||
|
}
|
||||||
|
if !strings.Contains(hits[0].Highlight, "gooattachmenttoken") {
|
||||||
|
t.Errorf("highlight = %q, want attachment token", hits[0].Highlight)
|
||||||
|
}
|
||||||
|
// Body text is indexed as before.
|
||||||
|
if hits, err := idx.Search(ctx, SearchQuery{Text: "goobodytoken"}); err != nil {
|
||||||
|
t.Fatalf("Search body token: %v", err)
|
||||||
|
} else if len(hits) != 1 {
|
||||||
|
t.Errorf("body-token hits = %d, want 1", len(hits))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,275 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestESTextSearchRequestShape(t *testing.T) {
|
||||||
|
got := esTextSearchRequest(SearchQuery{
|
||||||
|
Text: "S1021",
|
||||||
|
FolderID: "634",
|
||||||
|
Extensions: []string{".PDF", "pdf"},
|
||||||
|
Limit: 5,
|
||||||
|
})
|
||||||
|
if got.Size != 5 {
|
||||||
|
t.Errorf("size = %d, want 5", got.Size)
|
||||||
|
}
|
||||||
|
mm := got.Query.Bool.Must[0].MultiMatch
|
||||||
|
if mm == nil || !reflect.DeepEqual(mm.Fields, []string{"title^2", "content"}) {
|
||||||
|
t.Fatalf("multi_match = %+v, want title^2 + content", mm)
|
||||||
|
}
|
||||||
|
if _, ok := got.Highlight.Fields["content"]; !ok {
|
||||||
|
t.Error("content highlight missing")
|
||||||
|
}
|
||||||
|
if _, ok := got.Highlight.Fields["title"]; !ok {
|
||||||
|
t.Error("title highlight missing")
|
||||||
|
}
|
||||||
|
var folder, exts int
|
||||||
|
for _, f := range got.Query.Bool.Filter {
|
||||||
|
switch {
|
||||||
|
case f.Term != nil && f.Term["folder"] != nil:
|
||||||
|
folder++
|
||||||
|
if f.Term["folder"] != "634" {
|
||||||
|
t.Errorf("folder term = %+v", f.Term)
|
||||||
|
}
|
||||||
|
case f.Terms != nil:
|
||||||
|
exts++
|
||||||
|
if !reflect.DeepEqual(f.Terms["ext"], []string{"pdf"}) {
|
||||||
|
t.Errorf("ext terms = %+v, want deduped pdf", f.Terms)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if folder != 1 || exts != 1 {
|
||||||
|
t.Errorf("filters folder=%d exts=%d, want 1 each", folder, exts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestESTextBulkBody(t *testing.T) {
|
||||||
|
body := esTextBulkBody([]TextDoc{
|
||||||
|
{ID: "3578", Title: "S1021.pdf", FolderID: "634", Ext: "pdf", Content: "Begleitzettel <S1021> & mehr"},
|
||||||
|
{ID: "3579", Title: "S1023.pdf", Ext: "pdf", Content: "x"},
|
||||||
|
})
|
||||||
|
lines := strings.Split(strings.TrimRight(string(body), "\n"), "\n")
|
||||||
|
if len(lines) != 4 {
|
||||||
|
t.Fatalf("bulk body has %d lines, want 4:\n%s", len(lines), body)
|
||||||
|
}
|
||||||
|
if !strings.Contains(lines[0], `"index"`) || !strings.Contains(lines[0], `"_id":"3578"`) {
|
||||||
|
t.Errorf("action line = %q", lines[0])
|
||||||
|
}
|
||||||
|
if !strings.Contains(lines[1], `"content":"Begleitzettel <S1021> & mehr"`) {
|
||||||
|
t.Errorf("source line should keep HTML unescaped, got %q", lines[1])
|
||||||
|
}
|
||||||
|
if !strings.Contains(lines[2], `"_id":"3579"`) {
|
||||||
|
t.Errorf("second action line = %q", lines[2])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseESTextResponse(t *testing.T) {
|
||||||
|
raw := []byte(`{
|
||||||
|
"hits": {
|
||||||
|
"total": {"value": 1, "relation": "eq"},
|
||||||
|
"hits": [
|
||||||
|
{
|
||||||
|
"_id": "3578",
|
||||||
|
"_score": 3.21,
|
||||||
|
"_source": {"id": "3578", "title": "2026-07-28-S1021-Edelweiss-rechnung.pdf", "folder": "634", "ext": "pdf"},
|
||||||
|
"highlight": {"content": ["Begleitzettel … <em>S1021</em> …"]}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}`)
|
||||||
|
hits, err := parseESTextResponse(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parseESTextResponse: %v", err)
|
||||||
|
}
|
||||||
|
if len(hits) != 1 {
|
||||||
|
t.Fatalf("hits = %d, want 1", len(hits))
|
||||||
|
}
|
||||||
|
h := hits[0]
|
||||||
|
if h.ID != "3578" || h.Title != "2026-07-28-S1021-Edelweiss-rechnung.pdf" || h.Kind != File {
|
||||||
|
t.Errorf("entry = %+v", h.Entry)
|
||||||
|
}
|
||||||
|
if h.ParentID != "634" || !reflect.DeepEqual(h.Path, []string{"634"}) {
|
||||||
|
t.Errorf("path = %v parent = %q", h.Path, h.ParentID)
|
||||||
|
}
|
||||||
|
if h.Provider != "es-text" {
|
||||||
|
t.Errorf("provider = %q", h.Provider)
|
||||||
|
}
|
||||||
|
if h.Highlight != "Begleitzettel … S1021 …" {
|
||||||
|
t.Errorf("highlight = %q, want tags stripped", h.Highlight)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewESTextIndexDefaults(t *testing.T) {
|
||||||
|
if _, err := NewESTextIndex(ESTextConfig{}); err == nil {
|
||||||
|
t.Error("empty URL: want error")
|
||||||
|
}
|
||||||
|
x, err := NewESTextIndex(ESTextConfig{URL: "http://es:9200/"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewESTextIndex: %v", err)
|
||||||
|
}
|
||||||
|
if x.Index() != defaultESTextIndex {
|
||||||
|
t.Errorf("index = %q, want %q", x.Index(), defaultESTextIndex)
|
||||||
|
}
|
||||||
|
if x.cfg.URL != "http://es:9200" {
|
||||||
|
t.Errorf("url = %q, want trimmed", x.cfg.URL)
|
||||||
|
}
|
||||||
|
if x.Name() != "es-text" {
|
||||||
|
t.Errorf("Name() = %q", x.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestESTextConfigFromEnvIndexDefault(t *testing.T) {
|
||||||
|
t.Setenv("ONLYOFFICE_ES_URL", "http://es:9200/")
|
||||||
|
t.Setenv("ONLYOFFICE_ES_TEXT_INDEX", "")
|
||||||
|
cfg := ESTextConfigFromEnv()
|
||||||
|
if cfg.Index != defaultESTextIndex {
|
||||||
|
t.Errorf("index = %q, want %q", cfg.Index, defaultESTextIndex)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTextIndexerIndexEntries(t *testing.T) {
|
||||||
|
store := &textFakeStore{
|
||||||
|
files: map[string][]byte{"1": []byte("PDFBYTES")},
|
||||||
|
}
|
||||||
|
idx := &textFakeIndex{}
|
||||||
|
ix := NewTextIndexer(store, idx)
|
||||||
|
ix.Extractor = textFakeExtractor{prefix: "TEXT "}
|
||||||
|
|
||||||
|
res, err := ix.IndexEntries(context.Background(), []Entry{
|
||||||
|
{ID: "1", Title: "Rechnung.PDF", ParentID: "649", Kind: File},
|
||||||
|
{ID: "2", Title: "Tabelle.xlsx", ParentID: "649", Kind: File},
|
||||||
|
{ID: "3", Title: "Unterordner", Kind: Folder},
|
||||||
|
}, IndexOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("IndexEntries: %v", err)
|
||||||
|
}
|
||||||
|
if res.Scanned != 3 || res.Indexed != 1 || res.Skipped != 2 || res.Failed != 0 {
|
||||||
|
t.Errorf("result = %+v, want scanned=3 indexed=1 skipped=2 failed=0", res)
|
||||||
|
}
|
||||||
|
if len(idx.docs) != 1 {
|
||||||
|
t.Fatalf("indexed docs = %d, want 1", len(idx.docs))
|
||||||
|
}
|
||||||
|
got := idx.docs[0]
|
||||||
|
want := TextDoc{ID: "1", Title: "Rechnung.PDF", FolderID: "649", Ext: "pdf", Content: "TEXT PDFBYTES"}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("doc = %+v, want %+v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTextIndexerRecordsExtractionFailure(t *testing.T) {
|
||||||
|
store := &textFakeStore{files: map[string][]byte{"1": []byte("x")}}
|
||||||
|
idx := &textFakeIndex{}
|
||||||
|
ix := NewTextIndexer(store, idx)
|
||||||
|
ix.Extractor = textFailingExtractor{}
|
||||||
|
|
||||||
|
res, err := ix.IndexEntries(context.Background(), []Entry{{ID: "1", Title: "a.pdf", Kind: File}}, IndexOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("IndexEntries: %v", err)
|
||||||
|
}
|
||||||
|
if res.Indexed != 0 || res.Failed != 1 || len(res.Errors) != 1 {
|
||||||
|
t.Errorf("result = %+v, want one failure recorded", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTextIndexerPlanFolder(t *testing.T) {
|
||||||
|
store := &textFakeStore{dirs: map[string][]Entry{
|
||||||
|
"root": {
|
||||||
|
{ID: "10", Title: "a.pdf", Kind: File},
|
||||||
|
{ID: "11", Title: "sub", Kind: Folder},
|
||||||
|
},
|
||||||
|
"11": {
|
||||||
|
{ID: "12", Title: "b.PDF", Kind: File},
|
||||||
|
{ID: "13", Title: "c.xlsx", Kind: File},
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
ix := NewTextIndexer(store, &textFakeIndex{})
|
||||||
|
|
||||||
|
flat, err := ix.PlanFolder(context.Background(), "root", IndexOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PlanFolder: %v", err)
|
||||||
|
}
|
||||||
|
if len(flat) != 1 || flat[0].ID != "10" {
|
||||||
|
t.Errorf("flat plan = %+v, want only a.pdf", flat)
|
||||||
|
}
|
||||||
|
deep, err := ix.PlanFolder(context.Background(), "root", IndexOptions{Recursive: true, Limit: 10})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PlanFolder recursive: %v", err)
|
||||||
|
}
|
||||||
|
if len(deep) != 2 {
|
||||||
|
t.Errorf("recursive plan = %d entries, want 2", len(deep))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- fakes -----------------------------------------------------------------
|
||||||
|
|
||||||
|
type textFakeStore struct {
|
||||||
|
dirs map[string][]Entry
|
||||||
|
files map[string][]byte
|
||||||
|
stat map[string]Entry
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *textFakeStore) Name() string { return "fake" }
|
||||||
|
|
||||||
|
func (f *textFakeStore) List(_ context.Context, parentID string) ([]Entry, error) {
|
||||||
|
return f.dirs[parentID], nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *textFakeStore) Stat(_ context.Context, id string) (Entry, error) {
|
||||||
|
if e, ok := f.stat[id]; ok {
|
||||||
|
return e, nil
|
||||||
|
}
|
||||||
|
return Entry{}, fmt.Errorf("not found: %s", id)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *textFakeStore) Download(_ context.Context, id string, w io.Writer) (int64, error) {
|
||||||
|
b, ok := f.files[id]
|
||||||
|
if !ok {
|
||||||
|
return 0, fmt.Errorf("no bytes for %s", id)
|
||||||
|
}
|
||||||
|
n, err := w.Write(b)
|
||||||
|
return int64(n), err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *textFakeStore) CreateFolder(context.Context, string, string) (Entry, error) {
|
||||||
|
return Entry{}, nil
|
||||||
|
}
|
||||||
|
func (f *textFakeStore) Upload(context.Context, string, string, io.Reader) (Entry, error) {
|
||||||
|
return Entry{}, nil
|
||||||
|
}
|
||||||
|
func (f *textFakeStore) Move(context.Context, []string, string) error { return nil }
|
||||||
|
func (f *textFakeStore) Copy(context.Context, []string, string) error { return nil }
|
||||||
|
func (f *textFakeStore) Rename(context.Context, string, string) error { return nil }
|
||||||
|
func (f *textFakeStore) Delete(context.Context, []string) error { return nil }
|
||||||
|
|
||||||
|
type textFakeIndex struct{ docs []TextDoc }
|
||||||
|
|
||||||
|
func (f *textFakeIndex) Put(_ context.Context, docs []TextDoc) error {
|
||||||
|
f.docs = append(f.docs, docs...)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func (f *textFakeIndex) Delete(context.Context, []string) error { return nil }
|
||||||
|
func (f *textFakeIndex) Search(context.Context, SearchQuery) ([]SearchHit, error) { return nil, nil }
|
||||||
|
func (f *textFakeIndex) Name() string { return "fake" }
|
||||||
|
|
||||||
|
type textFakeExtractor struct{ prefix string }
|
||||||
|
|
||||||
|
func (f textFakeExtractor) Extract(path, _, _ string, _ int) (string, error) {
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return f.prefix + string(b), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type textFailingExtractor struct{}
|
||||||
|
|
||||||
|
func (textFailingExtractor) Extract(string, string, string, int) (string, error) {
|
||||||
|
return "", fmt.Errorf("boom")
|
||||||
|
}
|
||||||
+249
@@ -0,0 +1,249 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// Single file client (epic #34, F4 #38). FileClient composes the registered
|
||||||
|
// FileStore and Searcher backends and picks one per operation: REST/DAV for
|
||||||
|
// writes, PostgreSQL (when registered) for fast reads, Elasticsearch for name
|
||||||
|
// and content search. Client.Files returns the facade; it also implements
|
||||||
|
// FileStore, so existing callers keep compiling.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ProviderES is the composed Elasticsearch searcher. The SQL store owns
|
||||||
|
// ProviderPG/ProviderMySQL (file_pg.go); the facade references ProviderPG in
|
||||||
|
// readOrder.
|
||||||
|
const ProviderES = "elasticsearch"
|
||||||
|
|
||||||
|
var (
|
||||||
|
errNoReadBackend = errors.New("onlyoffice: no file backend registered for reads")
|
||||||
|
errNoWriteBackend = errors.New("onlyoffice: no file backend registered for writes")
|
||||||
|
errNoSearcher = errors.New("onlyoffice: no search backend registered (set ONLYOFFICE_ES_URL)")
|
||||||
|
)
|
||||||
|
|
||||||
|
// FileClient is the single entry point for file operations. It holds the
|
||||||
|
// registered backends and the order in which each operation tries them.
|
||||||
|
type FileClient struct {
|
||||||
|
stores map[string]FileStore
|
||||||
|
searchers map[string]Searcher
|
||||||
|
|
||||||
|
readOrder []string
|
||||||
|
writeOrder []string
|
||||||
|
searchOrder []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// newFileClient builds the facade over the built-in REST and DAV stores. The
|
||||||
|
// Elasticsearch searcher is registered when ONLYOFFICE_ES_URL is set; the
|
||||||
|
// missing-credential case is left to Search so read-only commands still work.
|
||||||
|
func (c *Client) newFileClient() *FileClient {
|
||||||
|
f := &FileClient{
|
||||||
|
stores: map[string]FileStore{
|
||||||
|
ProviderREST: &restStore{c: c},
|
||||||
|
ProviderDAV: &davStore{c: c},
|
||||||
|
},
|
||||||
|
searchers: map[string]Searcher{},
|
||||||
|
readOrder: []string{ProviderPG, ProviderREST, ProviderDAV},
|
||||||
|
writeOrder: []string{ProviderREST, ProviderDAV},
|
||||||
|
searchOrder: []string{ProviderES},
|
||||||
|
}
|
||||||
|
if cfg := ESConfigFromEnv(); cfg.URL != "" {
|
||||||
|
if es, err := NewESSearcher(cfg); err == nil {
|
||||||
|
f.searchers[ProviderES] = es
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterStore adds or replaces a named backend (for example the PostgreSQL
|
||||||
|
// read store). The name is matched case-insensitively.
|
||||||
|
func (f *FileClient) RegisterStore(name string, s FileStore) {
|
||||||
|
if f == nil || s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
name = normalizeProvider(name)
|
||||||
|
if name == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if f.stores == nil {
|
||||||
|
f.stores = map[string]FileStore{}
|
||||||
|
}
|
||||||
|
f.stores[name] = s
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterSearcher adds or replaces a named search backend.
|
||||||
|
func (f *FileClient) RegisterSearcher(name string, s Searcher) {
|
||||||
|
if f == nil || s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
name = normalizeProvider(name)
|
||||||
|
if name == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if f.searchers == nil {
|
||||||
|
f.searchers = map[string]Searcher{}
|
||||||
|
}
|
||||||
|
f.searchers[name] = s
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read returns the preferred backend for reads: PostgreSQL when registered,
|
||||||
|
// then REST, then WebDAV.
|
||||||
|
func (f *FileClient) Read() FileStore { return f.firstStore(f.readOrder) }
|
||||||
|
|
||||||
|
// Write returns the preferred backend for writes: REST, then WebDAV.
|
||||||
|
func (f *FileClient) Write() FileStore { return f.firstStore(f.writeOrder) }
|
||||||
|
|
||||||
|
// Search returns the preferred name/content searcher (Elasticsearch), or an
|
||||||
|
// error when no search backend is configured.
|
||||||
|
func (f *FileClient) Search() (Searcher, error) {
|
||||||
|
if f == nil {
|
||||||
|
return nil, errNoSearcher
|
||||||
|
}
|
||||||
|
for _, name := range f.searchOrder {
|
||||||
|
if s := f.searchers[normalizeProvider(name)]; s != nil {
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil, errNoSearcher
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstStore returns the first registered store in the order.
|
||||||
|
func (f *FileClient) firstStore(order []string) FileStore {
|
||||||
|
if f == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for _, name := range order {
|
||||||
|
if s := f.stores[normalizeProvider(name)]; s != nil {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// orderedStores returns the registered stores in the order.
|
||||||
|
func (f *FileClient) orderedStores(order []string) []FileStore {
|
||||||
|
if f == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]FileStore, 0, len(order))
|
||||||
|
for _, name := range order {
|
||||||
|
if s := f.stores[normalizeProvider(name)]; s != nil {
|
||||||
|
out = append(out, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeProvider(name string) string {
|
||||||
|
return strings.ToLower(strings.TrimSpace(name))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Name implements FileStore and reports the preferred read backend.
|
||||||
|
func (f *FileClient) Name() string {
|
||||||
|
if s := f.Read(); s != nil {
|
||||||
|
return s.Name()
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// List reads from the preferred backend, falling back to the next read backend
|
||||||
|
// only on a transient error (429/502/503/504).
|
||||||
|
func (f *FileClient) List(ctx context.Context, parentID string) ([]Entry, error) {
|
||||||
|
return fallbackRead(ctx, f.orderedStores(f.readOrder), func(s FileStore) ([]Entry, error) {
|
||||||
|
return s.List(ctx, parentID)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stat reads from the preferred backend, with the same transient fallback.
|
||||||
|
func (f *FileClient) Stat(ctx context.Context, id string) (Entry, error) {
|
||||||
|
return fallbackRead(ctx, f.orderedStores(f.readOrder), func(s FileStore) (Entry, error) {
|
||||||
|
return s.Stat(ctx, id)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Download streams file bytes. It does not fall back: a failed attempt may have
|
||||||
|
// already written partial bytes into w, so a second backend would append.
|
||||||
|
func (f *FileClient) Download(ctx context.Context, id string, w io.Writer) (int64, error) {
|
||||||
|
s := f.Read()
|
||||||
|
if s == nil {
|
||||||
|
return 0, errNoReadBackend
|
||||||
|
}
|
||||||
|
return s.Download(ctx, id, w)
|
||||||
|
}
|
||||||
|
|
||||||
|
// CreateFolder writes to the preferred write backend.
|
||||||
|
func (f *FileClient) CreateFolder(ctx context.Context, parentID, title string) (Entry, error) {
|
||||||
|
s := f.Write()
|
||||||
|
if s == nil {
|
||||||
|
return Entry{}, errNoWriteBackend
|
||||||
|
}
|
||||||
|
return s.CreateFolder(ctx, parentID, title)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload writes to the preferred write backend.
|
||||||
|
func (f *FileClient) Upload(ctx context.Context, parentID, title string, r io.Reader) (Entry, error) {
|
||||||
|
s := f.Write()
|
||||||
|
if s == nil {
|
||||||
|
return Entry{}, errNoWriteBackend
|
||||||
|
}
|
||||||
|
return s.Upload(ctx, parentID, title, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move writes to the preferred write backend.
|
||||||
|
func (f *FileClient) Move(ctx context.Context, ids []string, parentID string) error {
|
||||||
|
s := f.Write()
|
||||||
|
if s == nil {
|
||||||
|
return errNoWriteBackend
|
||||||
|
}
|
||||||
|
return s.Move(ctx, ids, parentID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy writes to the preferred write backend.
|
||||||
|
func (f *FileClient) Copy(ctx context.Context, ids []string, parentID string) error {
|
||||||
|
s := f.Write()
|
||||||
|
if s == nil {
|
||||||
|
return errNoWriteBackend
|
||||||
|
}
|
||||||
|
return s.Copy(ctx, ids, parentID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rename writes to the preferred write backend.
|
||||||
|
func (f *FileClient) Rename(ctx context.Context, id, title string) error {
|
||||||
|
s := f.Write()
|
||||||
|
if s == nil {
|
||||||
|
return errNoWriteBackend
|
||||||
|
}
|
||||||
|
return s.Rename(ctx, id, title)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete writes to the preferred write backend.
|
||||||
|
func (f *FileClient) Delete(ctx context.Context, ids []string) error {
|
||||||
|
s := f.Write()
|
||||||
|
if s == nil {
|
||||||
|
return errNoWriteBackend
|
||||||
|
}
|
||||||
|
return s.Delete(ctx, ids)
|
||||||
|
}
|
||||||
|
|
||||||
|
// fallbackRead runs op against each store in order, moving on only when the
|
||||||
|
// error is transient. Non-transient errors (not found, forbidden) are final.
|
||||||
|
func fallbackRead[T any](ctx context.Context, stores []FileStore, op func(FileStore) (T, error)) (T, error) {
|
||||||
|
var zero T
|
||||||
|
if len(stores) == 0 {
|
||||||
|
return zero, errNoReadBackend
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
for i, s := range stores {
|
||||||
|
var v T
|
||||||
|
v, err = op(s)
|
||||||
|
if err == nil {
|
||||||
|
return v, nil
|
||||||
|
}
|
||||||
|
if i == len(stores)-1 || !Transient(err) {
|
||||||
|
return zero, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return zero, err
|
||||||
|
}
|
||||||
@@ -0,0 +1,230 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeStore is a FileStore test double; it records which backend served a call
|
||||||
|
// and returns a canned result or error.
|
||||||
|
type fakeStore struct {
|
||||||
|
name string
|
||||||
|
entries []Entry
|
||||||
|
err error
|
||||||
|
calls *[]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) record(op string) {
|
||||||
|
if f.calls != nil {
|
||||||
|
*f.calls = append(*f.calls, op+":"+f.name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Name() string { return f.name }
|
||||||
|
|
||||||
|
func (f *fakeStore) List(_ context.Context, _ string) ([]Entry, error) {
|
||||||
|
f.record("list")
|
||||||
|
if f.err != nil {
|
||||||
|
return nil, f.err
|
||||||
|
}
|
||||||
|
return f.entries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Stat(_ context.Context, id string) (Entry, error) {
|
||||||
|
f.record("stat")
|
||||||
|
if f.err != nil {
|
||||||
|
return Entry{}, f.err
|
||||||
|
}
|
||||||
|
return Entry{ID: id, Title: "t-" + f.name, Provider: f.name}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) CreateFolder(_ context.Context, _, title string) (Entry, error) {
|
||||||
|
f.record("mkdir")
|
||||||
|
if f.err != nil {
|
||||||
|
return Entry{}, f.err
|
||||||
|
}
|
||||||
|
return Entry{ID: "new", Title: title, Provider: f.name}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Upload(_ context.Context, _, title string, _ io.Reader) (Entry, error) {
|
||||||
|
f.record("upload")
|
||||||
|
return Entry{ID: "up", Title: title, Provider: f.name}, f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Download(_ context.Context, _ string, _ io.Writer) (int64, error) {
|
||||||
|
f.record("download")
|
||||||
|
return 0, f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Move(_ context.Context, _ []string, _ string) error {
|
||||||
|
f.record("move")
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Copy(_ context.Context, _ []string, _ string) error {
|
||||||
|
f.record("copy")
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Rename(_ context.Context, _, _ string) error {
|
||||||
|
f.record("rename")
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) Delete(_ context.Context, _ []string) error {
|
||||||
|
f.record("delete")
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeSearcher struct{ name string }
|
||||||
|
|
||||||
|
func (s *fakeSearcher) Name() string { return s.name }
|
||||||
|
|
||||||
|
func (s *fakeSearcher) Search(_ context.Context, _ SearchQuery) ([]SearchHit, error) {
|
||||||
|
return []SearchHit{{Entry: Entry{Title: s.name}}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFacadeTestClient(stores map[string]FileStore, read, write []string) *FileClient {
|
||||||
|
return &FileClient{
|
||||||
|
stores: stores,
|
||||||
|
searchers: map[string]Searcher{},
|
||||||
|
readOrder: read,
|
||||||
|
writeOrder: write,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFileClientIsFileStore guarantees the facade can stand in for the
|
||||||
|
// interface anywhere a plain FileStore is expected.
|
||||||
|
func TestFileClientIsFileStore(t *testing.T) {
|
||||||
|
var _ FileStore = (*FileClient)(nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientFilesPrefersRESTForReadsAndWrites(t *testing.T) {
|
||||||
|
c := NewClient(Credentials{})
|
||||||
|
f := c.Files()
|
||||||
|
if got := f.Read().Name(); got != ProviderREST {
|
||||||
|
t.Errorf("Read().Name() = %q, want %q", got, ProviderREST)
|
||||||
|
}
|
||||||
|
if got := f.Write().Name(); got != ProviderREST {
|
||||||
|
t.Errorf("Write().Name() = %q, want %q", got, ProviderREST)
|
||||||
|
}
|
||||||
|
if got := f.Name(); got != ProviderREST {
|
||||||
|
t.Errorf("Name() = %q, want %q", got, ProviderREST)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientPostgresTakesReadPriority(t *testing.T) {
|
||||||
|
pg := &fakeStore{name: ProviderPG}
|
||||||
|
f := newFacadeTestClient(
|
||||||
|
map[string]FileStore{ProviderREST: &fakeStore{name: ProviderREST}, ProviderPG: pg},
|
||||||
|
[]string{ProviderPG, ProviderREST},
|
||||||
|
[]string{ProviderREST},
|
||||||
|
)
|
||||||
|
if got := f.Read().Name(); got != ProviderPG {
|
||||||
|
t.Errorf("Read().Name() = %q, want %q", got, ProviderPG)
|
||||||
|
}
|
||||||
|
if got := f.Write().Name(); got != ProviderREST {
|
||||||
|
t.Errorf("Write().Name() = %q, want %q (PG is read-only)", got, ProviderREST)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientRegisterStoreNormalizesName(t *testing.T) {
|
||||||
|
pg := &fakeStore{name: "pg"}
|
||||||
|
f := newFacadeTestClient(map[string]FileStore{}, []string{ProviderPG}, nil)
|
||||||
|
f.RegisterStore(" POSTGRES ", pg)
|
||||||
|
if got := f.Read(); got != pg {
|
||||||
|
t.Fatalf("Read() = %v, want registered postgres store", got)
|
||||||
|
}
|
||||||
|
f.RegisterStore("", pg)
|
||||||
|
f.RegisterStore("pg", nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientReadFallsBackOnlyOnTransient(t *testing.T) {
|
||||||
|
var calls []string
|
||||||
|
primary := &fakeStore{name: "primary", err: fmt.Errorf("onlyoffice: list: 503 unavailable"), calls: &calls}
|
||||||
|
secondary := &fakeStore{name: "secondary", entries: []Entry{{ID: "1"}}, calls: &calls}
|
||||||
|
f := newFacadeTestClient(
|
||||||
|
map[string]FileStore{"primary": primary, "secondary": secondary},
|
||||||
|
[]string{"primary", "secondary"},
|
||||||
|
nil,
|
||||||
|
)
|
||||||
|
got, err := f.List(context.Background(), "root")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("List: %v", err)
|
||||||
|
}
|
||||||
|
if len(got) != 1 || got[0].ID != "1" {
|
||||||
|
t.Fatalf("List() = %+v, want secondary entry", got)
|
||||||
|
}
|
||||||
|
want := []string{"list:primary", "list:secondary"}
|
||||||
|
if fmt.Sprint(calls) != fmt.Sprint(want) {
|
||||||
|
t.Fatalf("call order = %v, want %v", calls, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientReadStopsOnPermanentError(t *testing.T) {
|
||||||
|
var calls []string
|
||||||
|
primary := &fakeStore{name: "primary", err: errors.New("onlyoffice: not found"), calls: &calls}
|
||||||
|
secondary := &fakeStore{name: "secondary", entries: []Entry{{ID: "1"}}, calls: &calls}
|
||||||
|
f := newFacadeTestClient(
|
||||||
|
map[string]FileStore{"primary": primary, "secondary": secondary},
|
||||||
|
[]string{"primary", "secondary"},
|
||||||
|
nil,
|
||||||
|
)
|
||||||
|
if _, err := f.List(context.Background(), "root"); err == nil {
|
||||||
|
t.Fatal("expected permanent error to be returned")
|
||||||
|
}
|
||||||
|
if len(calls) != 1 || calls[0] != "list:primary" {
|
||||||
|
t.Fatalf("secondary backend must not run on a permanent error: %v", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientWriteUsesWriteBackend(t *testing.T) {
|
||||||
|
var calls []string
|
||||||
|
rest := &fakeStore{name: ProviderREST, calls: &calls}
|
||||||
|
dav := &fakeStore{name: ProviderDAV, calls: &calls}
|
||||||
|
f := newFacadeTestClient(
|
||||||
|
map[string]FileStore{ProviderREST: rest, ProviderDAV: dav},
|
||||||
|
[]string{ProviderREST},
|
||||||
|
[]string{ProviderREST, ProviderDAV},
|
||||||
|
)
|
||||||
|
if _, err := f.CreateFolder(context.Background(), "p", "t"); err != nil {
|
||||||
|
t.Fatalf("CreateFolder: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := f.Upload(context.Background(), "p", "t", nil); err != nil {
|
||||||
|
t.Fatalf("Upload: %v", err)
|
||||||
|
}
|
||||||
|
if len(calls) != 2 || calls[0] != "mkdir:rest" || calls[1] != "upload:rest" {
|
||||||
|
t.Fatalf("write calls = %v, want REST", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientWriteWithoutBackend(t *testing.T) {
|
||||||
|
f := newFacadeTestClient(map[string]FileStore{}, nil, nil)
|
||||||
|
if err := f.Delete(context.Background(), []string{"1"}); !errors.Is(err, errNoWriteBackend) {
|
||||||
|
t.Fatalf("Delete err = %v, want errNoWriteBackend", err)
|
||||||
|
}
|
||||||
|
if _, err := f.List(context.Background(), "root"); !errors.Is(err, errNoReadBackend) {
|
||||||
|
t.Fatalf("List err = %v, want errNoReadBackend", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileClientSearchSelection(t *testing.T) {
|
||||||
|
f := &FileClient{searchers: map[string]Searcher{}, searchOrder: []string{ProviderES}}
|
||||||
|
_, err := f.Search()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "ONLYOFFICE_ES_URL") {
|
||||||
|
t.Fatalf("Search without backend = %v, want ONLYOFFICE_ES_URL hint", err)
|
||||||
|
}
|
||||||
|
es := &fakeSearcher{name: "fake-es"}
|
||||||
|
f.RegisterSearcher(ProviderES, es)
|
||||||
|
got, err := f.Search()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Search: %v", err)
|
||||||
|
}
|
||||||
|
if got.Name() != "fake-es" {
|
||||||
|
t.Fatalf("searcher = %q, want fake-es", got.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
+516
@@ -0,0 +1,516 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// Read-only SQL backend of the unified file client (epic #34, F2 #36).
|
||||||
|
//
|
||||||
|
// The goal is to read files and folders straight from the Community Server
|
||||||
|
// database, without the REST layer. Research on the live portal (VM
|
||||||
|
// `onlyoffice-v2`) showed the server runs on **MySQL 8.0** (`files_file`,
|
||||||
|
// `files_folder`, `files_folder_tree`, tenant `tenants_tenants`), not
|
||||||
|
// PostgreSQL — see docs/community-server-db.md. The store below therefore
|
||||||
|
// speaks `database/sql` and selects its driver from the DSN, so it works
|
||||||
|
// against the live MySQL today and against PostgreSQL if the portal is ever
|
||||||
|
// migrated. Every query is a SELECT; the write methods of FileStore return
|
||||||
|
// ErrReadOnly.
|
||||||
|
//
|
||||||
|
// Downloads follow the portal's S3/MinIO object layout through the shared
|
||||||
|
// MinIO helper in storage_fallback.go — no HTTP file endpoint is used.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-sql-driver/mysql"
|
||||||
|
_ "github.com/jackc/pgx/v5/stdlib"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Provider names for the SQL backend. ProviderPG is the value Name reports for
|
||||||
|
// a PostgreSQL connection and ProviderMySQL for MySQL.
|
||||||
|
const (
|
||||||
|
ProviderPG = "postgres"
|
||||||
|
ProviderMySQL = "mysql"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrReadOnly is returned by every FileStore write method of the SQL backend.
|
||||||
|
var ErrReadOnly = errors.New("onlyoffice: sql file store is read-only")
|
||||||
|
|
||||||
|
const (
|
||||||
|
pgConnectTimeout = 10 * time.Second
|
||||||
|
pgSearchLimit = 50
|
||||||
|
pgSearchMaxLimit = 500
|
||||||
|
)
|
||||||
|
|
||||||
|
// PGConfig configures the read-only SQL store. DSN is a driver DSN:
|
||||||
|
// `user:pass@tcp(host:port)/onlyoffice?parseTime=true` for MySQL or a
|
||||||
|
// `postgres://` / libpq keyword string for PostgreSQL. Driver, when set,
|
||||||
|
// forces the engine ("postgres" or "mysql"); otherwise it is detected from the
|
||||||
|
// DSN. Tenant filters rows (empty means all tenants).
|
||||||
|
type PGConfig struct {
|
||||||
|
DSN string
|
||||||
|
Driver string
|
||||||
|
Tenant string
|
||||||
|
}
|
||||||
|
|
||||||
|
// PGConfigFromEnv reads ONLYOFFICE_DSN (or the ONLYOFFICE_PG_* parts),
|
||||||
|
// ONLYOFFICE_PG_DRIVER and the tenant from ONLYOFFICE_PG_TENANT /
|
||||||
|
// ONLYOFFICE_TENANT. The library never loads dotfiles — the CLI does that.
|
||||||
|
func PGConfigFromEnv() PGConfig {
|
||||||
|
dsn := strings.TrimSpace(os.Getenv("ONLYOFFICE_DSN"))
|
||||||
|
if dsn == "" {
|
||||||
|
dsn = pgDSNFromParts()
|
||||||
|
}
|
||||||
|
return PGConfig{
|
||||||
|
DSN: dsn,
|
||||||
|
Driver: strings.TrimSpace(os.Getenv("ONLYOFFICE_PG_DRIVER")),
|
||||||
|
Tenant: firstNonEmpty(os.Getenv("ONLYOFFICE_PG_TENANT"), os.Getenv("ONLYOFFICE_TENANT")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// pgDSNFromParts builds a libpq keyword DSN from ONLYOFFICE_PG_* variables.
|
||||||
|
// It returns "" unless a host is set, which keeps the MySQL path (ONLYOFFICE_DSN)
|
||||||
|
// the default.
|
||||||
|
func pgDSNFromParts() string {
|
||||||
|
host := strings.TrimSpace(os.Getenv("ONLYOFFICE_PG_HOST"))
|
||||||
|
if host == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
port := firstNonEmpty(os.Getenv("ONLYOFFICE_PG_PORT"), "5432")
|
||||||
|
dbname := firstNonEmpty(os.Getenv("ONLYOFFICE_PG_DBNAME"), "onlyoffice")
|
||||||
|
sslmode := firstNonEmpty(os.Getenv("ONLYOFFICE_PG_SSLMODE"), "disable")
|
||||||
|
return fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s",
|
||||||
|
host, port, os.Getenv("ONLYOFFICE_PG_USER"), os.Getenv("ONLYOFFICE_PG_PASSWORD"), dbname, sslmode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// pgStore is a read-only FileStore/Searcher over the Community Server database.
|
||||||
|
type pgStore struct {
|
||||||
|
db *sql.DB
|
||||||
|
driver string
|
||||||
|
tenantID int64
|
||||||
|
hasTenant bool
|
||||||
|
http *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ FileStore = (*pgStore)(nil)
|
||||||
|
_ Searcher = (*pgStore)(nil)
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewPGStore opens the database and verifies connectivity. It never writes.
|
||||||
|
func NewPGStore(cfg PGConfig) (*pgStore, error) {
|
||||||
|
dsn := strings.TrimSpace(cfg.DSN)
|
||||||
|
if dsn == "" {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql file store: empty DSN (set ONLYOFFICE_DSN)")
|
||||||
|
}
|
||||||
|
driver := pgDriver(dsn, cfg.Driver)
|
||||||
|
dsn, err := normalizeSQLDSN(driver, dsn)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
db, err := sql.Open(sqlDriverName(driver), dsn)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql file store: open %s: %w", driver, err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), pgConnectTimeout)
|
||||||
|
defer cancel()
|
||||||
|
if err := db.PingContext(ctx); err != nil {
|
||||||
|
db.Close()
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql file store: ping %s: %w", driver, err)
|
||||||
|
}
|
||||||
|
s := &pgStore{db: db, driver: driver, http: &http.Client{}}
|
||||||
|
if t := strings.TrimSpace(cfg.Tenant); t != "" {
|
||||||
|
n, err := strconv.ParseInt(t, 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
db.Close()
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql file store: non-numeric tenant %q", t)
|
||||||
|
}
|
||||||
|
s.tenantID, s.hasTenant = n, true
|
||||||
|
}
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close releases the database handle.
|
||||||
|
func (s *pgStore) Close() error { return s.db.Close() }
|
||||||
|
|
||||||
|
// Name implements FileStore and Searcher.
|
||||||
|
func (s *pgStore) Name() string { return s.driver }
|
||||||
|
|
||||||
|
// pgDriver resolves the engine: the explicit value wins, otherwise the DSN
|
||||||
|
// shape decides. A leading postgres:// scheme or a libpq keyword DSN (which
|
||||||
|
// always carries '=') selects PostgreSQL; anything else is MySQL.
|
||||||
|
func pgDriver(dsn, explicit string) string {
|
||||||
|
switch strings.ToLower(strings.TrimSpace(explicit)) {
|
||||||
|
case ProviderPG, "pg", "postgresql", "pgx":
|
||||||
|
return ProviderPG
|
||||||
|
case ProviderMySQL, "mariadb":
|
||||||
|
return ProviderMySQL
|
||||||
|
}
|
||||||
|
l := strings.ToLower(strings.TrimSpace(dsn))
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(l, "postgres://"), strings.HasPrefix(l, "postgresql://"):
|
||||||
|
return ProviderPG
|
||||||
|
case strings.HasPrefix(l, "mysql://"), strings.Contains(l, "@tcp("), strings.Contains(l, "@unix("):
|
||||||
|
return ProviderMySQL
|
||||||
|
case strings.Contains(l, "="):
|
||||||
|
return ProviderPG
|
||||||
|
default:
|
||||||
|
return ProviderMySQL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// sqlDriverName maps the engine to its registered database/sql driver.
|
||||||
|
func sqlDriverName(driver string) string {
|
||||||
|
if driver == ProviderPG {
|
||||||
|
return "pgx"
|
||||||
|
}
|
||||||
|
return "mysql"
|
||||||
|
}
|
||||||
|
|
||||||
|
// normalizeSQLDSN converts a mysql:// URL to the go-sql-driver form and forces
|
||||||
|
// parseTime so datetime columns scan into time.Time. PostgreSQL DSNs pass
|
||||||
|
// through untouched.
|
||||||
|
func normalizeSQLDSN(driver, dsn string) (string, error) {
|
||||||
|
if driver != ProviderMySQL {
|
||||||
|
return dsn, nil
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(strings.ToLower(dsn), "mysql://") {
|
||||||
|
converted, err := mysqlDSNFromURL(dsn)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
dsn = converted
|
||||||
|
}
|
||||||
|
cfg, err := mysql.ParseDSN(dsn)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("onlyoffice: sql file store: parse mysql DSN: %w", err)
|
||||||
|
}
|
||||||
|
cfg.ParseTime = true
|
||||||
|
return cfg.FormatDSN(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// mysqlDSNFromURL turns mysql://user:pass@host:port/db into the driver DSN.
|
||||||
|
func mysqlDSNFromURL(raw string) (string, error) {
|
||||||
|
u, err := url.Parse(raw)
|
||||||
|
if err != nil || u.Host == "" {
|
||||||
|
return "", fmt.Errorf("onlyoffice: sql file store: bad mysql URL %q", raw)
|
||||||
|
}
|
||||||
|
user := ""
|
||||||
|
if u.User != nil {
|
||||||
|
user = u.User.Username()
|
||||||
|
if p, ok := u.User.Password(); ok {
|
||||||
|
user += ":" + p
|
||||||
|
}
|
||||||
|
}
|
||||||
|
q := u.Query()
|
||||||
|
q.Set("parseTime", "true")
|
||||||
|
return fmt.Sprintf("%s@tcp(%s)/%s?%s", user, u.Host, strings.TrimPrefix(u.Path, "/"), q.Encode()), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// rebind rewrites '?' placeholders to PostgreSQL's $1..$n. MySQL keeps them.
|
||||||
|
func rebind(query, driver string) string {
|
||||||
|
if driver != ProviderPG {
|
||||||
|
return query
|
||||||
|
}
|
||||||
|
var b strings.Builder
|
||||||
|
b.Grow(len(query) + 8)
|
||||||
|
n := 0
|
||||||
|
for _, r := range query {
|
||||||
|
if r == '?' {
|
||||||
|
n++
|
||||||
|
b.WriteByte('$')
|
||||||
|
b.WriteString(strconv.Itoa(n))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
b.WriteRune(r)
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// List returns the folders and files directly below parentID, folders first.
|
||||||
|
func (s *pgStore) List(ctx context.Context, parentID string) ([]Entry, error) {
|
||||||
|
pid, err := parseEntryID(parentID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
folders, err := s.queryFolders(ctx, "parent_id = ?", pid)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
files, err := s.queryFiles(ctx, "folder_id = ? AND current_version = 1", pid)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := make([]Entry, 0, len(folders)+len(files))
|
||||||
|
for _, f := range folders {
|
||||||
|
out = append(out, folderRowToEntry(f, s.Name()))
|
||||||
|
}
|
||||||
|
for _, f := range files {
|
||||||
|
out = append(out, fileRowToEntry(f, s.Name()))
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stat resolves a folder or file id to an Entry. Folders win when both id
|
||||||
|
// spaces overlap (they never do on a real portal, but the lookup is cheap).
|
||||||
|
func (s *pgStore) Stat(ctx context.Context, id string) (Entry, error) {
|
||||||
|
n, err := parseEntryID(id)
|
||||||
|
if err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
folders, err := s.queryFolders(ctx, "id = ?", n)
|
||||||
|
if err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
if len(folders) > 0 {
|
||||||
|
return folderRowToEntry(folders[0], s.Name()), nil
|
||||||
|
}
|
||||||
|
files, err := s.queryFiles(ctx, "id = ? AND current_version = 1", n)
|
||||||
|
if err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
if len(files) == 0 {
|
||||||
|
return Entry{}, fmt.Errorf("onlyoffice: sql file store: id %s not found", id)
|
||||||
|
}
|
||||||
|
return fileRowToEntry(files[0], s.Name()), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Download streams the file's current version from the portal's S3/MinIO store.
|
||||||
|
// The object key is reconstructed from the file id and version; the parent
|
||||||
|
// folder id is not part of the key.
|
||||||
|
func (s *pgStore) Download(ctx context.Context, id string, w io.Writer) (int64, error) {
|
||||||
|
n, err := parseEntryID(id)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
files, err := s.queryFiles(ctx, "id = ? AND current_version = 1", n)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if len(files) == 0 {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: sql file store: file %s not found", id)
|
||||||
|
}
|
||||||
|
f := files[0]
|
||||||
|
key := csObjectKey(s.tenantID, f.id, f.version, filepath.Ext(f.title))
|
||||||
|
return downloadMinioObject(ctx, s.http, key, w)
|
||||||
|
}
|
||||||
|
|
||||||
|
// CreateFolder is unavailable: the SQL backend is read-only.
|
||||||
|
func (s *pgStore) CreateFolder(context.Context, string, string) (Entry, error) {
|
||||||
|
return Entry{}, ErrReadOnly
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload is unavailable: the SQL backend is read-only.
|
||||||
|
func (s *pgStore) Upload(context.Context, string, string, io.Reader) (Entry, error) {
|
||||||
|
return Entry{}, ErrReadOnly
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move is unavailable: the SQL backend is read-only.
|
||||||
|
func (s *pgStore) Move(context.Context, []string, string) error { return ErrReadOnly }
|
||||||
|
|
||||||
|
// Copy is unavailable: the SQL backend is read-only.
|
||||||
|
func (s *pgStore) Copy(context.Context, []string, string) error { return ErrReadOnly }
|
||||||
|
|
||||||
|
// Rename is unavailable: the SQL backend is read-only.
|
||||||
|
func (s *pgStore) Rename(context.Context, string, string) error { return ErrReadOnly }
|
||||||
|
|
||||||
|
// Delete is unavailable: the SQL backend is read-only.
|
||||||
|
func (s *pgStore) Delete(context.Context, []string) error { return ErrReadOnly }
|
||||||
|
|
||||||
|
// Search matches file titles by substring. Content search lives in the
|
||||||
|
// Elasticsearch backend; q.InContent is ignored here.
|
||||||
|
func (s *pgStore) Search(ctx context.Context, q SearchQuery) ([]SearchHit, error) {
|
||||||
|
text := strings.TrimSpace(q.Text)
|
||||||
|
if text == "" {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: empty search query")
|
||||||
|
}
|
||||||
|
limit := q.Limit
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = pgSearchLimit
|
||||||
|
}
|
||||||
|
if limit > pgSearchMaxLimit {
|
||||||
|
limit = pgSearchMaxLimit
|
||||||
|
}
|
||||||
|
|
||||||
|
where := "title LIKE ? AND current_version = 1"
|
||||||
|
args := []any{"%" + text + "%"}
|
||||||
|
if s.hasTenant {
|
||||||
|
where += " AND tenant_id = ?"
|
||||||
|
args = append(args, s.tenantID)
|
||||||
|
}
|
||||||
|
if fid := strings.TrimSpace(q.FolderID); fid != "" {
|
||||||
|
n, err := parseEntryID(fid)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
where += " AND folder_id = ?"
|
||||||
|
args = append(args, n)
|
||||||
|
}
|
||||||
|
for _, ext := range normalizeExtensions(q.Extensions) {
|
||||||
|
where += " AND LOWER(title) LIKE ?"
|
||||||
|
args = append(args, "%."+ext)
|
||||||
|
}
|
||||||
|
query := rebind(`SELECT id, folder_id, title, content_length, version, create_on, modified_on
|
||||||
|
FROM files_file WHERE `+where+` ORDER BY modified_on DESC, id DESC LIMIT ?`, s.driver)
|
||||||
|
args = append(args, limit)
|
||||||
|
|
||||||
|
rows, err := s.db.QueryContext(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql search: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var hits []SearchHit
|
||||||
|
for rows.Next() {
|
||||||
|
f, err := scanFileRow(rows)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
hits = append(hits, SearchHit{Entry: fileRowToEntry(f, s.Name())})
|
||||||
|
}
|
||||||
|
return hits, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// queryFolders runs a folder SELECT with the tenant filter applied.
|
||||||
|
func (s *pgStore) queryFolders(ctx context.Context, where string, arg any) ([]pgFolderRow, error) {
|
||||||
|
args := []any{arg}
|
||||||
|
if s.hasTenant {
|
||||||
|
where += " AND tenant_id = ?"
|
||||||
|
args = append(args, s.tenantID)
|
||||||
|
}
|
||||||
|
query := rebind(`SELECT id, parent_id, title, create_on, modified_on
|
||||||
|
FROM files_folder WHERE `+where+` ORDER BY title, id`, s.driver)
|
||||||
|
rows, err := s.db.QueryContext(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql list folders: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []pgFolderRow
|
||||||
|
for rows.Next() {
|
||||||
|
var r pgFolderRow
|
||||||
|
if err := rows.Scan(&r.id, &r.parentID, &r.title, &r.created, &r.modified); err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql folder row: %w", err)
|
||||||
|
}
|
||||||
|
out = append(out, r)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// queryFiles runs a file SELECT for the current version with the tenant filter.
|
||||||
|
func (s *pgStore) queryFiles(ctx context.Context, where string, arg any) ([]pgFileRow, error) {
|
||||||
|
args := []any{arg}
|
||||||
|
if s.hasTenant {
|
||||||
|
where += " AND tenant_id = ?"
|
||||||
|
args = append(args, s.tenantID)
|
||||||
|
}
|
||||||
|
query := rebind(`SELECT id, folder_id, title, content_length, version, create_on, modified_on
|
||||||
|
FROM files_file WHERE `+where+` ORDER BY title, id`, s.driver)
|
||||||
|
rows, err := s.db.QueryContext(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: sql list files: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []pgFileRow
|
||||||
|
for rows.Next() {
|
||||||
|
f, err := scanFileRow(rows)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, f)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// pgFileRow is one current files_file row.
|
||||||
|
type pgFileRow struct {
|
||||||
|
id int64
|
||||||
|
folderID int64
|
||||||
|
title string
|
||||||
|
size int64
|
||||||
|
version int
|
||||||
|
created time.Time
|
||||||
|
modified time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// pgFolderRow is one files_folder row.
|
||||||
|
type pgFolderRow struct {
|
||||||
|
id int64
|
||||||
|
parentID int64
|
||||||
|
title string
|
||||||
|
created time.Time
|
||||||
|
modified time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanFileRow reads the canonical file column order.
|
||||||
|
func scanFileRow(rows *sql.Rows) (pgFileRow, error) {
|
||||||
|
var f pgFileRow
|
||||||
|
if err := rows.Scan(&f.id, &f.folderID, &f.title, &f.size, &f.version, &f.created, &f.modified); err != nil {
|
||||||
|
return f, fmt.Errorf("onlyoffice: sql file row: %w", err)
|
||||||
|
}
|
||||||
|
return f, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileRowToEntry maps a files_file row to the canonical model.
|
||||||
|
func fileRowToEntry(f pgFileRow, provider string) Entry {
|
||||||
|
return Entry{
|
||||||
|
ID: strconv.FormatInt(f.id, 10),
|
||||||
|
ParentID: strconv.FormatInt(f.folderID, 10),
|
||||||
|
Title: f.title,
|
||||||
|
Kind: File,
|
||||||
|
Size: f.size,
|
||||||
|
MIME: mimeForTitle(f.title, ""),
|
||||||
|
Created: f.created.UTC(),
|
||||||
|
Modified: f.modified.UTC(),
|
||||||
|
Version: f.version,
|
||||||
|
Provider: provider,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// folderRowToEntry maps a files_folder row to the canonical model.
|
||||||
|
func folderRowToEntry(f pgFolderRow, provider string) Entry {
|
||||||
|
return Entry{
|
||||||
|
ID: strconv.FormatInt(f.id, 10),
|
||||||
|
ParentID: strconv.FormatInt(f.parentID, 10),
|
||||||
|
Title: f.title,
|
||||||
|
Kind: Folder,
|
||||||
|
Created: f.created.UTC(),
|
||||||
|
Modified: f.modified.UTC(),
|
||||||
|
Provider: provider,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// csObjectKey reconstructs the object key the portal's S3 consumer uses:
|
||||||
|
//
|
||||||
|
// 00/00/<tenant>/files/folder_<shard>/file_<id>/v<version>/content.<ext>
|
||||||
|
//
|
||||||
|
// The shard is the next thousand above the file id (file 3727 -> folder_4000),
|
||||||
|
// NOT the parent folder id — verified live against the MinIO bucket.
|
||||||
|
func csObjectKey(tenant int64, fileID int64, version int, ext string) string {
|
||||||
|
shard := (fileID/1000 + 1) * 1000
|
||||||
|
ext = strings.TrimPrefix(strings.ToLower(strings.TrimSpace(ext)), ".")
|
||||||
|
if ext == "" {
|
||||||
|
ext = "bin"
|
||||||
|
}
|
||||||
|
if version < 1 {
|
||||||
|
version = 1
|
||||||
|
}
|
||||||
|
if tenant <= 0 {
|
||||||
|
tenant = 1
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("00/00/%02d/files/folder_%d/file_%d/v%d/content.%s", tenant, shard, fileID, version, ext)
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseEntryID parses a numeric OnlyOffice id or returns a store error.
|
||||||
|
func parseEntryID(id string) (int64, error) {
|
||||||
|
n, err := strconv.ParseInt(strings.TrimSpace(id), 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: sql file store: non-numeric id %q", id)
|
||||||
|
}
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
//go:build integration
|
||||||
|
|
||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestIntegrationPGStore exercises the read-only SQL backend against the live
|
||||||
|
// Community Server database and cross-checks list/stat/download with the REST
|
||||||
|
// FileStore. It needs ONLYOFFICE_DSN plus the usual ONLYOFFICE_URL/USER/PASS;
|
||||||
|
// ONLYOFFICE_PG_TEST_FILE_ID / ONLYOFFICE_PG_TEST_FOLDER_ID pick a real file
|
||||||
|
// (a file reachable over REST too). Download streams from MinIO, so it also
|
||||||
|
// needs MINIO_ACCESS_KEY/MINIO_SECRET_KEY.
|
||||||
|
//
|
||||||
|
// The live Community Server runs on MySQL; PostgreSQL is supported by the same
|
||||||
|
// code path when the DSN says so.
|
||||||
|
func TestIntegrationPGStore(t *testing.T) {
|
||||||
|
cfg := PGConfigFromEnv()
|
||||||
|
if strings.TrimSpace(cfg.DSN) == "" {
|
||||||
|
t.Skip("ONLYOFFICE_DSN not set — skipping SQL store integration test")
|
||||||
|
}
|
||||||
|
store, err := NewPGStore(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPGStore: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = store.Close() })
|
||||||
|
t.Logf("sql store backend: %s", store.Name())
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
if err := testPGStoreReadOnly(ctx, store); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fileID := strings.TrimSpace(os.Getenv("ONLYOFFICE_PG_TEST_FILE_ID"))
|
||||||
|
folderID := strings.TrimSpace(os.Getenv("ONLYOFFICE_PG_TEST_FOLDER_ID"))
|
||||||
|
if fileID == "" || folderID == "" {
|
||||||
|
t.Skip("ONLYOFFICE_PG_TEST_FILE_ID / ONLYOFFICE_PG_TEST_FOLDER_ID not set — skipping live comparison")
|
||||||
|
}
|
||||||
|
|
||||||
|
c := liveClient(t)
|
||||||
|
rest := c.Files()
|
||||||
|
|
||||||
|
dbFile, err := store.Stat(ctx, fileID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sql Stat(%s): %v", fileID, err)
|
||||||
|
}
|
||||||
|
restFile, err := rest.Stat(ctx, fileID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("rest Stat(%s): %v", fileID, err)
|
||||||
|
}
|
||||||
|
if dbFile.Kind != File {
|
||||||
|
t.Errorf("sql kind = %v, want file", dbFile.Kind)
|
||||||
|
}
|
||||||
|
if dbFile.ID != restFile.ID || dbFile.Title != restFile.Title || dbFile.ParentID != restFile.ParentID {
|
||||||
|
t.Errorf("stat mismatch sql=%+v rest=%+v", dbFile, restFile)
|
||||||
|
}
|
||||||
|
// GetFile omits contentLength, so size is only comparable when REST has it.
|
||||||
|
if restFile.Size > 0 && dbFile.Size != restFile.Size {
|
||||||
|
t.Errorf("size sql=%d rest=%d", dbFile.Size, restFile.Size)
|
||||||
|
}
|
||||||
|
if d := dbFile.Modified.Sub(restFile.Modified); d > 2*time.Minute || d < -2*time.Minute {
|
||||||
|
t.Errorf("modified sql=%v rest=%v", dbFile.Modified, restFile.Modified)
|
||||||
|
}
|
||||||
|
|
||||||
|
list, err := store.List(ctx, folderID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sql List(%s): %v", folderID, err)
|
||||||
|
}
|
||||||
|
if entryByID(list, fileID) == nil {
|
||||||
|
t.Errorf("file %s not in sql List(%s)", fileID, folderID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every file the REST layer can see in the folder must be in the SQL list
|
||||||
|
// (the SQL store sees more, so only assert this direction).
|
||||||
|
restList, err := rest.List(ctx, folderID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("rest List(%s): %v", folderID, err)
|
||||||
|
}
|
||||||
|
dbIDs := make(map[string]bool, len(list))
|
||||||
|
for _, e := range list {
|
||||||
|
dbIDs[e.ID] = true
|
||||||
|
}
|
||||||
|
for _, e := range restList {
|
||||||
|
if e.Kind == File && !dbIDs[e.ID] {
|
||||||
|
t.Errorf("rest file %s (%q) missing from sql list", e.ID, e.Title)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
n, err := store.Download(ctx, fileID, &buf)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sql Download(%s): %v", fileID, err)
|
||||||
|
}
|
||||||
|
if n == 0 || n != dbFile.Size {
|
||||||
|
t.Errorf("sql Download = %d bytes, stat says %d", n, dbFile.Size)
|
||||||
|
}
|
||||||
|
if os.Getenv("MINIO_ACCESS_KEY") != "" && os.Getenv("MINIO_SECRET_KEY") != "" {
|
||||||
|
var restBuf bytes.Buffer
|
||||||
|
rn, err := rest.Download(ctx, fileID, &restBuf)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("rest Download(%s): %v", fileID, err)
|
||||||
|
}
|
||||||
|
if rn != n || !bytes.Equal(restBuf.Bytes(), buf.Bytes()) {
|
||||||
|
t.Errorf("download mismatch sql=%d rest=%d bytes", n, rn)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
t.Log("MINIO_ACCESS_KEY/MINIO_SECRET_KEY not set — REST download cross-check skipped")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// testPGStoreReadOnly asserts that every write method returns ErrReadOnly.
|
||||||
|
func testPGStoreReadOnly(ctx context.Context, s *pgStore) error {
|
||||||
|
if _, err := s.CreateFolder(ctx, "1", "x"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
return errors.New("CreateFolder did not return ErrReadOnly")
|
||||||
|
}
|
||||||
|
if _, err := s.Upload(ctx, "1", "x", strings.NewReader("x")); !errors.Is(err, ErrReadOnly) {
|
||||||
|
return errors.New("Upload did not return ErrReadOnly")
|
||||||
|
}
|
||||||
|
if err := s.Move(ctx, []string{"1"}, "2"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
return errors.New("Move did not return ErrReadOnly")
|
||||||
|
}
|
||||||
|
if err := s.Copy(ctx, []string{"1"}, "2"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
return errors.New("Copy did not return ErrReadOnly")
|
||||||
|
}
|
||||||
|
if err := s.Rename(ctx, "1", "x"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
return errors.New("Rename did not return ErrReadOnly")
|
||||||
|
}
|
||||||
|
if err := s.Delete(ctx, []string{"1"}); !errors.Is(err, ErrReadOnly) {
|
||||||
|
return errors.New("Delete did not return ErrReadOnly")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
+165
@@ -0,0 +1,165 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRebind(t *testing.T) {
|
||||||
|
mysqlQuery := "SELECT id FROM files_file WHERE folder_id = ? AND title = ? LIMIT ?"
|
||||||
|
if got := rebind(mysqlQuery, ProviderMySQL); got != mysqlQuery {
|
||||||
|
t.Errorf("mysql query changed: %q", got)
|
||||||
|
}
|
||||||
|
want := "SELECT id FROM files_file WHERE folder_id = $1 AND title = $2 LIMIT $3"
|
||||||
|
if got := rebind(mysqlQuery, ProviderPG); got != want {
|
||||||
|
t.Errorf("rebind = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCSPObjectKey(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
tenant int64
|
||||||
|
fileID int64
|
||||||
|
version int
|
||||||
|
ext string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{1, 2, 1, ".docx", "00/00/01/files/folder_1000/file_2/v1/content.docx"},
|
||||||
|
{1, 999, 1, ".pdf", "00/00/01/files/folder_1000/file_999/v1/content.pdf"},
|
||||||
|
{1, 1000, 1, ".xlsx", "00/00/01/files/folder_2000/file_1000/v1/content.xlsx"},
|
||||||
|
{1, 3727, 1, ".pdf", "00/00/01/files/folder_4000/file_3727/v1/content.pdf"},
|
||||||
|
{1, 22484, 1, ".PDF", "00/00/01/files/folder_23000/file_22484/v1/content.pdf"},
|
||||||
|
{1, 4, 6, "xlsx", "00/00/01/files/folder_1000/file_4/v6/content.xlsx"},
|
||||||
|
{0, 7, 0, "", "00/00/01/files/folder_1000/file_7/v1/content.bin"},
|
||||||
|
{2, 11, 3, ".doc", "00/00/02/files/folder_1000/file_11/v3/content.doc"},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
if got := csObjectKey(tc.tenant, tc.fileID, tc.version, tc.ext); got != tc.want {
|
||||||
|
t.Errorf("csObjectKey(%d,%d,%d,%q) = %q, want %q", tc.tenant, tc.fileID, tc.version, tc.ext, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPGDriverDetection(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
dsn, explicit, want string
|
||||||
|
}{
|
||||||
|
{"postgres://u:p@h:5432/onlyoffice", "", ProviderPG},
|
||||||
|
{"postgresql://u:p@h/db", "", ProviderPG},
|
||||||
|
{"host=h user=u password=p dbname=onlyoffice sslmode=disable", "", ProviderPG},
|
||||||
|
{"root:secret@tcp(127.0.0.1:3306)/onlyoffice?parseTime=true", "", ProviderMySQL},
|
||||||
|
{"mysql://root:secret@127.0.0.1:3306/onlyoffice", "", ProviderMySQL},
|
||||||
|
{"root:secret@tcp(h:3306)/db", "postgres", ProviderPG},
|
||||||
|
{"postgres://u:p@h/db", "mysql", ProviderMySQL},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
if got := pgDriver(tc.dsn, tc.explicit); got != tc.want {
|
||||||
|
t.Errorf("pgDriver(%q, %q) = %q, want %q", tc.dsn, tc.explicit, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeSQLDSNMySQL(t *testing.T) {
|
||||||
|
got, err := normalizeSQLDSN(ProviderMySQL, "mysql://root:secret@127.0.0.1:3306/onlyoffice")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("normalizeSQLDSN: %v", err)
|
||||||
|
}
|
||||||
|
want := "root:secret@tcp(127.0.0.1:3306)/onlyoffice?parseTime=true"
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("normalize = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A driver DSN keeps parseTime and gains it when missing.
|
||||||
|
got, err = normalizeSQLDSN(ProviderMySQL, "root:secret@tcp(127.0.0.1:3306)/onlyoffice")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("normalizeSQLDSN: %v", err)
|
||||||
|
}
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("normalize = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileRowToEntry(t *testing.T) {
|
||||||
|
created := time.Date(2026, 9, 12, 18, 0, 37, 0, time.UTC)
|
||||||
|
modified := time.Date(2026, 9, 13, 13, 50, 36, 0, time.UTC)
|
||||||
|
e := fileRowToEntry(pgFileRow{
|
||||||
|
id: 22484, folderID: 649, title: "Rechnung.pdf",
|
||||||
|
size: 123433, version: 2, created: created, modified: modified,
|
||||||
|
}, ProviderMySQL)
|
||||||
|
if e.ID != "22484" || e.ParentID != "649" {
|
||||||
|
t.Errorf("ids = %q/%q", e.ID, e.ParentID)
|
||||||
|
}
|
||||||
|
if e.Title != "Rechnung.pdf" || e.Kind != File {
|
||||||
|
t.Errorf("title/kind = %q/%v", e.Title, e.Kind)
|
||||||
|
}
|
||||||
|
if e.Size != 123433 || e.Version != 2 {
|
||||||
|
t.Errorf("size/version = %d/%d", e.Size, e.Version)
|
||||||
|
}
|
||||||
|
if e.MIME != "application/pdf" {
|
||||||
|
t.Errorf("mime = %q", e.MIME)
|
||||||
|
}
|
||||||
|
if !e.Created.Equal(created) || !e.Modified.Equal(modified) {
|
||||||
|
t.Errorf("times = %v/%v", e.Created, e.Modified)
|
||||||
|
}
|
||||||
|
if e.Provider != ProviderMySQL {
|
||||||
|
t.Errorf("provider = %q", e.Provider)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFolderRowToEntry(t *testing.T) {
|
||||||
|
modified := time.Date(2026, 8, 1, 10, 30, 0, 0, time.UTC)
|
||||||
|
e := folderRowToEntry(pgFolderRow{id: 649, parentID: 647, title: "2025", modified: modified}, ProviderMySQL)
|
||||||
|
if e.ID != "649" || e.ParentID != "647" || e.Title != "2025" {
|
||||||
|
t.Errorf("folder = %+v", e)
|
||||||
|
}
|
||||||
|
if e.Kind != Folder {
|
||||||
|
t.Errorf("kind = %v, want folder", e.Kind)
|
||||||
|
}
|
||||||
|
if e.Size != 0 || e.MIME != "" {
|
||||||
|
t.Errorf("folder size/mime = %d/%q", e.Size, e.MIME)
|
||||||
|
}
|
||||||
|
if !e.Modified.Equal(modified) {
|
||||||
|
t.Errorf("modified = %v", e.Modified)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPGStoreWriteMethodsReadOnly(t *testing.T) {
|
||||||
|
s := &pgStore{driver: ProviderPG}
|
||||||
|
ctx := context.Background()
|
||||||
|
if _, err := s.CreateFolder(ctx, "1", "x"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
t.Errorf("CreateFolder err = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.Upload(ctx, "1", "x", nil); !errors.Is(err, ErrReadOnly) {
|
||||||
|
t.Errorf("Upload err = %v", err)
|
||||||
|
}
|
||||||
|
if err := s.Move(ctx, nil, "1"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
t.Errorf("Move err = %v", err)
|
||||||
|
}
|
||||||
|
if err := s.Copy(ctx, nil, "1"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
t.Errorf("Copy err = %v", err)
|
||||||
|
}
|
||||||
|
if err := s.Rename(ctx, "1", "x"); !errors.Is(err, ErrReadOnly) {
|
||||||
|
t.Errorf("Rename err = %v", err)
|
||||||
|
}
|
||||||
|
if err := s.Delete(ctx, nil); !errors.Is(err, ErrReadOnly) {
|
||||||
|
t.Errorf("Delete err = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPGStoreName(t *testing.T) {
|
||||||
|
if got := (&pgStore{driver: ProviderPG}).Name(); got != ProviderPG {
|
||||||
|
t.Errorf("Name = %q, want %q", got, ProviderPG)
|
||||||
|
}
|
||||||
|
if got := (&pgStore{driver: ProviderMySQL}).Name(); got != ProviderMySQL {
|
||||||
|
t.Errorf("Name = %q, want %q", got, ProviderMySQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPGStoreStatRejectsNonNumeric(t *testing.T) {
|
||||||
|
s := &pgStore{driver: ProviderPG}
|
||||||
|
if _, err := s.Stat(context.Background(), "not-a-number"); err == nil {
|
||||||
|
t.Error("Stat accepted a non-numeric id")
|
||||||
|
}
|
||||||
|
}
|
||||||
+230
@@ -0,0 +1,230 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// restStore implements FileStore on top of the REST Documents methods in
|
||||||
|
// files.go. It is a thin adapter: no endpoint logic lives here, and every call
|
||||||
|
// is wrapped in DoRetry.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// restStore is a FileStore over the REST Documents API.
|
||||||
|
type restStore struct{ c *Client }
|
||||||
|
|
||||||
|
// Name reports the backend name.
|
||||||
|
func (s *restStore) Name() string { return ProviderREST }
|
||||||
|
|
||||||
|
// List returns the files and folders directly below parentID.
|
||||||
|
func (s *restStore) List(ctx context.Context, parentID string) ([]Entry, error) {
|
||||||
|
var out []Entry
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
raw, err := s.c.ListFolder(ctx, parentID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
entries, err := entriesFromFolderMap(raw, ProviderREST)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out = entries
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stat returns file metadata. The REST adapter resolves files only; folders
|
||||||
|
// are listed by their parent (use List).
|
||||||
|
func (s *restStore) Stat(ctx context.Context, id string) (Entry, error) {
|
||||||
|
var out Entry
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
f, err := s.c.GetFile(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out = FileEntryToEntry(f, ProviderREST)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// CreateFolder creates a subfolder under parentID.
|
||||||
|
func (s *restStore) CreateFolder(ctx context.Context, parentID, title string) (Entry, error) {
|
||||||
|
var out Entry
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
m, err := s.c.CreateFolder(ctx, parentID, title)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
e, err := folderEntryFromMap(m, parentID, ProviderREST)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if e.ParentID == "" {
|
||||||
|
e.ParentID = parentID
|
||||||
|
}
|
||||||
|
if e.Title == "" {
|
||||||
|
e.Title = title
|
||||||
|
}
|
||||||
|
out = e
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload streams r into parentID as title. UploadToFolder is path based, so
|
||||||
|
// the reader is spooled to a temporary file first (ponytail: OnlyOffice
|
||||||
|
// multipart upload buffers the whole body anyway).
|
||||||
|
func (s *restStore) Upload(ctx context.Context, parentID, title string, r io.Reader) (Entry, error) {
|
||||||
|
dir, err := os.MkdirTemp("", "oo-rest-upload-")
|
||||||
|
if err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(dir)
|
||||||
|
|
||||||
|
local := filepath.Join(dir, SafeLocalFileName(title))
|
||||||
|
f, err := os.Create(local)
|
||||||
|
if err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
if _, err := io.Copy(f, r); err != nil {
|
||||||
|
f.Close()
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
if err := f.Close(); err != nil {
|
||||||
|
return Entry{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var out Entry
|
||||||
|
err = retryStoreOp(ctx, func() error {
|
||||||
|
fe, err := s.c.UploadToFolder(ctx, parentID, local)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out = FileEntryToEntry(fe, ProviderREST)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Download streams the file bytes into w.
|
||||||
|
func (s *restStore) Download(ctx context.Context, id string, w io.Writer) (int64, error) {
|
||||||
|
var n int64
|
||||||
|
err := retryStoreOp(ctx, func() error {
|
||||||
|
var e error
|
||||||
|
n, e = s.c.DownloadFile(ctx, id, w)
|
||||||
|
return e
|
||||||
|
})
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Move moves file ids into parentID. The REST MoveFiles endpoint handles files
|
||||||
|
// only; folder moves are not exposed by this adapter.
|
||||||
|
func (s *restStore) Move(ctx context.Context, ids []string, parentID string) error {
|
||||||
|
dest, err := strconv.Atoi(strings.TrimSpace(parentID))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("onlyoffice: rest store: move: non-numeric destination folder id %q", parentID)
|
||||||
|
}
|
||||||
|
fileIDs, err := numericIDs(ids)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
_, err := s.c.MoveFiles(ctx, dest, fileIDs)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy copies file ids into parentID. files.go has no copy method, so the
|
||||||
|
// shared REST fileops copy endpoint (CopyDavItems) is used.
|
||||||
|
func (s *restStore) Copy(ctx context.Context, ids []string, parentID string) error {
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
return s.c.CopyDavItems(ctx, nil, ids, parentID)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rename sets a new title (including extension) for a file.
|
||||||
|
func (s *restStore) Rename(ctx context.Context, id, title string) error {
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
_, err := s.c.RenameFile(ctx, id, title)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete permanently deletes file ids.
|
||||||
|
func (s *restStore) Delete(ctx context.Context, ids []string) error {
|
||||||
|
fileIDs, err := numericIDs(ids)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(fileIDs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return retryStoreOp(ctx, func() error {
|
||||||
|
return s.c.DeleteFiles(ctx, fileIDs)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// entriesFromFolderMap converts a ListFolder response map into canonical
|
||||||
|
// entries, reusing the DavFile/DavFolder decoders for robust size handling.
|
||||||
|
func entriesFromFolderMap(m map[string]any, provider string) ([]Entry, error) {
|
||||||
|
if m == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(m)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var listing DavListing
|
||||||
|
if err := json.Unmarshal(b, &listing); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := make([]Entry, 0, len(listing.Folders)+len(listing.Files))
|
||||||
|
for _, f := range listing.Folders {
|
||||||
|
out = append(out, DavFolderToEntry(f, provider))
|
||||||
|
}
|
||||||
|
for _, f := range listing.Files {
|
||||||
|
out = append(out, DavFileToEntry(f, provider))
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// folderEntryFromMap converts a CreateFolder response map into a folder Entry.
|
||||||
|
func folderEntryFromMap(m map[string]any, parentID, provider string) (Entry, error) {
|
||||||
|
e := Entry{Kind: Folder, Provider: provider, ParentID: parentID}
|
||||||
|
if m == nil {
|
||||||
|
return e, nil
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(m)
|
||||||
|
if err != nil {
|
||||||
|
return e, err
|
||||||
|
}
|
||||||
|
var f DavFolder
|
||||||
|
if err := json.Unmarshal(b, &f); err != nil {
|
||||||
|
return e, err
|
||||||
|
}
|
||||||
|
e = DavFolderToEntry(f, provider)
|
||||||
|
return e, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// numericIDs parses Documents numeric ids from strings.
|
||||||
|
func numericIDs(ids []string) ([]int, error) {
|
||||||
|
out := make([]int, 0, len(ids))
|
||||||
|
for _, id := range ids {
|
||||||
|
n, err := strconv.Atoi(strings.TrimSpace(id))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: rest store: non-numeric id %q", id)
|
||||||
|
}
|
||||||
|
out = append(out, n)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,209 @@
|
|||||||
|
//go:build integration
|
||||||
|
|
||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"strconv"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestIntegrationFileStores runs the same operation set (create folder, upload,
|
||||||
|
// list, stat, download, move, copy, rename, delete) through the REST and DAV
|
||||||
|
// FileStore adapters against a throwaway project Documents folder. Destructive
|
||||||
|
// — only run against instances you own.
|
||||||
|
//
|
||||||
|
// The Documents fileops API is asynchronous: a move/copy/delete is accepted
|
||||||
|
// immediately and becomes visible a moment later, so effects are polled.
|
||||||
|
func TestIntegrationFileStores(t *testing.T) {
|
||||||
|
c := liveClient(t)
|
||||||
|
t.Cleanup(func() { cleanupTestProjects(t, c) })
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
suffix := time.Now().UTC().Format("20060102-150405")
|
||||||
|
project, err := c.CreateProject(NewProjectRequest{
|
||||||
|
Title: testProjectPrefix + "store-" + suffix,
|
||||||
|
Description: "go-onlyoffice file store integration",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateProject: %v", err)
|
||||||
|
}
|
||||||
|
if project.ID == nil {
|
||||||
|
t.Fatal("created project without id")
|
||||||
|
}
|
||||||
|
root, err := c.projectFolderID(ctx, strconv.Itoa(*project.ID))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("projectFolderID: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, backend := range []string{ProviderREST, ProviderDAV} {
|
||||||
|
t.Run(backend, func(t *testing.T) {
|
||||||
|
testFileStoreOps(t, ctx, c, c.FileStore(backend), root, suffix)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testFileStoreOps(t *testing.T, ctx context.Context, c *Client, store FileStore, root, suffix string) {
|
||||||
|
t.Helper()
|
||||||
|
content := []byte("file store " + store.Name() + " " + suffix + "\n")
|
||||||
|
|
||||||
|
src, err := store.CreateFolder(ctx, root, "fs-src-"+suffix)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateFolder src: %v", err)
|
||||||
|
}
|
||||||
|
if src.Kind != Folder || src.ID == "" {
|
||||||
|
t.Fatalf("created src folder: %+v", src)
|
||||||
|
}
|
||||||
|
dst, err := store.CreateFolder(ctx, root, "fs-dst-"+suffix)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateFolder dst: %v", err)
|
||||||
|
}
|
||||||
|
if dst.Kind != Folder || dst.ID == "" {
|
||||||
|
t.Fatalf("created dst folder: %+v", dst)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
if err := c.DeleteDavItems(ctx, []string{src.ID, dst.ID}, nil); err != nil {
|
||||||
|
t.Logf("cleanup folders: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
up, err := store.Upload(ctx, src.ID, "doc-"+suffix+".txt", bytes.NewReader(content))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Upload: %v", err)
|
||||||
|
}
|
||||||
|
if up.Kind != File || up.ID == "" {
|
||||||
|
t.Fatalf("uploaded entry: %+v", up)
|
||||||
|
}
|
||||||
|
if !waitEntry(ctx, store, src.ID, up.ID, 15*time.Second) {
|
||||||
|
t.Fatalf("uploaded %s not listed in src", up.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
st, err := store.Stat(ctx, up.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Stat: %v", err)
|
||||||
|
}
|
||||||
|
if st.ID != up.ID || st.Kind != File {
|
||||||
|
t.Fatalf("stat = %+v", st)
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
n, err := store.Download(ctx, up.ID, &buf)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Download: %v", err)
|
||||||
|
}
|
||||||
|
if n != int64(len(content)) || !bytes.Equal(buf.Bytes(), content) {
|
||||||
|
t.Fatalf("download mismatch: got %d bytes %q want %d", n, buf.String(), len(content))
|
||||||
|
}
|
||||||
|
|
||||||
|
moveEventually(t, ctx, store, up.ID, dst.ID)
|
||||||
|
if !waitEntry(ctx, store, dst.ID, up.ID, 20*time.Second) {
|
||||||
|
t.Fatalf("moved file %s not in dst", up.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := store.Copy(ctx, []string{up.ID}, src.ID); err != nil {
|
||||||
|
t.Fatalf("Copy: %v", err)
|
||||||
|
}
|
||||||
|
copied := waitOtherFile(ctx, store, src.ID, up.ID, 20*time.Second)
|
||||||
|
if copied == nil {
|
||||||
|
t.Fatalf("no copy found in src after Copy")
|
||||||
|
}
|
||||||
|
|
||||||
|
newTitle := "renamed-" + suffix + ".txt"
|
||||||
|
renameEventually(t, ctx, store, up.ID, newTitle)
|
||||||
|
|
||||||
|
if err := store.Delete(ctx, []string{up.ID, copied.ID}); err != nil {
|
||||||
|
t.Fatalf("Delete: %v", err)
|
||||||
|
}
|
||||||
|
if !waitNoEntry(ctx, store, dst.ID, up.ID, 20*time.Second) {
|
||||||
|
t.Fatalf("file %s still present in dst after delete", up.ID)
|
||||||
|
}
|
||||||
|
if !waitNoEntry(ctx, store, src.ID, copied.ID, 20*time.Second) {
|
||||||
|
t.Fatalf("copy %s still present in src after delete", copied.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// moveEventually issues Move and retries while the operation is not visible yet
|
||||||
|
// (the fileops API accepts asynchronously and occasionally rejects a move that
|
||||||
|
// raced the just-finished upload).
|
||||||
|
func moveEventually(t *testing.T, ctx context.Context, store FileStore, id, dstID string) {
|
||||||
|
t.Helper()
|
||||||
|
var lastErr error
|
||||||
|
for attempt := 0; attempt < 5; attempt++ {
|
||||||
|
if lastErr = store.Move(ctx, []string{id}, dstID); lastErr == nil {
|
||||||
|
if waitEntry(ctx, store, dstID, id, 6*time.Second) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(time.Second)
|
||||||
|
}
|
||||||
|
t.Fatalf("Move %s -> %s: %v", id, dstID, lastErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
func renameEventually(t *testing.T, ctx context.Context, store FileStore, id, title string) {
|
||||||
|
t.Helper()
|
||||||
|
var lastErr error
|
||||||
|
for attempt := 0; attempt < 5; attempt++ {
|
||||||
|
if lastErr = store.Rename(ctx, id, title); lastErr == nil {
|
||||||
|
if e, err := store.Stat(ctx, id); err == nil && e.Title == title {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(time.Second)
|
||||||
|
}
|
||||||
|
t.Fatalf("Rename %s -> %q: %v", id, title, lastErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitEntry(ctx context.Context, store FileStore, parentID, id string, d time.Duration) bool {
|
||||||
|
deadline := time.Now().Add(d)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if list, err := store.List(ctx, parentID); err == nil && entryByID(list, id) != nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitNoEntry(ctx context.Context, store FileStore, parentID, id string, d time.Duration) bool {
|
||||||
|
deadline := time.Now().Add(d)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if list, err := store.List(ctx, parentID); err == nil && entryByID(list, id) == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitOtherFile(ctx context.Context, store FileStore, parentID, id string, d time.Duration) *Entry {
|
||||||
|
deadline := time.Now().Add(d)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if list, err := store.List(ctx, parentID); err == nil {
|
||||||
|
if e := firstFileOtherThan(list, id); e != nil {
|
||||||
|
return e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func entryByID(entries []Entry, id string) *Entry {
|
||||||
|
for i := range entries {
|
||||||
|
if entries[i].ID == id {
|
||||||
|
return &entries[i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstFileOtherThan(entries []Entry, id string) *Entry {
|
||||||
|
for i := range entries {
|
||||||
|
if entries[i].Kind == File && entries[i].ID != id {
|
||||||
|
return &entries[i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,337 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// Text extraction pipeline for the own full-text index (epic #34, F6 #42).
|
||||||
|
//
|
||||||
|
// TextIndexer downloads stored documents, extracts text through docpipe
|
||||||
|
// (pdftotext; OCR for scans) and writes the result to a TextIndex. For PDFs it
|
||||||
|
// also indexes the text of embedded attachments (pdfdetach), so a scan filed
|
||||||
|
// as an attachment is searchable too. It is the write side of ESTextIndex and
|
||||||
|
// never touches the OnlyOffice server's own ES index.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"github.com/eslider/go-onlyoffice/internal/docpipe"
|
||||||
|
)
|
||||||
|
|
||||||
|
// defaultTextIndexExts are the formats extracted by default. The OnlyOffice
|
||||||
|
// index already covers docx/xlsx/pptx; F6 adds PDF.
|
||||||
|
var defaultTextIndexExts = []string{"pdf"}
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultTextIndexWorkers = 3
|
||||||
|
defaultTextIndexLang = "deu+eng"
|
||||||
|
maxTextIndexErrors = 20
|
||||||
|
)
|
||||||
|
|
||||||
|
// TextExtractor turns a local file into indexable plain text. The default uses
|
||||||
|
// docpipe (pdftotext + OCR); tests inject a fake to stay offline.
|
||||||
|
type TextExtractor interface {
|
||||||
|
Extract(path, workDir, lang string, minChars int) (string, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// docpipeExtractor is the production TextExtractor.
|
||||||
|
type docpipeExtractor struct{ tools docpipe.Tools }
|
||||||
|
|
||||||
|
// Extract renders the file as Markdown, OCRing PDFs/images with a weak text
|
||||||
|
// layer first and appending the text of embedded PDF attachments
|
||||||
|
// (docpipe.ToMarkdownWithAttachments).
|
||||||
|
func (d docpipeExtractor) Extract(path, workDir, lang string, minChars int) (string, error) {
|
||||||
|
text, err := d.tools.ToMarkdownWithAttachments(path, workDir, lang, minChars)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(text), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// IndexOptions controls a TextIndexer run.
|
||||||
|
type IndexOptions struct {
|
||||||
|
Recursive bool // IndexFolder: descend into subfolders
|
||||||
|
Extensions []string // empty = defaultTextIndexExts (pdf)
|
||||||
|
Limit int // max files to index, 0 = all
|
||||||
|
Lang string // OCR language(s), default deu+eng
|
||||||
|
MinChars int // OCR threshold, default docpipe.DefaultMinTextChars
|
||||||
|
Workers int // parallel downloads/extractions, default 3
|
||||||
|
}
|
||||||
|
|
||||||
|
// IndexResult summarises a run.
|
||||||
|
type IndexResult struct {
|
||||||
|
Scanned int
|
||||||
|
Indexed int
|
||||||
|
Skipped int
|
||||||
|
Failed int
|
||||||
|
Errors []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// TextIndexer wires a FileStore, a TextIndex and an extractor together.
|
||||||
|
type TextIndexer struct {
|
||||||
|
Store FileStore
|
||||||
|
Index TextIndex
|
||||||
|
Extractor TextExtractor // nil = local docpipe tools
|
||||||
|
WorkDir string // temp dir for downloads/extraction
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewTextIndexer returns a TextIndexer over the given store and index.
|
||||||
|
func NewTextIndexer(store FileStore, index TextIndex) *TextIndexer {
|
||||||
|
return &TextIndexer{Store: store, Index: index}
|
||||||
|
}
|
||||||
|
|
||||||
|
// textIndexEnsurer is implemented by indexes that can be created up front.
|
||||||
|
type textIndexEnsurer interface {
|
||||||
|
Ensure(ctx context.Context) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ensure creates the backing index when the TextIndex supports it.
|
||||||
|
func (ix *TextIndexer) Ensure(ctx context.Context) error {
|
||||||
|
if e, ok := ix.Index.(textIndexEnsurer); ok {
|
||||||
|
return e.Ensure(ctx)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// IndexFiles stats the given file ids and indexes them.
|
||||||
|
func (ix *TextIndexer) IndexFiles(ctx context.Context, ids []string, opts IndexOptions) (IndexResult, error) {
|
||||||
|
entries := make([]Entry, 0, len(ids))
|
||||||
|
for _, id := range ids {
|
||||||
|
e, err := ix.Store.Stat(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return IndexResult{}, fmt.Errorf("onlyoffice: stat %s: %w", id, err)
|
||||||
|
}
|
||||||
|
entries = append(entries, e)
|
||||||
|
}
|
||||||
|
return ix.IndexEntries(ctx, entries, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// IndexFolder lists a folder and indexes every matching file.
|
||||||
|
func (ix *TextIndexer) IndexFolder(ctx context.Context, folderID string, opts IndexOptions) (IndexResult, error) {
|
||||||
|
entries, err := ix.collect(ctx, folderID, opts.Recursive)
|
||||||
|
if err != nil {
|
||||||
|
return IndexResult{}, err
|
||||||
|
}
|
||||||
|
return ix.IndexEntries(ctx, entries, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// PlanFolder lists the files IndexFolder would process, without downloading or
|
||||||
|
// extracting anything.
|
||||||
|
func (ix *TextIndexer) PlanFolder(ctx context.Context, folderID string, opts IndexOptions) ([]Entry, error) {
|
||||||
|
entries, err := ix.collect(ctx, folderID, opts.Recursive)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return selectEntries(entries, opts), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PlanFiles stats the ids and returns those that would be indexed.
|
||||||
|
func (ix *TextIndexer) PlanFiles(ctx context.Context, ids []string, opts IndexOptions) ([]Entry, error) {
|
||||||
|
entries := make([]Entry, 0, len(ids))
|
||||||
|
for _, id := range ids {
|
||||||
|
e, err := ix.Store.Stat(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: stat %s: %w", id, err)
|
||||||
|
}
|
||||||
|
entries = append(entries, e)
|
||||||
|
}
|
||||||
|
return selectEntries(entries, opts), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// IndexEntries extracts and indexes the given files (folders are ignored).
|
||||||
|
func (ix *TextIndexer) IndexEntries(ctx context.Context, entries []Entry, opts IndexOptions) (IndexResult, error) {
|
||||||
|
opts = opts.withDefaults()
|
||||||
|
var res IndexResult
|
||||||
|
work := selectEntries(entries, opts)
|
||||||
|
res.Scanned = len(entries)
|
||||||
|
res.Skipped = len(entries) - len(work)
|
||||||
|
if len(work) == 0 {
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
workers := opts.Workers
|
||||||
|
if workers > len(work) {
|
||||||
|
workers = len(work)
|
||||||
|
}
|
||||||
|
if workers < 1 {
|
||||||
|
workers = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
type outcome struct {
|
||||||
|
doc TextDoc
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
jobs := make(chan Entry)
|
||||||
|
results := make(chan outcome, workers)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < workers; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for e := range jobs {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
results <- outcome{err: err}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
doc, err := ix.indexOne(ctx, e, opts)
|
||||||
|
results <- outcome{doc: doc, err: err}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
defer close(jobs)
|
||||||
|
for _, e := range work {
|
||||||
|
select {
|
||||||
|
case jobs <- e:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
go func() {
|
||||||
|
wg.Wait()
|
||||||
|
close(results)
|
||||||
|
}()
|
||||||
|
|
||||||
|
var docs []TextDoc
|
||||||
|
for r := range results {
|
||||||
|
if r.err != nil {
|
||||||
|
res.Failed++
|
||||||
|
if len(res.Errors) < maxTextIndexErrors {
|
||||||
|
res.Errors = append(res.Errors, r.err.Error())
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
docs = append(docs, r.doc)
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
if len(docs) > 0 {
|
||||||
|
if err := ix.Index.Put(ctx, docs); err != nil {
|
||||||
|
return res, fmt.Errorf("onlyoffice: index %d docs: %w", len(docs), err)
|
||||||
|
}
|
||||||
|
res.Indexed = len(docs)
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// indexOne downloads and extracts a single file.
|
||||||
|
func (ix *TextIndexer) indexOne(ctx context.Context, e Entry, opts IndexOptions) (TextDoc, error) {
|
||||||
|
ext := fileExt(e.Title)
|
||||||
|
dir := ix.WorkDir
|
||||||
|
if dir == "" {
|
||||||
|
dir = os.TempDir()
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||||
|
return TextDoc{}, err
|
||||||
|
}
|
||||||
|
tmp, err := os.CreateTemp(dir, "ooidx-*."+ext)
|
||||||
|
if err != nil {
|
||||||
|
return TextDoc{}, err
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
defer os.Remove(tmpPath)
|
||||||
|
|
||||||
|
if _, err := ix.Store.Download(ctx, e.ID, tmp); err != nil {
|
||||||
|
tmp.Close()
|
||||||
|
return TextDoc{}, fmt.Errorf("download %s (%s): %w", e.ID, e.Title, err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return TextDoc{}, err
|
||||||
|
}
|
||||||
|
text, err := ix.extractor().Extract(tmpPath, dir, opts.Lang, opts.MinChars)
|
||||||
|
if err != nil {
|
||||||
|
return TextDoc{}, fmt.Errorf("extract %s: %w", e.Title, err)
|
||||||
|
}
|
||||||
|
return TextDoc{ID: e.ID, Title: e.Title, FolderID: e.ParentID, Ext: ext, Content: text}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// collect lists files under folderID, breadth-first when recursive.
|
||||||
|
func (ix *TextIndexer) collect(ctx context.Context, folderID string, recursive bool) ([]Entry, error) {
|
||||||
|
var files []Entry
|
||||||
|
queue := []string{folderID}
|
||||||
|
for len(queue) > 0 {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
id := queue[0]
|
||||||
|
queue = queue[1:]
|
||||||
|
entries, err := ix.Store.List(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("onlyoffice: list folder %s: %w", id, err)
|
||||||
|
}
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.Kind == Folder {
|
||||||
|
if recursive {
|
||||||
|
queue = append(queue, e.ID)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if e.ParentID == "" {
|
||||||
|
e.ParentID = id
|
||||||
|
}
|
||||||
|
files = append(files, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// selectEntries filters files by extension and applies the limit.
|
||||||
|
func selectEntries(entries []Entry, opts IndexOptions) []Entry {
|
||||||
|
allowed := extensionSet(opts.Extensions)
|
||||||
|
work := make([]Entry, 0, len(entries))
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.Kind != File {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if opts.Limit > 0 && len(work) >= opts.Limit {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if !allowed[fileExt(e.Title)] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
work = append(work, e)
|
||||||
|
}
|
||||||
|
return work
|
||||||
|
}
|
||||||
|
|
||||||
|
// extensionSet normalises the extension allow-list (default: pdf).
|
||||||
|
func extensionSet(exts []string) map[string]bool {
|
||||||
|
if len(exts) == 0 {
|
||||||
|
exts = defaultTextIndexExts
|
||||||
|
}
|
||||||
|
set := make(map[string]bool, len(exts))
|
||||||
|
for _, e := range normalizeExtensions(exts) {
|
||||||
|
set[e] = true
|
||||||
|
}
|
||||||
|
return set
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileExt returns the lower-case extension without the dot.
|
||||||
|
func fileExt(title string) string {
|
||||||
|
return strings.ToLower(strings.TrimPrefix(filepath.Ext(strings.TrimSpace(title)), "."))
|
||||||
|
}
|
||||||
|
|
||||||
|
// withDefaults fills zero-valued options.
|
||||||
|
func (o IndexOptions) withDefaults() IndexOptions {
|
||||||
|
if o.Workers <= 0 {
|
||||||
|
o.Workers = defaultTextIndexWorkers
|
||||||
|
}
|
||||||
|
if o.MinChars <= 0 {
|
||||||
|
o.MinChars = docpipe.DefaultMinTextChars
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(o.Lang) == "" {
|
||||||
|
o.Lang = defaultTextIndexLang
|
||||||
|
}
|
||||||
|
return o
|
||||||
|
}
|
||||||
|
|
||||||
|
// extractor returns the configured extractor or the local docpipe default.
|
||||||
|
func (ix *TextIndexer) extractor() TextExtractor {
|
||||||
|
if ix.Extractor != nil {
|
||||||
|
return ix.Extractor
|
||||||
|
}
|
||||||
|
return docpipeExtractor{tools: docpipe.LookPath()}
|
||||||
|
}
|
||||||
@@ -247,6 +247,8 @@ func (c *Client) UploadProjectFileReplacing(ctx context.Context, projectID, loca
|
|||||||
}
|
}
|
||||||
|
|
||||||
// GetFile returns file metadata including viewUrl for download.
|
// GetFile returns file metadata including viewUrl for download.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Stat via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) GetFile(ctx context.Context, fileID string) (*FileEntry, error) {
|
func (c *Client) GetFile(ctx context.Context, fileID string) (*FileEntry, error) {
|
||||||
if fileID == "" {
|
if fileID == "" {
|
||||||
return nil, fmt.Errorf("file id is required")
|
return nil, fmt.Errorf("file id is required")
|
||||||
@@ -260,6 +262,8 @@ func (c *Client) GetFile(ctx context.Context, fileID string) (*FileEntry, error)
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RenameFile sets a new title (including extension) for the file.
|
// RenameFile sets a new title (including extension) for the file.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Rename via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) RenameFile(ctx context.Context, fileID, newTitle string) (*FileEntry, error) {
|
func (c *Client) RenameFile(ctx context.Context, fileID, newTitle string) (*FileEntry, error) {
|
||||||
if fileID == "" || newTitle == "" {
|
if fileID == "" || newTitle == "" {
|
||||||
return nil, fmt.Errorf("file id and new title are required")
|
return nil, fmt.Errorf("file id and new title are required")
|
||||||
@@ -275,6 +279,8 @@ func (c *Client) RenameFile(ctx context.Context, fileID, newTitle string) (*File
|
|||||||
// DeleteFiles permanently deletes files by numeric id (Documents module).
|
// DeleteFiles permanently deletes files by numeric id (Documents module).
|
||||||
// Uses per-file DELETE (DeleteDavItems); fileops/delete returns 200 on some
|
// Uses per-file DELETE (DeleteDavItems); fileops/delete returns 200 on some
|
||||||
// portals (e.g. produktor.io) without removing the file.
|
// portals (e.g. produktor.io) without removing the file.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Delete via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) DeleteFiles(ctx context.Context, fileIDs []int) error {
|
func (c *Client) DeleteFiles(ctx context.Context, fileIDs []int) error {
|
||||||
if len(fileIDs) == 0 {
|
if len(fileIDs) == 0 {
|
||||||
return fmt.Errorf("no file ids to delete")
|
return fmt.Errorf("no file ids to delete")
|
||||||
@@ -288,6 +294,8 @@ func (c *Client) DeleteFiles(ctx context.Context, fileIDs []int) error {
|
|||||||
|
|
||||||
// ListFolder returns the Documents module listing for a folder id
|
// ListFolder returns the Documents module listing for a folder id
|
||||||
// (GET /api/2.0/files/{folderId}).
|
// (GET /api/2.0/files/{folderId}).
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.List via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) ListFolder(ctx context.Context, folderID string) (map[string]any, error) {
|
func (c *Client) ListFolder(ctx context.Context, folderID string) (map[string]any, error) {
|
||||||
if folderID == "" {
|
if folderID == "" {
|
||||||
return nil, fmt.Errorf("folder id is required")
|
return nil, fmt.Errorf("folder id is required")
|
||||||
@@ -313,6 +321,8 @@ func (c *Client) CreateFolder(ctx context.Context, parentFolderID, title string)
|
|||||||
}
|
}
|
||||||
|
|
||||||
// MoveFiles moves file ids into destFolderID (Documents fileops/move).
|
// MoveFiles moves file ids into destFolderID (Documents fileops/move).
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Move via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) MoveFiles(ctx context.Context, destFolderID int, fileIDs []int) (map[string]any, error) {
|
func (c *Client) MoveFiles(ctx context.Context, destFolderID int, fileIDs []int) (map[string]any, error) {
|
||||||
if destFolderID == 0 || len(fileIDs) == 0 {
|
if destFolderID == 0 || len(fileIDs) == 0 {
|
||||||
return nil, fmt.Errorf("dest folder and file ids are required")
|
return nil, fmt.Errorf("dest folder and file ids are required")
|
||||||
@@ -321,15 +331,30 @@ func (c *Client) MoveFiles(ctx context.Context, destFolderID int, fileIDs []int)
|
|||||||
"folderIds": []int{},
|
"folderIds": []int{},
|
||||||
"fileIds": fileIDs,
|
"fileIds": fileIDs,
|
||||||
"destFolderId": destFolderID,
|
"destFolderId": destFolderID,
|
||||||
|
"resolveType": "Skip",
|
||||||
|
"holdResult": true,
|
||||||
}
|
}
|
||||||
out, err := c.putJSONObject(ctx, "/api/2.0/files/fileops/move.json", body)
|
// fileops/move answers an operations envelope (like MoveDavItems), not a
|
||||||
|
// single object, so parse the raw body before unwrapping and surface any
|
||||||
|
// per-operation error. Unwrapping first (putJSONObject) made fileopsError
|
||||||
|
// look for a "response" key that was already stripped.
|
||||||
|
raw, err := c.putJSON(ctx, "/api/2.0/files/fileops/move", body)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
out, err = c.putJSONObject(ctx, "/api/2.0/files/fileops/move", body)
|
raw, err = c.putJSON(ctx, "/api/2.0/files/fileops/move.json", body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return out, err
|
if ferr := fileopsError(raw); ferr != nil {
|
||||||
|
return nil, ferr
|
||||||
|
}
|
||||||
|
out, _ := unmarshalResponseObject(raw)
|
||||||
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// UploadToFolder uploads a local file into an arbitrary Documents folder id.
|
// UploadToFolder uploads a local file into an arbitrary Documents folder id.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Upload via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) UploadToFolder(ctx context.Context, folderID, localPath string) (*FileEntry, error) {
|
func (c *Client) UploadToFolder(ctx context.Context, folderID, localPath string) (*FileEntry, error) {
|
||||||
if folderID == "" || localPath == "" {
|
if folderID == "" || localPath == "" {
|
||||||
return nil, fmt.Errorf("folder id and local path are required")
|
return nil, fmt.Errorf("folder id and local path are required")
|
||||||
@@ -346,6 +371,35 @@ func (c *Client) UploadToFolder(ctx context.Context, folderID, localPath string)
|
|||||||
return decodeResponseFileEntry(raw)
|
return decodeResponseFileEntry(raw)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateFile uploads a new version of an existing file (same id, name and
|
||||||
|
// folder). It does not delete and does not create a second file.
|
||||||
|
//
|
||||||
|
// The Documents API method is PUT /api/2.0/files/{id}/update; POST is kept as
|
||||||
|
// a fallback for older servers. The path is tried with and without .json.
|
||||||
|
func (c *Client) UpdateFile(ctx context.Context, fileID, localPath string) (*FileEntry, error) {
|
||||||
|
if fileID == "" || localPath == "" {
|
||||||
|
return nil, fmt.Errorf("file id and local path are required")
|
||||||
|
}
|
||||||
|
base := fmt.Sprintf("/api/2.0/files/%s/update", url.PathEscape(fileID))
|
||||||
|
attempts := []struct {
|
||||||
|
method, path string
|
||||||
|
}{
|
||||||
|
{http.MethodPut, base},
|
||||||
|
{http.MethodPut, base + ".json"},
|
||||||
|
{http.MethodPost, base},
|
||||||
|
{http.MethodPost, base + ".json"},
|
||||||
|
}
|
||||||
|
var lastErr error
|
||||||
|
for _, a := range attempts {
|
||||||
|
raw, err := c.uploadMultipartMethod(ctx, a.method, a.path, "file", localPath)
|
||||||
|
if err == nil {
|
||||||
|
return decodeResponseFileEntry(raw)
|
||||||
|
}
|
||||||
|
lastErr = err
|
||||||
|
}
|
||||||
|
return nil, lastErr
|
||||||
|
}
|
||||||
|
|
||||||
// FileFolderID returns the parent folder id string for a file entry, if known.
|
// FileFolderID returns the parent folder id string for a file entry, if known.
|
||||||
func FileFolderID(f *FileEntry) string {
|
func FileFolderID(f *FileEntry) string {
|
||||||
if f == nil || f.FolderID == nil {
|
if f == nil || f.FolderID == nil {
|
||||||
@@ -355,36 +409,17 @@ func FileFolderID(f *FileEntry) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// DownloadFile streams file bytes from the file's viewUrl using the same auth
|
// DownloadFile streams file bytes from the file's viewUrl using the same auth
|
||||||
// as API calls. Writes into dst.
|
// as API calls. Writes into dst. When the portal serves the file from its stale
|
||||||
|
// AWS S3 consumer, the bytes are fetched from the local MinIO store instead
|
||||||
|
// (see storage_fallback.go).
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Download via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) DownloadFile(ctx context.Context, fileID string, dst io.Writer) (int64, error) {
|
func (c *Client) DownloadFile(ctx context.Context, fileID string, dst io.Writer) (int64, error) {
|
||||||
f, err := c.GetFile(ctx, fileID)
|
f, err := c.GetFile(ctx, fileID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
if f.ViewURL == nil || *f.ViewURL == "" {
|
return c.downloadFileEntry(ctx, f, dst)
|
||||||
return 0, fmt.Errorf("file %s has no viewUrl", fileID)
|
|
||||||
}
|
|
||||||
downloadURL := c.resolveAPIURL(*f.ViewURL)
|
|
||||||
auth, err := c.authHeader()
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
req.Header.Set("Authorization", auth)
|
|
||||||
resp, err := c.client.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
defer resp.Body.Close()
|
|
||||||
if resp.StatusCode >= 400 {
|
|
||||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
|
||||||
return 0, fmt.Errorf("GET viewUrl: %d %s", resp.StatusCode, truncate(string(b), 400))
|
|
||||||
}
|
|
||||||
n, err := io.Copy(dst, resp.Body)
|
|
||||||
return n, err
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Client) resolveAPIURL(ref string) string {
|
func (c *Client) resolveAPIURL(ref string) string {
|
||||||
|
|||||||
+89
-37
@@ -52,6 +52,8 @@ type DavListing struct {
|
|||||||
|
|
||||||
// ListDavFolder returns the contents of a folder by id, which may be a
|
// ListDavFolder returns the contents of a folder by id, which may be a
|
||||||
// symbolic root such as "@my". For "@root" use ListDavSections.
|
// symbolic root such as "@my". For "@root" use ListDavSections.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.List via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) ListDavFolder(ctx context.Context, id string) (*DavListing, error) {
|
func (c *Client) ListDavFolder(ctx context.Context, id string) (*DavListing, error) {
|
||||||
raw, err := c.getJSON(ctx, "/api/2.0/files/"+url.PathEscape(id))
|
raw, err := c.getJSON(ctx, "/api/2.0/files/"+url.PathEscape(id))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -111,6 +113,8 @@ func (c *Client) ListDavSections(ctx context.Context) ([]DavFolder, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// CreateDavFolder creates a folder titled title inside parentID.
|
// CreateDavFolder creates a folder titled title inside parentID.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.CreateFolder via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) CreateDavFolder(ctx context.Context, parentID, title string) (*DavFolder, error) {
|
func (c *Client) CreateDavFolder(ctx context.Context, parentID, title string) (*DavFolder, error) {
|
||||||
raw, err := c.postJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(parentID),
|
raw, err := c.postJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(parentID),
|
||||||
map[string]string{"title": title})
|
map[string]string{"title": title})
|
||||||
@@ -130,6 +134,8 @@ func (c *Client) CreateDavFolder(ctx context.Context, parentID, title string) (*
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RenameDavFolder renames a folder.
|
// RenameDavFolder renames a folder.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Rename via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) RenameDavFolder(ctx context.Context, id, title string) error {
|
func (c *Client) RenameDavFolder(ctx context.Context, id, title string) error {
|
||||||
_, err := c.putJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(id),
|
_, err := c.putJSON(ctx, "/api/2.0/files/folder/"+url.PathEscape(id),
|
||||||
map[string]string{"title": title})
|
map[string]string{"title": title})
|
||||||
@@ -137,6 +143,8 @@ func (c *Client) RenameDavFolder(ctx context.Context, id, title string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RenameDavFile renames a file (title includes the extension).
|
// RenameDavFile renames a file (title includes the extension).
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Rename via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) RenameDavFile(ctx context.Context, id, title string) error {
|
func (c *Client) RenameDavFile(ctx context.Context, id, title string) error {
|
||||||
_, err := c.putJSON(ctx, "/api/2.0/files/file/"+url.PathEscape(id),
|
_, err := c.putJSON(ctx, "/api/2.0/files/file/"+url.PathEscape(id),
|
||||||
map[string]string{"title": title})
|
map[string]string{"title": title})
|
||||||
@@ -144,30 +152,94 @@ func (c *Client) RenameDavFile(ctx context.Context, id, title string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// MoveDavItems moves the given folders and/or files into destFolderID.
|
// MoveDavItems moves the given folders and/or files into destFolderID.
|
||||||
|
// The fileops API answers 200 with per-operation error strings even when
|
||||||
|
// nothing moves (e.g. missing permission), so the response is parsed and the
|
||||||
|
// first operation error is returned instead of a silent nil.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Move via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) MoveDavItems(ctx context.Context, folderIDs, fileIDs []string, destFolderID string) error {
|
func (c *Client) MoveDavItems(ctx context.Context, folderIDs, fileIDs []string, destFolderID string) error {
|
||||||
_, err := c.putJSON(ctx, "/api/2.0/files/fileops/move", map[string]any{
|
raw, err := c.putJSON(ctx, "/api/2.0/files/fileops/move", map[string]any{
|
||||||
"folderIds": nums(folderIDs),
|
"folderIds": nums(folderIDs),
|
||||||
"fileIds": nums(fileIDs),
|
"fileIds": nums(fileIDs),
|
||||||
"destFolderId": num(destFolderID),
|
"destFolderId": num(destFolderID),
|
||||||
"resolveType": "Skip",
|
"resolveType": "Skip",
|
||||||
"holdResult": true,
|
"holdResult": true,
|
||||||
})
|
})
|
||||||
return err
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return fileopsError(raw)
|
||||||
}
|
}
|
||||||
|
|
||||||
// CopyDavItems copies the given folders and/or files into destFolderID.
|
// CopyDavItems copies the given folders and/or files into destFolderID.
|
||||||
|
// Per-operation errors are surfaced like in MoveDavItems.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Copy via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) CopyDavItems(ctx context.Context, folderIDs, fileIDs []string, destFolderID string) error {
|
func (c *Client) CopyDavItems(ctx context.Context, folderIDs, fileIDs []string, destFolderID string) error {
|
||||||
_, err := c.putJSON(ctx, "/api/2.0/files/fileops/copy", map[string]any{
|
raw, err := c.putJSON(ctx, "/api/2.0/files/fileops/copy", map[string]any{
|
||||||
"folderIds": nums(folderIDs),
|
"folderIds": nums(folderIDs),
|
||||||
"fileIds": nums(fileIDs),
|
"fileIds": nums(fileIDs),
|
||||||
"destFolderId": num(destFolderID),
|
"destFolderId": num(destFolderID),
|
||||||
"conflictResolveType": "Skip",
|
"conflictResolveType": "Skip",
|
||||||
"deleteAfter": true,
|
"deleteAfter": true,
|
||||||
})
|
})
|
||||||
return err
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return fileopsError(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListFileOps returns the currently active file operations
|
||||||
|
// (GET /api/2.0/files/fileops) for status polling.
|
||||||
|
func (c *Client) ListFileOps(ctx context.Context) ([]map[string]any, error) {
|
||||||
|
raw, err := c.getJSON(ctx, "/api/2.0/files/fileops")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
resp, err := responseField(raw, "response")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(resp) == 0 || string(resp) == "null" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
var ops []map[string]any
|
||||||
|
if err := json.Unmarshal(resp, &ops); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return ops, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileopsError extracts per-operation "error" strings from a fileops/move or
|
||||||
|
// fileops/copy envelope. A 200 with error entries means nothing moved.
|
||||||
|
func fileopsError(raw json.RawMessage) error {
|
||||||
|
resp, err := responseField(raw, "response")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var ops []struct {
|
||||||
|
Error *string `json:"error"`
|
||||||
|
Finished *bool `json:"finished"`
|
||||||
|
Progress *int `json:"progress"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(resp, &ops); err != nil {
|
||||||
|
return nil // not an operations envelope — nothing to report
|
||||||
|
}
|
||||||
|
var errs []string
|
||||||
|
for _, op := range ops {
|
||||||
|
if op.Error != nil && *op.Error != "" {
|
||||||
|
errs = append(errs, *op.Error)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(errs) > 0 {
|
||||||
|
return fmt.Errorf("onlyoffice: fileops: %s", strings.Join(errs, "; "))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// DeleteDavItems deletes the given folders and/or files.
|
// DeleteDavItems deletes the given folders and/or files.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Delete via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) DeleteDavItems(ctx context.Context, folderIDs, fileIDs []string) error {
|
func (c *Client) DeleteDavItems(ctx context.Context, folderIDs, fileIDs []string) error {
|
||||||
body := map[string]any{"DeleteAfter": true, "Immediately": true}
|
body := map[string]any{"DeleteAfter": true, "Immediately": true}
|
||||||
for _, id := range folderIDs {
|
for _, id := range folderIDs {
|
||||||
@@ -184,6 +256,8 @@ func (c *Client) DeleteDavItems(ctx context.Context, folderIDs, fileIDs []string
|
|||||||
}
|
}
|
||||||
|
|
||||||
// UploadDavFile uploads src (fileName) into folderID, streaming from src.
|
// UploadDavFile uploads src (fileName) into folderID, streaming from src.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Upload via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) UploadDavFile(ctx context.Context, folderID, fileName string, src io.Reader) (*DavFile, error) {
|
func (c *Client) UploadDavFile(ctx context.Context, folderID, fileName string, src io.Reader) (*DavFile, error) {
|
||||||
raw, err := c.uploadReader(ctx, "/api/2.0/files/"+url.PathEscape(folderID)+"/upload", "file", fileName, src)
|
raw, err := c.uploadReader(ctx, "/api/2.0/files/"+url.PathEscape(folderID)+"/upload", "file", fileName, src)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -201,34 +275,12 @@ func (c *Client) UploadDavFile(ctx context.Context, folderID, fileName string, s
|
|||||||
return env.Response, nil
|
return env.Response, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// DownloadDavFile streams the file identified by id to w, returning bytes copied.
|
// DownloadDavFile streams the file identified by id to w, returning bytes
|
||||||
|
// copied. It shares the MinIO stale-S3 fallback with DownloadFile.
|
||||||
|
//
|
||||||
|
// Deprecated: use FileStore.Download via Client.Files()/Client.FileStore.
|
||||||
func (c *Client) DownloadDavFile(ctx context.Context, id string, w io.Writer) (int64, error) {
|
func (c *Client) DownloadDavFile(ctx context.Context, id string, w io.Writer) (int64, error) {
|
||||||
file, err := c.GetFile(ctx, id)
|
return c.DownloadFile(ctx, id, w)
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
if file.ViewURL == nil || *file.ViewURL == "" {
|
|
||||||
return 0, fmt.Errorf("onlyoffice: file %s has no viewUrl", id)
|
|
||||||
}
|
|
||||||
u := c.resolveAPIURL(*file.ViewURL)
|
|
||||||
auth, err := c.authHeader()
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
req.Header.Set("Authorization", auth)
|
|
||||||
resp, err := c.client.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
defer resp.Body.Close()
|
|
||||||
if resp.StatusCode >= 400 {
|
|
||||||
return 0, fmt.Errorf("onlyoffice: download: %d", resp.StatusCode)
|
|
||||||
}
|
|
||||||
return io.Copy(w, resp.Body)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- internal helpers -------------------------------------------------------
|
// --- internal helpers -------------------------------------------------------
|
||||||
@@ -376,12 +428,12 @@ func (f *DavFolder) UnmarshalJSON(b []byte) error {
|
|||||||
// UnmarshalJSON decodes a file row, capturing size and timestamps.
|
// UnmarshalJSON decodes a file row, capturing size and timestamps.
|
||||||
func (f *DavFile) UnmarshalJSON(b []byte) error {
|
func (f *DavFile) UnmarshalJSON(b []byte) error {
|
||||||
var raw struct {
|
var raw struct {
|
||||||
ID *json.Number `json:"id"`
|
ID *json.Number `json:"id"`
|
||||||
Title *string `json:"title"`
|
Title *string `json:"title"`
|
||||||
PureSize *int64 `json:"pureContentLength"`
|
PureSize *int64 `json:"pureContentLength"`
|
||||||
SizeStr *string `json:"contentLength"`
|
SizeStr *string `json:"contentLength"`
|
||||||
Updated *string `json:"updated"`
|
Updated *string `json:"updated"`
|
||||||
ViewURL *string `json:"viewUrl"`
|
ViewURL *string `json:"viewUrl"`
|
||||||
}
|
}
|
||||||
if err := json.Unmarshal(b, &raw); err != nil {
|
if err := json.Unmarshal(b, &raw); err != nil {
|
||||||
return err
|
return err
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ go 1.25.0
|
|||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2
|
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2
|
||||||
|
github.com/aws/aws-sdk-go-v2 v1.41.1
|
||||||
github.com/charmbracelet/bubbles v0.18.0
|
github.com/charmbracelet/bubbles v0.18.0
|
||||||
github.com/charmbracelet/bubbletea v0.25.0
|
github.com/charmbracelet/bubbletea v0.25.0
|
||||||
github.com/charmbracelet/glamour v0.8.0
|
github.com/charmbracelet/glamour v0.8.0
|
||||||
@@ -12,7 +13,9 @@ require (
|
|||||||
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3
|
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3
|
||||||
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002
|
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002
|
||||||
github.com/eslider/go-xls/v2 v2.1.0
|
github.com/eslider/go-xls/v2 v2.1.0
|
||||||
|
github.com/go-sql-driver/mysql v1.10.1
|
||||||
github.com/google/go-querystring v1.2.0
|
github.com/google/go-querystring v1.2.0
|
||||||
|
github.com/jackc/pgx/v5 v5.11.0
|
||||||
github.com/joho/godotenv v1.5.1
|
github.com/joho/godotenv v1.5.1
|
||||||
github.com/mattn/go-runewidth v0.0.15
|
github.com/mattn/go-runewidth v0.0.15
|
||||||
github.com/muesli/termenv v0.16.0
|
github.com/muesli/termenv v0.16.0
|
||||||
@@ -23,9 +26,11 @@ require (
|
|||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
filippo.io/edwards25519 v1.2.0 // indirect
|
||||||
github.com/JohannesKaufmann/dom v0.3.1 // indirect
|
github.com/JohannesKaufmann/dom v0.3.1 // indirect
|
||||||
github.com/alecthomas/chroma/v2 v2.14.0 // indirect
|
github.com/alecthomas/chroma/v2 v2.14.0 // indirect
|
||||||
github.com/atotto/clipboard v0.1.4 // indirect
|
github.com/atotto/clipboard v0.1.4 // indirect
|
||||||
|
github.com/aws/smithy-go v1.24.0 // indirect
|
||||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||||
github.com/aymerick/douceur v0.2.0 // indirect
|
github.com/aymerick/douceur v0.2.0 // indirect
|
||||||
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
|
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
|
||||||
@@ -34,6 +39,10 @@ require (
|
|||||||
github.com/google/uuid v1.6.0 // indirect
|
github.com/google/uuid v1.6.0 // indirect
|
||||||
github.com/gorilla/css v1.0.1 // indirect
|
github.com/gorilla/css v1.0.1 // indirect
|
||||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||||
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||||
|
github.com/kr/text v0.2.0 // indirect
|
||||||
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
|
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
|
||||||
github.com/mattn/go-isatty v0.0.24 // indirect
|
github.com/mattn/go-isatty v0.0.24 // indirect
|
||||||
github.com/mattn/go-localereader v0.0.1 // indirect
|
github.com/mattn/go-localereader v0.0.1 // indirect
|
||||||
@@ -46,6 +55,7 @@ require (
|
|||||||
github.com/richardlehane/mscfb v1.0.7 // indirect
|
github.com/richardlehane/mscfb v1.0.7 // indirect
|
||||||
github.com/richardlehane/msoleps v1.0.6 // indirect
|
github.com/richardlehane/msoleps v1.0.6 // indirect
|
||||||
github.com/rivo/uniseg v0.4.7 // indirect
|
github.com/rivo/uniseg v0.4.7 // indirect
|
||||||
|
github.com/rogpeppe/go-internal v1.16.0 // indirect
|
||||||
github.com/spf13/pflag v1.0.9 // indirect
|
github.com/spf13/pflag v1.0.9 // indirect
|
||||||
github.com/tiendc/go-deepcopy v1.7.2 // indirect
|
github.com/tiendc/go-deepcopy v1.7.2 // indirect
|
||||||
github.com/xuri/efp v0.0.1 // indirect
|
github.com/xuri/efp v0.0.1 // indirect
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||||
|
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||||
github.com/JohannesKaufmann/dom v0.3.1 h1:J16l9JAHWgkFPR3VIPbQ1gvS0cWab6laK1q7PFL3qh0=
|
github.com/JohannesKaufmann/dom v0.3.1 h1:J16l9JAHWgkFPR3VIPbQ1gvS0cWab6laK1q7PFL3qh0=
|
||||||
github.com/JohannesKaufmann/dom v0.3.1/go.mod h1:BZPkf8ZeYrBgABjwJn9iiKt8aiCtkxpHkevms+Yp2DE=
|
github.com/JohannesKaufmann/dom v0.3.1/go.mod h1:BZPkf8ZeYrBgABjwJn9iiKt8aiCtkxpHkevms+Yp2DE=
|
||||||
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2 h1:XFJZFWESIWlUEHHjzBuv8RvrtCWnSGlimEX17ysSDb8=
|
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2 h1:XFJZFWESIWlUEHHjzBuv8RvrtCWnSGlimEX17ysSDb8=
|
||||||
@@ -10,6 +12,10 @@ github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc
|
|||||||
github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
|
github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
|
||||||
github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
|
github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
|
||||||
github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
|
github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
|
||||||
|
github.com/aws/aws-sdk-go-v2 v1.41.1 h1:ABlyEARCDLN034NhxlRUSZr4l71mh+T5KAeGh6cerhU=
|
||||||
|
github.com/aws/aws-sdk-go-v2 v1.41.1/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
|
||||||
|
github.com/aws/smithy-go v1.24.0 h1:LpilSUItNPFr1eY85RYgTIg5eIEPtvFbskaFcmmIUnk=
|
||||||
|
github.com/aws/smithy-go v1.24.0/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
|
||||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
||||||
github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWpi6yML8=
|
github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWpi6yML8=
|
||||||
@@ -31,6 +37,8 @@ github.com/charmbracelet/x/exp/golden v0.0.0-20240715153702-9ba8adf781c4/go.mod
|
|||||||
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 h1:q2hJAaP1k2wIvVRd/hEHD7lacgqrCPS+k8g1MndzfWY=
|
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 h1:q2hJAaP1k2wIvVRd/hEHD7lacgqrCPS+k8g1MndzfWY=
|
||||||
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
|
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
|
||||||
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||||
|
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
@@ -43,6 +51,8 @@ github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002 h1:LOFxQG4mxvlH7
|
|||||||
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002/go.mod h1:fIgfH/E1j3rU8du4X4+7mxTD0GPtPQibTzytgitdJWU=
|
github.com/eslider/go-hocr v0.2.2-0.20260827163626-8ff01582b002/go.mod h1:fIgfH/E1j3rU8du4X4+7mxTD0GPtPQibTzytgitdJWU=
|
||||||
github.com/eslider/go-xls/v2 v2.1.0 h1:HszWKqYQbXxACmAXXWdMsfNl1NDBfGVBnJUPtyUHQ7A=
|
github.com/eslider/go-xls/v2 v2.1.0 h1:HszWKqYQbXxACmAXXWdMsfNl1NDBfGVBnJUPtyUHQ7A=
|
||||||
github.com/eslider/go-xls/v2 v2.1.0/go.mod h1:xgxO6JrfuBr9jGUB+0z5l/yDmFFZ5diGk0ATGihxlMU=
|
github.com/eslider/go-xls/v2 v2.1.0/go.mod h1:xgxO6JrfuBr9jGUB+0z5l/yDmFFZ5diGk0ATGihxlMU=
|
||||||
|
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
|
||||||
|
github.com/go-sql-driver/mysql v1.10.1/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
|
||||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||||
github.com/google/go-querystring v1.2.0 h1:yhqkPbu2/OH+V9BfpCVPZkNmUXhb2gBxJArfhIxNtP0=
|
github.com/google/go-querystring v1.2.0 h1:yhqkPbu2/OH+V9BfpCVPZkNmUXhb2gBxJArfhIxNtP0=
|
||||||
@@ -59,8 +69,20 @@ github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUq
|
|||||||
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
|
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
|
||||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||||
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||||
|
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||||
|
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||||
|
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
||||||
|
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
||||||
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
||||||
|
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
|
||||||
|
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
|
||||||
|
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||||
|
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||||
github.com/lucasb-eyer/go-colorful v1.4.0 h1:UtrWVfLdarDgc44HcS7pYloGHJUjHV/4FwW4TvVgFr4=
|
github.com/lucasb-eyer/go-colorful v1.4.0 h1:UtrWVfLdarDgc44HcS7pYloGHJUjHV/4FwW4TvVgFr4=
|
||||||
github.com/lucasb-eyer/go-colorful v1.4.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
|
github.com/lucasb-eyer/go-colorful v1.4.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
|
||||||
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
|
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
|
||||||
@@ -94,6 +116,8 @@ github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJ
|
|||||||
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
|
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
|
||||||
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
|
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
|
||||||
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
||||||
|
github.com/rogpeppe/go-internal v1.16.0 h1:O9DK+vNMDVGLr2BeZqmpLeMjiMNkuXfcqntWbZV6S5g=
|
||||||
|
github.com/rogpeppe/go-internal v1.16.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
|
||||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||||
github.com/sebdah/goldie/v2 v2.8.0 h1:dZb9wR8q5++oplmEiJT+U/5KyotVD+HNGCAc5gNr8rc=
|
github.com/sebdah/goldie/v2 v2.8.0 h1:dZb9wR8q5++oplmEiJT+U/5KyotVD+HNGCAc5gNr8rc=
|
||||||
github.com/sebdah/goldie/v2 v2.8.0/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
|
github.com/sebdah/goldie/v2 v2.8.0/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
|
||||||
@@ -103,6 +127,9 @@ github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
|
|||||||
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
|
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
|
||||||
github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
|
github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
|
||||||
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||||
github.com/tiendc/go-deepcopy v1.7.2 h1:Ut2yYR7W9tWjTQitganoIue4UGxZwCcJy3orjrrIj44=
|
github.com/tiendc/go-deepcopy v1.7.2 h1:Ut2yYR7W9tWjTQitganoIue4UGxZwCcJy3orjrrIj44=
|
||||||
@@ -138,8 +165,10 @@ golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
|
|||||||
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
|
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
|
||||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||||
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI=
|
modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI=
|
||||||
|
|||||||
@@ -346,6 +346,14 @@ func (c *Client) putJSON(ctx context.Context, path string, body any) (json.RawMe
|
|||||||
|
|
||||||
// uploadMultipart posts a single file to path under the given form field name.
|
// uploadMultipart posts a single file to path under the given form field name.
|
||||||
func (c *Client) uploadMultipart(ctx context.Context, path, fieldName, filePath string) (json.RawMessage, error) {
|
func (c *Client) uploadMultipart(ctx context.Context, path, fieldName, filePath string) (json.RawMessage, error) {
|
||||||
|
return c.uploadMultipartMethod(ctx, http.MethodPost, path, fieldName, filePath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// uploadMultipartMethod sends a single-file multipart request with the given
|
||||||
|
// HTTP method. The OnlyOffice Documents API needs PUT for /update (a new
|
||||||
|
// version) and POST for /upload (a new file); sending POST to /update answers
|
||||||
|
// 500 on current servers.
|
||||||
|
func (c *Client) uploadMultipartMethod(ctx context.Context, method, path, fieldName, filePath string) (json.RawMessage, error) {
|
||||||
auth, err := c.authHeader()
|
auth, err := c.authHeader()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -368,7 +376,7 @@ func (c *Client) uploadMultipart(ctx context.Context, path, fieldName, filePath
|
|||||||
if err := mw.Close(); err != nil {
|
if err := mw.Close(); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL()+path, &buf)
|
req, err := http.NewRequestWithContext(ctx, method, c.baseURL()+path, &buf)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
// - pandoc — md↔docx
|
// - pandoc — md↔docx
|
||||||
// - ocrmypdf — OCR into a searchable PDF
|
// - ocrmypdf — OCR into a searchable PDF
|
||||||
// - pdftotext — extract text layer
|
// - pdftotext — extract text layer
|
||||||
|
// - pdfdetach — list/save embedded PDF attachments
|
||||||
// - tesseract — OCR single images when ocrmypdf is unsuitable
|
// - tesseract — OCR single images when ocrmypdf is unsuitable
|
||||||
// - ghostscript (gs) — PDF rewrite/optimize via PostScript (pdfwrite)
|
// - ghostscript (gs) — PDF rewrite/optimize via PostScript (pdfwrite)
|
||||||
package docpipe
|
package docpipe
|
||||||
@@ -26,6 +27,7 @@ type Tools struct {
|
|||||||
Pandoc string
|
Pandoc string
|
||||||
OCRMyPDF string
|
OCRMyPDF string
|
||||||
PDFToText string
|
PDFToText string
|
||||||
|
PDFDetach string
|
||||||
Tesseract string
|
Tesseract string
|
||||||
Ghostscript string
|
Ghostscript string
|
||||||
}
|
}
|
||||||
@@ -44,6 +46,7 @@ func LookPath() Tools {
|
|||||||
Pandoc: find("pandoc"),
|
Pandoc: find("pandoc"),
|
||||||
OCRMyPDF: find("ocrmypdf"),
|
OCRMyPDF: find("ocrmypdf"),
|
||||||
PDFToText: find("pdftotext"),
|
PDFToText: find("pdftotext"),
|
||||||
|
PDFDetach: find("pdfdetach"),
|
||||||
Tesseract: find("tesseract"),
|
Tesseract: find("tesseract"),
|
||||||
Ghostscript: find("gs", "ghostscript"),
|
Ghostscript: find("gs", "ghostscript"),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,231 @@
|
|||||||
|
package docpipe
|
||||||
|
|
||||||
|
// Embedded PDF attachments (F6 #42). Digitised invoices often carry the
|
||||||
|
// original scan as a PDF attachment; the searchable body may hold only a
|
||||||
|
// summary. pdfdetach (poppler) lists/saves them; each saved attachment is run
|
||||||
|
// through the normal docpipe extraction (pdftotext/OCR).
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/xml"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// PDFAttachment is one embedded file in a PDF.
|
||||||
|
type PDFAttachment struct {
|
||||||
|
Index int // 1-based number as `pdfdetach -list` reports it
|
||||||
|
Name string // embedded file name
|
||||||
|
}
|
||||||
|
|
||||||
|
// AttachmentText is the extracted text of one embedded attachment.
|
||||||
|
type AttachmentText struct {
|
||||||
|
Name string
|
||||||
|
Text string
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseAttachmentList parses `pdfdetach -list` output. The first line is a
|
||||||
|
// count ("N embedded files"); every following line is "<index>: <name>".
|
||||||
|
// Pure, so it is unit-tested.
|
||||||
|
func parseAttachmentList(out string) []PDFAttachment {
|
||||||
|
var atts []PDFAttachment
|
||||||
|
for _, line := range strings.Split(out, "\n") {
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
if line == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
colon := strings.Index(line, ":")
|
||||||
|
if colon <= 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
n, err := strconv.Atoi(strings.TrimSpace(line[:colon]))
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
name := strings.TrimSpace(line[colon+1:])
|
||||||
|
if name == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
atts = append(atts, PDFAttachment{Index: n, Name: name})
|
||||||
|
}
|
||||||
|
return atts
|
||||||
|
}
|
||||||
|
|
||||||
|
// safeAttachmentName strips directories and leading dots so a hostile
|
||||||
|
// attachment name cannot escape the extraction directory.
|
||||||
|
func safeAttachmentName(name string) string {
|
||||||
|
name = strings.ReplaceAll(strings.TrimSpace(name), "\\", "/")
|
||||||
|
name = filepath.Base(name)
|
||||||
|
name = strings.TrimLeft(name, ".")
|
||||||
|
if name == "" || name == "." || name == "/" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
|
||||||
|
// JoinWithAttachments appends attachment text to the document body, each
|
||||||
|
// section preceded by an "[attachment: <name>]" marker so a search hit shows
|
||||||
|
// its source. Empty attachments are skipped. Pure, so it is unit-tested.
|
||||||
|
func JoinWithAttachments(body string, atts []AttachmentText) string {
|
||||||
|
var b strings.Builder
|
||||||
|
b.WriteString(strings.TrimRight(body, "\n"))
|
||||||
|
for _, a := range atts {
|
||||||
|
text := strings.TrimSpace(a.Text)
|
||||||
|
if text == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
b.WriteString("\n\n[attachment: ")
|
||||||
|
b.WriteString(a.Name)
|
||||||
|
b.WriteString("]\n\n")
|
||||||
|
b.WriteString(text)
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListAttachments returns the embedded files of a PDF. A PDF without
|
||||||
|
// attachments yields an empty slice and no error.
|
||||||
|
func (t Tools) ListAttachments(pdfPath string) ([]PDFAttachment, error) {
|
||||||
|
if t.PDFDetach == "" {
|
||||||
|
return nil, fmt.Errorf("pdfdetach not found on PATH")
|
||||||
|
}
|
||||||
|
cmd := exec.Command(t.PDFDetach, "-list", pdfPath)
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
cmd.Stderr = &stderr
|
||||||
|
out, err := cmd.Output()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("pdfdetach -list %s: %w (%s)", filepath.Base(pdfPath), err, strings.TrimSpace(stderr.String()))
|
||||||
|
}
|
||||||
|
return parseAttachmentList(string(out)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SaveAttachment writes the n-th embedded file (1-based) to outPath.
|
||||||
|
func (t Tools) SaveAttachment(pdfPath string, index int, outPath string) error {
|
||||||
|
if t.PDFDetach == "" {
|
||||||
|
return fmt.Errorf("pdfdetach not found on PATH")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outPath) == "" {
|
||||||
|
return fmt.Errorf("output path required")
|
||||||
|
}
|
||||||
|
if err := EnsureDir(outPath); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
cmd := exec.Command(t.PDFDetach, "-save", strconv.Itoa(index), "-o", outPath, pdfPath)
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
cmd.Stderr = &stderr
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
return fmt.Errorf("pdfdetach -save %d: %w (%s)", index, err, strings.TrimSpace(stderr.String()))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ToMarkdownWithAttachments extracts the file as ToMarkdown does, then — for
|
||||||
|
// PDFs — appends the text of every embedded attachment under an
|
||||||
|
// "[attachment: <name>]" marker. Attachment failures are non-fatal: the body
|
||||||
|
// is returned unchanged.
|
||||||
|
func (t Tools) ToMarkdownWithAttachments(path, workDir, lang string, minChars int) (string, error) {
|
||||||
|
res, err := t.ToMarkdown(path, workDir, lang, minChars)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if Ext(path) != ".pdf" {
|
||||||
|
return res.Markdown, nil
|
||||||
|
}
|
||||||
|
atts, err := t.attachmentTexts(path, workDir, lang, minChars)
|
||||||
|
if err != nil {
|
||||||
|
return res.Markdown, nil
|
||||||
|
}
|
||||||
|
return JoinWithAttachments(res.Markdown, atts), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// attachmentTexts saves and extracts every embedded attachment, skipping the
|
||||||
|
// ones that cannot be read. It returns an error only when the attachment list
|
||||||
|
// itself cannot be obtained.
|
||||||
|
func (t Tools) attachmentTexts(pdfPath, workDir, lang string, minChars int) ([]AttachmentText, error) {
|
||||||
|
list, err := t.ListAttachments(pdfPath)
|
||||||
|
if err != nil || len(list) == 0 {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if workDir == "" {
|
||||||
|
workDir = os.TempDir()
|
||||||
|
}
|
||||||
|
dir := filepath.Join(workDir, "att-"+trimExt(filepath.Base(pdfPath)))
|
||||||
|
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(dir)
|
||||||
|
|
||||||
|
out := make([]AttachmentText, 0, len(list))
|
||||||
|
for _, a := range list {
|
||||||
|
name := safeAttachmentName(a.Name)
|
||||||
|
if name == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
saved := filepath.Join(dir, fmt.Sprintf("%d-%s", a.Index, name))
|
||||||
|
if err := t.SaveAttachment(pdfPath, a.Index, saved); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
text, err := t.attachmentMarkdown(saved, dir, lang, minChars)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, AttachmentText{Name: a.Name, Text: text})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// attachmentMarkdown extracts a saved attachment with the regular pipeline.
|
||||||
|
// Structured attachments that docpipe does not convert (e-invoice XML,
|
||||||
|
// CuraSoft JSON, CSV/HTML) fall back to their text content, so the embedded
|
||||||
|
// original is still searchable. Other unreadable formats return an error and
|
||||||
|
// the caller skips them.
|
||||||
|
func (t Tools) attachmentMarkdown(path, workDir, lang string, minChars int) (string, error) {
|
||||||
|
if res, err := t.ToMarkdown(path, workDir, lang, minChars); err == nil {
|
||||||
|
return res.Markdown, nil
|
||||||
|
}
|
||||||
|
switch Ext(path) {
|
||||||
|
case ".xml", ".html", ".htm":
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return xmlToText(raw), nil
|
||||||
|
case ".json", ".csv":
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return string(raw), nil
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("unsupported attachment type %q", Ext(path))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// xmlToText returns the character data of an XML/HTML document: element text
|
||||||
|
// values with decoded entities, one per line. Used for invoice XML (EN 16931
|
||||||
|
// CII / ZUGFeRD) and HTML attachments. Pure, so it is unit-tested.
|
||||||
|
func xmlToText(raw []byte) string {
|
||||||
|
dec := xml.NewDecoder(bytes.NewReader(raw))
|
||||||
|
dec.Strict = false
|
||||||
|
var b strings.Builder
|
||||||
|
for {
|
||||||
|
tok, err := dec.Token()
|
||||||
|
if err != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
cd, ok := tok.(xml.CharData)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
s := strings.TrimSpace(string(cd))
|
||||||
|
if s == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
b.WriteString(s)
|
||||||
|
b.WriteByte('\n')
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
package docpipe
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseAttachmentList(t *testing.T) {
|
||||||
|
out := "2 embedded files\n1: original.pdf\n2: scan_001.png\n"
|
||||||
|
got := parseAttachmentList(out)
|
||||||
|
want := []PDFAttachment{{Index: 1, Name: "original.pdf"}, {Index: 2, Name: "scan_001.png"}}
|
||||||
|
if len(got) != len(want) {
|
||||||
|
t.Fatalf("got %+v, want %+v", got, want)
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if got[i] != want[i] {
|
||||||
|
t.Errorf("att[%d] = %+v, want %+v", i, got[i], want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseAttachmentListEmptyAndMalformed(t *testing.T) {
|
||||||
|
for _, in := range []string{"", "0 embedded files\n", "garbage\n\n \n"} {
|
||||||
|
if got := parseAttachmentList(in); len(got) != 0 {
|
||||||
|
t.Errorf("parseAttachmentList(%q) = %+v, want empty", in, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSafeAttachmentName(t *testing.T) {
|
||||||
|
cases := map[string]string{
|
||||||
|
"note.txt": "note.txt",
|
||||||
|
"../../evil.pdf": "evil.pdf",
|
||||||
|
`..\..\evil.pdf`: "evil.pdf",
|
||||||
|
"/abs/scan_001.pdf": "scan_001.pdf",
|
||||||
|
".hidden": "hidden",
|
||||||
|
" spaced name.txt ": "spaced name.txt",
|
||||||
|
"..": "",
|
||||||
|
"": "",
|
||||||
|
}
|
||||||
|
for in, want := range cases {
|
||||||
|
if got := safeAttachmentName(in); got != want {
|
||||||
|
t.Errorf("safeAttachmentName(%q) = %q, want %q", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJoinWithAttachments(t *testing.T) {
|
||||||
|
body := "# scan.pdf\n\nbody token\n"
|
||||||
|
atts := []AttachmentText{
|
||||||
|
{Name: "original.pdf", Text: " original token "},
|
||||||
|
{Name: "empty.txt", Text: " "},
|
||||||
|
}
|
||||||
|
got := JoinWithAttachments(body, atts)
|
||||||
|
if !strings.Contains(got, "body token") {
|
||||||
|
t.Errorf("body text lost: %q", got)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got, "[attachment: original.pdf]") {
|
||||||
|
t.Errorf("marker missing: %q", got)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got, "original token") {
|
||||||
|
t.Errorf("attachment text missing: %q", got)
|
||||||
|
}
|
||||||
|
if strings.Contains(got, "empty.txt") {
|
||||||
|
t.Errorf("empty attachment must be skipped: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJoinWithAttachmentsNoAttachments(t *testing.T) {
|
||||||
|
got := JoinWithAttachments("# a.pdf\n\ntext\n\n", nil)
|
||||||
|
if got != "# a.pdf\n\ntext" {
|
||||||
|
t.Errorf("got %q, want trimmed body only", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestListAttachmentsWithoutTool(t *testing.T) {
|
||||||
|
if _, err := (Tools{}).ListAttachments("x.pdf"); err == nil || !strings.Contains(err.Error(), "pdfdetach") {
|
||||||
|
t.Fatalf("want pdfdetach error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestToMarkdownWithAttachmentsFixture exercises the real pdfdetach + pdftotext
|
||||||
|
// pipeline on testdata/pdf-with-attachment.pdf (body token + embedded
|
||||||
|
// goo-note.txt). Skips when poppler is not installed.
|
||||||
|
func TestToMarkdownWithAttachmentsFixture(t *testing.T) {
|
||||||
|
tools := LookPath()
|
||||||
|
if tools.PDFDetach == "" || tools.PDFToText == "" {
|
||||||
|
t.Skip("pdfdetach/pdftotext not on PATH — skipping attachment extraction test")
|
||||||
|
}
|
||||||
|
fixture := filepath.Join("..", "..", "testdata", "pdf-with-attachment.pdf")
|
||||||
|
got, err := tools.ToMarkdownWithAttachments(fixture, t.TempDir(), "eng", 1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ToMarkdownWithAttachments: %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"goobodytoken", "[attachment: goo-note.txt]", "gooattachmenttoken"} {
|
||||||
|
if !strings.Contains(got, want) {
|
||||||
|
t.Errorf("result missing %q:\n%s", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestXMLToText(t *testing.T) {
|
||||||
|
raw := []byte(`<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<rsm:CrossIndustryInvoice><rsm:ExchangedDocument>
|
||||||
|
<ram:ID>S1063</ram:ID></rsm:ExchangedDocument>
|
||||||
|
<ram:Name>Edelweiss & Co</ram:Name><ram:GrandTotalAmount>42.00</ram:GrandTotalAmount>
|
||||||
|
</rsm:CrossIndustryInvoice>`)
|
||||||
|
got := xmlToText(raw)
|
||||||
|
for _, want := range []string{"S1063", "Edelweiss & Co", "42.00"} {
|
||||||
|
if !strings.Contains(got, want) {
|
||||||
|
t.Errorf("xmlToText missing %q:\n%s", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.ContainsAny(got, "<>") {
|
||||||
|
t.Errorf("xmlToText left markup: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestAttachmentMarkdownFallback verifies structured attachments that docpipe
|
||||||
|
// cannot convert are still reduced to searchable text, and unknown binary
|
||||||
|
// formats error (so the caller skips them).
|
||||||
|
func TestAttachmentMarkdownFallback(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
xmlPath := filepath.Join(dir, "factur-x.xml")
|
||||||
|
if err := os.WriteFile(xmlPath, []byte(`<Invoice><Number>S1063</Number></Invoice>`), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got, err := (Tools{}).attachmentMarkdown(xmlPath, dir, "", 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("attachmentMarkdown(xml): %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got, "S1063") {
|
||||||
|
t.Errorf("xml attachment text = %q, want S1063", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
binPath := filepath.Join(dir, "data.bin")
|
||||||
|
if err := os.WriteFile(binPath, []byte{0, 1, 2, 3}, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := (Tools{}).attachmentMarkdown(binPath, dir, "", 0); err == nil {
|
||||||
|
t.Error("unsupported attachment: want error, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestToMarkdownWithAttachmentsPlainPDF ensures a PDF without attachments
|
||||||
|
// returns just the body (pdfdetach prints "0 embedded files").
|
||||||
|
func TestToMarkdownWithAttachmentsPlainPDF(t *testing.T) {
|
||||||
|
tools := LookPath()
|
||||||
|
if tools.PDFDetach == "" || tools.PDFToText == "" {
|
||||||
|
t.Skip("pdfdetach/pdftotext not on PATH")
|
||||||
|
}
|
||||||
|
// The fixture itself is a PDF with one attachment; strip it by extracting
|
||||||
|
// the body only through ToMarkdown and compare JoinWithAttachments(nil).
|
||||||
|
res, err := tools.ToMarkdown(filepath.Join("..", "..", "testdata", "pdf-with-attachment.pdf"), t.TempDir(), "eng", 1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ToMarkdown: %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(res.Markdown, "gooattachmenttoken") {
|
||||||
|
t.Fatalf("body must not contain attachment text: %q", res.Markdown)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"regexp"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RetryPolicy controls deterministic retries against OnlyOffice: fixed linear
|
||||||
|
// backoff without jitter, so repeated runs wait exactly the same schedule.
|
||||||
|
// OnlyOffice throttles bulk reads/writes with 429 (and occasional 502/503/504
|
||||||
|
// from openresty), so every bulk tool routes API calls through DoRetry.
|
||||||
|
type RetryPolicy struct {
|
||||||
|
Attempts int // total attempts, including the first try
|
||||||
|
Base time.Duration // wait before retry N is N*Base
|
||||||
|
Max time.Duration // per-wait cap
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultRetryPolicy retries up to 5 times with 1s, 2s, 3s, 4s waits.
|
||||||
|
func DefaultRetryPolicy() RetryPolicy {
|
||||||
|
return RetryPolicy{Attempts: 5, Base: time.Second, Max: 30 * time.Second}
|
||||||
|
}
|
||||||
|
|
||||||
|
var transientRe = regexp.MustCompile(`:\s*(429|502|503|504)\b`)
|
||||||
|
|
||||||
|
// Transient reports whether err looks like a transient OnlyOffice answer
|
||||||
|
// (an HTTP 429/502/503/504 surfaced as "...: <code> ...").
|
||||||
|
func Transient(err error) bool {
|
||||||
|
if err == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return transientRe.MatchString(err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
// DoRetry runs fn until it succeeds, fails non-transiently, or attempts run
|
||||||
|
// out. Waits are deterministic: N*Base capped at Max, no jitter.
|
||||||
|
func DoRetry(ctx context.Context, p RetryPolicy, fn func() error) error {
|
||||||
|
if p.Attempts < 1 {
|
||||||
|
p.Attempts = 1
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
for attempt := 1; attempt <= p.Attempts; attempt++ {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if err = fn(); err == nil || !Transient(err) || attempt == p.Attempts {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
wait := time.Duration(attempt) * p.Base
|
||||||
|
if wait > p.Max {
|
||||||
|
wait = p.Max
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-time.After(wait):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
// MinIO download fallback for the portal's stale AWS S3 consumer.
|
||||||
|
//
|
||||||
|
// On the Fibu EDL portal some older Documents files live in S3/MinIO, but the
|
||||||
|
// portal's storage consumer still points at s3.us-east-1.amazonaws.com with
|
||||||
|
// access key "minio". Downloads of those files answer 403 InvalidAccessKeyId.
|
||||||
|
// The bytes are present in the local MinIO store under a deterministic object
|
||||||
|
// key, so the client retries the GET there.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws/signer/v4"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultMinioEndpoint = "http://192.168.188.10:9000"
|
||||||
|
defaultMinioBucket = "office"
|
||||||
|
minioRegion = "us-east-1"
|
||||||
|
)
|
||||||
|
|
||||||
|
// minioObjectKey is the fallback object key layout the portal's S3 consumer
|
||||||
|
// writes for Documents files: 00/00/01/files/folder_<folderId>/file_<fileId>/v1/content.pdf.
|
||||||
|
// Prefer minioObjectKeyFromURL: the portal stores all files below its storage
|
||||||
|
// root folder, which is not the API folderId returned by GetFile.
|
||||||
|
func minioObjectKey(fileID, folderID string) string {
|
||||||
|
return "00/00/01/files/folder_" + folderID + "/file_" + fileID + "/v1/content.pdf"
|
||||||
|
}
|
||||||
|
|
||||||
|
// minioObjectKeyFromURL extracts the object key from an S3 download URL. Path
|
||||||
|
// style URLs (bucket as first path segment) have that segment removed; virtual
|
||||||
|
// host style URLs are returned as-is. This is authoritative: the portal signs
|
||||||
|
// the exact key, so no folder-id guessing is needed.
|
||||||
|
func minioObjectKeyFromURL(rawURL, bucket string) (string, bool) {
|
||||||
|
u, err := url.Parse(strings.TrimSpace(rawURL))
|
||||||
|
if err != nil || u.Path == "" {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
segs := strings.Split(strings.Trim(u.Path, "/"), "/")
|
||||||
|
// Path-style URLs carry the bucket as leading segment; the portal's S3
|
||||||
|
// consumer can emit it twice (serviceurl already includes the bucket), so
|
||||||
|
// strip every leading segment equal to the bucket.
|
||||||
|
for len(segs) > 0 && bucket != "" && segs[0] == bucket {
|
||||||
|
segs = segs[1:]
|
||||||
|
}
|
||||||
|
if len(segs) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
for _, s := range segs {
|
||||||
|
if s == "" || s == "." || s == ".." {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return strings.Join(segs, "/"), true
|
||||||
|
}
|
||||||
|
|
||||||
|
// isStaleS3Redirect reports whether a download landed on the portal's stale AWS
|
||||||
|
// S3 consumer. Such responses either carry an S3 InvalidAccessKeyId XML body or
|
||||||
|
// point at amazonaws.com with the "minio" access key id in the query.
|
||||||
|
func isStaleS3Redirect(rawURL string, body []byte) bool {
|
||||||
|
if strings.Contains(strings.ToLower(string(body)), "invalidaccesskeyid") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
u, err := url.Parse(strings.TrimSpace(rawURL))
|
||||||
|
if err != nil || u.Host == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
host := strings.ToLower(u.Host)
|
||||||
|
if !strings.Contains(host, "amazonaws.com") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
q := strings.ToLower(u.RawQuery)
|
||||||
|
return strings.Contains(q, "accesskeyid=minio") || strings.Contains(q, "x-amz-credential=minio")
|
||||||
|
}
|
||||||
|
|
||||||
|
// minioConfig is the runtime configuration for the local MinIO fallback.
|
||||||
|
type minioConfig struct {
|
||||||
|
Endpoint string
|
||||||
|
Bucket string
|
||||||
|
AccessKey string
|
||||||
|
SecretKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadMinioConfig reads the fallback configuration from the environment.
|
||||||
|
// Secrets are never defaulted; without access/secret keys the fallback is off.
|
||||||
|
func loadMinioConfig() minioConfig {
|
||||||
|
return minioConfig{
|
||||||
|
Endpoint: strings.TrimRight(firstNonEmpty(os.Getenv("MINIO_ENDPOINT"), defaultMinioEndpoint), "/"),
|
||||||
|
Bucket: firstNonEmpty(os.Getenv("MINIO_BUCKET"), defaultMinioBucket),
|
||||||
|
AccessKey: os.Getenv("MINIO_ACCESS_KEY"),
|
||||||
|
SecretKey: os.Getenv("MINIO_SECRET_KEY"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// downloadFileEntry downloads f's bytes to dst. It transparently falls back to
|
||||||
|
// the local MinIO store when the portal redirects the download to its stale AWS
|
||||||
|
// S3 consumer.
|
||||||
|
func (c *Client) downloadFileEntry(ctx context.Context, f *FileEntry, dst io.Writer) (int64, error) {
|
||||||
|
if f == nil {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: download: nil file entry")
|
||||||
|
}
|
||||||
|
if f.ViewURL == nil || *f.ViewURL == "" {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: file has no viewUrl")
|
||||||
|
}
|
||||||
|
downloadURL := c.resolveAPIURL(*f.ViewURL)
|
||||||
|
auth, err := c.authHeader()
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Authorization", auth)
|
||||||
|
resp, err := c.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode >= 400 {
|
||||||
|
b, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
|
||||||
|
finalURL := downloadURL
|
||||||
|
if resp.Request != nil && resp.Request.URL != nil {
|
||||||
|
finalURL = resp.Request.URL.String()
|
||||||
|
}
|
||||||
|
if isStaleS3Redirect(finalURL, b) {
|
||||||
|
key, ok := minioObjectKeyFromURL(finalURL, loadMinioConfig().Bucket)
|
||||||
|
if !ok {
|
||||||
|
fileID := ""
|
||||||
|
if f.ID != nil {
|
||||||
|
fileID = f.ID.String()
|
||||||
|
}
|
||||||
|
key = minioObjectKey(fileID, FileFolderID(f))
|
||||||
|
}
|
||||||
|
n, merr := c.downloadFromMinio(ctx, key, dst)
|
||||||
|
if merr == nil {
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
|
return 0, fmt.Errorf("GET viewUrl: %d (stale S3) and minio fallback: %w", resp.StatusCode, merr)
|
||||||
|
}
|
||||||
|
return 0, fmt.Errorf("GET viewUrl: %d %s", resp.StatusCode, truncate(string(b), 400))
|
||||||
|
}
|
||||||
|
return io.Copy(dst, resp.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// downloadFromMinio streams objectKey from the configured MinIO bucket.
|
||||||
|
func (c *Client) downloadFromMinio(ctx context.Context, objectKey string, dst io.Writer) (int64, error) {
|
||||||
|
return downloadMinioObject(ctx, c.client, objectKey, dst)
|
||||||
|
}
|
||||||
|
|
||||||
|
// downloadMinioObject streams objectKey from the configured MinIO bucket using
|
||||||
|
// hc. It is the shared implementation behind (*Client).downloadFromMinio and
|
||||||
|
// the read-only SQL store (file_pg.go), which has no *Client.
|
||||||
|
func downloadMinioObject(ctx context.Context, hc *http.Client, objectKey string, dst io.Writer) (int64, error) {
|
||||||
|
if objectKey == "" {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: minio fallback: empty object key")
|
||||||
|
}
|
||||||
|
cfg := loadMinioConfig()
|
||||||
|
if cfg.AccessKey == "" || cfg.SecretKey == "" {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: minio fallback: MINIO_ACCESS_KEY/MINIO_SECRET_KEY not set")
|
||||||
|
}
|
||||||
|
base, err := url.Parse(cfg.Endpoint)
|
||||||
|
if err != nil || base.Host == "" {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: minio fallback: bad MINIO_ENDPOINT %q", cfg.Endpoint)
|
||||||
|
}
|
||||||
|
u := *base
|
||||||
|
u.Path = "/" + cfg.Bucket + "/" + objectKey
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if err := signMinioRequest(ctx, cfg, req); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
resp, err := hc.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("onlyoffice: minio fallback: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode >= 400 {
|
||||||
|
b, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||||
|
return 0, fmt.Errorf("onlyoffice: minio fallback: %d %s", resp.StatusCode, truncate(string(b), 300))
|
||||||
|
}
|
||||||
|
return io.Copy(dst, resp.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// signMinioRequest signs req with AWS Signature V4 for the S3 service.
|
||||||
|
func signMinioRequest(ctx context.Context, cfg minioConfig, req *http.Request) error {
|
||||||
|
sum := sha256.Sum256(nil)
|
||||||
|
creds := aws.Credentials{AccessKeyID: cfg.AccessKey, SecretAccessKey: cfg.SecretKey}
|
||||||
|
if err := v4.NewSigner().SignHTTP(ctx, creds, req, hex.EncodeToString(sum[:]), "s3", minioRegion, time.Now()); err != nil {
|
||||||
|
return fmt.Errorf("onlyoffice: minio fallback: sign: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
//go:build integration
|
||||||
|
|
||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestIntegrationMinioFallback downloads known stale-S3 files through the local
|
||||||
|
// MinIO fallback. Requires ONLYOFFICE_URL/USER/PASS (as all integration tests),
|
||||||
|
// MINIO_ACCESS_KEY/MINIO_SECRET_KEY and MINIO_TEST_FILE_IDS="3785,3859,3666";
|
||||||
|
// skips when any of those are missing.
|
||||||
|
func TestIntegrationMinioFallback(t *testing.T) {
|
||||||
|
if os.Getenv("MINIO_ACCESS_KEY") == "" || os.Getenv("MINIO_SECRET_KEY") == "" {
|
||||||
|
t.Skip("MINIO_ACCESS_KEY/MINIO_SECRET_KEY not set — skipping integration test")
|
||||||
|
}
|
||||||
|
raw := strings.TrimSpace(os.Getenv("MINIO_TEST_FILE_IDS"))
|
||||||
|
if raw == "" {
|
||||||
|
t.Skip("MINIO_TEST_FILE_IDS not set — skipping integration test")
|
||||||
|
}
|
||||||
|
c := liveClient(t)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
for _, id := range strings.Split(raw, ",") {
|
||||||
|
id = strings.TrimSpace(id)
|
||||||
|
if id == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
n, err := c.DownloadFile(ctx, id, io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("DownloadFile(%s): %v", id, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if n == 0 {
|
||||||
|
t.Errorf("DownloadFile(%s): 0 bytes", id)
|
||||||
|
} else {
|
||||||
|
t.Logf("DownloadFile(%s): %d bytes", id, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,219 @@
|
|||||||
|
package onlyoffice
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMinioObjectKey(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
fileID string
|
||||||
|
folderID string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"3785", "652", "00/00/01/files/folder_652/file_3785/v1/content.pdf"},
|
||||||
|
{"1", "2", "00/00/01/files/folder_2/file_1/v1/content.pdf"},
|
||||||
|
{"3666", "4000", "00/00/01/files/folder_4000/file_3666/v1/content.pdf"},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
if got := minioObjectKey(tc.fileID, tc.folderID); got != tc.want {
|
||||||
|
t.Errorf("minioObjectKey(%q, %q) = %q, want %q", tc.fileID, tc.folderID, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMinioObjectKeyFromURL(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
url string
|
||||||
|
bucket string
|
||||||
|
want string
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "path style drops bucket segment",
|
||||||
|
url: "https://s3.us-east-1.amazonaws.com/office/00/00/01/files/folder_4000/file_3785/v1/content.pdf?AWSAccessKeyId=minio",
|
||||||
|
bucket: "office",
|
||||||
|
want: "00/00/01/files/folder_4000/file_3785/v1/content.pdf",
|
||||||
|
ok: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "doubled bucket segment (portal serviceurl includes bucket)",
|
||||||
|
url: "https://s3.us-east-1.amazonaws.com/office/office/00/00/01/files/folder_4000/file_3785/v1/content.pdf?AWSAccessKeyId=minio",
|
||||||
|
bucket: "office",
|
||||||
|
want: "00/00/01/files/folder_4000/file_3785/v1/content.pdf",
|
||||||
|
ok: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "virtual host style keeps path",
|
||||||
|
url: "https://office.s3.us-east-1.amazonaws.com/00/00/01/files/folder_4000/file_3785/v1/content.pdf",
|
||||||
|
bucket: "office",
|
||||||
|
want: "00/00/01/files/folder_4000/file_3785/v1/content.pdf",
|
||||||
|
ok: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "foreign first segment kept",
|
||||||
|
url: "https://example.com/other/file_1/v1/content.pdf",
|
||||||
|
bucket: "office",
|
||||||
|
want: "other/file_1/v1/content.pdf",
|
||||||
|
ok: true,
|
||||||
|
},
|
||||||
|
{name: "empty path", url: "https://example.com", bucket: "office", ok: false},
|
||||||
|
{name: "traversal", url: "https://example.com/office/../etc/passwd", bucket: "office", ok: false},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
got, ok := minioObjectKeyFromURL(tc.url, tc.bucket)
|
||||||
|
if ok != tc.ok || got != tc.want {
|
||||||
|
t.Errorf("minioObjectKeyFromURL(%q, %q) = (%q, %v), want (%q, %v)", tc.url, tc.bucket, got, ok, tc.want, tc.ok)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsStaleS3Redirect(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
url string
|
||||||
|
body []byte
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "aws redirect with minio access key",
|
||||||
|
url: "https://s3.us-east-1.amazonaws.com/office/x/file_1?AWSAccessKeyId=minio&Expires=1",
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "aws redirect with minio x-amz-credential",
|
||||||
|
url: "https://office.s3.us-east-1.amazonaws.com/00/00/01/files/folder_1/file_1?X-Amz-Credential=minio%2F20260914",
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid access key xml body",
|
||||||
|
url: "https://portal.internal/download/1",
|
||||||
|
body: []byte(`<?xml version="1.0"?><Error><Code>InvalidAccessKeyId</Code><AWSAccessKeyId>minio</AWSAccessKeyId></Error>`),
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "regular pdf from portal",
|
||||||
|
url: "https://portal.internal/download/1",
|
||||||
|
body: []byte("%PDF-1.7 data"),
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "aws redirect with foreign key",
|
||||||
|
url: "https://s3.us-east-1.amazonaws.com/office/x?AWSAccessKeyId=other",
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "amazonaws in path but foreign host",
|
||||||
|
url: "https://example.com/amazonaws.com/file?AWSAccessKeyId=minio",
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty",
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := isStaleS3Redirect(tc.url, tc.body); got != tc.want {
|
||||||
|
t.Errorf("isStaleS3Redirect(%q, %q) = %v, want %v", tc.url, tc.body, got, tc.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const staleS3Body = `<?xml version="1.0" encoding="UTF-8"?>` +
|
||||||
|
`<Error><Code>InvalidAccessKeyId</Code>` +
|
||||||
|
`<Message>The AWS Access Key Id you provided does not exist in our records.</Message>` +
|
||||||
|
`<AWSAccessKeyId>minio</AWSAccessKeyId></Error>`
|
||||||
|
|
||||||
|
func TestDownloadFileMinioFallback(t *testing.T) {
|
||||||
|
const payload = "PDFDATA-3785"
|
||||||
|
|
||||||
|
portal := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
switch r.URL.Path {
|
||||||
|
case "/api/2.0/files/file/3785.json":
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
io.WriteString(w, `{"response":{"id":3785,"title":"04.pdf","folderId":655,"viewUrl":"/download/3785"}}`)
|
||||||
|
case "/download/3785":
|
||||||
|
http.Redirect(w, r, "/office/00/00/01/files/folder_4000/file_3785/v1/content.pdf?AWSAccessKeyId=minio", http.StatusTemporaryRedirect)
|
||||||
|
case "/office/00/00/01/files/folder_4000/file_3785/v1/content.pdf":
|
||||||
|
w.WriteHeader(http.StatusForbidden)
|
||||||
|
io.WriteString(w, staleS3Body)
|
||||||
|
default:
|
||||||
|
http.NotFound(w, r)
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
defer portal.Close()
|
||||||
|
|
||||||
|
var minioPath, minioAuth string
|
||||||
|
minio := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
minioPath, minioAuth = r.URL.Path, r.Header.Get("Authorization")
|
||||||
|
io.WriteString(w, payload)
|
||||||
|
}))
|
||||||
|
defer minio.Close()
|
||||||
|
|
||||||
|
t.Setenv("MINIO_ENDPOINT", minio.URL)
|
||||||
|
t.Setenv("MINIO_BUCKET", "office")
|
||||||
|
t.Setenv("MINIO_ACCESS_KEY", "testkey")
|
||||||
|
t.Setenv("MINIO_SECRET_KEY", "testsecret")
|
||||||
|
|
||||||
|
c := &Client{
|
||||||
|
client: portal.Client(),
|
||||||
|
credentials: &Credentials{Url: portal.URL},
|
||||||
|
token: &Token{Value: "Bearer test", Expires: Time(time.Now().Add(time.Hour))},
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
n, err := c.DownloadFile(context.Background(), "3785", &buf)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DownloadFile: %v", err)
|
||||||
|
}
|
||||||
|
if n != int64(len(payload)) || buf.String() != payload {
|
||||||
|
t.Fatalf("got %d bytes %q, want %d bytes %q", n, buf.String(), len(payload), payload)
|
||||||
|
}
|
||||||
|
if want := "/office/00/00/01/files/folder_4000/file_3785/v1/content.pdf"; minioPath != want {
|
||||||
|
t.Errorf("minio path = %q, want %q", minioPath, want)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(minioAuth, "AWS4-HMAC-SHA256") {
|
||||||
|
t.Errorf("minio request not SigV4-signed; Authorization=%q", minioAuth)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadFileMinioFallbackWithoutCreds(t *testing.T) {
|
||||||
|
portal := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
switch r.URL.Path {
|
||||||
|
case "/api/2.0/files/file/3785.json":
|
||||||
|
io.WriteString(w, `{"response":{"id":3785,"folderId":652,"viewUrl":"/download/3785"}}`)
|
||||||
|
default:
|
||||||
|
w.WriteHeader(http.StatusForbidden)
|
||||||
|
io.WriteString(w, staleS3Body)
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
defer portal.Close()
|
||||||
|
|
||||||
|
t.Setenv("MINIO_ACCESS_KEY", "")
|
||||||
|
t.Setenv("MINIO_SECRET_KEY", "")
|
||||||
|
|
||||||
|
c := &Client{
|
||||||
|
client: portal.Client(),
|
||||||
|
credentials: &Credentials{Url: portal.URL},
|
||||||
|
token: &Token{Value: "Bearer test", Expires: Time(time.Now().Add(time.Hour))},
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := c.DownloadFile(context.Background(), "3785", io.Discard)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error without minio credentials")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "MINIO_ACCESS_KEY/MINIO_SECRET_KEY not set") {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
Vendored
+114
@@ -0,0 +1,114 @@
|
|||||||
|
%PDF-1.7
|
||||||
|
%Çì�¢
|
||||||
|
%%Invocation: gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite ? ? ?
|
||||||
|
5 0 obj
|
||||||
|
<</Length 6 0 R/Filter /FlateDecode>>
|
||||||
|
stream
|
||||||
|
xœ-ŠA
|
||||||
|
ƒ0÷ÿoÙnìO7q]èZ*ÿ�IˆmJ½½M
|
||||||
|
³f7
|
||||||
|
\¨6‘žfRÿŠ*ñºõª…8:g}‡f†Dºø”ÞiØsúØ ÝôÝ;炱(Ùn.ly]ìUFz
|
||||||
|
½~Aë øendstream
|
||||||
|
endobj
|
||||||
|
6 0 obj
|
||||||
|
110
|
||||||
|
endobj
|
||||||
|
4 0 obj
|
||||||
|
<</Type/Page/MediaBox [0 0 612 792]
|
||||||
|
/Rotate 0/Parent 3 0 R
|
||||||
|
/Resources<</ProcSet[/PDF /Text]
|
||||||
|
/Font 8 0 R
|
||||||
|
>>
|
||||||
|
/Contents 5 0 R
|
||||||
|
>>
|
||||||
|
endobj
|
||||||
|
3 0 obj
|
||||||
|
<< /Type /Pages /Kids [
|
||||||
|
4 0 R
|
||||||
|
] /Count 1
|
||||||
|
>>
|
||||||
|
endobj
|
||||||
|
1 0 obj
|
||||||
|
<</Type /Catalog /Pages 3 0 R
|
||||||
|
/Metadata 9 0 R
|
||||||
|
>>
|
||||||
|
endobj
|
||||||
|
8 0 obj
|
||||||
|
<</R7
|
||||||
|
7 0 R>>
|
||||||
|
endobj
|
||||||
|
7 0 obj
|
||||||
|
<</BaseFont/Helvetica/Type/Font
|
||||||
|
/Subtype/Type1>>
|
||||||
|
endobj
|
||||||
|
9 0 obj
|
||||||
|
<</Type/Metadata
|
||||||
|
/Subtype/XML/Length 1173>>stream
|
||||||
|
<?xpacket begin='' id='W5M0MpCehiHzreSzNTczkc9d'?>
|
||||||
|
<?adobe-xap-filters esc="CRLF"?>
|
||||||
|
<x:xmpmeta xmlns:x='adobe:ns:meta/' x:xmptk='XMP toolkit 2.9.1-13, framework 1.6'>
|
||||||
|
<rdf:RDF xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#' xmlns:iX='http://ns.adobe.com/iX/1.0/'>
|
||||||
|
<rdf:Description rdf:about="" xmlns:pdf='http://ns.adobe.com/pdf/1.3/' pdf:Producer='GPL Ghostscript 10.02.1'/>
|
||||||
|
<rdf:Description rdf:about="" xmlns:xmp='http://ns.adobe.com/xap/1.0/'><xmp:ModifyDate>2026-09-16T17:29:12Z</xmp:ModifyDate>
|
||||||
|
<xmp:CreateDate>2026-09-16T17:29:12Z</xmp:CreateDate>
|
||||||
|
<xmp:CreatorTool>UnknownApplication</xmp:CreatorTool></rdf:Description>
|
||||||
|
<rdf:Description rdf:about="" xmlns:xapMM='http://ns.adobe.com/xap/1.0/mm/' xapMM:DocumentID='uuid:ae8d7575-ea10-11fc-0000-3a98742e0b71'/>
|
||||||
|
<rdf:Description rdf:about="" xmlns:dc='http://purl.org/dc/elements/1.1/' dc:format='application/pdf'><dc:title><rdf:Alt><rdf:li xml:lang='x-default'>Untitled</rdf:li></rdf:Alt></dc:title></rdf:Description>
|
||||||
|
</rdf:RDF>
|
||||||
|
</x:xmpmeta>
|
||||||
|
|
||||||
|
|
||||||
|
<?xpacket end='w'?>
|
||||||
|
endstream
|
||||||
|
endobj
|
||||||
|
2 0 obj
|
||||||
|
<</Producer(GPL Ghostscript 10.02.1)
|
||||||
|
/CreationDate(D:20260916172912Z00'00')
|
||||||
|
/ModDate(D:20260916172912Z00'00')>>endobj
|
||||||
|
xref
|
||||||
|
0 10
|
||||||
|
0000000000 65535 f
|
||||||
|
0000000476 00000 n
|
||||||
|
0000001882 00000 n
|
||||||
|
0000000417 00000 n
|
||||||
|
0000000276 00000 n
|
||||||
|
0000000077 00000 n
|
||||||
|
0000000257 00000 n
|
||||||
|
0000000569 00000 n
|
||||||
|
0000000540 00000 n
|
||||||
|
0000000633 00000 n
|
||||||
|
trailer
|
||||||
|
<< /Size 10 /Root 1 0 R /Info 2 0 R
|
||||||
|
/ID [<5A562476FBF11852133AA873452909F9><5A562476FBF11852133AA873452909F9>]
|
||||||
|
>>
|
||||||
|
startxref
|
||||||
|
2008
|
||||||
|
%%EOF
|
||||||
|
1 0 obj
|
||||||
|
<</Type /Catalog /Pages 3 0 R /Metadata 9 0 R /Names <</EmbeddedFiles 12 0 R >> >>
|
||||||
|
endobj
|
||||||
|
10 0 obj
|
||||||
|
<</Length 44 /Params <</Size 44 >> >> stream
|
||||||
|
gooattachmenttoken embedded attachment text
|
||||||
|
|
||||||
|
endstream
|
||||||
|
|
||||||
|
endobj
|
||||||
|
11 0 obj
|
||||||
|
<</Type /Filespec /UF <feff0067006f006f002d006e006f00740065002e007400780074> /EF <</F 10 0 R >> >>
|
||||||
|
endobj
|
||||||
|
12 0 obj
|
||||||
|
<</Names [<feff0067006f006f002d006e006f00740065002e007400780074> 11 0 R ] >>
|
||||||
|
endobj
|
||||||
|
xref
|
||||||
|
0 2
|
||||||
|
0000000002 65535 f
|
||||||
|
0000002361 00000 n
|
||||||
|
10 3
|
||||||
|
0000002463 00000 n
|
||||||
|
0000002586 00000 n
|
||||||
|
0000002705 00000 n
|
||||||
|
trailer
|
||||||
|
<</Size 13 /ID [(ZV$vûñR:¨sE\) ù) (P…õrˆ‚ Åð¡Ï) ] /Root 1 0 R /Prev 2008 /Info 2 0 R >>
|
||||||
|
startxref
|
||||||
|
2802
|
||||||
Reference in New Issue
Block a user