From b96c5c5361ff9707637600252c40b60535a04fd4 Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Wed, 16 Sep 2026 21:41:46 +0000 Subject: [PATCH] =?UTF-8?q?docs(files):=20unified=20file=20client=20+=20ba?= =?UTF-8?q?ckends=20=D0=BF=D0=BE=D1=81=D0=BB=D0=B5=20#58=E2=80=93#61=20(#5?= =?UTF-8?q?3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AGENTS.md | 21 ++++++++++++++++++--- README.md | 32 ++++++++++++++++++++++++++------ docs/README.md | 23 +++++++++++++++++++++++ docs/unified-file-client.md | 34 +++++++++++++++++++++++++++------- 4 files changed, 94 insertions(+), 16 deletions(-) create mode 100644 docs/README.md diff --git a/AGENTS.md b/AGENTS.md index b37c558..ab7ce32 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,8 +9,8 @@ Canonical Go client for OnlyOffice Workspace (Projects + Calendar + CRM) and the - `request.go` — `Request`, `Query`, `Time`, `Token`, `MetaResponse`, `Permissions`. - `auth.go` — `Authenticate`, `AuthenticateContext`, `InvalidateToken`, `Auth`, token lifecycle. - `http.go` — transport + DRY response decoders (`ResponseArray`/`ResponseObject`/`postFormObject`/`putFormObject`/`deleteObject`). - - `projects.go`, `tasks.go`, `users.go`, `calendar.go`, `crm.go`, `files.go`, `files_webdav.go`, `files_stem.go`, `retry.go`, `mails.go`, `invoices.go` — typed / untyped domain methods. **`files.go`** — CRM opportunity upload plus **project/task Documents** (`UpdateFile`, `UploadToFolderReplacing`). **`files_webdav.go`** — Documents module by id (`ListDavFolder`, `MoveDavItems`/`CopyDavItems` with per-operation error surfacing, `ListFileOps`). **`retry.go`** — `DoRetry`: deterministic linear backoff (no jitter) on 429/502/503/504; every bulk tool routes API calls through it, and the HTTP transport + auth (`retryRaw`, `AuthenticateContext`) retry transient answers centrally. **`mails.go`** — OnlyOffice Workspace Mail. **`invoices.go`** — CRM invoices, PDF regen/cleanup, status. Association rules: [`docs/crm-associations.md`](docs/crm-associations.md). - - **Unified file client (epic #34) — `file_core.go`, `file_rest.go`, `file_dav.go`, `file_pg.go`, `file_es.go`, `file_es_text.go`, `file_text_index.go`, `file_facade.go`.** `file_core.go` — model (`Entry`, `Kind`) + `FileStore`/`Searcher`; `file_rest.go`/`file_dav.go` — REST/WebDAV adapters; `file_pg.go` — **read-only** SQL store (PostgreSQL/MySQL, `ErrReadOnly` on writes); `file_es.go` — OnlyOffice Elasticsearch searcher; `file_es_text.go`/`file_text_index.go` — own PDF/scan index (`oo_docs_text`, PDF attachments via pdfdetach); `file_facade.go` — `FileClient` with read/write/search order and transient fallback. Use `c.Files()` (facade), `c.FileStore("rest"|"dav"|"pg"|"sql")` or `c.SQLFileStore()`; contract and how to add a backend: [`docs/unified-file-client.md`](docs/unified-file-client.md). + - `projects.go`, `tasks.go`, `users.go`, `calendar.go`, `crm.go`, `files.go`, `files_webdav.go`, `files_stem.go`, `files_dedupe.go`, `retry.go`, `mails.go`, `invoices.go` — typed / untyped domain methods. **`files.go`** — CRM opportunity upload plus **project/task Documents** (`UpdateFile`, `UploadToFolderReplacing`). **`files_webdav.go`** — Documents module by id (`ListDavFolder`, `MoveDavItems`/`CopyDavItems` with per-operation error surfacing, `ListFileOps`). **`files_dedupe.go`** — stem|ext duplicate scans (`FindProjectDuplicates`, `DedupeProject`, `ApplyDedupGroups`; backs `oo projects files dedupe`). **`retry.go`** — `DoRetry`: deterministic linear backoff (no jitter) on 429/502/503/504; every bulk tool routes API calls through it, and the HTTP transport + auth (`retryRaw`, `AuthenticateContext`) retry transient answers centrally. **`mails.go`** — OnlyOffice Workspace Mail. **`invoices.go`** — CRM invoices, PDF regen/cleanup, status. Association rules: [`docs/crm-associations.md`](docs/crm-associations.md). + - **Unified file client (epic #34) — `file_core.go`, `file_rest.go`, `file_dav.go`, `file_pg.go`, `file_es.go`, `file_es_text.go`, `file_text_index.go`, `file_facade.go`.** `file_core.go` — model (`Entry`, `Kind`) + `FileStore`/`Searcher`; `file_rest.go`/`file_dav.go` — REST/WebDAV adapters; `file_pg.go` — **read-only** SQL store (PostgreSQL/MySQL, `ErrReadOnly` on writes); `file_es.go` — OnlyOffice Elasticsearch searcher; `file_es_text.go`/`file_text_index.go` — own PDF/scan index (`oo_docs_text`, PDF attachments via pdfdetach); `file_facade.go` — `FileClient` with read/write/search order and transient fallback. Use `c.Files()` (facade), `c.FileStore("rest"|"dav"|"webdav"|"pg"|"sql"|"postgres"|"mysql")` or `c.SQLFileStore()`. Integration: `file_store_integration_test.go` runs the full read+write set through the REST and DAV stores. Contract and how to add a backend: [`docs/unified-file-client.md`](docs/unified-file-client.md); the rclone WebDAV FUSE mount: [`docs/rclone-webdav.md`](docs/rclone-webdav.md). - Pure stdlib + `google/go-querystring`; no UI, no dotenv. - **CLI — `cmd/oo/` as `package main`.** Cobra wrapper that loads `.env` via `godotenv` at startup. **Subject-based command tree** mirroring [`tea`](https://gitea.com/gitea/tea): - `main.go` — entry point (docstring lists the command tree). @@ -51,7 +51,22 @@ write `mux.HandleFunc("/api/2.0/...")` to emulate OnlyOffice, we write an `ONLYOFFICE_USER`, `ONLYOFFICE_PASS` (aliases `_HOST`/`_NAME`/`_PASSWORD` also accepted). Tests **skip** cleanly when credentials are missing so `go test ./...` remains green in CI. -- Run integration with: `go test -tags=integration ./...`. +- Backend-specific env (each test skips when its env is missing): + - REST/DAV store: `ONLYOFFICE_URL/USER/PASS` — `file_store_integration_test.go` + (`TestIntegrationFileStores`, destructive, throwaway project). + - SQL store: `ONLYOFFICE_DSN` (+ `ONLYOFFICE_PG_TENANT`, + `ONLYOFFICE_PG_TEST_FILE_ID`, `ONLYOFFICE_PG_TEST_FOLDER_ID`); + `MINIO_*` to compare `Download` with REST — `file_pg_integration_test.go` + (`TestIntegrationPGStore`, `TestIntegrationSQLFacade`). + - Elasticsearch: `ONLYOFFICE_ES_URL` (+ `ONLYOFFICE_ES_INDEX`, + `ONLYOFFICE_TENANT`) — `file_es_integration_test.go` + (`TestIntegrationESFacadeUsesES` proves `c.Files().Search()` is the ES + backend), `file_es_text_integration_test.go` (own `oo_docs_text`). +- Run unit with `-race`: `go test -race ./...`. Run integration with: + `go test -tags=integration ./...`. +- The HTTP transport and auth retry transient answers (429/502/503/504) + centrally via `retryRaw`/`DoRetry`, so parallel integration runs survive the + shared openresty rate limit (#57). - New endpoints **must** ship with an integration test before merge. ## Related diff --git a/README.md b/README.md index dfb9f2e..81e8617 100644 --- a/README.md +++ b/README.md @@ -690,12 +690,17 @@ See [`docs/elasticsearch.md`](docs/elasticsearch.md) for the tunnel setup. ```bash oo search "Rechnung" # names only oo search "Mahngebühr" --content # names + document text +oo search "Rechnung 2025" --substring # case-insensitive *term*, words ANDed oo search "Rechnung" --folder 649 --limit 50 oo search "Rechnung" --json # shorthand for -o json ``` +Flags: `--content`, `--substring`, `--folder ID` (folder subtree), `--limit N`, +`--backend oo|own`, `--json`. + Requires `ONLYOFFICE_ES_URL` (plus optional `ONLYOFFICE_ES_INDEX`, -`ONLYOFFICE_TENANT`). +`ONLYOFFICE_TENANT`). `TestIntegrationESFacadeUsesES` pins +`c.Files().Search()` to the ES backend (the REST `@search` only looks at names). #### PDF/scans: own index (`oo index` + `--backend own`) @@ -722,13 +727,22 @@ trade-offs. All file backends (REST, WebDAV, read-only SQL, Elasticsearch) sit behind one facade: `c.Files()` returns a `*FileClient` that also implements `FileStore`, -so old call sites keep working. Pick a transport per call with -`c.FileStore("rest"|"dav"|"pg"|"sql")` (SQL is read-only), open the SQL store -with `c.SQLFileStore()`, or register a backend on the facade +so old call sites keep working. Reads prefer SQL, then REST, then DAV; writes +go to REST/DAV. Pick a transport per call with +`c.FileStore("rest"|"dav"|"webdav"|"pg"|"sql"|"postgres"|"mysql")` (SQL is +read-only, `ErrReadOnly` on writes), open the SQL store with +`c.SQLFileStore()`, or register a backend on the facade (`RegisterStore`/`RegisterSearcher`). Contract, model (`Entry`/`Kind`), fallback rules, env names and how to add a backend: [`docs/unified-file-client.md`](docs/unified-file-client.md). +- SQL (PostgreSQL/MySQL): `ONLYOFFICE_DSN` (or `ONLYOFFICE_PG_*` parts) — + [`docs/community-server-db.md`](docs/community-server-db.md). +- Elasticsearch: `ONLYOFFICE_ES_URL` — [`docs/elasticsearch.md`](docs/elasticsearch.md). +- Duplicate cleanup: `oo projects files dedupe PROJECT_ID` (`--apply`, `--cross`). +- Mount the Documents tree as a local FS (docker compose + FUSE): + `deploy/docker-compose.rclone-webdav.yml` — [`docs/rclone-webdav.md`](docs/rclone-webdav.md). + ### Bulk tools (`cmd/`) Small single-purpose binaries for bulk Documents work. All of them pace @@ -765,8 +779,8 @@ kontolink IN.xlsx oo-index.tsv OUT.xlsx [FILE_ID] [AMOUNTS_TSV] | `docs` | `tools`, `convert`, `optimize`, `ocr`, `hocr`, `as-md`, `put-md`, `put-txt`, `put-xlsx` | | `catalog` | `match`, `merge`, `apply`, `scan-contacts`, `scan-projects`, `scan-thunderbird` | | `dav` | `ls`, `move`, `copy`, `mkdir`, `rename-file`, `rename-folder`, `download`, `fileops` | -| `search` | `QUERY` (`--content`, `--folder ID`, `--limit N`, `--backend oo\|own`, `--json`) | -| `index` | `folder FOLDER_ID`, `files FILE_ID...` (`--recursive`, `--exts pdf`, `--limit N`, `--dry-run`) | +| `search` | `QUERY` (`--content`, `--substring`, `--folder ID`, `--limit N`, `--backend oo\|own`, `--json`) | +| `index` | `folder FOLDER_ID`, `files FILE_ID...` (`--recursive`, `--exts pdf`, `--limit N`, `--workers N`, `--lang deu+eng`, `--min-chars N`, `--work-dir DIR`, `--backend rest\|dav`, `--dry-run`, `--json`) | The CLI reads only `.env` from the current working directory (godotenv is a CLI-only concern — the library itself never loads dotfiles). @@ -999,6 +1013,12 @@ oo projects files list 33 | `ONLYOFFICE_CALENDAR_ID` | Default calendar id used when omitted (default `1`) | | `ONLYOFFICE_PROJECT_ID` | Default project id used when omitted (default `33`) | | `OO_URL`, `OO_USER`, `OO_PASS` | Optional CLI-only aliases for `ONLYOFFICE_*` | +| `ONLYOFFICE_ES_URL`, `ONLYOFFICE_ES_INDEX`, `ONLYOFFICE_TENANT` | Elasticsearch search (`oo search`, `oo index`) — [docs/elasticsearch.md](docs/elasticsearch.md) | +| `ONLYOFFICE_ES_TEXT_INDEX` | Own PDF/scan index (default `oo_docs_text`) | +| `ONLYOFFICE_DSN`, `ONLYOFFICE_PG_DRIVER`, `ONLYOFFICE_PG_TENANT`, `ONLYOFFICE_PG_*` | Read-only SQL store — [docs/community-server-db.md](docs/community-server-db.md) | +| `MINIO_ENDPOINT`, `MINIO_BUCKET`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY` | SQL `Download` / portal S3 fallback | +| `ONLYOFFICE_DOCS_URL`, `ONLYOFFICE_DOCS_SECRET` | Optional Document Server for `office` DOCX preview | +| `ONLYOFFICE_WEBDAV_URL`, `ONLYOFFICE_USER`, `ONLYOFFICE_PASSWORD` | rclone WebDAV mount — [docs/rclone-webdav.md](docs/rclone-webdav.md) | Mail and CRM cleanup are documented in [oo CLI use cases](#oo-cli-use-cases) above. Personal disk inventory / dossier sync lives in the private `oo-workspace` (`oow`) tooling. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..faba3cc --- /dev/null +++ b/docs/README.md @@ -0,0 +1,23 @@ +--- +type: index +status: current +related: + - README.md +--- + +# docs — индекс + +Документация `go-onlyoffice`. Код — источник истины, доки только описывают. + +| файл | о чём | +|------|-------| +| [unified-file-client.md](unified-file-client.md) | Единый файловый клиент: `FileStore`/`Searcher`, `c.Files()`, фасад, порядок чтения/записи, SQL read-only, как добавить backend | +| [community-server-db.md](community-server-db.md) | Прямой read-only SQL-доступ к БД Community Server (MySQL/Postgres), `ONLYOFFICE_DSN`, схема, MinIO-download | +| [elasticsearch.md](elasticsearch.md) | Полнотекстовый поиск: индекс OnlyOffice `files_file` и свой `oo_docs_text` (PDF/сканы) | +| [rclone-webdav.md](rclone-webdav.md) | Монтирование WebDAV-дерева Documents как ФС (docker compose + FUSE) | +| [crm-associations.md](crm-associations.md) | Связи CRM: company ↔ person ↔ deal ↔ project ↔ invoice ↔ mail | + +## См. также + +- [`../README.md`](../README.md) — установка, CLI-команды, env. +- [`../AGENTS.md`](../AGENTS.md) — топология репо и правила. diff --git a/docs/unified-file-client.md b/docs/unified-file-client.md index 5547ec1..9ff0769 100644 --- a/docs/unified-file-client.md +++ b/docs/unified-file-client.md @@ -25,8 +25,10 @@ REST, WebDAV, SQL (PostgreSQL/MySQL), Elasticsearch. Правило одно: `Size`, `MIME`, `Created`, `Modified`, `Updated` (сырая строка API), `Version`, `Provider`, `FilesCount`/`FoldersCount` (папки). Чего бэкенд не даёт — остаётся в нуле. -- `SearchQuery` — `Text`, `InContent`, `FolderID`, `Extensions`, `Limit`. -- `SearchHit` — `Entry` + `Score`, `Highlight`, `Path`. +- `SearchQuery` — `Text`, `InContent`, `FolderID`, `Extensions`, `Limit`, + `Substring` (регистронезависимое `*term*` по имени, несколько слов AND). +- `SearchHit` — `Entry` + `Score`, `Highlight`, `Path []string` (предки + root → leaf). ## Интерфейсы @@ -113,11 +115,12 @@ e, _ := f.Stat(ctx, "19423") // e.Provider == "mysql" entries, _ := f.List(ctx, "676") // пойдёт в SQL ``` -`Client.FileStore("pg"|"sql"|"postgres"|"mysql")` — одноразовый доступ к -SQL-стору без фасада: открывает из env; при ошибке возвращает заглушку, -которая отдаёт ошибку открытия на каждом вызове (не `nil`). `SQLFileStore()` -— тот же открыватель, но с ошибкой. Отвечавший бэкенд видно по -`Entry.Provider` (`mysql` / `postgres` у SQL, `rest` у REST). +`Client.FileStore("rest"|"dav"|"webdav"|"pg"|"sql"|"postgres"|"mysql")` — +прямой доступ к одному бэкенду без фасада. SQL открывается из env; при ошибке +возвращается заглушка, которая отдаёт ошибку открытия на каждом вызове (не +`nil`). `SQLFileStore()` — тот же открыватель, но с ошибкой. Пустое/неизвестное +имя — REST. Отвечавший бэкенд видно по `Entry.Provider` (`mysql`/`postgres` у +SQL, `rest`/`dav` у HTTP). ## CLI @@ -125,6 +128,7 @@ SQL-стору без фасада: открывает из env; при ошиб # поиск: --backend oo (индекс OnlyOffice) | own (свой oo_docs_text) oo search "Rechnung" oo search "Mahngebühr" --content +oo search "Rechnung 2025" --substring # *term* по имени, слова AND oo search "S1021" --content --backend own --folder 634 --limit 50 --json # наполнение своего индекса (PDF/сканы, idempotent upsert по file id) @@ -193,11 +197,27 @@ res, _ := ti.IndexFolder(ctx, "634", onlyoffice.IndexOptions{Recursive: true}) ```bash go test ./... # unit, без сети +go test -race ./... go test -tags=integration ./... # live (креды в .env) go test ./ -run 'FileStore|Facade|ESText|PG' ``` +- `file_store_integration_test.go` — `TestIntegrationFileStores`: полный набор + read+write (create/upload/list/stat/download/move/copy/rename/delete) через + REST и DAV. Нужен `ONLYOFFICE_URL/USER/PASS`. +- `file_pg_integration_test.go` — `TestIntegrationPGStore`, + `TestIntegrationSQLFacade`: SQL против REST, `ErrReadOnly`. Нужен + `ONLYOFFICE_DSN` (+ `MINIO_*` для `Download`). +- `file_es_integration_test.go` — `TestIntegrationESFacadeUsesES`: доказывает, + что `c.Files().Search()` идёт в ES, а не в REST. Нужен `ONLYOFFICE_ES_URL`. +- `file_es_text_integration_test.go` — свой индекс `oo_docs_text`. + +Каждый тест делает чистый `skip` без своего env. HTTP-транспорт и auth ретраят +transient-ответы (429/502/503/504) централизованно (`retryRaw`/`DoRetry`), см. +[`retry.go`](../retry.go). + ## См. также - [elasticsearch.md](elasticsearch.md) — индекс OnlyOffice и свой `oo_docs_text`. - [community-server-db.md](community-server-db.md) — SQL-стор и схема БД. +- [rclone-webdav.md](rclone-webdav.md) — монтирование Documents через FUSE.