feat(files): SQL backend via Client.FileStore + live MySQL facade test (#55) #60

Merged
eSlider merged 1 commits from feat/db-integration#55 into main 2026-09-16 22:30:05 +01:00
9 changed files with 269 additions and 34 deletions
+1 -1
View File
@@ -10,7 +10,7 @@ Canonical Go client for OnlyOffice Workspace (Projects + Calendar + CRM) and the
- `auth.go` — `Authenticate`, `AuthenticateContext`, `InvalidateToken`, `Auth`, token lifecycle.
- `http.go` — transport + DRY response decoders (`ResponseArray`/`ResponseObject`/`postFormObject`/`putFormObject`/`deleteObject`).
- `projects.go`, `tasks.go`, `users.go`, `calendar.go`, `crm.go`, `files.go`, `files_webdav.go`, `files_stem.go`, `retry.go`, `mails.go`, `invoices.go` — typed / untyped domain methods. **`files.go`** — CRM opportunity upload plus **project/task Documents** (`UpdateFile`, `UploadToFolderReplacing`). **`files_webdav.go`** — Documents module by id (`ListDavFolder`, `MoveDavItems`/`CopyDavItems` with per-operation error surfacing, `ListFileOps`). **`retry.go`** — `DoRetry`: deterministic linear backoff (no jitter) on 429/502/503/504; every bulk tool routes API calls through it. **`mails.go`** — OnlyOffice Workspace Mail. **`invoices.go`** — CRM invoices, PDF regen/cleanup, status. Association rules: [`docs/crm-associations.md`](docs/crm-associations.md).
- **Unified file client (epic #34) — `file_core.go`, `file_rest.go`, `file_dav.go`, `file_pg.go`, `file_es.go`, `file_es_text.go`, `file_text_index.go`, `file_facade.go`.** `file_core.go` — model (`Entry`, `Kind`) + `FileStore`/`Searcher`; `file_rest.go`/`file_dav.go` — REST/WebDAV adapters; `file_pg.go` — **read-only** SQL store (PostgreSQL/MySQL, `ErrReadOnly` on writes); `file_es.go` — OnlyOffice Elasticsearch searcher; `file_es_text.go`/`file_text_index.go` — own PDF/scan index (`oo_docs_text`, PDF attachments via pdfdetach); `file_facade.go` — `FileClient` with read/write/search order and transient fallback. Use `c.Files()` (facade) or `c.FileStore("rest"|"dav")`; contract and how to add a backend: [`docs/unified-file-client.md`](docs/unified-file-client.md).
- **Unified file client (epic #34) — `file_core.go`, `file_rest.go`, `file_dav.go`, `file_pg.go`, `file_es.go`, `file_es_text.go`, `file_text_index.go`, `file_facade.go`.** `file_core.go` — model (`Entry`, `Kind`) + `FileStore`/`Searcher`; `file_rest.go`/`file_dav.go` — REST/WebDAV adapters; `file_pg.go` — **read-only** SQL store (PostgreSQL/MySQL, `ErrReadOnly` on writes); `file_es.go` — OnlyOffice Elasticsearch searcher; `file_es_text.go`/`file_text_index.go` — own PDF/scan index (`oo_docs_text`, PDF attachments via pdfdetach); `file_facade.go` — `FileClient` with read/write/search order and transient fallback. Use `c.Files()` (facade), `c.FileStore("rest"|"dav"|"pg"|"sql")` or `c.SQLFileStore()`; contract and how to add a backend: [`docs/unified-file-client.md`](docs/unified-file-client.md).
- Pure stdlib + `google/go-querystring`; no UI, no dotenv.
- **CLI — `cmd/oo/` as `package main`.** Cobra wrapper that loads `.env` via `godotenv` at startup. **Subject-based command tree** mirroring [`tea`](https://gitea.com/gitea/tea):
- `main.go` — entry point (docstring lists the command tree).
+2 -1
View File
@@ -723,7 +723,8 @@ trade-offs.
All file backends (REST, WebDAV, read-only SQL, Elasticsearch) sit behind one
facade: `c.Files()` returns a `*FileClient` that also implements `FileStore`,
so old call sites keep working. Pick a transport per call with
`c.FileStore("rest"|"dav")`, or register a backend on the facade
`c.FileStore("rest"|"dav"|"pg"|"sql")` (SQL is read-only), open the SQL store
with `c.SQLFileStore()`, or register a backend on the facade
(`RegisterStore`/`RegisterSearcher`). Contract, model (`Entry`/`Kind`),
fallback rules, env names and how to add a backend:
[`docs/unified-file-client.md`](docs/unified-file-client.md).
+58 -9
View File
@@ -65,14 +65,22 @@ MySQL слушает только `127.0.0.1:3306` внутри VM. Снаруж
```bash
ssh -f -N -o ControlMaster=no -o ControlPath=none \
-p 32 -i ~/.ssh/id_ed25519 \
-L 13306:127.0.0.1:3306 root@127.0.0.1
-L 3306:127.0.0.1:3306 root@127.0.0.1
# MySQL DSN затем:
# root:<pw>@tcp(127.0.0.1:13306)/onlyoffice?parseTime=true
# root:<pw>@tcp(127.0.0.1:3306)/onlyoffice?parseTime=true
```
`-o ControlMaster=no -o ControlPath=none` обязательны: иначе forward уходит в
persistent master из `~/.ssh/config`.
Любой свободный локальный порт подойдёт (напр. `13306`); тогда тот же порт —
в DSN. `-o ControlMaster=no -o ControlPath=none` обязательны: иначе forward
уходит в persistent master из `~/.ssh/config`.
Креды MySQL — в конфиге Community Server внутри VM:
`/etc/onlyoffice/communityserver/appsettings.production.json` →
`ConnectionStrings.connectionString` (поля `User ID`, `Password`), база
`onlyoffice`. В самом MySQL-контейнере (`onlyoffice-mysql-server`) база пустая;
рабочий сервер — host-mysqld на `127.0.0.1:3306` (207 таблиц). Не печатать
пароль.
## Переменные
@@ -85,6 +93,31 @@ persistent master из `~/.ssh/config`.
Имена — в [`.env.example`](../.env.example). Секретов нет.
## Использование
Напрямую: `NewPGStore(PGConfigFromEnv())`.
Через фасад (эпик #34): SQL-стор регистрируется на `FileClient`. После этого
`Read()` и все чтения (`Stat`/`List`) идут в БД, `Write()` остаётся REST/DAV.
```go
c := onlyoffice.NewClient(onlyoffice.GetEnvironmentCredentials())
sql, err := c.SQLFileStore() // открыть из env; caller закрывает
if err != nil { /* нет DSN / нет связи */ }
if closer, ok := sql.(interface{ Close() error }); ok { defer closer.Close() }
f := c.Files()
f.RegisterStore(onlyoffice.ProviderPG, sql)
e, _ := f.Stat(ctx, "19423") // e.Provider == "mysql" — ответил SQL
```
`Client.FileStore("pg"|"sql"|"postgres"|"mysql")` тоже отдаёт SQL-стор
(открывает из env). Если DSN нет/битый — возвращается не `nil`, а заглушка,
чей метод отдаёт ошибку открытия; ошибку как таковую даёт `SQLFileStore()`.
Различить бэкенд в ответе можно по `Entry.Provider` (`mysql` у SQL, `rest` у
REST).
## Download (MinIO)
`Download` не ходит в REST. Ключ объекта собирается из строки `files_file`:
@@ -100,6 +133,12 @@ shard = (id/1000 + 1) * 1000
Стриминг переиспользует `downloadMinioObject` из `storage_fallback.go`
(та же подпись SigV4 и `MINIO_*`), без дублирования.
Ограничение: схема валидна только для файлов, лежащих в **MinIO/S3** (старые
папки). Файлы в **Disc**-хранилище портала (`Data/Products/Files/...`, новые
папки) по этому ключу недоступны — `Download` вернёт `404`. Если
`MINIO_ACCESS_KEY`/`MINIO_SECRET_KEY` не заданы, `Download` вернёт явную
ошибку; `Stat`/`List`/`Search` работают и без них.
## Тесты
```bash
@@ -107,16 +146,26 @@ go test ./... # unit: rebind, csObjectKey, мапп
go test -race ./...
# integration (нужен DSN; skip без него)
ONLYOFFICE_DSN='root:<pw>@tcp(127.0.0.1:13306)/onlyoffice?parseTime=true' \
ONLYOFFICE_DSN='root:<pw>@tcp(127.0.0.1:3306)/onlyoffice?parseTime=true' \
ONLYOFFICE_PG_TENANT=1 \
ONLYOFFICE_PG_TEST_FILE_ID=22484 \
ONLYOFFICE_PG_TEST_FOLDER_ID=649 \
ONLYOFFICE_PG_TEST_FILE_ID=19423 \
ONLYOFFICE_PG_TEST_FOLDER_ID=676 \
go test -tags=integration -run 'TestIntegrationPGStore|TestIntegrationSQLFacade' -v ./...
# плюс MINIO_* для сверки Download с REST (иначе этот шаг skip)
MINIO_ENDPOINT=http://127.0.0.1:9000 MINIO_BUCKET=office \
MINIO_ACCESS_KEY=... MINIO_SECRET_KEY=... \
go test -tags=integration -run TestIntegrationPGStore -v ./...
```
Integration сверяет `Stat`/`List`/`Download` с REST (`c.Files()`) и проверяет,
что write-методы дают `ErrReadOnly`.
- `TestIntegrationPGStore` — `Stat`/`List`/`Download` SQL против REST и
`ErrReadOnly` у write-методов.
- `TestIntegrationSQLFacade` — SQL-стор, зарегистрированный на фасаде, реально
обслуживает чтения: `Read().Name()` = SQL-бэкенд, `Entry.Provider == "mysql"`
(у REST — `"rest"`), сверка `Stat`/`List` с REST, и прямой
`Client.FileStore("pg")`.
Без `ONLYOFFICE_DSN` оба теста делают чистый `skip`.
## Грабли
+14 -6
View File
@@ -84,7 +84,7 @@ type Searcher interface {
компилируется.
- `Read()` — первый зарегистрированный из `readOrder`:
`postgres` → `rest` → `dav`.
`postgres` → `mysql` → `rest` → `dav`.
- `Write()` — первый из `writeOrder`: `rest` → `dav`. SQL не пишет.
- `Search()` — первый из `searchOrder`: `elasticsearch`. Нет бэкенда →
ошибка (`ONLYOFFICE_ES_URL`).
@@ -103,14 +103,22 @@ Fallback:
каждый `c.Files()`, регистрируй на том же экземпляре.
```go
sql, err := c.SQLFileStore() // открыть из env (ONLYOFFICE_DSN)
if err != nil { /* нет DSN */ }
if closer, ok := sql.(interface{ Close() error }); ok { defer closer.Close() }
f := c.Files()
if pg, err := onlyoffice.NewPGStore(onlyoffice.PGConfigFromEnv()); err == nil {
defer pg.Close()
f.RegisterStore(onlyoffice.ProviderPG, pg)
}
entries, _ := f.List(ctx, "649") // пойдёт в SQL
f.RegisterStore(onlyoffice.ProviderPG, sql) // или sql.Name() == "mysql"
e, _ := f.Stat(ctx, "19423") // e.Provider == "mysql"
entries, _ := f.List(ctx, "676") // пойдёт в SQL
```
`Client.FileStore("pg"|"sql"|"postgres"|"mysql")` — одноразовый доступ к
SQL-стору без фасада: открывает из env; при ошибке возвращает заглушку,
которая отдаёт ошибку открытия на каждом вызове (не `nil`). `SQLFileStore()`
— тот же открыватель, но с ошибкой. Отвечавший бэкенд видно по
`Entry.Provider` (`mysql` / `postgres` у SQL, `rest` у REST).
## CLI
```bash
+48 -3
View File
@@ -108,18 +108,63 @@ type Searcher interface {
Name() string
}
// FileStore returns the adapter for a backend name: ProviderREST (default) or
// ProviderDAV. Unknown or empty names select the REST backend. The composed
// facade (backend selection/fallback) lives on FileClient in file_facade.go.
// FileStore returns the adapter for a backend name: ProviderREST (default),
// ProviderDAV (alias "webdav") or the read-only SQL store (ProviderPG,
// ProviderMySQL and the aliases "pg"/"sql"). The SQL store is opened from the
// environment (ONLYOFFICE_DSN / ONLYOFFICE_PG_*); when it cannot be opened the
// returned store surfaces that error on every operation instead of returning
// nil. Use SQLFileStore when the open error itself is needed. Unknown or empty
// names select the REST backend. The composed facade (backend
// selection/fallback) lives on FileClient in file_facade.go.
func (c *Client) FileStore(backend string) FileStore {
switch strings.ToLower(strings.TrimSpace(backend)) {
case ProviderDAV, "webdav":
return &davStore{c: c}
case ProviderPG, ProviderMySQL, "pg", "sql":
s, err := c.SQLFileStore()
if err != nil {
return &errStore{name: strings.ToLower(strings.TrimSpace(backend)), err: err}
}
return s
default:
return &restStore{c: c}
}
}
// errStore is the FileStore placeholder returned when a backend cannot be
// opened (for example SQL without a DSN). Every operation returns the recorded
// error instead of panicking on a nil interface.
type errStore struct {
name string
err error
}
func (s *errStore) Name() string { return s.name }
func (s *errStore) List(context.Context, string) ([]Entry, error) { return nil, s.err }
func (s *errStore) Stat(context.Context, string) (Entry, error) { return Entry{}, s.err }
func (s *errStore) CreateFolder(context.Context, string, string) (Entry, error) {
return Entry{}, s.err
}
func (s *errStore) Upload(context.Context, string, string, io.Reader) (Entry, error) {
return Entry{}, s.err
}
func (s *errStore) Download(context.Context, string, io.Writer) (int64, error) {
return 0, s.err
}
func (s *errStore) Move(context.Context, []string, string) error { return s.err }
func (s *errStore) Copy(context.Context, []string, string) error { return s.err }
func (s *errStore) Rename(context.Context, string, string) error { return s.err }
func (s *errStore) Delete(context.Context, []string) error { return s.err }
// Files returns the composed file facade. The returned *FileClient implements
// FileStore, so callers that used Files() as the plain REST store keep working.
func (c *Client) Files() *FileClient { return c.newFileClient() }
+3 -3
View File
@@ -45,7 +45,7 @@ func (c *Client) newFileClient() *FileClient {
ProviderDAV: &davStore{c: c},
},
searchers: map[string]Searcher{},
readOrder: []string{ProviderPG, ProviderREST, ProviderDAV},
readOrder: []string{ProviderPG, ProviderMySQL, ProviderREST, ProviderDAV},
writeOrder: []string{ProviderREST, ProviderDAV},
searchOrder: []string{ProviderES},
}
@@ -88,8 +88,8 @@ func (f *FileClient) RegisterSearcher(name string, s Searcher) {
f.searchers[name] = s
}
// Read returns the preferred backend for reads: PostgreSQL when registered,
// then REST, then WebDAV.
// Read returns the preferred backend for reads: the SQL store (PostgreSQL or
// MySQL) when registered, then REST, then WebDAV.
func (f *FileClient) Read() FileStore { return f.firstStore(f.readOrder) }
// Write returns the preferred backend for writes: REST, then WebDAV.
+38
View File
@@ -212,6 +212,44 @@ func TestFileClientWriteWithoutBackend(t *testing.T) {
}
}
func TestNewFileClientReadOrderIncludesSQL(t *testing.T) {
f := NewClient(Credentials{}).newFileClient()
want := []string{ProviderPG, ProviderMySQL, ProviderREST, ProviderDAV}
if fmt.Sprint(f.readOrder) != fmt.Sprint(want) {
t.Fatalf("readOrder = %v, want %v", f.readOrder, want)
}
}
// TestClientFileStoreSQLRoutingWithoutDSN checks that the SQL backend names are
// recognised and never yield nil: without a DSN the returned store surfaces the
// open error on use.
func TestClientFileStoreSQLRoutingWithoutDSN(t *testing.T) {
t.Setenv("ONLYOFFICE_DSN", "")
t.Setenv("ONLYOFFICE_PG_HOST", "")
c := NewClient(Credentials{})
for _, name := range []string{"pg", "sql", ProviderPG, ProviderMySQL} {
s := c.FileStore(name)
if s == nil {
t.Fatalf("FileStore(%q) = nil", name)
}
if _, err := s.Stat(context.Background(), "1"); err == nil {
t.Errorf("FileStore(%q).Stat without DSN: want error", name)
}
}
}
func TestFileClientMySQLStoreIsPreferredForReads(t *testing.T) {
mysql := &fakeStore{name: ProviderMySQL}
f := newFacadeTestClient(
map[string]FileStore{ProviderREST: &fakeStore{name: ProviderREST}, ProviderMySQL: mysql},
[]string{ProviderPG, ProviderMySQL, ProviderREST},
[]string{ProviderREST},
)
if got := f.Read().Name(); got != ProviderMySQL {
t.Errorf("Read().Name() = %q, want %q", got, ProviderMySQL)
}
}
func TestFileClientSearchSelection(t *testing.T) {
f := &FileClient{searchers: map[string]Searcher{}, searchOrder: []string{ProviderES}}
_, err := f.Search()
+9
View File
@@ -90,6 +90,15 @@ func pgDSNFromParts() string {
host, port, os.Getenv("ONLYOFFICE_PG_USER"), os.Getenv("ONLYOFFICE_PG_PASSWORD"), dbname, sslmode)
}
// SQLFileStore opens the read-only SQL store from the environment
// (PGConfigFromEnv: ONLYOFFICE_DSN or the ONLYOFFICE_PG_* parts). It is the
// error-aware counterpart of Client.FileStore("pg"/"sql"), which returns an
// errStore when the open fails. The caller owns the returned store and should
// close it (the concrete type has a Close method).
func (c *Client) SQLFileStore() (FileStore, error) {
return NewPGStore(PGConfigFromEnv())
}
// pgStore is a read-only FileStore/Searcher over the Community Server database.
type pgStore struct {
db *sql.DB
+96 -11
View File
@@ -93,6 +93,14 @@ func TestIntegrationPGStore(t *testing.T) {
}
}
// Download reads the object store, not the database, so it only runs with
// the MinIO credentials configured (the portal's S3 layout). Without them
// the DSN-only assertions above still prove the SQL reads.
if os.Getenv("MINIO_ACCESS_KEY") == "" || os.Getenv("MINIO_SECRET_KEY") == "" {
t.Log("MINIO_ACCESS_KEY/MINIO_SECRET_KEY not set — SQL download cross-check skipped")
return
}
var buf bytes.Buffer
n, err := store.Download(ctx, fileID, &buf)
if err != nil {
@@ -101,17 +109,94 @@ func TestIntegrationPGStore(t *testing.T) {
if n == 0 || n != dbFile.Size {
t.Errorf("sql Download = %d bytes, stat says %d", n, dbFile.Size)
}
if os.Getenv("MINIO_ACCESS_KEY") != "" && os.Getenv("MINIO_SECRET_KEY") != "" {
var restBuf bytes.Buffer
rn, err := rest.Download(ctx, fileID, &restBuf)
if err != nil {
t.Fatalf("rest Download(%s): %v", fileID, err)
}
if rn != n || !bytes.Equal(restBuf.Bytes(), buf.Bytes()) {
t.Errorf("download mismatch sql=%d rest=%d bytes", n, rn)
}
} else {
t.Log("MINIO_ACCESS_KEY/MINIO_SECRET_KEY not set — REST download cross-check skipped")
var restBuf bytes.Buffer
rn, err := rest.Download(ctx, fileID, &restBuf)
if err != nil {
t.Fatalf("rest Download(%s): %v", fileID, err)
}
if rn != n || !bytes.Equal(restBuf.Bytes(), buf.Bytes()) {
t.Errorf("download mismatch sql=%d rest=%d bytes", n, rn)
}
}
// TestIntegrationSQLFacade proves that reads are served by the SQL store when
// it is part of the file client, not by REST. It needs ONLYOFFICE_DSN plus
// ONLYOFFICE_PG_TEST_FILE_ID / ONLYOFFICE_PG_TEST_FOLDER_ID and the usual REST
// credentials (for the cross-check). Every Entry served by SQL carries
// Provider "mysql"; REST entries carry "rest", so the provider is the proof of
// which backend answered.
func TestIntegrationSQLFacade(t *testing.T) {
cfg := PGConfigFromEnv()
if strings.TrimSpace(cfg.DSN) == "" {
t.Skip("ONLYOFFICE_DSN not set — skipping SQL facade integration test")
}
fileID := strings.TrimSpace(os.Getenv("ONLYOFFICE_PG_TEST_FILE_ID"))
folderID := strings.TrimSpace(os.Getenv("ONLYOFFICE_PG_TEST_FOLDER_ID"))
if fileID == "" || folderID == "" {
t.Skip("ONLYOFFICE_PG_TEST_FILE_ID / ONLYOFFICE_PG_TEST_FOLDER_ID not set — skipping SQL facade integration test")
}
c := liveClient(t)
ctx := context.Background()
sqlStore, err := c.SQLFileStore()
if err != nil {
t.Fatalf("SQLFileStore: %v", err)
}
if closer, ok := sqlStore.(interface{ Close() error }); ok {
t.Cleanup(func() { _ = closer.Close() })
}
if sqlStore.Name() == ProviderREST {
t.Fatalf("SQLFileStore returned REST")
}
// Direct constructor: Client.FileStore("pg") must not be REST either.
direct := c.FileStore("pg")
if direct == nil || direct.Name() == ProviderREST {
t.Fatalf("FileStore(\"pg\") = %v, want SQL backend", direct)
}
if closer, ok := direct.(interface{ Close() error }); ok {
t.Cleanup(func() { _ = closer.Close() })
}
f := c.Files()
f.RegisterStore(ProviderPG, sqlStore)
if got := f.Read().Name(); got != sqlStore.Name() {
t.Fatalf("facade read backend = %q, want %q (SQL)", got, sqlStore.Name())
}
got, err := f.Stat(ctx, fileID)
if err != nil {
t.Fatalf("facade Stat(%s): %v", fileID, err)
}
if got.Provider != ProviderMySQL {
t.Errorf("facade Stat provider = %q, want %q (SQL, not REST)", got.Provider, ProviderMySQL)
}
want, err := c.FileStore(ProviderREST).Stat(ctx, fileID)
if err != nil {
t.Fatalf("rest Stat(%s): %v", fileID, err)
}
if got.ID != want.ID || got.Title != want.Title || got.ParentID != want.ParentID {
t.Errorf("facade SQL stat %+v != REST %+v", got, want)
}
list, err := f.List(ctx, folderID)
if err != nil {
t.Fatalf("facade List(%s): %v", folderID, err)
}
entry := entryByID(list, fileID)
if entry == nil {
t.Fatalf("file %s not in facade List(%s)", fileID, folderID)
}
if entry.Provider != ProviderMySQL {
t.Errorf("facade List provider = %q, want %q", entry.Provider, ProviderMySQL)
}
d, err := direct.Stat(ctx, fileID)
if err != nil {
t.Fatalf("FileStore(\"pg\").Stat(%s): %v", fileID, err)
}
if d.Provider != ProviderMySQL {
t.Errorf("FileStore(\"pg\") provider = %q, want %q", d.Provider, ProviderMySQL)
}
}