eSliderandGitHub
a331042488
OQ3: internal/duckstats + bin/qa/stats.go. Zig stays Ladybug-only. mikefarah/yq for small structured slices. Gitea #30.
39 lines
1.3 KiB
Markdown
39 lines
1.3 KiB
Markdown
---
|
|
name: duckdb
|
|
description: >-
|
|
Use https://github.com/duckdb/duckdb-go in-process for columnar analytics
|
|
(quantiles, GROUP BY, JSON/CSV/Parquet/JSONL scans) when that is faster than
|
|
nested Go loops. Not Ladybug. Not the web-search sqlite cache. Use when
|
|
aggregating samples, counting JSONL, or SQL over tabular files.
|
|
---
|
|
|
|
# duckdb-go
|
|
|
|
Use https://github.com/duckdb/duckdb-go where it makes sense to get better performance in code.
|
|
|
|
In-process DuckDB (`internal/duckstats`, `database/sql` driver `duckdb`).
|
|
Vectorized SQL over tables, JSONL, CSV, Parquet. CGO with bundled libs
|
|
(linux/darwin amd64/arm64). Links with **gcc/g++** (libstdc++), not Zig.
|
|
D21 Zig (`bin/cgo/zcc`) is Ladybug/tokenizers only. After
|
|
`eval "$(bin/cgo/zig env)"`:
|
|
|
|
```bash
|
|
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= ./bin/qa/stats.go <<< '[1,2,3,4,5]'
|
|
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go test ./internal/duckstats
|
|
```
|
|
|
|
| Store | Job |
|
|
|-------|-----|
|
|
| Ladybug | graph + FTS + HNSW (facts/info) |
|
|
| modernc sqlite | web-search KV cache + throttle |
|
|
| duckdb-go | OLAP: quantiles, counts, scans of many rows/files |
|
|
| mikefarah/yq | small YAML/JSON/XML/CSV/TOML/HCL slice, not bulk |
|
|
|
|
```bash
|
|
./bin/qa/stats.go <<< '[1,2,3,4,5]'
|
|
./bin/qa/stats.go --jsonl path/to/rows.jsonl
|
|
```
|
|
|
|
Do not open Ladybug through DuckDB. Do not put secrets or client PII into
|
|
DuckDB files under the repo.
|