feat: DuckDB quantiles in-process (gcc CGO), not Ladybug. (#34)
OQ3: internal/duckstats + bin/qa/stats.go. Zig stays Ladybug-only. mikefarah/yq for small structured slices. Gitea #30.
This commit is contained in:
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: duckdb
|
||||
description: >-
|
||||
Use https://github.com/duckdb/duckdb-go in-process for columnar analytics
|
||||
(quantiles, GROUP BY, JSON/CSV/Parquet/JSONL scans) when that is faster than
|
||||
nested Go loops. Not Ladybug. Not the web-search sqlite cache. Use when
|
||||
aggregating samples, counting JSONL, or SQL over tabular files.
|
||||
---
|
||||
|
||||
# duckdb-go
|
||||
|
||||
Use https://github.com/duckdb/duckdb-go where it makes sense to get better performance in code.
|
||||
|
||||
In-process DuckDB (`internal/duckstats`, `database/sql` driver `duckdb`).
|
||||
Vectorized SQL over tables, JSONL, CSV, Parquet. CGO with bundled libs
|
||||
(linux/darwin amd64/arm64). Links with **gcc/g++** (libstdc++), not Zig.
|
||||
D21 Zig (`bin/cgo/zcc`) is Ladybug/tokenizers only. After
|
||||
`eval "$(bin/cgo/zig env)"`:
|
||||
|
||||
```bash
|
||||
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= ./bin/qa/stats.go <<< '[1,2,3,4,5]'
|
||||
CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go test ./internal/duckstats
|
||||
```
|
||||
|
||||
| Store | Job |
|
||||
|-------|-----|
|
||||
| Ladybug | graph + FTS + HNSW (facts/info) |
|
||||
| modernc sqlite | web-search KV cache + throttle |
|
||||
| duckdb-go | OLAP: quantiles, counts, scans of many rows/files |
|
||||
| mikefarah/yq | small YAML/JSON/XML/CSV/TOML/HCL slice, not bulk |
|
||||
|
||||
```bash
|
||||
./bin/qa/stats.go <<< '[1,2,3,4,5]'
|
||||
./bin/qa/stats.go --jsonl path/to/rows.jsonl
|
||||
```
|
||||
|
||||
Do not open Ladybug through DuckDB. Do not put secrets or client PII into
|
||||
DuckDB files under the repo.
|
||||
@@ -1,15 +1,15 @@
|
||||
---
|
||||
name: picoclaw
|
||||
description: >-
|
||||
2dph is the memory/fact gate, not the agent loop. Use when wiring PicoClaw
|
||||
or any MCP client: call brain search/get/audit before a factual reply.
|
||||
throttled is not a negative finding.
|
||||
2dph is the memory/fact gate. Compose runs the official PicoClaw gateway.
|
||||
Use when wiring PicoClaw or any MCP client: call brain search/get/audit
|
||||
before a factual reply. throttled is not a negative finding.
|
||||
---
|
||||
|
||||
# PicoClaw — fact-check before assert
|
||||
|
||||
PicoClaw (or any agent) speaks MCP at `POST /mcp` on `bin/brain/serve.go`.
|
||||
2dph does not run the agent loop. Compose: `docker compose --profile picoclaw up brain-mcp`
|
||||
PicoClaw speaks MCP at `POST /mcp` on `bin/brain/serve.go`. Compose profile
|
||||
`picoclaw` runs the official `sipeed/picoclaw` gateway plus `brain-mcp`
|
||||
(see [docs/picoclaw.md](../../docs/picoclaw.md)).
|
||||
|
||||
## Tool order (before a factual reply)
|
||||
|
||||
@@ -9,7 +9,7 @@ description: >-
|
||||
# postgres
|
||||
|
||||
`bin/postgres/query.go` wraps vendored `bin/db/psql-yq`. Output is YAML
|
||||
(cheaper than psql ASCII, easy to slice with `yq`).
|
||||
(cheaper than psql ASCII, easy to slice with mikefarah/yq).
|
||||
|
||||
```bash
|
||||
bin/postgres/query.go --profile onlyoffice -s document_asset # column list
|
||||
|
||||
@@ -11,7 +11,7 @@ description: >-
|
||||
```bash
|
||||
bin/web/search.go "LadybugDB vector index"
|
||||
bin/web/search.go "model2vec multilingual" --category it
|
||||
bin/web/search.go "hypervisor" --site example.com --json | jq -r '.results[].url'
|
||||
bin/web/search.go "hypervisor" --site example.com --json | yq -r '.results[].url'
|
||||
bin/web/search.go "postgres partial index" --lang en --fresh year
|
||||
```
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ found" unless the client refuses to call it absence.
|
||||
|
||||
```bash
|
||||
for i in $(seq 10); do
|
||||
bin/web/search.go "test $i" -n 1 --refresh --json | jq -r .status
|
||||
bin/web/search.go "test $i" -n 1 --refresh --json | yq -r '.status'
|
||||
done
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
name: yq
|
||||
description: >-
|
||||
Use https://github.com/mikefarah/yq to work with YAML, JSON, XML, CSV,
|
||||
TOML, HCL where it's efficient and less code. Use when slicing compose,
|
||||
config, --json tool output, CSV/TOML/HCL/XML, or converting between those
|
||||
formats. Not kislyuk Python yq. Not jq when yq already does the job.
|
||||
---
|
||||
|
||||
# yq (mikefarah)
|
||||
|
||||
Use https://github.com/mikefarah/yq to work with YAML, JSON, XML, CSV, TOML, HCL where it's efficient and less code.
|
||||
|
||||
This is the Go `yq` (`yq --version` contains `mikefarah`). It is not
|
||||
kislyuk/yq (Python, jq-syntax, YAML-only wrapper). `bin/db/psql-yq` already
|
||||
calls this binary.
|
||||
|
||||
Prefer `yq` over `python3 -c`, `jq`, or ad-hoc parsers when one expression
|
||||
reads or converts the file. Keep Python/Go for HTTP, binary protocols, and
|
||||
in-process tests.
|
||||
|
||||
```bash
|
||||
yq '.services.picoclaw.image' compose.yaml
|
||||
yq -P . deploy/picoclaw/config.json # JSON → YAML
|
||||
yq -o=json '.gates' # JSON stdin (qa/system_perf.py --json)
|
||||
yq -p=csv -o=json .
|
||||
yq -p=xml -o=json .
|
||||
yq -p=toml '.package.name' file.toml
|
||||
bin/brain/search.go "LadybugDB" --json | yq '.[].ref'
|
||||
bin/web/search.go "hypervisor" --json | yq -r '.results[].url'
|
||||
```
|
||||
|
||||
Do not print secrets, PII, or `$HOME/.config/brain/` through `yq`.
|
||||
Reference in New Issue
Block a user