From d4a88eead73d35c8db806a1e2e489cba23014069 Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Mon, 10 Aug 2026 20:50:25 +0100 Subject: [PATCH] feat(skills): integrate web-search, db-yaml, agent-cost, diataxis-docs, kb-search; retire searxng --- bin/db/psql-yq | 1 + bin/web/search | 1 + skills/agent-cost/SKILL.md | 34 ++++++++++ skills/db-yaml/SKILL.md | 40 +++++++++++ skills/diataxis-docs/SKILL.md | 55 +++++++++++++++ skills/kb-search/SKILL.md | 45 +++++++++++++ skills/web-search/SKILL.md | 51 ++++++++++++++ .../web-search/reference/instance-tuning.md | 67 +++++++++++++++++++ 8 files changed, 294 insertions(+) create mode 120000 bin/db/psql-yq create mode 120000 bin/web/search create mode 100644 skills/agent-cost/SKILL.md create mode 100644 skills/db-yaml/SKILL.md create mode 100644 skills/diataxis-docs/SKILL.md create mode 100644 skills/kb-search/SKILL.md create mode 100644 skills/web-search/SKILL.md create mode 100644 skills/web-search/reference/instance-tuning.md diff --git a/bin/db/psql-yq b/bin/db/psql-yq new file mode 120000 index 0000000..f625f1e --- /dev/null +++ b/bin/db/psql-yq @@ -0,0 +1 @@ +/mnt/8TB/projects/ai/agent-skills/bin/db/psql-yq \ No newline at end of file diff --git a/bin/web/search b/bin/web/search new file mode 120000 index 0000000..db76e68 --- /dev/null +++ b/bin/web/search @@ -0,0 +1 @@ +/mnt/8TB/projects/ai/agent-skills/bin/web/search \ No newline at end of file diff --git a/skills/agent-cost/SKILL.md b/skills/agent-cost/SKILL.md new file mode 100644 index 0000000..cef3394 --- /dev/null +++ b/skills/agent-cost/SKILL.md @@ -0,0 +1,34 @@ +--- +name: agent-cost +description: >- + Measure what an agent session actually costs in tokens using bin/agents/cost. + Use before and after changing documentation, skills or context layout, and when + a session feels unexpectedly expensive. +--- + +# agent-cost + +```bash +bin/agents/cost # every project, YAML +bin/agents/cost --repo brain-ui # only sessions whose cwd matches +bin/agents/cost --json | jq .cursor.by_tool +bin/agents/cost --snapshot after-x --repo brain-ui # append a row to docs/CONTEXT-BUDGET.md +``` + +Reads local session storage from OpenCode and Cursor transcripts. Reports the +always-loaded baseline, cache hit/miss/thrash, and which tools moved the most +bytes. + +## How to read it + +- **Baseline** is what every single message pays for: `AGENTS.md` plus anything + eagerly linked from it. Keep it small; it multiplies by message count. +- **Cache thrash** matters more than raw size. Editing a file that sits early in + the context invalidates the prompt cache for the whole session. +- **by_tool bytes** shows where the real spend is. Usually it is unfiltered + command output, not documentation. + +## Rule + +Measure before and after. A claim that something "reduces tokens" without a +before and an after number is an opinion, not a result. diff --git a/skills/db-yaml/SKILL.md b/skills/db-yaml/SKILL.md new file mode 100644 index 0000000..c27c1bc --- /dev/null +++ b/skills/db-yaml/SKILL.md @@ -0,0 +1,40 @@ +--- +name: db-yaml +description: >- + Read any Postgres as compact YAML through db/psql-yq, with a read-only guard and + named profiles. Use when a task needs table contents, column types or a SELECT + against cs_brain or another project database. +--- + +# db-yaml + +`agent-skills/bin/db/psql-yq` talks to Postgres and returns YAML, which is far +cheaper than a psql ASCII table and easy to slice with `yq`. + +```bash +bin/db/offline -s kunde # column list (brain-ui wrapper) +bin/db/offline -t tour -l 20 # 20 sample rows as YAML +bin/db/offline -c 'SELECT ...' # query -> YAML +bin/db/live -c 'SELECT ...' # live database instead of the snapshot +``` + +Ad-hoc targets without a profile: + +```bash +agent-skills/bin/db/psql-yq --container my-pg --db app -c 'SELECT 1' +agent-skills/bin/db/psql-yq --dsn 'postgres://user@host:5432/db' -c 'SELECT 1' +``` + +## Profiles + +Connection details live in `~/.config/brain/db-profiles.yml` (mode 600), never in a +project repo. A profile names either a `container` or a `host`; passwords are read +from a separate `password_env_file` and never appear in argv. + +## Rules + +- **Read-only.** Any `insert|update|delete|drop|truncate|alter|create|grant| + revoke|vacuum|copy` is rejected with exit 3. Do not work around it. +- **PII.** `cs_brain` holds client data. Aggregate and count freely; never copy + names or addresses into chat, issues or docs. +- Use `-l` to keep samples small. Twenty rows answer most questions. diff --git a/skills/diataxis-docs/SKILL.md b/skills/diataxis-docs/SKILL.md new file mode 100644 index 0000000..8a0dcdd --- /dev/null +++ b/skills/diataxis-docs/SKILL.md @@ -0,0 +1,55 @@ +--- +name: diataxis-docs +description: >- + Classify and structure documentation by Diataxis (tutorial, howto, reference, + explanation) with PARA status, using frontmatter that the knowledge-base index + reads. Use when creating a document, when one file tries to do two jobs, or when + auditing a docs folder. +--- + +# diataxis-docs + +Every markdown file answers exactly one of four questions. A file that answers +two should be split; that is the whole method. + +| Type | Answers | Reader is | +|------|---------|-----------| +| `tutorial` | "teach me by doing" | learning | +| `howto` | "help me do this task" | working | +| `reference` | "tell me the facts" | looking up | +| `explanation` | "help me understand why" | studying | + +## Frontmatter + +```yaml +--- +type: reference # one of the four above +status: current # current | archive (PARA) +related: + - docs/OPS.md +--- +``` + +`bin/kb/index` reads this. `type` becomes a searchable column and `related` +becomes a graph edge: + +```bash +bin/kb/search search "deploy" --type howto +bin/kb/search search "Stecktafel" --hop 1 # follow links and related +``` + +## Audit checklist + +1. Does the title match the type? A reference that starts "first, open..." is a + howto wearing a disguise. +2. More than one H2 topic that a reader would search separately? Split the file. + The indexer chunks on H2, so split files also search better. +3. Is `status: archive` set on anything superseded? Archived files stay indexed + but stop competing with current ones for a reader's attention. +4. Does every explanation link the reference it explains, and vice versa? That + link is what `--hop 1` walks. + +## Rule + +Do not invent a fifth type. If a document does not fit, it is usually two +documents. diff --git a/skills/kb-search/SKILL.md b/skills/kb-search/SKILL.md new file mode 100644 index 0000000..3800da3 --- /dev/null +++ b/skills/kb-search/SKILL.md @@ -0,0 +1,45 @@ +--- +name: kb-search +description: >- + Deduction search over the 2dph brain (Ladybug graph: ops corpus, portfolio, + ssh hosts) with bin/kb/search instead of reading files or grepping repos. + Use whenever a question starts with "where is", "what runs on", "which file + describes", "who is", "how is X done", before opening any documentation. +--- + +# kb-search — deduction over facts and info + +One embedded Ladybug graph (`var/kb.lbug`, read-only when queried) holding two +roots: + +- **facts** — assertions backed by ≥2 independent sources (docker ps × compose + × ssh-config × docs), `confidence: confirmed`. +- **info** — descriptive/narrative leafs, searchable, never asserted. + +Search = deduction: facts root first, info root second, `web-search` as the +second independent source when local roots cannot confirm. An answer is +`confirmed` only if it comes off the facts root; anything else is +`(not confirmed)`. + +```bash +bin/kb/search "Matrix federation" # pointers + snippets, YAML +bin/kb/search "what runs on arc-2" --hop 1 # follow graph edges +bin/kb/search "onlyoffice postgres" --root facts # restrict to confirmed +bin/kb/search "where is cs-lexicon" --json | yq '.[].ref' +bin/kb/get --body # full chunk only when needed +bin/kb/stats # index health +bin/kb/eval # recall@5 >= 0.95 gate +``` + +## Rules + +- Search before you read. Never grep a repo for a concept the graph covers. +- `--root facts` returns only confirmed evidence-linked answers. Default shows + facts first, then info leafs clearly marked `(not confirmed)`. +- If recall looks wrong, run `bin/kb/eval`; it gates control questions and + should stay at or above 95% recall@5. +- `--hop N` follows sibling leaves, owning files, `related:` links and + vector-neighbours — that is the deduction walk, not random expansion. +- Escalate to `web-search` (the `web-search` skill) as the independent second + source when both local roots cannot confirm; never report an unconfirmed + single-source local answer as fact. \ No newline at end of file diff --git a/skills/web-search/SKILL.md b/skills/web-search/SKILL.md new file mode 100644 index 0000000..b22c1c4 --- /dev/null +++ b/skills/web-search/SKILL.md @@ -0,0 +1,51 @@ +--- +name: web-search +description: >- + Search the public web through the self-hosted SearXNG at search.ops.io + using bin/web/search. Use for German care law, SGB paragraphs, vendor + documentation and any fact that is not in our own repos - and as the second + independent source the detective method requires. +--- + +# web-search + +```bash +bin/web/search "Pflegegrad SGB XI Einstufung" +bin/web/search "Toureffizienz" --site ticket.detective.de +bin/web/search "sqlite-vec" --category it -n 3 --json | jq -r '.results[].url' +bin/web/search "Pflegereform" --lang de --fresh year +``` + +## Web or knowledge base + +`bin/kb/search` holds our own facts: product docs, detective business logic, the +lexicon. Go there first. Reach for `bin/web/search` when the answer is outside +our repos: legislation, vendor documentation, upstream library behaviour. + +Keep the two apart. A finding is stronger when the reader can see that one +source was ours and one was not. + +## PII: this query leaves the host + +The search goes to external engines. Never put a client or staff identifier in +it. The tool refuses long digit runs, `Personalnummer`, `KV-Nr`, dates of birth +and street-with-number, and exits 2. Rephrase rather than reaching for `--force`. + +## Read the status, not just the results + +The instance answers HTTP 200 with an empty list when it throttles, so an empty +answer is ambiguous by construction. The tool resolves that for you: + +| status | exit | meaning | +|--------|------|---------| +| `ok` | 0 | engines answered | +| `throttled` | 3 | nobody answered; say nothing about what exists | + +Never turn a `throttled` result into "there is no information about X". + +## Etiquette + +Calls are serialised host-wide and kept ten seconds apart, and answers are +cached for seven days. Do not loop over queries, and do not use `--refresh` +unless the cached answer is genuinely stale: a burst suspends the engines for +several minutes for everyone. diff --git a/skills/web-search/reference/instance-tuning.md b/skills/web-search/reference/instance-tuning.md new file mode 100644 index 0000000..510d0a2 --- /dev/null +++ b/skills/web-search/reference/instance-tuning.md @@ -0,0 +1,67 @@ +--- +type: howto +status: current +--- + +# Tuning the SearXNG instance + +The client works around a fragile instance. These changes fix the cause, and +they need shell access to the host behind `search.ops.io` +(`90.169.228.16` / `ops.mywire.org`), which is a different machine from +the one the agents run on. + +## Why it is needed + +Measured on 2026-08-10 from this host: + +- The default engine set for the `general` category is only `duckduckgo`, + `brave` and `startpage`. `brave` and `startpage` sit in + `Suspended: too many requests` or `Suspended: CAPTCHA` almost permanently, so + in practice a single engine carries every query. +- About 25 probe requests over a few minutes pushed `duckduckgo` into `CAPTCHA` + as well. The instance then answered HTTP 200 with `results: []` and an empty + `unresponsive_engines` - indistinguishable from "nothing found" without the + client-side handling we added. +- Recovery took roughly six minutes. + +## Changes + +1. **Allow our egress IP through the limiter.** In `limiter.toml`: + + ```toml + [botdetection.ip_lists] + pass_ip = ["77.7.46.234"] + ``` + +2. **Shorten the suspensions.** In `settings.yml` the defaults are 24 hours for + a CAPTCHA and one hour for too-many-requests, which is far longer than the + condition lasts: + + ```yaml + search: + suspended_times: + SearxEngineCaptcha: 300 + SearxEngineTooManyRequests: 120 + SearxEngineAccessDenied: 300 + ``` + +3. **Give `general` more than one working engine.** `google` and `wikipedia` + report `enabled: true` in `/config` yet never appear in a `general` response, + so they are not in the default set. Put them in it; one live engine per + category is a single point of failure. + +4. **Keep the JSON API on.** `formats: [html, json]` must stay, otherwise every + client here breaks. + +## Verifying + +Ten requests in a row used to suspend the instance for minutes. After the +change they should all answer: + +```bash +for i in $(seq 10); do + bin/web/search "test $i" -n 1 --refresh --json | jq -r .status +done +``` + +Ten lines of `ok` means it is fixed.