From 368a757612ef4dbce11e4ab33a7b1d883ea5ef23 Mon Sep 17 00:00:00 2001 From: Andrey Oblivantsev Date: Thu, 13 Aug 2026 19:53:31 +0100 Subject: [PATCH] feat: Go SearXNG client; throttled is not absence (#16) --- AGENTS.md | 4 +- PLAN.md | 5 +- README.md | 8 + bin/tools/test_bin_layout.py | 5 + bin/tools/test_published_docs.py | 14 ++ bin/web/search | 148 +---------- bin/web/search.go | 232 ++++++++++++++++++ compose.yaml | 15 ++ deploy/searxng/limiter.toml | 7 + deploy/searxng/settings.yml | 28 +++ docs/design.md | 1 + go.mod | 10 +- go.sum | 50 +++- internal/websearch/cache.go | 97 ++++++++ internal/websearch/fetch.go | 90 +++++++ internal/websearch/fetch_test.go | 77 ++++++ internal/websearch/websearch.go | 205 ++++++++++++++++ internal/websearch/websearch_test.go | 203 +++++++++++++++ skills/web-search/SKILL.md | 23 +- .../web-search/reference/instance-tuning.md | 72 +++--- 20 files changed, 1099 insertions(+), 195 deletions(-) create mode 100755 bin/web/search.go create mode 100644 deploy/searxng/limiter.toml create mode 100644 deploy/searxng/settings.yml create mode 100644 internal/websearch/cache.go create mode 100644 internal/websearch/fetch.go create mode 100644 internal/websearch/fetch_test.go create mode 100644 internal/websearch/websearch.go create mode 100644 internal/websearch/websearch_test.go diff --git a/AGENTS.md b/AGENTS.md index 404a758..0dd5be5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,7 +42,8 @@ bin/mail/ sync.go import.go (index_mail → brain/index.go) bin/markdown/ import.go (mistune leafs) bin/postgres/ query.go (read-only YAML) bin/git/ import.go (go-git history; Python shim execs it) -internal/ shared Go (brain/rank is cgo-free; chats parsers; gitlog) +bin/web/ search.go (SearXNG; Python shim execs it) +internal/ shared Go (brain/rank is cgo-free; chats parsers; gitlog; websearch) bin/watch/ corpus watcher (used by bin/brain/watch.go) bin/tools/ vendored python libs behind bin/* (kblib, yamlout, websearch) bin/docker-entrypoint container entrypoint (brain index|search|serve|watch) @@ -83,6 +84,7 @@ bin/brain/search.go "query" [--root facts|info] # deduction search → YAML bin/brain/get.go [--body] bin/markdown/import.go [dir] # mistune leaves → YAML bin/git/import.go [REPO] [--json] [--limit N] # go-git history → commit leafs +bin/web/search.go "query" [--json] # SearXNG; throttled ≠ absence bin/postgres/query.go --profile onlyoffice -c 'SELECT 1' bin/md/tables # what the graph holds → YAML bin/brain/deduce "question" # thinking wrapper diff --git a/PLAN.md b/PLAN.md index 4f4052d..44a26d2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -26,7 +26,7 @@ detective method: **a fact needs ≥2 independent sources or it is |---|----------|--------| | D1 | RAG corpus | ops stack (chat, onlyoffice, gitea/NPM, searchxng, observability, ai-bot, mcp-servers, `~/.ssh/config`) + portfolio. Exclude `office.dev` + jobs/applications. | | D2 | skill merging | integrate skills **in this project** `skills/`; skip gitea / brain-dependent skills. | -| D3 | web search | Vendored client; SearXNG URL is config. Optional Compose instance (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. | +| D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. | | D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. | | D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). | | D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`); Python remains for index/write until the Go write path is safe. | @@ -63,11 +63,12 @@ detective method: **a fact needs ≥2 independent sources or it is markdown/import.go mistune leaves postgres/query.go read-only YAML (wraps bin/db/psql-yq) git/import.go go-git history (no git binary; conversion only) + web/search.go SearXNG client (throttled ≠ absence) chats/sync.go import.go facts.go apply.go (libs in internal/chats; no chats index) md/import (deprecated; bin/markdown/import.go) brain/extract brain/audit brain/deduce (thinking wrapper) - web/search (vendored) + web/search (deprecated shim → web/search.go) db/psql-yq (vendored) ssh-tunnel onlyoffice pg tunnel 5433 var/kb.lbug single embedded store (gitignored) diff --git a/README.md b/README.md index 6a92435..8480efa 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,14 @@ bin/git/import.go --root "$PROJECTS_ROOT" --json # one pass per .git under root Conversion only. Graph write (`File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`) stays with `bin/brain/index.go`. +Web search (second independent source) goes through SearXNG. Empty results mean **throttled**, not “nothing exists”: + +```bash +bin/web/search.go "LadybugDB vector index" --json +# Optional local instance (skip if BRAIN_SEARCH_URL already points at one): +# SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d +``` + Mail is a first-class corpus (retrievable through the same search): ```bash diff --git a/bin/tools/test_bin_layout.py b/bin/tools/test_bin_layout.py index b239dfb..db79ebc 100644 --- a/bin/tools/test_bin_layout.py +++ b/bin/tools/test_bin_layout.py @@ -102,6 +102,11 @@ class BinLayoutTest(unittest.TestCase): ) self.assertIn("bin/git/import.go", py) + def test_web_search_is_shebang(self) -> None: + self._assert_shebang("bin/web/search.go") + py = (ROOT / "bin" / "web" / "search").read_text() + self.assertIn("bin/web/search.go", py) + def test_gitimport_py_has_no_git_binary(self) -> None: py = (ROOT / "bin" / "tools" / "gitimport.py").read_text() self.assertNotIn("subprocess", py) diff --git a/bin/tools/test_published_docs.py b/bin/tools/test_published_docs.py index ef4408f..345af9b 100644 --- a/bin/tools/test_published_docs.py +++ b/bin/tools/test_published_docs.py @@ -45,6 +45,20 @@ class PublishedDocsTest(unittest.TestCase): self.assertIn("go-git", text) self.assertIn("D19", (ROOT / "PLAN.md").read_text()) + def test_web_search_is_go_not_ops_host(self) -> None: + readme = (ROOT / "README.md").read_text() + self.assertIn("bin/web/search.go", readme) + skill = (ROOT / "skills" / "web-search" / "SKILL.md").read_text() + self.assertIn("bin/web/search.go", skill) + self.assertNotIn("search.ops.io", skill) + self.assertNotIn("search.ops.io", readme) + compose = (ROOT / "compose.yaml").read_text() + self.assertIn("searxng", compose) + self.assertNotIn("search.ops.io", compose) + settings = (ROOT / "deploy" / "searxng" / "settings.yml").read_text() + self.assertNotIn("password", settings.lower()) + self.assertIn("json", settings) + def test_docs_do_not_claim_hop_walks(self) -> None: paths = [ ROOT / "README.md", diff --git a/bin/web/search b/bin/web/search index 87984b8..05218f9 100755 --- a/bin/web/search +++ b/bin/web/search @@ -1,150 +1,26 @@ #!/usr/bin/env python3 -"""web/search - web search through the self-hosted SearXNG at search.ops.io. +"""web/search — deprecated. Use bin/web/search.go (SearXNG, no Python client). - bin/web/search "LadybugDB vector search" - bin/web/search "model2vec multilingual" --site github.com - bin/web/search "uclancy" --category it -n 3 --json | jq -r '.results[].url' - bin/web/search "sqlite-vec" --refresh # ignore the cached answer - -This complements bin/kb/search: the knowledge base holds our own facts, this -reaches the public web. Use it as the second, independent source that the -detective method asks for. - -Exit codes: 0 results, 2 refused as possible PII, 3 throttled (not "nothing -found" - the instance answers 200 with an empty list when it throttles). + bin/web/search.go QUERY [--json] [-n N] [--site HOST] """ from __future__ import annotations -import argparse -import fcntl -import json import os import sys -import time -import urllib.parse -import urllib.request from pathlib import Path -TOOLS = Path(__file__).resolve().parents[1] / "tools" -sys.path.insert(0, str(TOOLS)) -sys.path.insert(0, str(TOOLS / "web-search")) - -import websearch as ws # noqa: E402 -from yamlout import to_yaml # noqa: E402 - -CONFIG = Path(os.environ.get("BRAIN_SEARCH_ENV", Path.home() / ".config/brain/search.env")) -CACHE = Path(os.environ.get("BRAIN_SEARCH_CACHE", Path.home() / ".cache/brain/web-search.sqlite")) -LOCK = CACHE.with_suffix(".lock") +ROOT = Path(__file__).resolve().parents[2] -def load_config() -> dict: - if not CONFIG.exists(): - sys.exit(f"no credentials at {CONFIG} (mode 600, BRAIN_SEARCH_URL/USER/PASS)") - conf = {} - for line in CONFIG.read_text().splitlines(): - line = line.strip() - if not line or line.startswith("#") or "=" not in line: - continue - key, _, value = line.partition("=") - conf[key.strip()] = value.strip().strip("\"'") - missing = {"BRAIN_SEARCH_URL", "BRAIN_SEARCH_USER", "BRAIN_SEARCH_PASS"} - conf.keys() - if missing: - sys.exit(f"{CONFIG} is missing {', '.join(sorted(missing))}") - return conf - - -def fetch(conf: dict, query: str, params: dict, timeout: int) -> dict: - args = {"q": query, "format": "json", **params} - url = f"{conf['BRAIN_SEARCH_URL'].rstrip('/')}/search?{urllib.parse.urlencode(args)}" - request = urllib.request.Request(url) - token = f"{conf['BRAIN_SEARCH_USER']}:{conf['BRAIN_SEARCH_PASS']}".encode() - import base64 - request.add_header("Authorization", "Basic " + base64.b64encode(token).decode()) - with urllib.request.urlopen(request, timeout=timeout) as response: - return json.loads(response.read().decode()) - - -def main() -> int: - parser = argparse.ArgumentParser(description="web search via SearXNG") - parser.add_argument("query") - parser.add_argument("-n", "--limit", type=int, default=ws.DEFAULT_LIMIT) - parser.add_argument("--site", help="restrict to one domain") - parser.add_argument("--lang", help="language code, e.g. de") - parser.add_argument("--fresh", choices=["day", "week", "month", "year"], - help="time range") - parser.add_argument("--category", help="SearXNG category, e.g. it, science, news") - parser.add_argument("--engines", help="comma separated engine list") - parser.add_argument("--json", action="store_true") - parser.add_argument("--refresh", action="store_true", help="bypass the cache") - parser.add_argument("--ttl", type=float, default=ws.CACHE_TTL) - parser.add_argument("--timeout", type=int, default=25) - parser.add_argument("--force", action="store_true", - help="send even if the query looks like PII") - args = parser.parse_args() - - query = f"site:{args.site} {args.query}" if args.site else args.query - - reason = ws.phi_reason(query) - if reason and not args.force: - print(f"refused: {reason}. This query would leave the host.", file=sys.stderr) - print("Rephrase without identifiers, or pass --force if it is genuinely public.", - file=sys.stderr) - return 2 - - params = {} - if args.lang: - params["language"] = args.lang - if args.fresh: - params["time_range"] = args.fresh - if args.category: - params["categories"] = args.category - if args.engines: - params["engines"] = args.engines - - key = ws.cache_key(query, params) - conn = ws.open_cache(CACHE) - - if not args.refresh: - cached = ws.cache_get(conn, key, ttl=args.ttl) - if cached is not None: - out = ws.project(cached, limit=args.limit) - out["cached"] = True - sys.stdout.write(json.dumps(out, indent=2, ensure_ascii=False) + "\n" - if args.json else to_yaml(out)) - return 0 - - conf = load_config() - LOCK.parent.mkdir(parents=True, exist_ok=True) - - # One request at a time across every agent on this host: the instance - # suspends engines for minutes when several of us ask at once. - with open(LOCK, "w") as lock: - fcntl.flock(lock, fcntl.LOCK_EX) - - payload = None - for attempt in range(1 + len(ws.RETRY_BACKOFF)): - delay = ws.wait_for(ws.last_call(conn), time.time()) - if delay: - time.sleep(delay) - ws.mark_call(conn) - try: - payload = fetch(conf, query, params, args.timeout) - except Exception as error: # noqa: BLE001 - report, do not crash - print(f"request failed: {error}", file=sys.stderr) - return 3 - if ws.classify(payload) == "ok": - break - if attempt < len(ws.RETRY_BACKOFF): - time.sleep(ws.RETRY_BACKOFF[attempt]) - - if ws.classify(payload) == "ok": - ws.cache_put(conn, key, payload) - - out = ws.project(payload, limit=args.limit) - sys.stdout.write(json.dumps(out, indent=2, ensure_ascii=False) + "\n" - if args.json else to_yaml(out)) - return 0 if out["status"] == "ok" else 3 +def main(argv: list[str]) -> int: + print( + "bin/web/search is deprecated; use bin/web/search.go", + file=sys.stderr, + ) + target = ROOT / "bin" / "web" / "search.go" + os.execvp("go", ["go", "run", str(target), *argv]) + return 1 if __name__ == "__main__": - sys.exit(main()) + sys.exit(main(sys.argv[1:])) diff --git a/bin/web/search.go b/bin/web/search.go new file mode 100755 index 0000000..7b3518b --- /dev/null +++ b/bin/web/search.go @@ -0,0 +1,232 @@ +//usr/bin/env go run "$0" "$@"; exit +// +// bin/web/search.go - SearXNG as the second independent source (D3). +// +// ./bin/web/search.go "LadybugDB vector search" +// ./bin/web/search.go "model2vec" --category it --json +// ./bin/web/search.go "postgres" --site github.com --fresh year +// +// Empty results mean throttled, not "nothing exists". Exit 2 = PII refuse, 3 = throttled. +// Config: $BRAIN_SEARCH_ENV (default $HOME/.config/brain/search.env). +// NOTE: never run `gofmt -w` on this file — it breaks the shebang. +package main + +import ( + "encoding/json" + "fmt" + "net/http" + "os" + "strconv" + "time" + + "github.com/eSlider/2dph/internal/websearch" + "golang.org/x/sys/unix" +) + +func main() { + os.Exit(run(os.Args[1:])) +} + +func run(args []string) int { + var ( + query, site, lang, fresh, category, engines string + limit = websearch.DefaultLimit + jsonOut, refresh, force bool + ttl = float64(websearch.CacheTTL) + timeout = 25 + ) + i := 0 + for i < len(args) { + a := args[i] + switch { + case a == "--json": + jsonOut = true + case a == "--refresh": + refresh = true + case a == "--force": + force = true + case (a == "-n" || a == "--limit") && i+1 < len(args): + i++ + n, err := strconv.Atoi(args[i]) + if err != nil || n < 0 { + fmt.Fprintln(os.Stderr, "web/search: --limit must be a non-negative integer") + return 2 + } + limit = n + case a == "--site" && i+1 < len(args): + i++ + site = args[i] + case a == "--lang" && i+1 < len(args): + i++ + lang = args[i] + case a == "--fresh" && i+1 < len(args): + i++ + fresh = args[i] + case a == "--category" && i+1 < len(args): + i++ + category = args[i] + case a == "--engines" && i+1 < len(args): + i++ + engines = args[i] + case a == "--ttl" && i+1 < len(args): + i++ + v, err := strconv.ParseFloat(args[i], 64) + if err != nil { + fmt.Fprintln(os.Stderr, "web/search: --ttl must be a number") + return 2 + } + ttl = v + case a == "--timeout" && i+1 < len(args): + i++ + n, err := strconv.Atoi(args[i]) + if err != nil || n <= 0 { + fmt.Fprintln(os.Stderr, "web/search: --timeout must be a positive integer") + return 2 + } + timeout = n + case a == "-h" || a == "--help": + fmt.Fprintln(os.Stderr, `usage: bin/web/search.go QUERY [--json] [-n N] [--site HOST] [--lang LANG] [--fresh day|week|month|year] [--category CAT] [--engines LIST] [--refresh] [--force]`) + return 0 + case len(a) > 0 && a[0] != '-' && query == "": + query = a + default: + fmt.Fprintf(os.Stderr, "web/search: unknown flag %s\n", a) + return 2 + } + i++ + } + if query == "" { + fmt.Fprintln(os.Stderr, "web/search: query required") + return 2 + } + if site != "" { + query = "site:" + site + " " + query + } + if reason := websearch.PHIReason(query); reason != "" && !force { + fmt.Fprintf(os.Stderr, "refused: %s. This query would leave the host.\n", reason) + fmt.Fprintln(os.Stderr, "Rephrase without identifiers, or pass --force if it is genuinely public.") + return 2 + } + + params := map[string]string{} + if lang != "" { + params["language"] = lang + } + if fresh != "" { + params["time_range"] = fresh + } + if category != "" { + params["categories"] = category + } + if engines != "" { + params["engines"] = engines + } + + cachePath := os.Getenv("BRAIN_SEARCH_CACHE") + if cachePath == "" { + cachePath = os.Getenv("HOME") + "/.cache/brain/web-search.sqlite" + } + cache, err := websearch.OpenCache(cachePath) + if err != nil { + fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err) + return 1 + } + defer cache.Close() + + key := websearch.CacheKey(query, params) + now := float64(time.Now().Unix()) + if !refresh { + if cached, err := cache.Get(key, ttl, now); err != nil { + fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err) + return 1 + } else if cached != nil { + out := websearch.Project(*cached, limit, websearch.DefaultSnippetChars) + out.Cached = true + return writeOut(out, jsonOut) + } + } + + envPath := os.Getenv("BRAIN_SEARCH_ENV") + if envPath == "" { + envPath = os.Getenv("HOME") + "/.config/brain/search.env" + } + conf, err := websearch.LoadConfig(envPath) + if err != nil { + fmt.Fprintf(os.Stderr, "web/search: %v\n", err) + return 1 + } + + lockPath := cachePath + ".lock" + lock, err := os.OpenFile(lockPath, os.O_CREATE|os.O_RDWR, 0o600) + if err != nil { + fmt.Fprintf(os.Stderr, "web/search: lock: %v\n", err) + return 1 + } + defer lock.Close() + if err := unix.Flock(int(lock.Fd()), unix.LOCK_EX); err != nil { + fmt.Fprintf(os.Stderr, "web/search: lock: %v\n", err) + return 1 + } + defer unix.Flock(int(lock.Fd()), unix.LOCK_UN) + + var payload websearch.Payload + attempts := 1 + len(websearch.RetryBackoff) + client := &http.Client{} + for attempt := 0; attempt < attempts; attempt++ { + last, err := cache.LastCall() + if err != nil { + fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err) + return 1 + } + if delay := websearch.WaitFor(last, float64(time.Now().Unix()), websearch.MinInterval); delay > 0 { + time.Sleep(time.Duration(delay * float64(time.Second))) + } + if err := cache.MarkCall(float64(time.Now().Unix())); err != nil { + fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err) + return 1 + } + payload, err = websearch.Fetch(client, conf, query, params, time.Duration(timeout)*time.Second) + if err != nil { + fmt.Fprintf(os.Stderr, "request failed: %v\n", err) + return 3 + } + if websearch.Classify(payload) == websearch.StatusOK { + break + } + if attempt < len(websearch.RetryBackoff) { + time.Sleep(time.Duration(websearch.RetryBackoff[attempt] * float64(time.Second))) + } + } + + if websearch.Classify(payload) == websearch.StatusOK { + if err := cache.Put(key, payload, float64(time.Now().Unix())); err != nil { + fmt.Fprintf(os.Stderr, "web/search: cache: %v\n", err) + } + } + out := websearch.Project(payload, limit, websearch.DefaultSnippetChars) + code := writeOut(out, jsonOut) + if out.Status != websearch.StatusOK && code == 0 { + return 3 + } + return code +} + +func writeOut(out websearch.Output, jsonOut bool) int { + if jsonOut { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + enc.SetEscapeHTML(false) + if err := enc.Encode(out); err != nil { + return 1 + } + if out.Status != websearch.StatusOK { + return 3 + } + return 0 + } + fmt.Print(out.YAML()) + if out.Status != websearch.StatusOK { + return 3 + } + return 0 +} diff --git a/compose.yaml b/compose.yaml index 106ee0a..f9460ed 100644 --- a/compose.yaml +++ b/compose.yaml @@ -61,6 +61,21 @@ services: restart: unless-stopped stop_grace_period: 20s + # Optional local SearXNG (D3). Skip if BRAIN_SEARCH_URL already points at a + # live instance — do not run a second copy on that host. + # SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d + searxng: + profiles: ["searxng"] + image: docker.io/searxng/searxng:2026.8.10-0a118066d + ports: + - "127.0.0.1:8888:8080" + environment: + SEARXNG_SECRET: ${SEARXNG_SECRET:-} + volumes: + - ./deploy/searxng/settings.yml:/etc/searxng/settings.yml:ro + - ./deploy/searxng/limiter.toml:/etc/searxng/limiter.toml:ro + restart: unless-stopped + volumes: kb-model: kb-var: \ No newline at end of file diff --git a/deploy/searxng/limiter.toml b/deploy/searxng/limiter.toml new file mode 100644 index 0000000..b333ce7 --- /dev/null +++ b/deploy/searxng/limiter.toml @@ -0,0 +1,7 @@ +[botdetection.ip_lists] +# RFC1918 only. Do not copy a live instance egress IP into git. +pass_ip = [ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16", +] diff --git a/deploy/searxng/settings.yml b/deploy/searxng/settings.yml new file mode 100644 index 0000000..f549fb0 --- /dev/null +++ b/deploy/searxng/settings.yml @@ -0,0 +1,28 @@ +use_default_settings: true + +general: + instance_name: "2dph" + +search: + formats: + - html + - json + suspended_times: + SearxEngineCaptcha: 300 + SearxEngineTooManyRequests: 120 + SearxEngineAccessDenied: 300 + +server: + limiter: true + image_proxy: false + # secret_key comes from SEARXNG_SECRET (never commit a real secret) + +engines: + - name: bing + disabled: false + - name: google + disabled: false + - name: duckduckgo + disabled: false + - name: wikipedia + disabled: false diff --git a/docs/design.md b/docs/design.md index 8e330e5..ab5f284 100644 --- a/docs/design.md +++ b/docs/design.md @@ -21,6 +21,7 @@ bin/brain/search.go "question" 1. facts root — confirmed answers only → return with evidence links 2. info root — supporting narrative → snippets, marked (not confirmed) 3. web-search — second independent source → upgrade hypothesis to confirmed + (`bin/web/search.go`; status `throttled` is not evidence of absence) ``` `--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an diff --git a/go.mod b/go.mod index 0a8e6a7..53081f4 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,9 @@ require ( github.com/chewxy/math32 v1.11.2 github.com/daulet/tokenizers v1.27.0 github.com/go-git/go-git/v5 v5.19.2 + golang.org/x/sys v0.47.0 golang.org/x/text v0.40.0 + modernc.org/sqlite v1.56.0 ) require ( @@ -18,6 +20,7 @@ require ( github.com/apache/arrow-go/v18 v18.6.0 // indirect github.com/cloudflare/circl v1.6.3 // indirect github.com/cyphar/filepath-securejoin v0.6.1 // indirect + github.com/dustin/go-humanize v1.0.1 // indirect github.com/emirpasic/gods v1.18.1 // indirect github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect github.com/go-git/go-billy/v5 v5.9.0 // indirect @@ -29,8 +32,11 @@ require ( github.com/kevinburke/ssh_config v1.2.0 // indirect github.com/klauspost/compress v1.18.5 // indirect github.com/klauspost/cpuid/v2 v2.3.0 // indirect + github.com/mattn/go-isatty v0.0.24 // indirect + github.com/ncruces/go-strftime v1.0.0 // indirect github.com/pierrec/lz4/v4 v4.1.26 // indirect github.com/pjbgf/sha1cd v0.6.0 // indirect + github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect github.com/shopspring/decimal v1.4.0 // indirect github.com/skeema/knownhosts v1.3.1 // indirect @@ -39,6 +45,8 @@ require ( golang.org/x/crypto v0.53.0 // indirect golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect golang.org/x/net v0.56.0 // indirect - golang.org/x/sys v0.46.0 // indirect gopkg.in/warnings.v0 v0.1.2 // indirect + modernc.org/libc v1.74.4 // indirect + modernc.org/mathutil v1.7.1 // indirect + modernc.org/memory v1.11.0 // indirect ) diff --git a/go.sum b/go.sum index 0bee964..7c26588 100644 --- a/go.sum +++ b/go.sum @@ -31,6 +31,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o= github.com/elazarl/goproxy v1.7.2/go.mod h1:82vkLNir0ALaW14Rc399OTTjyNREgmdL2cVoIbS6XaE= github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc= @@ -53,8 +55,12 @@ github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9U github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo= +github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= +github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4= @@ -70,6 +76,10 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI= +github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= +github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w= +github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= github.com/onsi/gomega v1.34.1 h1:EUMJIKUjM8sKjYbtxQI9A4z2o+rruxnzNvpknOXie6k= github.com/onsi/gomega v1.34.1/go.mod h1:kU1QgUvBDLXBJq618Xvm2LUX6rSAfRaFRTcdOeDLwwY= github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY= @@ -81,6 +91,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= +github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 h1:n661drycOFuPLCN3Uc8sB6B/s6Z4t2xvBgU1htSHuq8= @@ -106,17 +118,21 @@ golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto= golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio= golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM= golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80= +golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ= +golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o= golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec= +golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= +golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= -golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc= golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y= @@ -124,6 +140,8 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q= +golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -136,3 +154,31 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI= +modernc.org/cc/v4 v4.29.1/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI= +modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU= +modernc.org/ccgo/v4 v4.34.6/go.mod h1:SZ8YcN9NG7XVsQYdm6jYBvi8PQP1qi+kqB6OhjqI3Fk= +modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM= +modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU= +modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI= +modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito= +modernc.org/gc/v3 v3.1.4 h1:2g65LGVSmFQrXeITAw97x7hCRvZFcyE1uDP+7Vng7JI= +modernc.org/gc/v3 v3.1.4/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY= +modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks= +modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI= +modernc.org/libc v1.74.4 h1:fX1Omw4o2/1C2iRkkIsrQTasJQldLhRmuPreXLoWs9k= +modernc.org/libc v1.74.4/go.mod h1:eeQAS9W3sZeKYMFubydxJpII9ybHWshk+7or7bLG9co= +modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU= +modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg= +modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI= +modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw= +modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg= +modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns= +modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w= +modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE= +modernc.org/sqlite v1.56.0 h1:/D8e2RfFqoy/Zc6PuC76U28zFwmI/sYx1Kjm4yEn9e0= +modernc.org/sqlite v1.56.0/go.mod h1:yCJ2cmAaIkHQ25oXWrF8H4O1lIfPYPR26yCEDj2P3pQ= +modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0= +modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A= +modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y= +modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM= diff --git a/internal/websearch/cache.go b/internal/websearch/cache.go new file mode 100644 index 0000000..5ad0ee7 --- /dev/null +++ b/internal/websearch/cache.go @@ -0,0 +1,97 @@ +package websearch + +import ( + "database/sql" + "encoding/json" + "os" + "path/filepath" + + _ "modernc.org/sqlite" +) + +const cacheSchema = ` +CREATE TABLE IF NOT EXISTS responses ( + key TEXT PRIMARY KEY, + fetched REAL NOT NULL, + payload TEXT NOT NULL +); +CREATE TABLE IF NOT EXISTS meta ( + key TEXT PRIMARY KEY, + value REAL NOT NULL +); +` + +type Cache struct { + db *sql.DB +} + +func OpenCache(path string) (*Cache, error) { + if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { + return nil, err + } + db, err := sql.Open("sqlite", path) + if err != nil { + return nil, err + } + if _, err := db.Exec(cacheSchema); err != nil { + db.Close() + return nil, err + } + return &Cache{db: db}, nil +} + +func (c *Cache) Close() error { + if c == nil || c.db == nil { + return nil + } + return c.db.Close() +} + +func (c *Cache) Get(key string, ttl, now float64) (*Payload, error) { + var fetched float64 + var raw string + err := c.db.QueryRow("SELECT fetched, payload FROM responses WHERE key = ?", key).Scan(&fetched, &raw) + if err == sql.ErrNoRows { + return nil, nil + } + if err != nil { + return nil, err + } + if now-fetched > ttl { + return nil, nil + } + var p Payload + if err := json.Unmarshal([]byte(raw), &p); err != nil { + return nil, err + } + return &p, nil +} + +func (c *Cache) Put(key string, p Payload, now float64) error { + raw, err := json.Marshal(p) + if err != nil { + return err + } + _, err = c.db.Exec( + "INSERT OR REPLACE INTO responses (key, fetched, payload) VALUES (?, ?, ?)", + key, now, string(raw), + ) + return err +} + +func (c *Cache) LastCall() (*float64, error) { + var v float64 + err := c.db.QueryRow("SELECT value FROM meta WHERE key = 'last_call'").Scan(&v) + if err == sql.ErrNoRows { + return nil, nil + } + if err != nil { + return nil, err + } + return &v, nil +} + +func (c *Cache) MarkCall(now float64) error { + _, err := c.db.Exec("INSERT OR REPLACE INTO meta (key, value) VALUES ('last_call', ?)", now) + return err +} diff --git a/internal/websearch/fetch.go b/internal/websearch/fetch.go new file mode 100644 index 0000000..0a7bde5 --- /dev/null +++ b/internal/websearch/fetch.go @@ -0,0 +1,90 @@ +package websearch + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "os" + "strings" + "time" +) + +type Config struct { + URL string + User string + Pass string +} + +func LoadConfig(path string) (Config, error) { + raw, err := os.ReadFile(path) + if err != nil { + return Config{}, fmt.Errorf("no credentials at %s (mode 600, BRAIN_SEARCH_URL)", path) + } + conf := map[string]string{} + for _, line := range strings.Split(string(raw), "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") || !strings.Contains(line, "=") { + continue + } + k, v, _ := strings.Cut(line, "=") + v = strings.TrimSpace(v) + v = strings.Trim(v, `"'`) + conf[strings.TrimSpace(k)] = v + } + out := Config{ + URL: conf["BRAIN_SEARCH_URL"], + User: conf["BRAIN_SEARCH_USER"], + Pass: conf["BRAIN_SEARCH_PASS"], + } + if out.URL == "" { + return Config{}, fmt.Errorf("%s is missing BRAIN_SEARCH_URL", path) + } + return out, nil +} + +func Fetch(client *http.Client, conf Config, query string, params map[string]string, timeout time.Duration) (Payload, error) { + if client == nil { + client = &http.Client{Timeout: timeout} + } else if timeout > 0 { + c := *client + c.Timeout = timeout + client = &c + } + q := url.Values{} + q.Set("q", query) + q.Set("format", "json") + for k, v := range params { + if v != "" { + q.Set(k, v) + } + } + u := strings.TrimRight(conf.URL, "/") + "/search?" + q.Encode() + req, err := http.NewRequest(http.MethodGet, u, nil) + if err != nil { + return Payload{}, err + } + if conf.User != "" || conf.Pass != "" { + token := base64.StdEncoding.EncodeToString([]byte(conf.User + ":" + conf.Pass)) + req.Header.Set("Authorization", "Basic "+token) + } + resp, err := client.Do(req) + if err != nil { + return Payload{}, err + } + defer resp.Body.Close() + body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return Payload{}, err + } + if resp.StatusCode >= 400 { + return Payload{}, fmt.Errorf("HTTP %d", resp.StatusCode) + } + var p Payload + if err := json.Unmarshal(body, &p); err != nil { + return Payload{}, err + } + return p, nil +} diff --git a/internal/websearch/fetch_test.go b/internal/websearch/fetch_test.go new file mode 100644 index 0000000..1e4f9a7 --- /dev/null +++ b/internal/websearch/fetch_test.go @@ -0,0 +1,77 @@ +package websearch + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + "time" +) + +func TestLoadConfigRequiresURL(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "search.env") + if err := os.WriteFile(p, []byte("BRAIN_SEARCH_USER=x\n"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := LoadConfig(p); err == nil { + t.Fatal("expected missing URL error") + } +} + +func TestLoadConfigOptionalAuth(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "search.env") + if err := os.WriteFile(p, []byte("BRAIN_SEARCH_URL=http://127.0.0.1:8080\n"), 0o600); err != nil { + t.Fatal(err) + } + c, err := LoadConfig(p) + if err != nil { + t.Fatal(err) + } + if c.URL != "http://127.0.0.1:8080" || c.User != "" || c.Pass != "" { + t.Fatalf("%+v", c) + } +} + +func TestFetchJSONNoBasicAuth(t *testing.T) { + payload := Payload{Query: "x", Results: []RawHit{{Title: "t", URL: "http://example.com", Content: "c", Engine: "bing"}}} + var sawAuth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + sawAuth = r.Header.Get("Authorization") + if r.URL.Query().Get("format") != "json" || r.URL.Query().Get("q") != "x" { + t.Errorf("query = %s", r.URL.RawQuery) + } + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(payload) + })) + defer srv.Close() + got, err := Fetch(srv.Client(), Config{URL: srv.URL}, "x", nil, 2*time.Second) + if err != nil { + t.Fatal(err) + } + if sawAuth != "" { + t.Fatalf("Authorization = %q, want empty for local instance", sawAuth) + } + if Classify(got) != StatusOK { + t.Fatalf("classify = %s", Classify(got)) + } +} + +func TestFetchSendsBasicAuthWhenConfigured(t *testing.T) { + var sawAuth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + sawAuth = r.Header.Get("Authorization") + w.Write([]byte(`{"query":"x","results":[]}`)) + })) + defer srv.Close() + _, err := Fetch(srv.Client(), Config{URL: srv.URL, User: "u", Pass: "p"}, "x", nil, 2*time.Second) + if err != nil { + t.Fatal(err) + } + if sawAuth == "" { + t.Fatal("expected Basic auth") + } +} diff --git a/internal/websearch/websearch.go b/internal/websearch/websearch.go new file mode 100644 index 0000000..19fda24 --- /dev/null +++ b/internal/websearch/websearch.go @@ -0,0 +1,205 @@ +// Package websearch is the SearXNG client used as the second independent source. +// +// An empty result list from this instance is throttling, not evidence of absence. +package websearch + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "regexp" + "strings" + "unicode" + "unicode/utf8" +) + +const ( + StatusOK = "ok" + StatusThrottled = "throttled" + + DefaultLimit = 5 + DefaultSnippetChars = 150 + MinInterval = 10.0 + CacheTTL = 7 * 24 * 3600 +) + +var RetryBackoff = []float64{20, 60} + +type Payload struct { + Query string `json:"query"` + Results []RawHit `json:"results"` + UnresponsiveEngines [][]string `json:"unresponsive_engines"` +} + +type RawHit struct { + Title string `json:"title"` + URL string `json:"url"` + Content string `json:"content"` + Engine string `json:"engine"` +} + +type Hit struct { + Rank int `json:"rank"` + Title string `json:"title"` + URL string `json:"url"` + Snippet string `json:"snippet"` + Engine string `json:"engine"` +} + +type Output struct { + Query string `json:"query"` + Status string `json:"status"` + Results []Hit `json:"results"` + Unresponsive []string `json:"unresponsive,omitempty"` + Note string `json:"note,omitempty"` + Cached bool `json:"cached,omitempty"` +} + +func Classify(p Payload) string { + if len(p.Results) > 0 { + return StatusOK + } + return StatusThrottled +} + +func Project(p Payload, limit, snippetChars int) Output { + if limit <= 0 { + limit = DefaultLimit + } + if snippetChars <= 0 { + snippetChars = DefaultSnippetChars + } + status := Classify(p) + n := limit + if n > len(p.Results) { + n = len(p.Results) + } + hits := make([]Hit, 0, n) + for i := 0; i < n; i++ { + item := p.Results[i] + hits = append(hits, Hit{ + Rank: i + 1, + Title: item.Title, + URL: item.URL, + Snippet: trimSnippet(item.Content, snippetChars), + Engine: item.Engine, + }) + } + out := Output{ + Query: p.Query, + Status: status, + Results: hits, + } + for _, pair := range p.UnresponsiveEngines { + if len(pair) >= 2 { + out.Unresponsive = append(out.Unresponsive, pair[0]+": "+pair[1]) + } else if len(pair) == 1 { + out.Unresponsive = append(out.Unresponsive, pair[0]) + } + } + if status == StatusThrottled { + out.Note = "no engine answered - this is a throttled instance, not evidence that nothing exists" + } + return out +} + +var spaceRE = regexp.MustCompile(`\s+`) + +func trimSnippet(s string, max int) string { + s = strings.TrimSpace(spaceRE.ReplaceAllString(s, " ")) + if utf8.RuneCountInString(s) <= max { + return s + } + runes := []rune(s) + cut := strings.TrimRightFunc(string(runes[:max]), unicode.IsSpace) + return cut + "..." +} + +func CacheKey(query string, params map[string]string) string { + norm := strings.Join(strings.Fields(strings.ToLower(query)), " ") + if params == nil { + params = map[string]string{} + } + stable, _ := json.Marshal(params) + sum := sha256.Sum256([]byte(norm + "\x00" + string(stable))) + return hex.EncodeToString(sum[:]) +} + +func WaitFor(last *float64, now, interval float64) float64 { + if last == nil { + return 0 + } + d := interval - (now - *last) + if d < 0 { + return 0 + } + return d +} + +func PHIReason(query string) string { + for _, p := range phiPatterns { + if p.re.MatchString(query) { + return p.reason + } + } + return "" +} + +type phiPat struct { + re *regexp.Regexp + reason string +} + +var phiPatterns = []phiPat{ + {regexp.MustCompile(`\d{6,}`), "a run of six or more digits looks like an ID"}, + {regexp.MustCompile(`(?i)\bpersonalnummer\b`), "Personalnummer is staff data"}, + {regexp.MustCompile(`(?i)\bkv[-\s]?nr\b`), "KV-Nr is an insurance number"}, + {regexp.MustCompile(`(?i)\bversichertennummer\b`), "insurance number"}, + {regexp.MustCompile(`(?i)\b[A-Za-zÄÖÜäöüß]+(?:stra(?:ss|ß)e|str\.)\s*\d+`), "a street with a house number looks like an address"}, + {regexp.MustCompile(`(?i)\bgeb(?:urtsdatum)?\.?\s*\d{1,2}[./]\d{1,2}[./]\d{2,4}`), "a date of birth"}, +} + +func (o Output) YAML() string { + var b strings.Builder + fmt.Fprintf(&b, "query: %s\n", yamlScalar(o.Query)) + fmt.Fprintf(&b, "status: %s\n", yamlScalar(o.Status)) + if len(o.Results) == 0 { + b.WriteString("results: []\n") + } else { + b.WriteString("results:\n") + for _, r := range o.Results { + b.WriteString("-\n") + fmt.Fprintf(&b, " rank: %d\n", r.Rank) + fmt.Fprintf(&b, " title: %s\n", yamlScalar(r.Title)) + fmt.Fprintf(&b, " url: %s\n", yamlScalar(r.URL)) + fmt.Fprintf(&b, " snippet: %s\n", yamlScalar(r.Snippet)) + fmt.Fprintf(&b, " engine: %s\n", yamlScalar(r.Engine)) + } + } + if len(o.Unresponsive) > 0 { + b.WriteString("unresponsive:\n") + for _, u := range o.Unresponsive { + fmt.Fprintf(&b, "- %s\n", yamlScalar(u)) + } + } + if o.Note != "" { + fmt.Fprintf(&b, "note: %s\n", yamlScalar(o.Note)) + } + if o.Cached { + b.WriteString("cached: true\n") + } + return b.String() +} + +func yamlScalar(s string) string { + if strings.Contains(s, "\n") { + b, _ := json.Marshal(s) + return string(b) + } + if s == "" || strings.ContainsAny(s, ":#'\"[]{}&*!|>%@`") || s != strings.TrimSpace(s) { + b, _ := json.Marshal(s) + return string(b) + } + return s +} diff --git a/internal/websearch/websearch_test.go b/internal/websearch/websearch_test.go new file mode 100644 index 0000000..8cb3098 --- /dev/null +++ b/internal/websearch/websearch_test.go @@ -0,0 +1,203 @@ +package websearch + +import ( + "encoding/json" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "unicode/utf8" +) + +func loadFixture(t *testing.T, name string) Payload { + t.Helper() + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("runtime.Caller") + } + path := filepath.Join(filepath.Dir(file), "..", "..", "bin", "tools", "web-search", "fixtures", name) + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + var p Payload + if err := json.Unmarshal(raw, &p); err != nil { + t.Fatal(err) + } + return p +} + +func TestClassifyHealthyIsOK(t *testing.T) { + if got := Classify(loadFixture(t, "healthy.json")); got != StatusOK { + t.Fatalf("classify healthy = %q, want ok", got) + } +} + +func TestClassifyEmptyIsThrottledNotEmpty(t *testing.T) { + got := Classify(loadFixture(t, "throttled.json")) + if got != StatusThrottled { + t.Fatalf("classify empty = %q, want throttled", got) + } + if got == "empty" || got == "no_results" { + t.Fatal("status must never sound like absence") + } +} + +func TestProjectKeepsContextFields(t *testing.T) { + out := Project(loadFixture(t, "healthy.json"), 3, DefaultSnippetChars) + if out.Status != StatusOK { + t.Fatalf("status = %q", out.Status) + } + if len(out.Results) != 3 { + t.Fatalf("len = %d, want 3", len(out.Results)) + } + r := out.Results[0] + if r.Rank != 1 || r.Title == "" || r.URL == "" { + t.Fatalf("hit = %+v", r) + } +} + +func TestProjectTrimsSnippet(t *testing.T) { + out := Project(loadFixture(t, "healthy.json"), 5, 40) + for _, r := range out.Results { + n := utf8.RuneCountInString(r.Snippet) + if n > 43 { + t.Fatalf("snippet len %d > 43: %q", n, r.Snippet) + } + } +} + +func TestProjectIsCheaperThanRaw(t *testing.T) { + raw, err := os.ReadFile(filepath.Join(fixtureDir(t), "healthy.json")) + if err != nil { + t.Fatal(err) + } + out, err := json.Marshal(Project(loadFixture(t, "healthy.json"), 5, DefaultSnippetChars)) + if err != nil { + t.Fatal(err) + } + if len(out)*3 >= len(raw) { + t.Fatalf("projected %d not cheaper than raw %d", len(out), len(raw)) + } +} + +func TestThrottledProjectionCarriesEngineReasons(t *testing.T) { + out := Project(loadFixture(t, "throttled.json"), 5, DefaultSnippetChars) + if out.Status != StatusThrottled { + t.Fatalf("status = %q", out.Status) + } + if len(out.Results) != 0 { + t.Fatalf("results = %v", out.Results) + } + if len(out.Unresponsive) == 0 { + t.Fatal("unresponsive empty") + } + if !strings.Contains(out.Note, "not evidence that nothing exists") { + t.Fatalf("note = %q", out.Note) + } +} + +func TestCacheKeyStable(t *testing.T) { + if CacheKey("Pflegegrad", nil) != CacheKey("Pflegegrad", map[string]string{}) { + t.Fatal("nil vs empty params") + } + if CacheKey(" Pflegegrad ", nil) != CacheKey("pflegegrad", nil) { + t.Fatal("case/padding") + } + if CacheKey("x", map[string]string{"lang": "de"}) == CacheKey("x", nil) { + t.Fatal("params must change key") + } + a := CacheKey("x", map[string]string{"a": "1", "b": "2"}) + b := CacheKey("x", map[string]string{"b": "2", "a": "1"}) + if a != b { + t.Fatal("param order must not change key") + } +} + +func TestPHIGuard(t *testing.T) { + if PHIReason("Pflegegrad SGB XI Einstufung") != "" { + t.Fatal("technical query refused") + } + if PHIReason("site:example.com technical query") != "" { + t.Fatal("site query refused") + } + if PHIReason("SGB XI Paragraph 45b") != "" { + t.Fatal("short numbers refused") + } + if PHIReason("Kunde 4711220385 Adresse") == "" { + t.Fatal("long digit run allowed") + } + if PHIReason("KV-Nr A123456789") == "" { + t.Fatal("KV-Nr allowed") + } + if PHIReason("Hauptstraße 14 Berlin") == "" { + t.Fatal("street allowed") + } + if PHIReason("Lindenstr. 7") == "" { + t.Fatal("str. allowed") + } + if PHIReason("Personalnummer 12") == "" { + t.Fatal("Personalnummer allowed") + } +} + +func TestWaitFor(t *testing.T) { + last := 100.0 + if got := WaitFor(&last, 104.0, 10); got != 6 { + t.Fatalf("wait = %v, want 6", got) + } + if got := WaitFor(&last, 130.0, 10); got != 0 { + t.Fatalf("wait = %v, want 0", got) + } + if got := WaitFor(nil, 130.0, 10); got != 0 { + t.Fatalf("first call wait = %v", got) + } +} + +func TestSQLiteCacheRoundTrip(t *testing.T) { + dir := t.TempDir() + c, err := OpenCache(filepath.Join(dir, "web-search.sqlite")) + if err != nil { + t.Fatal(err) + } + defer c.Close() + p := loadFixture(t, "healthy.json") + key := CacheKey("pflegegrad", nil) + if got, err := c.Get(key, CacheTTL, 1_000); err != nil || got != nil { + t.Fatalf("empty get = %v %v", got, err) + } + if err := c.Put(key, p, 1_000); err != nil { + t.Fatal(err) + } + got, err := c.Get(key, CacheTTL, 1_001) + if err != nil || got == nil { + t.Fatalf("get = %v %v", got, err) + } + if Classify(*got) != StatusOK { + t.Fatalf("cached classify = %s", Classify(*got)) + } + expired, err := c.Get(key, 10, 2_000) + if err != nil || expired != nil { + t.Fatalf("expired = %v %v", expired, err) + } + if v, err := c.LastCall(); err != nil || v != nil { + t.Fatalf("last = %v %v", v, err) + } + if err := c.MarkCall(50); err != nil { + t.Fatal(err) + } + v, err := c.LastCall() + if err != nil || v == nil || *v != 50 { + t.Fatalf("last after mark = %v %v", v, err) + } +} + +func fixtureDir(t *testing.T) string { + t.Helper() + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("runtime.Caller") + } + return filepath.Join(filepath.Dir(file), "..", "..", "bin", "tools", "web-search", "fixtures") +} diff --git a/skills/web-search/SKILL.md b/skills/web-search/SKILL.md index 0e6c707..276ef72 100644 --- a/skills/web-search/SKILL.md +++ b/skills/web-search/SKILL.md @@ -1,25 +1,30 @@ --- name: web-search description: >- - Search the public web through the self-hosted SearXNG at search.ops.io - using bin/web/search. Use for German care law, SGB paragraphs, vendor - documentation and any fact that is not in our own repos - and as the second - independent source the detective method requires. + Search the public web through SearXNG using bin/web/search.go. Use for vendor + documentation, public standards, and any fact that is not in our own repos — + and as the second independent source the detective method requires. --- # web-search ```bash -bin/web/search "LadybugDB vector index" -bin/web/search "model2vec multilingual" --category it -bin/web/search "hypervisor" --site ops.io --json | jq -r '.results[].url' -bin/web/search "postgres partial index" --lang en --fresh year +bin/web/search.go "LadybugDB vector index" +bin/web/search.go "model2vec multilingual" --category it +bin/web/search.go "hypervisor" --site example.com --json | jq -r '.results[].url' +bin/web/search.go "postgres partial index" --lang en --fresh year ``` +URL and optional Basic Auth live in `$BRAIN_SEARCH_ENV` (default +`$HOME/.config/brain/search.env`): `BRAIN_SEARCH_URL` is required; +`BRAIN_SEARCH_USER` / `BRAIN_SEARCH_PASS` only if the instance uses Basic Auth. +A host that already runs SearXNG should set `BRAIN_SEARCH_URL` and not start +the Compose profile. + ## Web or knowledge base `bin/brain/search.go` holds our own facts: the ops stack, portfolio, ssh hosts, -the lexicon. Go there first. Reach for `bin/web/search` when the answer is +the lexicon. Go there first. Reach for `bin/web/search.go` when the answer is outside our repos: upstream library behaviour, vendor documentation, public standards. diff --git a/skills/web-search/reference/instance-tuning.md b/skills/web-search/reference/instance-tuning.md index 510d0a2..5e73030 100644 --- a/skills/web-search/reference/instance-tuning.md +++ b/skills/web-search/reference/instance-tuning.md @@ -5,63 +5,47 @@ status: current # Tuning the SearXNG instance -The client works around a fragile instance. These changes fix the cause, and -they need shell access to the host behind `search.ops.io` -(`90.169.228.16` / `ops.mywire.org`), which is a different machine from -the one the agents run on. +The client treats HTTP 200 + `results: []` as **throttled**, not as absence. +Fix the cause on the instance you point `BRAIN_SEARCH_URL` at, or use the +optional Compose profile in this repo. -## Why it is needed +## Optional Compose profile -Measured on 2026-08-10 from this host: +Do not start this on a host that already runs SearXNG — set `BRAIN_SEARCH_URL` +instead (D3). -- The default engine set for the `general` category is only `duckduckgo`, - `brave` and `startpage`. `brave` and `startpage` sit in - `Suspended: too many requests` or `Suspended: CAPTCHA` almost permanently, so - in practice a single engine carries every query. -- About 25 probe requests over a few minutes pushed `duckduckgo` into `CAPTCHA` - as well. The instance then answered HTTP 200 with `results: []` and an empty - `unresponsive_engines` - indistinguishable from "nothing found" without the - client-side handling we added. -- Recovery took roughly six minutes. +```bash +SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d +``` -## Changes +Pinned image: `docker.io/searxng/searxng:2026.8.10-0a118066d`. +Settings: `deploy/searxng/settings.yml` + `limiter.toml` (RFC1918 `pass_ip`, +short `suspended_times`, `formats: [html, json]`). No secrets in git. Bind is +`127.0.0.1:8888`. -1. **Allow our egress IP through the limiter.** In `limiter.toml`: +## Why the client classifies empty as throttled - ```toml - [botdetection.ip_lists] - pass_ip = ["77.7.46.234"] - ``` +A default engine set under load answers HTTP 200 with `results: []` (sometimes +with empty `unresponsive_engines`). That is indistinguishable from "nothing +found" unless the client refuses to call it absence. -2. **Shorten the suspensions.** In `settings.yml` the defaults are 24 hours for - a CAPTCHA and one hour for too-many-requests, which is far longer than the - condition lasts: +## Instance-side levers - ```yaml - search: - suspended_times: - SearxEngineCaptcha: 300 - SearxEngineTooManyRequests: 120 - SearxEngineAccessDenied: 300 - ``` - -3. **Give `general` more than one working engine.** `google` and `wikipedia` - report `enabled: true` in `/config` yet never appear in a `general` response, - so they are not in the default set. Put them in it; one live engine per - category is a single point of failure. - -4. **Keep the JSON API on.** `formats: [html, json]` must stay, otherwise every - client here breaks. +1. **Allow the callers through the limiter** (`limiter.toml` `pass_ip`). The + Compose file uses RFC1918 only. +2. **Shorten suspensions** (`settings.yml` `search.suspended_times`) so a + CAPTCHA does not last a day. +3. **More than one engine in `general`.** One live engine is a single point of + failure. This repo enables bing, google, duckduckgo, wikipedia. +4. **Keep the JSON API on.** `formats: [html, json]` must stay. ## Verifying -Ten requests in a row used to suspend the instance for minutes. After the -change they should all answer: - ```bash for i in $(seq 10); do - bin/web/search "test $i" -n 1 --refresh --json | jq -r .status + bin/web/search.go "test $i" -n 1 --refresh --json | jq -r .status done ``` -Ten lines of `ok` means it is fixed. +Ten lines of `ok` means the instance is healthy. Any `throttled` means say +nothing about whether the subject exists.