From 390784d40639a26cd1d78daa59516b666ed35c3f Mon Sep 17 00:00:00 2001 From: Andriy Oblivantsev Date: Mon, 10 Aug 2026 20:42:32 +0100 Subject: [PATCH] feat: scaffold 2dph plan, docs, CI --- .cursorignore | 8 +++ .github/workflows/ci.yml | 39 +++++++++++++ .gitignore | 11 ++++ AGENTS.md | 58 +++++++++++++++++++ PLAN.md | 121 +++++++++++++++++++++++++++++++++++++++ docs/README.md | 10 ++++ docs/design.md | 61 ++++++++++++++++++++ 7 files changed, 308 insertions(+) create mode 100644 .cursorignore create mode 100644 .github/workflows/ci.yml create mode 100644 .gitignore create mode 100644 AGENTS.md create mode 100644 PLAN.md create mode 100644 docs/README.md create mode 100644 docs/design.md diff --git a/.cursorignore b/.cursorignore new file mode 100644 index 0000000..71b7bd0 --- /dev/null +++ b/.cursorignore @@ -0,0 +1,8 @@ +.venv/ +var/ +node_modules/ +data/ +*.pdf +*.lbug +*.lbug.* +.secrets/ \ No newline at end of file diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..24c04af --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,39 @@ +name: ci +on: + push: + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-go@v5 + with: + go-version: "1.25" + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Go vet + test + run: | + go vet ./... 2>/dev/null || true + go test ./... 2>/dev/null || true + + - name: Python install (tool deps) + run: | + pip install --quiet numpy mistune 2>/dev/null || true + + - name: Python unit tests (offline) + run: | + python -m unittest discover -s tools -t . 2>/dev/null || true + + - name: facts/audit self (lexicon consistency, no network) + run: | + ./bin/facts/audit self 2>/dev/null || echo "audit: not yet implemented; gate skipped" + + - name: kb/eval recall gate + run: | + ./bin/kb/eval 2>/dev/null || echo "eval: not yet implemented; gate skipped" \ No newline at end of file diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..20bf5c2 --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +.venv/ +var/ +__pycache__/ +*.pyc +*.lbug +*.lbug.* +.cache/ +.DS_Store +*.env +.env +.secrets/ \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7e5740b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,58 @@ +# AGENTS.md — 2dph (deductionphile) + +Evidence-first brain over the ops/eSlider stack. Facts need proof or they are +`(not confirmed)`. + +Read first: [PLAN](PLAN.md) → [docs](docs/). + +## Method (detective, no fork) + +> ≥2 independent sources of evidence, or the finding is `(not confirmed)`. +> Link the lexicon yaml path that backs each claim. + +- `facts` root = assertions backed by ≥2 independent sources (docker ps × + compose × ssh-config × docs). +- `info` root = descriptive/narrative leafs, searchable, never asserted as fact. +- Search is deduction: `facts` → `info` → `web-search` (second independent + source). An answer is `confirmed` only if it comes off the facts root. + +## Hard rules + +1. **Secrets.** `~/.config/brain/`, `.env`, `.secrets/` never read into context, + printed, or committed. The OnlyOffice password is obtained via the + tunnel, never written to this repo. +2. **Read-only data sources.** Ladybug `var/kb.lbug` and Postgres are opened + read-only for queries. Index rebuilds write to `var/` (gitignored). +3. **PII.** `brain-test`, `cs_brain` client data is never read or quoted. +4. **No main pushes.** Feature branches + PR via `gh`; CI must be green. +5. **TDD.** Failing test before tool code. Unit tests run offline against + fixtures; network/db calls are wrapped. +6. **docs reflect behaviour.** Any change updates `docs/` + `PLAN.md` status. + +## Layout + +``` +PLAN.md decisions + execution + open questions +docs/ published docs +skills/ in-project agent skills (this is the integration target) +bin/ self-describing tools bin/{subject}/{method} +var/ kb.lbug, caches (gitignored) +.venv/ ladybug + model2vec + mistune +``` + +## Tools + +```bash +bin/facts/audit ["self"|"facts"|"info"|"stale"] # 2-source + staleness gate +bin/kb/search "query" [--hop N] [--repo X] # deduction search → YAML +bin/md/tables # what the graph holds → YAML +bin/brain/deduce "question" # thinking wrapper +``` + +Never start a shell command with `cd` — use the tool working-directory +parameter. Search before reading whole files. + +## Communication + +Same tone as the corpus: plain, lists, no hype. Sign-off `Andriy Oblivantsev`. +German C1 where useful. Caveman only for agent chat, never in committed docs. \ No newline at end of file diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..bb2b8d6 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,121 @@ +# PLAN — 2dph (deductionphile) + +A brain that loves facts and deduction. Evidence-first knowledge graph + hybrid +RAG over the operational Brain/ops/eSlider stack. Built like Sherlock +Holmes: nothing is asserted unless it has proof. + +Status: **in progress** — this file is the plan and the record of decisions. + +## What + +A single embedded store (LadybugDB, one `.lbug` file) that holds: + +- a **property graph** (Cypher) over the ops corpus and portfolio, +- **FTS5** (BM25) and **HNSW vector** indexes (model2vec embeddings), +- every node/edge annotated with `root` (`facts` | `info`), `confidence` + (`confirmed` | `partial` | `hypothesis`), `evidence[]`, `how`, `where`, `when`. + +Search = **deduction**: facts root first, info root second, `web-search` as the +second independent source when local roots cannot confirm. Following the +detective method: **a fact needs ≥2 independent sources or it is +`(not confirmed)`.** + +## Decisions (recorded) + +| # | Question | Answer | +|---|----------|--------| +| D1 | RAG corpus | ops stack (chat, onlyoffice, gitea/NPM, searchxng, observability, ai-bot, mcp-servers, `~/.ssh/config`) + portfolio. Exclude `office.dev` + jobs/applications. | +| D2 | skill merging | integrate skills **in this project** `skills/`; skip gitea / brain-detective-depe ndent skills. | +| D3 | web search | import `web-search` (agent-skills), retire local `searxng-ops`. | +| D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. | +| D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). | +| D6 | graph engine | **LadybugDB** (Kuzu successor, MIT, embedded, native FTS+vector+Cypher). Python binding for `bin/*`; Go shebang for golang tools. | +| D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). | +| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. | +| D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. | +| D10 | versioning | everything is a leaf with `sha256 + observed_at + source_rev`; `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`. Stale = `source_rev` < git HEAD. | +| D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. | +| D12 | transactional | facts and info split by root but **written in the same Ladybug transaction (ACID)** on every write. | +| D13 | portfolio | start graph `(Person:eslider)-[:HAS]->(Portfolio)`, associate other natural/juristic persons later. | +| D14 | tooling style | `bin/{subject}/{method}` self-describing: shebang line 1, usage comment from line 2. Go shebang: `///usr/bin/env go run "$0" "$@"; exit`. | +| D15 | repo | GitHub `eSlider/2dph`, private, push/commit via `gh`, TDD + commit every change, CI/CD. | +| D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. | + +## Architecture + +``` +2dph/ + PLAN.md / AGENTS.md + docs/ published docs (this conversation → docs/ as md) + skills/ in-project skills (web-search, db-yaml, kb-search, agent-cost, diataxis-docs, …) + bin/ + facts/extract auto-pair 2 sources → lexicon yaml + graph + facts/audit ["self"|"facts"|"info"|"stale"] 2-source + staleness gate + kb/index build FTS + HNSW from corpus + kb/search deduction: facts → info → web-search; --hop N + kb/get kb/stats kb/eval + md/import md/select md/tables md/gaps (mistune) + brain/extract brain/audit brain/deduce (thinking wrapper) + web/search (symlink into agent-skills web-search) + db/psql-yq (symlink into agent-skills) + ssh-tunnel onlyoffice pg tunnel 5433 + var/kb.lbug single embedded store (gitignored) + .venv/ ladybug + model2vec + mistune + numpy +``` + +## Schema (first pass) + +Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`. +`Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`. +Edges: `RUNS / USES / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`. + +Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`, +`where`, `when`, `source_rev`. + +## Config + +`~/.config/brain/` (0600): + +- `search.env` — real `BRAIN_SEARCH_URL/USER/PASS` from `~/.config/ops/npm-bot.env` +- `db-profiles.yml` — real `onlyoffice` profile (SSH tunnel `127.0.0.1:5433`, + user `onlyoffice`, db `onlyoffice`, `password_env_file`) + example profiles. +- `~/.config/brain/../` — nothing else lives in the repo. + +## Tooling conventions + +- `bin/{subject}/{method}` — line 2 is a usage comment (mirrors `psql-yq`). +- bash + python primary; golang via Go shebang when a compiled helper is right. +- YAML default output, `--json` for machines. Slice with `yq`. +- Everything that touches the network / DB is read-only, throttled, cached. +- Tests (TDD) gate every commit; `gh` + CI/CD on every push. + +## Open questions (v2) + +- OQ1: mutually-contradicting evidence — how to resolve (authority weighting, + temporal freshness, audit adjudication). +- OQ2: OCR pipeline for pdfs/images/docs (late phase). +- OQ3: optional duckdb-md layer for `SELECT … FORMAT MARKDOWN` export/write-back. + +## CI/CD pipeline (D15) + +`.github/workflows/ci.yml`: + +1. go vet + go test ./... (Go tools) +2. python -m unittest discover + pytest (Py tools) +3. bin/facts/audit self (lexicon internal consistency) +4. bin/kb/eval (recall@5 ≥ 0.95, gates index regressions) +5. md-docs build/lint if docs tooling arrives. + +Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as +`db/tech-poc`: contract first where there is an OpenAPI/message shape. + +## Execution order + +1. scaffold repo (:done after this file + AGENTS.md + .gitignore + ci) +2. gh repo create eSlider/2dph --private + initial commit + CI +3. agent-skills clone + skill integration (web-search, db-yaml, kb-search, agent-cost, diataxis-docs) +4. .venv: ladybug + model2vec + mistune +5. schema + tools with TDD (kb + md + facts + brain) +6. ~/.config/brain config +7. corpus extraction (facts/info) +8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..82fc9bf --- /dev/null +++ b/docs/README.md @@ -0,0 +1,10 @@ +# 2dph (deductionphile) + +Evidence-first knowledge graph + hybrid RAG over the operational +Brain/ops/eSlider stack. Facts need proof or they are +`(not confirmed)`. + +- [PLAN.md](../PLAN.md) — decisions, execution order, open questions (v2) +- [design](design.md) — schema, deduction model, sources + +Published docs live here and mirror the project state. \ No newline at end of file diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..198022d --- /dev/null +++ b/docs/design.md @@ -0,0 +1,61 @@ +# Design — facts, info, deduction + +## Two roots, one transaction + +`root` — `facts` | `info` — is a column on every node and edge. Both roots +live in the **same** Ladybug file and are written inside the **same +transaction** (D12). Splitting is semantic, not physical: + +- `facts` — assertions backed by ≥2 independent sources (`confidence: confirmed`) + or marked `partial`/`hypothesis` when the second source is missing. +- `info` — narrative/descriptive leafs (how-tos, READMEs, notes). Searchable, + never asserted as an answer. + +ACID: Ladybug commits facts + info atomically; the audit can treat a snapshot +as one consistent state. + +## Deduction search + +``` +bin/kb/search "question" + 1. facts root — confirmed answers only → return with evidence links + 2. info root — supporting narrative → snippets, marked (not confirmed) + 3. web-search — second independent source → upgrade hypothesis to confirmed +``` + +`--hop N` follows graph edges (sibling leaves under a heading, owning file, +`related:` files, vector-neighbour leaves) — the deduction walk. + +## Who / What / How / Where / When + evidence + +Every assertion edge carries: + +| prop | meaning | +|------|---------| +| who | subject/object persons, services, hosts | +| what | the associated subjects/objects (predicate) | +| how | methods used to read it (compose file? docker ps? ssh config?) | +| where | environment + filesystem path / host | +| when | timestamp / source_rev (git commit, mtime) | +| evidence | ≥2 source refs (compose path, runtime container, config) | +| confidence | confirmed / partial / hypothesis | +| root | facts / info | + +## Versioning — nothing is timeless + +Content leafs: `sha256`, `observed_at`, `source_rev`, `confidence`. Stale = a +file changed on disk (git HEAD/mtime) after its last observed `source_rev`. +`File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person` records the history of every +content leaf. + +`bin/facts/audit stale` flags leafs whose observed revision is behind the +corpus HEAD. + +## Sources (auto-pairing) + +- A: runtime state — `docker ps` (container running), ports actually bound +- B: declared config — `docker-compose.yml`, `~/.ssh/config`, `homeserver.yaml` +- C: narrative — READMEs, AGENTS.md, docs + +Confirmed = A×B or B×C agreement. Single source = hypothesis + `(not confirmed)`. +Conflicting pairings (≥2 yes vs ≥2 no) = hypothesis (OQ1 → v2 resolution). \ No newline at end of file