Merge pull request 'docs: public-ready English repo (README, docs/benchmarks, MIT)' (#7) from docs/public-en into main

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 vado
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,27 +1,114 @@
-# 1b — Gensyn open-1b
+# open-1b — independent verification & local inference
 
-Независимая верификация *auditable training* Gensyn **open-1b** и локальный запуск модели.
+An independent, reproducible check of Gensyn's **open-1b** "auditable training"
+claim — plus a **pure-Go** path to convert and run the model locally.
 
-- **Epic:** vado/1b#1
+> Reference: [Introducing open-1b](https://www.gensyn.ai/news/introducing-open-1b-auditable-training)
-- **Отчёты:** [`reports/`](reports/)
+> · harness: [gensyn-ai/open-transformers](https://github.com/gensyn-ai/open-transformers)
-- **Тела issue:** [`docs/issues/`](docs/issues/)
+> · audit app: [open1b.gensyn.ai](https://open1b.gensyn.ai/)
 
-## Итоги (кратко)
+Everything we wrote is **Go** (no Python in our code or pipeline). For the audit we
+used Gensyn's official harness, which is Python-only and the sole vendor-supplied
+path; for inference we wrote our own Go GGUF converter.
 
-| Задача | Статус |
+## Results
-|--------|--------|
-| Аудит init-unit open-1b (CPU) | ✅ **MATCH** — `state_hash 16554a11…`; см. [`reports/audit-open1b-init.md`](reports/audit-open1b-init.md) |
-| Аудит тренировочного интервала | ⛔ заблокирован (нет опубликованного interval-unit + RAM/диск) |
-| Локальный inference на Arc B580 | ✅ **работает** — Go-конвертер, GGUF байт-идентичен; **~114 tok/s** decode, ~892 prefill, ~3.6 GiB VRAM; см. [`reports/open1b-local-inference.md`](reports/open1b-local-inference.md) |
 
-## Ссылки
+| Item | Result |
+|------|--------|
+| **Init-unit audit** (Linux x86-64 CPU, no NVIDIA) | ✅ **MATCH** — published `state_hash 16554a11…` reproduced bit-for-bit |
+| **Training-interval audit** | ⛔ blocked — no interval unit is published; replay needs ~24 GB RAM / ≥60 GB disk |
+| **GGUF conversion** (Go, `tools/hf2gguf`) | ✅ **byte-identical** to the vendor GGUF (same SHA-256) |
+| **Local inference** (1× Intel Arc B580, Vulkan) | ✅ ~114 tok/s decode · ~892 tok/s prefill · ~3.6 GiB VRAM |
 
-- Анонс: https://www.gensyn.ai/news/introducing-open-1b-auditable-training
+Details: [`docs/audit-open1b-init.md`](docs/audit-open1b-init.md),
-- Harness: https://github.com/gensyn-ai/open-transformers
+[`docs/open1b-local-inference.md`](docs/open1b-local-inference.md),
-- Audit app: https://open1b.gensyn.ai/
+[`docs/benchmarks.md`](docs/benchmarks.md).
 
-## Ограничения (по состоянию на 2026-09)
+## Repository layout
 
-- `repop` (воспроизводимые ядра) — **не open source**, только wheel под «Gensyn Reproducibility License v1.0».
+```
-- v3-хэш **не покрывает** RNG, курсор data-stream, spike state и ключи `meta.json`.
+docs/
-- В self-serve audit-ките опубликованы **только init-units**; interval-units — «added per run once … published».
+  audit-open1b-init.md      # init-unit audit report (raw evidence)
+  open1b-local-inference.md # local bring-up report
+  benchmarks.md             # speed benchmarks + reference comparison
+tools/hf2gguf/              # pure-Go safetensors → GGUF converter (dump/convert/verify)
+```
+
+## Audit — what it proves, and what it does not
+
+**Proves.** The installed artifacts are the ones the kit's `kit.json` published
+(all SHA-256s match); the `repop` build commit matches the manifest; and running
+the vendor harness on a plain CPU regenerates the `1b_repop_v2` initial state hash
+**bit-for-bit** — an independent third party reproduced the run's init commitment.
+
+**Does not prove.**
+
+- **No training interval.** An init-only unit says nothing about any forward/backward/AdamW/data-stream step. The claim about bitwise-reproducible *optimizer steps* is not exercised here.
+- **`repop` is closed source.** Only compiled wheels ship, under the "Gensyn Reproducibility License v1.0". The kernels that make training reproducible cannot be read or independently re-derived.
+- **The v3 hash omits** RNG, the data-stream cursor, spike state, and `meta.json` descriptor keys.
+- **No interval unit is published**, so "pick any step, replay it" is not self-serve checkable yet.
+- Provenance of the *published run* is not established — only that the pinned artifacts reproduce the published commitment on our machine.
+
+## GGUF conversion (pure Go)
+
+`tools/hf2gguf` reads the HF checkpoint (`config.json`, `model.safetensors`,
+`tokenizer.json`) and writes a GGUF file, using
+[`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf) for GGUF
+read/write. The reference GGUF produced by Gensyn's converter is used **read-only
+as the schema oracle**, so the output is validated byte-for-byte.
+
+```bash
+cd tools/hf2gguf
+go build -o hf2gguf .
+
+./hf2gguf dump   -gguf models/open-1b-sft-f16.gguf -o manifest.json
+./hf2gguf convert -ref models/open-1b-sft-f16.gguf \
+    -safetensors models/open-1b-sft/model.safetensors \
+    -config models/open-1b-sft/config.json \
+    -tokenizer-config models/open-1b-sft/tokenizer_config.json \
+    -tokenizer models/open-1b-sft/tokenizer.json \
+    -o models/open-1b-sft-f16-go.gguf
+./hf2gguf verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
+```
+
+`verify` reports `39/39 metadata keys` and `220/220 tensors` byte-identical; the
+two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
+
+## Running the model locally
+
+Serve the (or our Go-built) GGUF on one Arc B580 with llama.cpp's Vulkan backend:
+
+```bash
+llama-server \
+  -m models/open-1b-sft-f16-go.gguf \
+  --device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \
+  --spec-type ngram-map-k,ngram-cache \
+  -t 16 --parallel 1 --alias open-1b --host 127.0.0.1 --port 8085
+```
+
+Notes: `-fa off` is required on these Arc/Vulkan builds (so KV stays f16); the
+requested `-c 32768` is capped to the model's **4096**-token context.
+
+## Caveats
+
+- **Chat-template quirk.** With the exact vendor tail `…assistant<|end_header|>\n\n`,
+  the released SFT checkpoint emits `<|eot|>` as its first token (empty reply).
+  Our GGUF is byte-identical to the vendor one, so this is a property of the
+  released checkpoint, not of the conversion. Workarounds: seed the assistant
+  turn, use `ignore_eos`, or fix the trailing newline.
+- **Not a strong model.** open-1b scores 25.4 on the OLMo2 suite (vs 31.9 for
+  OLMo2-1B). Its value is **verifiability**, not quality — see
+  [`docs/benchmarks.md`](docs/benchmarks.md).
+- **Scope.** We only run/inspect the released weights and replay one published
+  commitment; we do not train or fine-tune anything.
+
+## License
+
+MIT — see [`LICENSE`](LICENSE). The vendor harness (`open-transformers`) is
+Apache-2.0; `repop`, the checkpoints and the corpus are distributed separately by
+Gensyn under their own terms.
+
+## Credits
+
+Built on Gensyn's open-1b release and audit tooling. Independent work; not
+affiliated with Gensyn.

docs/benchmarks.md

@@ -0,0 +1,48 @@
+# Benchmarks
+
+Speed of **open-1b** on our hardware, with a reference comparison against the
+model we run in production (`gemma-4-E2B`). All numbers are single-stream
+(one request at a time) on one GPU.
+
+## Hardware / setup
+
+- **GPU:** 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp `build-vulkan` (b10893).
+- **CPU:** AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM.
+- **Serving flags:** `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`.
+- `-fa off` is mandatory on these Arc/Vulkan builds → KV cache is f16 (no `-ctk q8_0`).
+
+## open-1b (our Go-built GGUF, f16)
+
+| Metric | Value |
+|---|---|
+| Decode | **~112–117 tok/s** (e.g. 256 tok @ 111.7; 96 tok @ 117.2) |
+| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) |
+| VRAM | **~3.6 GiB** (10971 → 7258 MiB free) |
+| Context | 4096 (model limit; the requested 32k is capped) |
+| Weights | f16, ~3.2 GB |
+
+## Reference comparison (1× Arc B580)
+
+| Model | Weights | Context | Decode | Prefill |
+|---|---|---|---|---|
+| **open-1b-sft** (this repo) | f16 ~3.2 GB | 4096 | ~114 tok/s | ~892 tok/s |
+| **gemma-4-E2B** (prod) | Q4 ~1.5 GiB | 64k | **~214 tok/s** | **~1659 tok/s** |
+
+*gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok /
+prefill 1698 tok) on the same card and build.*
+
+**Reading:** gemma is roughly **2× faster**, ~**2.4× smaller**, and has **16×
+the context** of open-1b. For any practical workload (classification, extraction,
+translation, tool-calling) gemma wins outright. open-1b's reason to exist is not
+throughput or quality but **auditable training**.
+
+## Caveats
+
+- This is **not** a controlled A/B: the open-1b figures are a short first-run
+  measurement with different prompts; the gemma figures come from a separate
+  controlled benchmark. Use them as an order-of-magnitude orientation.
+- The `ngram` speculative decoder helps task-dependent amounts; open-1b's
+  baseline (no-spec) number was not measured here.
+- Quality: open-1b scores **25.4** on the OLMo2 suite vs **31.9** for OLMo2-1B,
+  i.e. below an ordinary 1B model. It is a proof of *verifiability*, not a strong
+  assistant.

docs/issues/01-epic.md

@@ -1,26 +0,0 @@
-# Epic: Gensyn open-1b — верификация auditable training и локальный inference
-
-**Цель.** Независимо проверить заявление Gensyn о *bitwise auditable training* (open-1b) на нашем железе и запустить модель локально; зафиксировать факты, ограничения и решения.
-
-## Контекст
-
-- Анонс 2026-09-15: https://www.gensyn.ai/news/introducing-open-1b-auditable-training (+ PRNewswire 302878850)
-- Harness: https://github.com/gensyn-ai/open-transformers (Apache-2.0)
-- Audit CLI: https://github.com/gensyn-ai/gensyn-audit
-- Audit app: https://open1b.gensyn.ai/ (manifest: https://open1b.gensyn.ai/manifest.json)
-- Модель: HF `Gensyn/open-1b-base`, `-midtrained-93B`, `-sft`
-- Модель: 1.61B total / 1.08B non-emb, 24 слоя, 400B токенов, 80 957 шагов, 48×H100
-- 2dph info-leaf (PO, 2026-09-28): `6703767319c6e2f9fa6d63465934d72e`
-
-## Дети эпика
-
-- `#AUDIT-INIT` — Аудит init-unit open-1b (CPU, Linux x86-64)
-- `#AUDIT-INTERVAL` — Аудит тренировочного интервала open-1b (заблокировано)
-- `#INFERENCE` — Локальный запуск open-1b на Arc B580 (llama.cpp Vulkan)
-- `#REPORT` — Сводный отчёт и запись фактов в 2dph
-
-## Критерий закрытия эпика
-
-Все дочерние issues закрыты с доказательствами (тест/лог/хэш). Заблокированный
-интервальный аудит может остаться открытым, если блокер явно задокументирован
-(нет опубликованного interval-unit + ресурсы хоста).

docs/issues/02-audit-init.md

@@ -1,25 +0,0 @@
-Epic: #EPIC
-
-## Цель
-
-Независимо воспроизвести опубликованный **init-хэш** open-1b официальным harness'ом
-на CPU (Linux x86-64, без NVIDIA).
-
-## Критерии приёмки
-
-- [x] kit `pt-f91110a0d0c8_rp-c9ca6e71e673` скачан; sha256 **всех** файлов сверены с `kit.json`
-- [x] `repop.build_info().commit == kit.json.repop_commit` (`c9ca6e71e673`)
-- [x] `pretrain-audit-replay --from-init --until-step 0 --config-name 1b_repop_v2 --device cpu` → `match=true`
-- [x] отчёт с raw-выводом и таблицей sha256
-
-## Результат (2026-09-28)
-
-✅ **MATCH**. `state_hash = 16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed`,
-wall 58.3 s, peak RSS 16.8 GB, exit 0.
-
-Отчёт: `audit/REPORT.md`.
-
-## Ограничение
-
-Init-unit по README **не** проверяет тренировочный интервал — это отдельная задача
-`#AUDIT-INTERVAL`.

docs/issues/03-audit-interval.md

@@ -1,29 +0,0 @@
-Epic: #EPIC
-
-## Цель
-
-Воспроизвести **тренировочный интервал** (replay шага против опубликованного hash) —
-именно это доказывает заявление «bitwise auditable training», в отличие от init-unit.
-
-## Критерии приёмки
-
-- [ ] получен interval-unit с canonical target hash (published commitment на шаг)
-- [ ] скачан чекпоинт + шарды, потреблённые интервалом
-- [ ] `pretrain-audit-replay --checkpoint … --until-step N --gcs-root … --expect-hash …` → `MATCH`
-- [ ] отчёт с raw-выводом
-
-## Блокер (2026-09-28)
-
-- ⛔ В kit `pt-f91110a0d0c8_rp-c9ca6e71e673` `trajectory.json` содержит **только init-units**
-  (`100m_smoke_repop`, `1b_repop_run3`, `1b_repop_v2`). Interval-units «added per run once its
-  checkpoints and state_hash files are published» — **не опубликованы**. Self-serve кит
-  не определяет commit на шаг.
-- ⛔ Ресурсы хоста: нужно ~24 ГБ RAM (есть ~11 ГБ) и ≥60 ГБ диска (есть 39 ГБ);
-  один чекпоинт `ckpt/step_000000100/` = 19.31 ГБ.
-- Решение по разблокировке (владелец — PO/пользователь): запросить у audit-app interval-unit,
-  либо выполнять на хосте с ≥32 ГБ RAM / ≥60 ГБ диска (предпочтительно NVIDIA `--device cuda`).
-
-## Примечание
-
-Per-step log доступен на `gs://gensyn-open-1b/logs/state_hashes.jsonl`; harness не умеет
-автоподстановку, а audit-app гейтит сабмиты — это тоже часть блокера.

docs/issues/04-inference.md

@@ -1,23 +0,0 @@
-Epic: #EPIC
-
-## Цель
-
-Поднять open-1b локально и доказать работоспособность на нашем железе
-(Intel Arc B580, Vulkan; без NVIDIA/CUDA).
-
-## Критерии приёмки
-
-- [ ] llama.cpp собран с Vulkan; бэкенд активен на Arc B580
-- [ ] веса HF (`Gensyn/open-1b-sft`, fallback `-base`) скачаны, provenance = точная ревизия
-- [ ] конвертация в GGUF; собран/выбран квант
-- [ ] `llama-cli`/`llama-server` даёт связный ответ; замер tok/s и VRAM
-- [ ] отчёт с точными командами и caveat'ами
-
-## Прочее
-
-- Сервисы — только docker compose (не systemd).
-- Обвязка — Go; Python только через `uv` (канонический `convert_hf_to_gguf.py`).
-
-## Статус
-
-🔄 выполняется фоновым SE-субагентом (2026-09-28).

docs/issues/05-report.md

@@ -1,16 +0,0 @@
-Epic: #EPIC
-
-## Цель
-
-Свести результаты в отчёт и записать проверенные факты в 2dph (brain), пометив
-что подтверждено, а что остаётся ограничением.
-
-## Критерии приёмки
-
-- [ ] сводный отчёт (что доказано / что нет) в репо
-- [ ] info-leaf в 2dph с итогом (source: `po/gensyn`)
-- [ ] явно зафиксированы ограничения: `repop` closed-source, v3-hash не покрывает RNG/cursor/spike/meta, interval-unit не опубликован
-
-## Статус
-
-info-leaf записан: `6703767319c6e2f9fa6d63465934d72e`. Отчёт — по завершении #INFERENCE и #AUDIT-INTERVAL.