diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..3d65908 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 vado + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index d25b9ba..f2cd973 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,114 @@ -# 1b — Gensyn open-1b +# open-1b — independent verification & local inference -Независимая верификация *auditable training* Gensyn **open-1b** и локальный запуск модели. +An independent, reproducible check of Gensyn's **open-1b** "auditable training" +claim — plus a **pure-Go** path to convert and run the model locally. -- **Epic:** vado/1b#1 -- **Отчёты:** [`reports/`](reports/) -- **Тела issue:** [`docs/issues/`](docs/issues/) +> Reference: [Introducing open-1b](https://www.gensyn.ai/news/introducing-open-1b-auditable-training) +> · harness: [gensyn-ai/open-transformers](https://github.com/gensyn-ai/open-transformers) +> · audit app: [open1b.gensyn.ai](https://open1b.gensyn.ai/) -## Итоги (кратко) +Everything we wrote is **Go** (no Python in our code or pipeline). For the audit we +used Gensyn's official harness, which is Python-only and the sole vendor-supplied +path; for inference we wrote our own Go GGUF converter. -| Задача | Статус | -|--------|--------| -| Аудит init-unit open-1b (CPU) | ✅ **MATCH** — `state_hash 16554a11…`; см. [`reports/audit-open1b-init.md`](reports/audit-open1b-init.md) | -| Аудит тренировочного интервала | ⛔ заблокирован (нет опубликованного interval-unit + RAM/диск) | -| Локальный inference на Arc B580 | ✅ **работает** — Go-конвертер, GGUF байт-идентичен; **~114 tok/s** decode, ~892 prefill, ~3.6 GiB VRAM; см. [`reports/open1b-local-inference.md`](reports/open1b-local-inference.md) | +## Results -## Ссылки +| Item | Result | +|------|--------| +| **Init-unit audit** (Linux x86-64 CPU, no NVIDIA) | ✅ **MATCH** — published `state_hash 16554a11…` reproduced bit-for-bit | +| **Training-interval audit** | ⛔ blocked — no interval unit is published; replay needs ~24 GB RAM / ≥60 GB disk | +| **GGUF conversion** (Go, `tools/hf2gguf`) | ✅ **byte-identical** to the vendor GGUF (same SHA-256) | +| **Local inference** (1× Intel Arc B580, Vulkan) | ✅ ~114 tok/s decode · ~892 tok/s prefill · ~3.6 GiB VRAM | -- Анонс: https://www.gensyn.ai/news/introducing-open-1b-auditable-training -- Harness: https://github.com/gensyn-ai/open-transformers -- Audit app: https://open1b.gensyn.ai/ +Details: [`docs/audit-open1b-init.md`](docs/audit-open1b-init.md), +[`docs/open1b-local-inference.md`](docs/open1b-local-inference.md), +[`docs/benchmarks.md`](docs/benchmarks.md). -## Ограничения (по состоянию на 2026-09) +## Repository layout -- `repop` (воспроизводимые ядра) — **не open source**, только wheel под «Gensyn Reproducibility License v1.0». -- v3-хэш **не покрывает** RNG, курсор data-stream, spike state и ключи `meta.json`. -- В self-serve audit-ките опубликованы **только init-units**; interval-units — «added per run once … published». +``` +docs/ + audit-open1b-init.md # init-unit audit report (raw evidence) + open1b-local-inference.md # local bring-up report + benchmarks.md # speed benchmarks + reference comparison +tools/hf2gguf/ # pure-Go safetensors → GGUF converter (dump/convert/verify) +``` + +## Audit — what it proves, and what it does not + +**Proves.** The installed artifacts are the ones the kit's `kit.json` published +(all SHA-256s match); the `repop` build commit matches the manifest; and running +the vendor harness on a plain CPU regenerates the `1b_repop_v2` initial state hash +**bit-for-bit** — an independent third party reproduced the run's init commitment. + +**Does not prove.** + +- **No training interval.** An init-only unit says nothing about any forward/backward/AdamW/data-stream step. The claim about bitwise-reproducible *optimizer steps* is not exercised here. +- **`repop` is closed source.** Only compiled wheels ship, under the "Gensyn Reproducibility License v1.0". The kernels that make training reproducible cannot be read or independently re-derived. +- **The v3 hash omits** RNG, the data-stream cursor, spike state, and `meta.json` descriptor keys. +- **No interval unit is published**, so "pick any step, replay it" is not self-serve checkable yet. +- Provenance of the *published run* is not established — only that the pinned artifacts reproduce the published commitment on our machine. + +## GGUF conversion (pure Go) + +`tools/hf2gguf` reads the HF checkpoint (`config.json`, `model.safetensors`, +`tokenizer.json`) and writes a GGUF file, using +[`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf) for GGUF +read/write. The reference GGUF produced by Gensyn's converter is used **read-only +as the schema oracle**, so the output is validated byte-for-byte. + +```bash +cd tools/hf2gguf +go build -o hf2gguf . + +./hf2gguf dump -gguf models/open-1b-sft-f16.gguf -o manifest.json +./hf2gguf convert -ref models/open-1b-sft-f16.gguf \ + -safetensors models/open-1b-sft/model.safetensors \ + -config models/open-1b-sft/config.json \ + -tokenizer-config models/open-1b-sft/tokenizer_config.json \ + -tokenizer models/open-1b-sft/tokenizer.json \ + -o models/open-1b-sft-f16-go.gguf +./hf2gguf verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf +``` + +`verify` reports `39/39 metadata keys` and `220/220 tensors` byte-identical; the +two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`. + +## Running the model locally + +Serve the (or our Go-built) GGUF on one Arc B580 with llama.cpp's Vulkan backend: + +```bash +llama-server \ + -m models/open-1b-sft-f16-go.gguf \ + --device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \ + --spec-type ngram-map-k,ngram-cache \ + -t 16 --parallel 1 --alias open-1b --host 127.0.0.1 --port 8085 +``` + +Notes: `-fa off` is required on these Arc/Vulkan builds (so KV stays f16); the +requested `-c 32768` is capped to the model's **4096**-token context. + +## Caveats + +- **Chat-template quirk.** With the exact vendor tail `…assistant<|end_header|>\n\n`, + the released SFT checkpoint emits `<|eot|>` as its first token (empty reply). + Our GGUF is byte-identical to the vendor one, so this is a property of the + released checkpoint, not of the conversion. Workarounds: seed the assistant + turn, use `ignore_eos`, or fix the trailing newline. +- **Not a strong model.** open-1b scores 25.4 on the OLMo2 suite (vs 31.9 for + OLMo2-1B). Its value is **verifiability**, not quality — see + [`docs/benchmarks.md`](docs/benchmarks.md). +- **Scope.** We only run/inspect the released weights and replay one published + commitment; we do not train or fine-tune anything. + +## License + +MIT — see [`LICENSE`](LICENSE). The vendor harness (`open-transformers`) is +Apache-2.0; `repop`, the checkpoints and the corpus are distributed separately by +Gensyn under their own terms. + +## Credits + +Built on Gensyn's open-1b release and audit tooling. Independent work; not +affiliated with Gensyn. diff --git a/reports/audit-open1b-init.md b/docs/audit-open1b-init.md similarity index 100% rename from reports/audit-open1b-init.md rename to docs/audit-open1b-init.md diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..b943f5a --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,48 @@ +# Benchmarks + +Speed of **open-1b** on our hardware, with a reference comparison against the +model we run in production (`gemma-4-E2B`). All numbers are single-stream +(one request at a time) on one GPU. + +## Hardware / setup + +- **GPU:** 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp `build-vulkan` (b10893). +- **CPU:** AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM. +- **Serving flags:** `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`. +- `-fa off` is mandatory on these Arc/Vulkan builds → KV cache is f16 (no `-ctk q8_0`). + +## open-1b (our Go-built GGUF, f16) + +| Metric | Value | +|---|---| +| Decode | **~112–117 tok/s** (e.g. 256 tok @ 111.7; 96 tok @ 117.2) | +| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) | +| VRAM | **~3.6 GiB** (10971 → 7258 MiB free) | +| Context | 4096 (model limit; the requested 32k is capped) | +| Weights | f16, ~3.2 GB | + +## Reference comparison (1× Arc B580) + +| Model | Weights | Context | Decode | Prefill | +|---|---|---|---|---| +| **open-1b-sft** (this repo) | f16 ~3.2 GB | 4096 | ~114 tok/s | ~892 tok/s | +| **gemma-4-E2B** (prod) | Q4 ~1.5 GiB | 64k | **~214 tok/s** | **~1659 tok/s** | + +*gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok / +prefill 1698 tok) on the same card and build.* + +**Reading:** gemma is roughly **2× faster**, ~**2.4× smaller**, and has **16× +the context** of open-1b. For any practical workload (classification, extraction, +translation, tool-calling) gemma wins outright. open-1b's reason to exist is not +throughput or quality but **auditable training**. + +## Caveats + +- This is **not** a controlled A/B: the open-1b figures are a short first-run + measurement with different prompts; the gemma figures come from a separate + controlled benchmark. Use them as an order-of-magnitude orientation. +- The `ngram` speculative decoder helps task-dependent amounts; open-1b's + baseline (no-spec) number was not measured here. +- Quality: open-1b scores **25.4** on the OLMo2 suite vs **31.9** for OLMo2-1B, + i.e. below an ordinary 1B model. It is a proof of *verifiability*, not a strong + assistant. diff --git a/docs/issues/01-epic.md b/docs/issues/01-epic.md deleted file mode 100644 index 39a9c49..0000000 --- a/docs/issues/01-epic.md +++ /dev/null @@ -1,26 +0,0 @@ -# Epic: Gensyn open-1b — верификация auditable training и локальный inference - -**Цель.** Независимо проверить заявление Gensyn о *bitwise auditable training* (open-1b) на нашем железе и запустить модель локально; зафиксировать факты, ограничения и решения. - -## Контекст - -- Анонс 2026-09-15: https://www.gensyn.ai/news/introducing-open-1b-auditable-training (+ PRNewswire 302878850) -- Harness: https://github.com/gensyn-ai/open-transformers (Apache-2.0) -- Audit CLI: https://github.com/gensyn-ai/gensyn-audit -- Audit app: https://open1b.gensyn.ai/ (manifest: https://open1b.gensyn.ai/manifest.json) -- Модель: HF `Gensyn/open-1b-base`, `-midtrained-93B`, `-sft` -- Модель: 1.61B total / 1.08B non-emb, 24 слоя, 400B токенов, 80 957 шагов, 48×H100 -- 2dph info-leaf (PO, 2026-09-28): `6703767319c6e2f9fa6d63465934d72e` - -## Дети эпика - -- `#AUDIT-INIT` — Аудит init-unit open-1b (CPU, Linux x86-64) -- `#AUDIT-INTERVAL` — Аудит тренировочного интервала open-1b (заблокировано) -- `#INFERENCE` — Локальный запуск open-1b на Arc B580 (llama.cpp Vulkan) -- `#REPORT` — Сводный отчёт и запись фактов в 2dph - -## Критерий закрытия эпика - -Все дочерние issues закрыты с доказательствами (тест/лог/хэш). Заблокированный -интервальный аудит может остаться открытым, если блокер явно задокументирован -(нет опубликованного interval-unit + ресурсы хоста). diff --git a/docs/issues/02-audit-init.md b/docs/issues/02-audit-init.md deleted file mode 100644 index c24011c..0000000 --- a/docs/issues/02-audit-init.md +++ /dev/null @@ -1,25 +0,0 @@ -Epic: #EPIC - -## Цель - -Независимо воспроизвести опубликованный **init-хэш** open-1b официальным harness'ом -на CPU (Linux x86-64, без NVIDIA). - -## Критерии приёмки - -- [x] kit `pt-f91110a0d0c8_rp-c9ca6e71e673` скачан; sha256 **всех** файлов сверены с `kit.json` -- [x] `repop.build_info().commit == kit.json.repop_commit` (`c9ca6e71e673`) -- [x] `pretrain-audit-replay --from-init --until-step 0 --config-name 1b_repop_v2 --device cpu` → `match=true` -- [x] отчёт с raw-выводом и таблицей sha256 - -## Результат (2026-09-28) - -✅ **MATCH**. `state_hash = 16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed`, -wall 58.3 s, peak RSS 16.8 GB, exit 0. - -Отчёт: `audit/REPORT.md`. - -## Ограничение - -Init-unit по README **не** проверяет тренировочный интервал — это отдельная задача -`#AUDIT-INTERVAL`. diff --git a/docs/issues/03-audit-interval.md b/docs/issues/03-audit-interval.md deleted file mode 100644 index ba58a4b..0000000 --- a/docs/issues/03-audit-interval.md +++ /dev/null @@ -1,29 +0,0 @@ -Epic: #EPIC - -## Цель - -Воспроизвести **тренировочный интервал** (replay шага против опубликованного hash) — -именно это доказывает заявление «bitwise auditable training», в отличие от init-unit. - -## Критерии приёмки - -- [ ] получен interval-unit с canonical target hash (published commitment на шаг) -- [ ] скачан чекпоинт + шарды, потреблённые интервалом -- [ ] `pretrain-audit-replay --checkpoint … --until-step N --gcs-root … --expect-hash …` → `MATCH` -- [ ] отчёт с raw-выводом - -## Блокер (2026-09-28) - -- ⛔ В kit `pt-f91110a0d0c8_rp-c9ca6e71e673` `trajectory.json` содержит **только init-units** - (`100m_smoke_repop`, `1b_repop_run3`, `1b_repop_v2`). Interval-units «added per run once its - checkpoints and state_hash files are published» — **не опубликованы**. Self-serve кит - не определяет commit на шаг. -- ⛔ Ресурсы хоста: нужно ~24 ГБ RAM (есть ~11 ГБ) и ≥60 ГБ диска (есть 39 ГБ); - один чекпоинт `ckpt/step_000000100/` = 19.31 ГБ. -- Решение по разблокировке (владелец — PO/пользователь): запросить у audit-app interval-unit, - либо выполнять на хосте с ≥32 ГБ RAM / ≥60 ГБ диска (предпочтительно NVIDIA `--device cuda`). - -## Примечание - -Per-step log доступен на `gs://gensyn-open-1b/logs/state_hashes.jsonl`; harness не умеет -автоподстановку, а audit-app гейтит сабмиты — это тоже часть блокера. diff --git a/docs/issues/04-inference.md b/docs/issues/04-inference.md deleted file mode 100644 index 250dafe..0000000 --- a/docs/issues/04-inference.md +++ /dev/null @@ -1,23 +0,0 @@ -Epic: #EPIC - -## Цель - -Поднять open-1b локально и доказать работоспособность на нашем железе -(Intel Arc B580, Vulkan; без NVIDIA/CUDA). - -## Критерии приёмки - -- [ ] llama.cpp собран с Vulkan; бэкенд активен на Arc B580 -- [ ] веса HF (`Gensyn/open-1b-sft`, fallback `-base`) скачаны, provenance = точная ревизия -- [ ] конвертация в GGUF; собран/выбран квант -- [ ] `llama-cli`/`llama-server` даёт связный ответ; замер tok/s и VRAM -- [ ] отчёт с точными командами и caveat'ами - -## Прочее - -- Сервисы — только docker compose (не systemd). -- Обвязка — Go; Python только через `uv` (канонический `convert_hf_to_gguf.py`). - -## Статус - -🔄 выполняется фоновым SE-субагентом (2026-09-28). diff --git a/docs/issues/05-report.md b/docs/issues/05-report.md deleted file mode 100644 index 9cd2777..0000000 --- a/docs/issues/05-report.md +++ /dev/null @@ -1,16 +0,0 @@ -Epic: #EPIC - -## Цель - -Свести результаты в отчёт и записать проверенные факты в 2dph (brain), пометив -что подтверждено, а что остаётся ограничением. - -## Критерии приёмки - -- [ ] сводный отчёт (что доказано / что нет) в репо -- [ ] info-leaf в 2dph с итогом (source: `po/gensyn`) -- [ ] явно зафиксированы ограничения: `repop` closed-source, v3-hash не покрывает RNG/cursor/spike/meta, interval-unit не опубликован - -## Статус - -info-leaf записан: `6703767319c6e2f9fa6d63465934d72e`. Отчёт — по завершении #INFERENCE и #AUDIT-INTERVAL. diff --git a/reports/open1b-local-inference.md b/docs/open1b-local-inference.md similarity index 100% rename from reports/open1b-local-inference.md rename to docs/open1b-local-inference.md diff --git a/tools/hf2gguf/convert_test.go b/tools/hf2gguf/convert_test.go index bb257c7..d9b6f42 100644 --- a/tools/hf2gguf/convert_test.go +++ b/tools/hf2gguf/convert_test.go @@ -26,8 +26,8 @@ func TestF32BitsToF16(t *testing.T) { {"neginf", 0xff800000, 0xfc00}, {"qnan", 0x7fc00000, 0x7e00}, {"snan", 0x7f800001, 0x7e00}, - {"min_normal", 0x38800000, 0x0400}, // 2^-14 - {"min_subnormal", 0x33800000, 0x0001}, // 2^-24 + {"min_normal", 0x38800000, 0x0400}, // 2^-14 + {"min_subnormal", 0x33800000, 0x0001}, // 2^-24 {"tie_to_even_zero", 0x33000000, 0x0000}, // 2^-25 (half of min subnormal) {"tie_to_even_two", 0x33c00000, 0x0002}, // 3 * 2^-25 {"one_tenth", 0x3dcccccd, 0x2e66}, diff --git a/tools/hf2gguf/hfconfig.go b/tools/hf2gguf/hfconfig.go index 81cf19e..d0436dc 100644 --- a/tools/hf2gguf/hfconfig.go +++ b/tools/hf2gguf/hfconfig.go @@ -14,23 +14,23 @@ import ( ) type hfConfigFile struct { - ModelType string `json:"model_type"` - HiddenSize int64 `json:"hidden_size"` - NumHiddenLayers int64 `json:"num_hidden_layers"` - NumAttentionHeads int64 `json:"num_attention_heads"` - NumKeyValueHeads int64 `json:"num_key_value_heads"` - HeadDim int64 `json:"head_dim"` - IntermediateSize int64 `json:"intermediate_size"` - VocabSize int64 `json:"vocab_size"` - MaxPositionEmbeddings int64 `json:"max_position_embeddings"` + ModelType string `json:"model_type"` + HiddenSize int64 `json:"hidden_size"` + NumHiddenLayers int64 `json:"num_hidden_layers"` + NumAttentionHeads int64 `json:"num_attention_heads"` + NumKeyValueHeads int64 `json:"num_key_value_heads"` + HeadDim int64 `json:"head_dim"` + IntermediateSize int64 `json:"intermediate_size"` + VocabSize int64 `json:"vocab_size"` + MaxPositionEmbeddings int64 `json:"max_position_embeddings"` RopeTheta float64 `json:"rope_theta"` RMSNormEps float64 `json:"rms_norm_eps"` - TieWordEmbeddings bool `json:"tie_word_embeddings"` - QKNorm bool `json:"qk_norm"` - QKNormGain bool `json:"qk_norm_gain"` - EmbeddingNorm bool `json:"embedding_norm"` - SlidingWindow int64 `json:"sliding_window"` - SWAFullEvery int64 `json:"swa_full_every"` + TieWordEmbeddings bool `json:"tie_word_embeddings"` + QKNorm bool `json:"qk_norm"` + QKNormGain bool `json:"qk_norm_gain"` + EmbeddingNorm bool `json:"embedding_norm"` + SlidingWindow int64 `json:"sliding_window"` + SWAFullEvery int64 `json:"swa_full_every"` } func findKV(kvs []rawKV, key string) (rawKV, bool) { @@ -100,9 +100,9 @@ func validateConfig(path string, kvs []rawKV) error { } type tokConfigFile struct { - EOSToken string `json:"eos_token"` - ModelMaxLen int64 `json:"model_max_length"` - ChatTemplate string `json:"chat_template"` + EOSToken string `json:"eos_token"` + ModelMaxLen int64 `json:"model_max_length"` + ChatTemplate string `json:"chat_template"` AddedDecoders map[string]struct { Content string `json:"content"` } `json:"added_tokens_decoder"` diff --git a/tools/hf2gguf/rawkv.go b/tools/hf2gguf/rawkv.go index f373221..474a5c9 100644 --- a/tools/hf2gguf/rawkv.go +++ b/tools/hf2gguf/rawkv.go @@ -90,9 +90,9 @@ func (t ggufBType) size() int { type rawKV struct { Key string BType ggufBType - Int int64 // integer/bool scalars - Float float64 // float scalars - Str string // string scalars + Int int64 // integer/bool scalars + Float float64 // float scalars + Str string // string scalars ElemType ggufBType // arrays: element type Count uint64 // arrays: element count Raw []byte // arrays: concatenated element bytes (excludes elem_type+count)