diff --git a/README.md b/README.md index cb1d45e..be487ce 100644 --- a/README.md +++ b/README.md @@ -1,54 +1,49 @@ -# open-1b — independent verification & local inference +# open-1b: an independent check, and a local run -An independent, reproducible check of Gensyn's **open-1b** "auditable training" -claim — plus a **pure-Go** path to convert and run the model locally. +We looked at Gensyn's open-1b and its claim of auditable training, then ran the +model on our own hardware. The audit uses Gensyn's official harness. Everything +we wrote is Go. -> Reference: [Introducing open-1b](https://www.gensyn.ai/news/introducing-open-1b-auditable-training) -> · harness: [gensyn-ai/open-transformers](https://github.com/gensyn-ai/open-transformers) -> · audit app: [open1b.gensyn.ai](https://open1b.gensyn.ai/) - -Everything we wrote is **Go** (no Python in our code or pipeline). For the audit we -used Gensyn's official harness, which is Python-only and the sole vendor-supplied -path; for inference we wrote our own Go GGUF converter. +Links: [announcement](https://www.gensyn.ai/news/introducing-open-1b-auditable-training), +[harness](https://github.com/gensyn-ai/open-transformers), +[audit app](https://open1b.gensyn.ai/). ## Results -| Item | Result | -|------|--------| -| **Init-unit audit** (Linux x86-64 CPU, no NVIDIA) | ✅ **MATCH** — published `state_hash 16554a11…` reproduced bit-for-bit | -| **Training-interval audit** | ⛔ blocked — no interval unit is published; replay needs ~24 GB RAM / ≥60 GB disk | -| **GGUF conversion** (Go, `tools/hf2gguf`) | ✅ **byte-identical** to the vendor GGUF (same SHA-256) | -| **Local inference** (1× Intel Arc B580, Vulkan) | ✅ ~114 tok/s decode · ~892 tok/s prefill · ~3.6 GiB VRAM | +| Check | Result | +| --- | --- | +| Init-unit audit, x86-64 CPU, no NVIDIA | Match. The published state hash `16554a11...` is reproduced exactly. | +| Training-interval audit | Not run. No interval unit is published, and a replay needs about 24 GB RAM and 60 GB disk. | +| GGUF conversion (Go) | Byte-identical to the reference conversion, same SHA-256. | +| Local inference, one Intel Arc B580 | 114 tok/s decode, 892 tok/s prefill, about 3.6 GiB VRAM. | -Details: [`docs/audit-open1b-init.md`](docs/audit-open1b-init.md), -[`docs/open1b-local-inference.md`](docs/open1b-local-inference.md), -[`docs/benchmarks.md`](docs/benchmarks.md), -[`docs/inference-stack.md`](docs/inference-stack.md). +Reports: [audit](docs/audit-open1b-init.md), [local run](docs/open1b-local-inference.md), +[benchmarks](docs/benchmarks.md), [inference stack](docs/inference-stack.md). ## How it works -**Audit — replay one published commitment and compare.** +Audit. Replay one published commitment and compare hashes. ```mermaid flowchart LR - KIT["Gensyn audit kit"] --> SHA{"sha256\nvs kit.json"} - SHA --> WHL["repop + pretrain wheels"] - WHL --> REPLAY["pretrain-audit-replay\n--from-init --device cpu"] - REPLAY --> H1["regenerated\ninit state hash"] - KIT --> H2["published\ncommitment"] - H1 --> CMP{"equal?"} + KIT["Gensyn audit kit"] --> SHA{"sha256
against kit.json"} + SHA --> WHL["repop and pretrain wheels"] + WHL --> REPLAY["pretrain-audit-replay
--from-init --device cpu"] + REPLAY --> H1["regenerated init hash"] + KIT --> H2["published commitment"] + H1 --> CMP{"equal"} H2 --> CMP - CMP -->|yes| OK["MATCH"] - CMP -->|no| BAD["MISMATCH"] + CMP -->|yes| OK["match"] + CMP -->|no| BAD["mismatch"] ``` -**Inference — convert in Go, serve on one Arc (the engine is a patched llama.cpp).** +Inference. Convert in Go, serve on one Arc. The engine is a patched llama.cpp. ```mermaid flowchart LR - HF["HF checkpoint\nmodel.safetensors (fp32)"] --> GO["tools/hf2gguf\n(pure Go)"] - GO --> GGUF["GGUF (f16)\nbyte-identical to reference"] - GGUF --> LC["llama.cpp Vulkan\nupstream + open1b.patch"] + HF["HF checkpoint
model.safetensors, fp32"] --> GO["tools/hf2gguf
Go"] + GO --> GGUF["GGUF, f16
identical to reference"] + GGUF --> LC["llama.cpp Vulkan
upstream plus open1b patch"] LC --> GPU["Intel Arc B580"] GPU --> API["llama-server HTTP"] ``` @@ -57,43 +52,44 @@ flowchart LR ``` docs/ - audit-open1b-init.md # init-unit audit report (raw evidence) - open1b-local-inference.md # local bring-up report - benchmarks.md # speed benchmarks + reference comparison - inference-stack.md # how llama.cpp was built/launched (NOT stock) - llama-cpp-open1b.patch # local Open1B patch applied to llama.cpp -tools/hf2gguf/ # pure-Go safetensors → GGUF converter (dump/convert/verify) + audit-open1b-init.md + open1b-local-inference.md + benchmarks.md + inference-stack.md + llama-cpp-open1b.patch +tools/hf2gguf/ ``` -## Audit — what it proves, and what it does not +## The audit, and its limits -**Proves.** The installed artifacts are the ones the kit's `kit.json` published -(all SHA-256s match); the `repop` build commit matches the manifest; and running -the vendor harness on a plain CPU regenerates the `1b_repop_v2` initial state hash -**bit-for-bit** — an independent third party reproduced the run's init commitment. +The init unit regenerates the published initial state hash exactly on a plain +CPU. The installed wheels match the kit manifest, and the repop build commit +matches the manifest. -**Does not prove.** +It does not cover a training step. An init-only unit says nothing about the +forward pass, the backward pass, the optimizer, or the data stream, so the +per-step claim stays untested here. Three more reasons to be careful: -- **No training interval.** An init-only unit says nothing about any forward/backward/AdamW/data-stream step. The claim about bitwise-reproducible *optimizer steps* is not exercised here. -- **`repop` is closed source.** Only compiled wheels ship, under the "Gensyn Reproducibility License v1.0". The kernels that make training reproducible cannot be read or independently re-derived. -- **The v3 hash omits** RNG, the data-stream cursor, spike state, and `meta.json` descriptor keys. -- **No interval unit is published**, so "pick any step, replay it" is not self-serve checkable yet. -- Provenance of the *published run* is not established — only that the pinned artifacts reproduce the published commitment on our machine. +- repop ships as compiled wheels only, under a custom licence. The kernels that + make training reproducible cannot be read or checked. +- The v3 state hash omits the RNG, the data-stream cursor, spike state, and the + meta.json descriptor keys. +- No interval unit is published, so "replay any step" is not yet something an + outsider can do without permission. -## GGUF conversion (pure Go) +A match shows that the pinned artifacts reproduce the published commitment on +our machine. It does not show how the original run was carried out. -`tools/hf2gguf` reads the HF checkpoint (`config.json`, `model.safetensors`, -`tokenizer.json`) and writes a GGUF file, using -[`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf) for GGUF -read/write. A reference GGUF produced by llama.cpp's `convert_hf_to_gguf.py` -**with our [Open1B patch](docs/llama-cpp-open1b.patch)** is used **read-only as -the schema oracle**, so the output is validated byte-for-byte. +## Conversion in Go + +`tools/hf2gguf` reads the Hugging Face checkpoint (config.json, +model.safetensors, tokenizer.json) and writes a GGUF file, using +[go-gguf](https://github.com/cymertek/go-gguf). A reference GGUF from llama.cpp's +converter, built with our open1b patch, is read only, as the schema to match. ```bash cd tools/hf2gguf go build -o hf2gguf . - -./hf2gguf dump -gguf models/open-1b-sft-f16.gguf -o manifest.json ./hf2gguf convert -ref models/open-1b-sft-f16.gguf \ -safetensors models/open-1b-sft/model.safetensors \ -config models/open-1b-sft/config.json \ @@ -103,48 +99,40 @@ go build -o hf2gguf . ./hf2gguf verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf ``` -`verify` reports `39/39 metadata keys` and `220/220 tensors` byte-identical; the -two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`. +`verify` reports 39 of 39 metadata keys and 220 of 220 tensors identical. The two +files share SHA-256 +`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`. -## Running the model locally +## Running the model -> **Engine caveat:** open-1b needs an `open1b` architecture that upstream llama.cpp -> does not have. We run a build with a local patch — see -> [`docs/inference-stack.md`](docs/inference-stack.md). - -Serve the (or our Go-built) GGUF on one Arc B580 with llama.cpp's Vulkan backend: +Upstream llama.cpp has no open1b architecture, so the model needs a patched +build. Details in the [inference stack](docs/inference-stack.md). ```bash -llama-server \ - -m models/open-1b-sft-f16-go.gguf \ +llama-server -m models/open-1b-sft-f16-go.gguf \ --device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \ --spec-type ngram-map-k,ngram-cache \ -t 16 --parallel 1 --alias open-1b --host 127.0.0.1 --port 8085 ``` -Notes: `-fa off` is required on these Arc/Vulkan builds (so KV stays f16); the -requested `-c 32768` is capped to the model's **4096**-token context. +`-fa off` is required on these Arc/Vulkan builds, which keeps the KV cache in +f16. The requested 32k context is capped to the model's 4096. ## Caveats -- **Chat-template quirk.** With the exact vendor tail `…assistant<|end_header|>\n\n`, - the released SFT checkpoint emits `<|eot|>` as its first token (empty reply). - Our GGUF is byte-identical to the vendor one, so this is a property of the - released checkpoint, not of the conversion. Workarounds: seed the assistant - turn, use `ignore_eos`, or fix the trailing newline. -- **Not a strong model.** open-1b scores 25.4 on the OLMo2 suite (vs 31.9 for - OLMo2-1B). Its value is **verifiability**, not quality — see - [`docs/benchmarks.md`](docs/benchmarks.md). -- **Scope.** We only run/inspect the released weights and replay one published - commitment; we do not train or fine-tune anything. +- Chat template. With the vendor tail `...assistant<|end_header|>` followed by a + blank line, the released SFT checkpoint emits an end-of-turn token first and + returns an empty reply. Our GGUF is identical to the reference, so this is a + property of the released weights. Seed the assistant turn, set ignore_eos, or + adjust the trailing newline. +- Model quality. open-1b scores 25.4 on the OLMo2 suite, against 31.9 for + OLMo2-1B. It is a test of verifiability, not a strong model. +- Scope. We run and inspect released weights and replay one published + commitment. We do not train or fine-tune anything. ## License -MIT — see [`LICENSE`](LICENSE). The vendor harness (`open-transformers`) is -Apache-2.0; `repop`, the checkpoints and the corpus are distributed separately by -Gensyn under their own terms. +MIT, see [LICENSE](LICENSE). The harness is Apache-2.0. repop, the checkpoints, +and the corpus are distributed by Gensyn under their own terms. -## Credits - -Built on Gensyn's open-1b release and audit tooling. Independent work; not -affiliated with Gensyn. +Built on Gensyn's release. Not affiliated with Gensyn. diff --git a/docs/audit-open1b-init.md b/docs/audit-open1b-init.md index 12be3dc..8a7c9cb 100644 --- a/docs/audit-open1b-init.md +++ b/docs/audit-open1b-init.md @@ -1,214 +1,132 @@ -# Gensyn OPEN-1B audit — independent replay (CPU, Linux x86-64) +# Audit: open-1b init unit on CPU -- **Date:** 2026-09-28 -- **Host:** `brainstation`, Linux x86-64, kernel `7.3.0-070300rc1-generic` (Ubuntu), **no NVIDIA GPU** -- **Auditor:** SE sub-agent (independent) -- **Workdir:** `/root/projects/gensyn/audit` -- **Harness:** `gensyn-ai/open-transformers` @ `d7b7b674cd851f7a16815229f060658180cb772b` - ("Initial commit: the OPEN-1B pretraining and audit-replay harness", 2026-09-15), cloned via git. -- **Verdict:** ✅ **MATCH** on the `1b_repop_v2` init unit. ❌ No training-interval - replay run (not defined by this kit and not feasible on this host — see §7). +- Date: 2026-09-28 +- Host: brainstation, Linux x86-64, kernel 7.3.0, no NVIDIA GPU +- Harness: `gensyn-ai/open-transformers` at `d7b7b674` +- Workdir: `audit/` +- Verdict: match on the `1b_repop_v2` init unit. No training interval was replayed. ---- +## Kit -## 1. Kit selection +The audit app publishes the mapping at `https://open1b.gensyn.ai/manifest.json`: -Four kits were listed. All four `trajectory.json` files declare the **same** -`1b_repop_v2` init commitment -`16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed`, so the choice -of kit does not change the init hash under test. The authoritative mapping is -published by the audit app itself at `https://open1b.gensyn.ai/manifest.json`: - -```json -"run": { "id": "20260722-213626-ad3276b", "model": "1b_repop_v2", - "initStateHash": "16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed" }, -"audit": { "kit": { "id": "pt-f91110a0d0c8_rp-c9ca6e71e673", - "urlBase": "https://storage.googleapis.com/gensyn-audit-public/audit-kit/pt-f91110a0d0c8_rp-c9ca6e71e673" } } +``` +run: 1b_repop_v2 + initStateHash 16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed +audit: kit pt-f91110a0d0c8_rp-c9ca6e71e673 ``` -**Kit under test: `pt-f91110a0d0c8_rp-c9ca6e71e673`.** +All four published kits declare the same init commitment, so the choice of kit +does not change the hash under test. -- `pretrain_commit`: `f91110a0d0c84fa3bad808f365567d9c655d10b2` -- `repop_commit`: `c9ca6e71e673660afc6c87de2592fa6f62c87b15` -- `torch_requirement`: `torch==2.10.0` +- pretrain commit: `f91110a0d0c84fa3bad808f365567d9c655d10b2` +- repop commit: `c9ca6e71e673660afc6c87de2592fa6f62c87b15` +- torch: `2.10.0` -(The other three kits — `pt-cb4ac29438bc_rp-…`, `pt-f6bc086dfebd_rp-…`, -`pt-f5eece09abe6_rp-…` — carry the same init commitment but are older -pretrain/repop pins. Two share the `c9ca6e71e673` repop but a different pretrain -commit; the app manifest picks `f91110a0d0c8`.) +## Toolchain -## 2. Toolchain +| Component | Version | +| --- | --- | +| Python | 3.11.16, via uv. System Python untouched. | +| uv | 0.12.17 | +| PyTorch | 2.10.0+cpu | +| repop | 0.1.5, build commit `c9ca6e71e673`, matches kit.json | +| pretrain | 0.1.0, kit wheel | +| Go | 1.27, for all glue and checks | -| Component | Version / identity | -|---|---| -| Python | 3.11.16 (managed by `uv`; system Python 3.14 untouched) | -| `uv` | 0.12.17 | -| PyTorch | `2.10.0+cpu` (`torch.version.cuda == None`) | -| `repop` | 0.1.5, `build_info().commit == c9ca6e71e673660afc6c87de2592fa6f62c87b15` ✅ (== kit.json `repop_commit`); backends `["cpu","cuda"]` | -| `pretrain` | 0.1.0 (kit wheel) | -| Go (glue tools) | go1.27.0 (fabric rule: all glue/parsing in Go) | +The runbook targets Linux with an NVIDIA GPU and points at the CUDA 12.9 index. +This host has no GPU. The README names CPU as the reference device, so the pinned +torch version was installed from the CPU index. Init is device-independent by +construction, so the result is unaffected. -**Deviation, stated plainly:** the RUNBOOK targets Linux hosts with an NVIDIA GPU -and points them at the CUDA 12.9 PyTorch index. This host has no GPU, and the -README names CPU as the *reference* device, so the exact pin `torch==2.10.0` was -installed from PyTorch's official **CPU** index (`.../whl/cpu`, wheel -`2.10.0+cpu`). The pinned version is unchanged; only the CPU build variant was -chosen deliberately, not by an accidental PyPI fallback. Init is device-independent -by construction (see §6), so this does not weaken the init result. - -Python (`uv`-managed) is used **only** because the vendor harness is Python and -ships as wheels; all download/verification/listing glue is Go under `tools/` -(`kitinspect`, `trajinspect`, `kitfetch`, `gslist`). - -## 3. Commands run +## Commands ```bash -# workdir + docs -mkdir -p /root/projects/gensyn/audit && cd /root/projects/gensyn/audit git clone --depth 1 https://github.com/gensyn-ai/open-transformers - -# kit manifests + trajectories (Go inspector) go run ./tools/kitinspect kits/*/ -go run ./tools/trajinspect kits/*/trajectory.json +go run ./tools/kitfetch kits/pt-f91110a0d0c8_rp-c9ca6e71e673 kit-pt-f91110a0d0c8_rp-c9ca6e71e673 -# download + per-file sha256 verification against kit.json (Go tool, -# downloads to .part, verifies, removes on mismatch) -go run ./tools/kitfetch kits/pt-f91110a0d0c8_rp-c9ca6e71e673 \ - kit-pt-f91110a0d0c8_rp-c9ca6e71e673 - -# Python 3.11 env uv venv --python 3.11 .venv uv pip install --python .venv --index-url https://download.pytorch.org/whl/cpu "torch==2.10.0" uv pip install --python .venv \ - ./kit-.../repop-0.1.5-cp311-cp311-linux_x86_64.whl \ - ./kit-.../pretrain-0.1.0-cp311-cp311-linux_x86_64.whl + ./kit-pt-f91110a0d0c8_rp-c9ca6e71e673/repop-0.1.5-cp311-cp311-linux_x86_64.whl \ + ./kit-pt-f91110a0d0c8_rp-c9ca6e71e673/pretrain-0.1.0-cp311-cp311-linux_x86_64.whl -# provenance assertions -.venv/bin/python -c "import repop,json; print(repop.build_info())" - -# THE AUDIT (init unit) -PRETRAIN_AUDIT_MEMLOG=1 /usr/bin/time -v .venv/bin/pretrain-audit-replay \ - --from-init --until-step 0 \ - --config-name 1b_repop_v2 \ - --device cpu \ +.venv/bin/pretrain-audit-replay --from-init --until-step 0 \ + --config-name 1b_repop_v2 --device cpu \ --expect-hash 16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed ``` -## 4. SHA-256 verification table (all 5 kit files) +## Checksums -Every file was hashed **before** install; the Go fetcher adopts a file only on a -full 64-hex match and deletes rejected bytes. Expected values are read from the -kit's own `kit.json`. +All five kit files were hashed before install. The Go fetcher accepts a file only +on a full match and deletes rejected bytes. Expected values come from kit.json. -| File | expected (`kit.json`) | computed | result | -|---|---|---|---| -| `pretrain-0.1.0-cp311-cp311-linux_x86_64.whl` | `ecc49d1c05c0e2876a59d98d877795769d6e9620eb66e4b4c866b16f695fb174` | same | ✅ OK | -| `pretrain-0.1.0-cp311-cp311-macosx_14_0_arm64.whl` | `fbd23a54aa0bf87775e86493d67f759d1c1f4cc1dcd63ad991fba59dcb92491c` | same | ✅ OK | -| `repop-0.1.5-cp311-cp311-linux_x86_64.whl` | `15f171643871211b2a772ee83c94833c0be2eef0495790cedc33813c95147ff0` | same | ✅ OK | -| `repop-0.1.5-cp311-cp311-macosx_14_0_arm64.whl` | `d8658e0327c7278812ecb6dd455cd4cf8723a974a78788adba41d5f747289c69` | same | ✅ OK | -| `trajectory.json` | `9435515c37d8956d0c81d5bc78c0a383bb6bf27c57b5dbaf1f8be564df0b324a` | same | ✅ OK | +| File | SHA-256 | Result | +| --- | --- | --- | +| `pretrain-0.1.0-cp311-cp311-linux_x86_64.whl` | `ecc49d1c05c0e2876a59d98d877795769d6e9620eb66e4b4c866b16f695fb174` | match | +| `pretrain-0.1.0-cp311-cp311-macosx_14_0_arm64.whl` | `fbd23a54aa0bf87775e86493d67f759d1c1f4cc1dcd63ad991fba59dcb92491c` | match | +| `repop-0.1.5-cp311-cp311-linux_x86_64.whl` | `15f171643871211b2a772ee83c94833c0be2eef0495790cedc33813c95147ff0` | match | +| `repop-0.1.5-cp311-cp311-macosx_14_0_arm64.whl` | `d8658e0327c7278812ecb6dd455cd4cf8723a974a78788adba41d5f747289c69` | match | +| `trajectory.json` | `9435515c37d8956d0c81d5bc78c0a383bb6bf27c57b5dbaf1f8be564df0b324a` | match | -Byte sizes also matched `kit.json`. Only the two **linux_x86_64** wheels were -installed; the macOS arm64 wheels were downloaded and verified but not installed. +Only the two linux_x86_64 wheels were installed. -## 5. Init-unit result — MATCH - -Raw result object printed by the harness (stdout, exit status 0): +## Result ```json { "step": 0, - "consumed_tokens": 0, "state_hash": "16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed", "mode": "init", - "repop": { "commit": "c9ca6e71e673660afc6c87de2592fa6f62c87b15", - "backends": ["cpu", "cuda"], "cuda_arch_list": "7.5 8.0 9.0" }, "device": "cpu", "expected": "16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed", - "match": true, - "cpu_threads": { "requested": "auto", "source": "runtime-default", - "torch_num_threads": 8, "omp_num_threads": null, "mkl_num_threads": null } + "match": true } ``` Log line: `from-init: regenerated init state_hash=16554a1119745f1b expected=16554a1119745f1b MATCH=True`. -Config `1b_repop_v2`, seed `42`, loss path `fused CE+z-loss`. The only warning is -the expected `not distributed + no CUDA — parallelize applies AC only (FSDP -skipped; dev mode)`, i.e. the documented CPU reference path. -**Resource notes:** wall **58.3 s**, user 78.5 s / sys 61.1 s, **peak RSS -16.80 GB** (16,804,956 KB), **0 swaps**, exit 0. (README/RUNBOOK size this unit at -~21.9 GB peak on CPU; we measured 16.8 GB. Free RAM was ~11 GB and the unit still -completed without swapping, so the doc figure is conservative.) +Resources: 58.3 s wall, peak RSS 16.8 GB, no swap, exit 0. -## 6. What this proves — and what it does not +## Limits -**Proves.** The bytes installed are the bytes `kit.json` published (all five -SHA-256s match). The `repop` kernel build is the one the kit claims -(`build_info().commit == repop_commit`). Running the vendor harness on this -CPU, from seed, the `1b_repop_v2` **initial state** — seeded device-independent -trunc-normal init plus zeroed optimizer moments — reproduces the published -canonical hash `16554a11…` **bit-for-bit**. Independent third-party -reproduction of the run's init commitment: ✅. +1. No training interval. An init-only unit says nothing about the forward pass, + the backward pass, the optimizer, or the data stream. The per-step claim is + not exercised. +2. repop is closed source. Only compiled wheels ship, under the Gensyn + Reproducibility Licence. The kernels that make training reproducible cannot be + read or checked. The init unit does not even run the kernels changed by the + recent repop repin. +3. Hash coverage. The v3 hash covers weights, optimizer state, gradients, and the + batch digest. It omits the RNG, the data-stream cursor, spike state, and the + meta.json descriptor keys. +4. Init is the easy case. init.py samples Philox on CPU and byte-copies, so it is + device-independent by design. The hard cross-device claim lives in the + training kernels, which this unit does not touch. +5. Provenance of the published run is not established. We only show that the + pinned artifacts reproduce the published commitment on this machine. -**Does not prove.** +## Interval replay was not attempted -1. **No training interval.** Per the README: "An initialization-only unit does - not verify a training interval." A match here says nothing about any forward, - backward, AdamW or data-stream step. The claimed bitwise reproducibility of - *optimizer steps* is **not** tested by this run. -2. **`repop` is closed source.** Only compiled wheels are shipped; the kernel - arithmetic that makes training bitwise reproducible cannot be read or - independently re-derived from this kit. The init unit does not even exercise - the kernels that the recent `c9ca6e71e673` repin changed (the trajectory notes - say exactly this), so a match is *not* evidence for the kernel-level BFR - contract at that commit. -3. **v3-hash coverage.** The README states the v3 hash covers weights, optimizer - state, gradients and the batch digest, but **not** RNG, the data-stream - cursor, spike state, or the `meta.json` descriptor keys. Even a matching - interval hash would leave those unverified. -4. **Init is the easy case by design.** `init.py` samples Philox on CPU and - byte-copies, so it is device-independent; it is expected to match across - devices. The hard cross-device claim lives in the training kernels, which - this unit does not touch. -5. Provenance of the *published run* is not established; only that the pinned - artifacts reproduce the published commitment on this machine. +This is a decision, not a stall. -## 7. Step-interval replay — deliberately not attempted +- Not defined by the kit. This kit's trajectory.json contains only three init + units (`100m_smoke_repop`, `1b_repop_run3`, `1b_repop_v2`). No interval unit + with a canonical target hash is published, so there is no commitment to check. +- Memory. A step replay needs about 24 GB. This host had about 11 GB free, and + the init unit already peaked at 16.8 GB. +- Disk. One checkpoint `ckpt/step_000000100/` is 19.31 GB; the full `ckpt/` tree + is 15.64 TB. With 39 GB free, a single checkpoint is tight. -Stopping after the init unit is a considered decision, not a stall: +Re-attempt on a host with 32 GB or more RAM and 60 GB or more disk, ideally with +a CUDA GPU. -- **Not defined by the kit.** This kit's `trajectory.json` (`init-units-v1`) - contains **only 3 init units** — `100m_smoke_repop`, `1b_repop_run3`, - `1b_repop_v2`. Its own note says: "Step-interval units are added per run once - its checkpoints and state_hash files are published." **No interval unit with a - canonical target hash is published here**, so the kit defines no - step-interval commitment to check. (The run's per-step log exists at - `gs://gensyn-open-1b/logs/state_hashes.jsonl`, but the harness has no automatic - lookup and the audit app gates submissions.) -- **Memory.** A step-interval replay needs ~24 GB (README), and the audit's - auto-offload only reaches "every published unit" at that figure. This host had - ~11 GB RAM available (29 GB total, 18 GB in use); an init unit already peaked at - 16.8 GB. The heavy replay would swap-thrash and would not fit honestly. -- **Disk/size.** One checkpoint `ckpt/step_000000100/` is **19.31 GB** - (150 objects); the full `ckpt/` prefix is **15.64 TB** (810 checkpoint dirs). - With 39 GB free, a single checkpoint is tight against free space and the - documented `--from-init + --gcs-root` combination is broken (wrong shard - anchor), so a from-init interval would first require staging shards via - `--data-root`. +## Files -Re-attempting an interval audit should be done on a host with ≥32 GB free RAM -(NVIDIA GPU preferred: `--device cuda`) and ≥60 GB free disk. - -## 8. Files produced - -- `open-transformers/` — harness clone (docs read: `README.md`, - `docs/audit-replay-usage.md`, `scripts/audit_volunteer/RUNBOOK.md`) -- `kits//kit.json`, `kits//trajectory.json` — manifests -- `kit-pt-f91110a0d0c8_rp-c9ca6e71e673/` — verified wheels + trajectory -- `.venv/` — Python 3.11 audit env -- `init-replay.log` — full captured replay output + `/usr/bin/time -v` -- `tools/{kitinspect,trajinspect,kitfetch,gslist}` — Go glue -- `REPORT.md` — this file - -No secrets were written. Nothing outside the workdir was modified or deleted. +- `open-transformers/`: harness clone +- `kits/`: manifests +- `kit-pt-f91110a0d0c8_rp-c9ca6e71e673/`: verified wheels +- `.venv/`: Python 3.11 audit environment +- `tools/`: Go glue (`kitinspect`, `trajinspect`, `kitfetch`, `gslist`) +- `init-replay.log`: full replay output diff --git a/docs/benchmarks.md b/docs/benchmarks.md index b943f5a..6f5451d 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -1,48 +1,41 @@ # Benchmarks -Speed of **open-1b** on our hardware, with a reference comparison against the -model we run in production (`gemma-4-E2B`). All numbers are single-stream -(one request at a time) on one GPU. +open-1b on our hardware, with the production model, gemma-4-E2B, as reference. +Single stream, one GPU. -## Hardware / setup +## Setup -- **GPU:** 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp `build-vulkan` (b10893). -- **CPU:** AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM. -- **Serving flags:** `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`. -- `-fa off` is mandatory on these Arc/Vulkan builds → KV cache is f16 (no `-ctk q8_0`). +- GPU: one Intel Arc B580 (BMG G21, 12 GB), Vulkan, llama.cpp build-vulkan (b10893). +- CPU: AMD Ryzen 7 7700X, 29 GB RAM. +- Flags: `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`. +- `-fa off` is required on these Arc/Vulkan builds, so the KV cache stays in f16. -## open-1b (our Go-built GGUF, f16) +## open-1b, Go-built GGUF, f16 | Metric | Value | -|---|---| -| Decode | **~112–117 tok/s** (e.g. 256 tok @ 111.7; 96 tok @ 117.2) | -| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) | -| VRAM | **~3.6 GiB** (10971 → 7258 MiB free) | -| Context | 4096 (model limit; the requested 32k is capped) | -| Weights | f16, ~3.2 GB | +| --- | --- | +| Decode | 112 to 117 tok/s | +| Prefill | about 892 tok/s | +| VRAM | about 3.6 GiB | +| Context | 4096, the model limit | +| Weights | f16, about 3.2 GB | -## Reference comparison (1× Arc B580) +## Reference, one Arc B580 | Model | Weights | Context | Decode | Prefill | -|---|---|---|---|---| -| **open-1b-sft** (this repo) | f16 ~3.2 GB | 4096 | ~114 tok/s | ~892 tok/s | -| **gemma-4-E2B** (prod) | Q4 ~1.5 GiB | 64k | **~214 tok/s** | **~1659 tok/s** | +| --- | --- | --- | --- | --- | +| open-1b-sft | f16, about 3.2 GB | 4096 | about 114 tok/s | about 892 tok/s | +| gemma-4-E2B | Q4, about 1.5 GiB | 64k | about 214 tok/s | about 1659 tok/s | -*gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok / -prefill 1698 tok) on the same card and build.* +The gemma figures come from our 2026-09-20 benchmark on the same card and build. -**Reading:** gemma is roughly **2× faster**, ~**2.4× smaller**, and has **16× -the context** of open-1b. For any practical workload (classification, extraction, -translation, tool-calling) gemma wins outright. open-1b's reason to exist is not -throughput or quality but **auditable training**. +gemma is about twice as fast, about 2.4 times smaller, and has sixteen times the +context. For classification, extraction, translation, or tool calls, gemma is the +better choice. open-1b is interesting for its training audit, not for its speed. -## Caveats +## Notes -- This is **not** a controlled A/B: the open-1b figures are a short first-run - measurement with different prompts; the gemma figures come from a separate - controlled benchmark. Use them as an order-of-magnitude orientation. -- The `ngram` speculative decoder helps task-dependent amounts; open-1b's - baseline (no-spec) number was not measured here. -- Quality: open-1b scores **25.4** on the OLMo2 suite vs **31.9** for OLMo2-1B, - i.e. below an ordinary 1B model. It is a proof of *verifiability*, not a strong - assistant. +- Not a controlled comparison. The open-1b numbers are a short first run with + different prompts; the gemma numbers come from a separate controlled benchmark. +- No speculative-decoder-off measurement for open-1b. +- Quality: open-1b scores 25.4 on the OLMo2 suite, against 31.9 for OLMo2-1B. diff --git a/docs/inference-stack.md b/docs/inference-stack.md index 97e7e1f..ea76b6a 100644 --- a/docs/inference-stack.md +++ b/docs/inference-stack.md @@ -1,38 +1,38 @@ -# Inference stack — how llama.cpp was built and launched +# Inference stack: how llama.cpp was built and launched -This documents the exact engine used to run **open-1b** locally, and answers the -fair question: *is the llama.cpp build stock?* **No.** +This records the engine behind the local run. Is the llama.cpp build stock? No. -## Engine provenance — not stock +## Provenance | | | -|---|---| -| **Upstream base** | `ggerganov/llama.cpp` @ `c32d1dabe` — "tests : increase tolerance for Add fusion tests (#28691)", 2026-09-10 | -| **Local changes** | uncommitted patches adding the **`open1b`** architecture (9 files, +299 lines) | -| **Exported patch** | [`llama-cpp-open1b.patch`](llama-cpp-open1b.patch) | -| **Build** | `-DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release` | -| **Binary** | `build-vulkan/bin/llama-server` — `0.4.0-dev (build 10893, commit c32d1dabe)` | +| --- | --- | +| Upstream base | `ggerganov/llama.cpp` at `c32d1dabe`, 2026-09-10 | +| Local changes | uncommitted patches adding the `open1b` architecture, 9 files, +299 lines | +| Patch | [llama-cpp-open1b.patch](llama-cpp-open1b.patch) | +| Build | `-DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release` | +| Binary | `build-vulkan/bin/llama-server`, `0.4.0-dev`, build 10893 | -**Why the patch is required.** Upstream llama.cpp has no `open1b` architecture. open-1b -is Llama-3-derived but differs in four ways that the loader/graph must know about: +Upstream has no `open1b` architecture, so the patch is required. open-1b is +Llama-3-derived, with four differences the loader and graph must know about: -- gain-free per-head RMSNorm on Q and K before RoPE (`qk_norm`, no learnable gain); -- RMSNorm on the **token-embedding output** (`embedding_norm`); -- **hybrid sliding-window attention** (512-token window, full causal on the last layer of each group of 5 and on the final layer); -- **untied** input embedding / LM head, and interleaved-pair RoPE (`LLAMA_ROPE_TYPE_NORM`). +- gain-free per-head RMSNorm on Q and K before RoPE, +- RMSNorm on the token-embedding output, +- hybrid sliding-window attention, 512 tokens, full causal on the last layer of + each group of five and on the final layer, +- untied embedding and LM head, with interleaved-pair RoPE. ## Patch inventory | File | Change | -|---|---| -| `src/models/open1b.cpp` | **new** — `llama_model_open1b`: hparams, tensor mapping, graph | -| `src/models/models.h` | declare `llama_model_open1b` | -| `src/llama-arch.h` / `.cpp` | add `LLM_ARCH_OPEN1B` (enum + name) | -| `src/llama-model.cpp` | dispatch `LLM_ARCH_OPEN1B`; RoPE type `NORM` | -| `gguf-py/gguf/constants.py` | `MODEL_ARCH.OPEN1B`, arch name, tensor list | -| `gguf-py/gguf/tensor_mapping.py` | HF → GGUF tensor name map | +| --- | --- | +| `src/models/open1b.cpp` | new: `llama_model_open1b`, hparams, tensors, graph | +| `src/models/models.h` | declare the model | +| `src/llama-arch.h`, `.cpp` | add `LLM_ARCH_OPEN1B` | +| `src/llama-model.cpp` | dispatch the arch, set the RoPE type | +| `gguf-py/gguf/constants.py` | arch enum, name, tensor list | +| `gguf-py/gguf/tensor_mapping.py` | HF to GGUF tensor names | | `conversion/__init__.py` | register the converter | -| `conversion/open1b.py` | **new** — `Open1BModel` HF→GGUF converter | +| `conversion/open1b.py` | new: `Open1BModel` converter | ## Build @@ -57,35 +57,33 @@ setsid --fork ./build-vulkan/bin/llama-server \ --host 127.0.0.1 --port 8085 >/tmp/open1b.log 2>&1 GO["tools/hf2gguf\n(pure Go)"] - GO --> GGUF["open-1b-sft-f16-go.gguf\n(f16, byte-identical to reference)"] - GGUF --> LC["llama.cpp Vulkan\nupstream c32d1dabe\n+ open1b.patch"] - LC --> GPU["Intel Arc B580\n(Vulkan2)"] - GPU --> API["llama-server\nHTTP /completion"] + HF["HF checkpoint
model.safetensors, fp32"] --> GO["tools/hf2gguf
Go"] + GO --> GGUF["open-1b-sft-f16-go.gguf
f16"] + GGUF --> LC["llama.cpp Vulkan
upstream plus open1b patch"] + LC --> GPU["Intel Arc B580"] + GPU --> API["llama-server
HTTP /completion"] ``` -## Caveats +## Notes -- **The patch is ours and uncommitted.** It lives only in the working tree of the - build source; `llama-cpp-open1b.patch` is its export. For reproducibility it should - be committed and, ideally, proposed upstream. -- **The "reference" GGUF is not an independent vendor artifact.** It was produced by - llama.cpp's `convert_hf_to_gguf.py` **with this same Open1B patch**. Our Go converter - reproduces it byte-for-byte — a genuine independent reimplementation of the GGUF - serialization — but "byte-identical to the reference" means *identical to our patched - Python path*, not to a Gensyn-published file. -- Patch quality: written for bring-up and validated by a working inference run; not - reviewed upstream. +- The patch is ours and uncommitted. The exported patch file makes the build + reproducible. Committing it, or proposing it upstream, would be better. +- The reference GGUF is not an independent artifact. It came from llama.cpp's + `convert_hf_to_gguf.py` with this same patch. The Go converter reproduces it + byte for byte, which is a real reimplementation of the GGUF writer, but the + comparison is against our own patched Python output, not a file published by + Gensyn. +- The patch was written for bring-up. It is not upstream-reviewed. diff --git a/docs/open1b-local-inference.md b/docs/open1b-local-inference.md index 6d30bc3..f81f0cb 100644 --- a/docs/open1b-local-inference.md +++ b/docs/open1b-local-inference.md @@ -1,332 +1,177 @@ -# Gensyn open-1b (SFT) — local bring-up & proof of run +# Local run: open-1b on one Arc B580 -Epic: `vado/1b#1` / task #4 (SE). Workdir: `/root/projects/gensyn/inference/`. -Engine: existing shared llama.cpp **Vulkan** build (`/root/projects/inference/build-vulkan/bin/llama-server`). -Language: **Go 1.27 only** (see "Process note" re one accidental `python3` call). +- Workdir: `inference/` +- Engine: the shared llama.cpp Vulkan build, with the open1b patch +- Language: Go 1.27 -## TL;DR +## Summary -- Built a pure-Go tool `hf2gguf` (module `hf2gguf`) that **dumps** the reference GGUF - (go-gguf + a minimal raw-KV walker for arrays), **converts** `model.safetensors` - → `models/open-1b-sft-f16-go.gguf`, and **verifies** the two files. -- **Validation: PASS — 39/39 metadata KV byte-identical, 220/220 tensors byte-identical.** - Whole files are byte-for-byte identical (`cmp` + SHA-256 match). -- **Served** on one Arc (Vulkan2 / `07:00.0`) with the existing Vulkan build; coherent - output, **~114 tok/s decode**, **~892 tok/s prefill**, ~**3.6 GiB VRAM**. -- No commit made (all under gitignored `inference/`). +- A Go tool, `hf2gguf`, with three commands: `dump`, `convert`, `verify`. +- Validation: 39 of 39 metadata keys and 220 of 220 tensors are byte-identical. + The two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`. +- Served on one Arc (Vulkan2, PCI `07:00.0`): about 114 tok/s decode, about 892 + tok/s prefill, about 3.6 GiB VRAM. -Reference SHA-256 (and our output — identical): -`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae` - ---- - -## 1. Inputs & topology +## Inputs | Item | Path | Notes | -|------|------|-------| -| HF checkpoint (fp32) | `models/open-1b-sft/model.safetensors` | 6,432,081,672 B; dtype **F32**; header 34560 B | -| Reference GGUF (f16) | `models/open-1b-sft-f16.gguf` | 3,220,153,856 B; produced by llama.cpp `convert_hf_to_gguf.py` + our local Open1B patch; **read-only oracle** | -| HF config | `models/open-1b-sft/config.json` | `model_type=open1b`, 24 layers, hidden 2048, heads 16/4, head_dim 128, ffn 5632, vocab 128256, rope_theta 5e5, rms_eps 1e-5, swa 512 every 5, qk_norm gain-free, embedding_norm, untied | -| Vendor Python | `models/open-1b-sft/modeling_open1b.py` | read only, never executed | -| Engine | `/root/projects/inference/build-vulkan/bin/llama-server` | symlink → `/mnt/2TB/projects/build-vulkan`; shared prod tree, untouched | +| --- | --- | --- | +| HF checkpoint | `models/open-1b-sft/model.safetensors` | 6.43 GB, F32 | +| Reference GGUF | `models/open-1b-sft-f16.gguf` | 3.22 GB, f16, read-only | +| HF config | `models/open-1b-sft/config.json` | see below | +| Engine | `build-vulkan/bin/llama-server` | shared prod tree, not modified | -GPU devices as reported by the engine (`--list-devices`): +The config: `model_type=open1b`, 24 layers, hidden 2048, heads 16 and key/value +heads 4, head_dim 128, feed-forward 5632, vocab 128256, rope_theta 500000, +rms_eps 1e-5, hybrid sliding window 512 every 5 layers, gain-free qk_norm, +embedding_norm, untied embeddings. + +Devices as the engine lists them: ``` -Vulkan0: AMD Ryzen 7 7700X (RADV RAPHAEL_MENDOCINO) (15835 MiB) -Vulkan1: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB) -Vulkan2: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB) -Vulkan3: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB) +Vulkan0: AMD Ryzen 7 7700X (RADV RAPHAEL_MENDOCINO), 15835 MiB +Vulkan1: Intel Arc B580 (BMG G21), 12216 MiB +Vulkan2: Intel Arc B580 (BMG G21), 12216 MiB +Vulkan3: Intel Arc B580 (BMG G21), 12216 MiB ``` -`Vulkan2` was confirmed to map to PCI `0000:07:00.0` (Arc #2) via +Vulkan2 maps to PCI `0000:07:00.0`, confirmed through `/sys/class/drm/renderD129/device`. ---- - -## 2. Tool: `hf2gguf` (Go 1.27, `github.com/cymertek/go-gguf`) +## Tool ``` inference/hf2gguf/ - go.mod, go.sum - main.go # subcommand dispatch + JSON manifest types - rawkv.go # minimal read-only GGUF KV walker (fills go-gguf's array gap) - safetensors.go # safetensors header + tensor byte access - hfconfig.go # config.json / tokenizer_config.json / tokenizer.json validation - convert.go # metadata reproduction + HF→GGUF tensor mapping + F32→F16 - verify.go # byte-level metadata & tensor comparison - convert_test.go + main.go dispatch, manifest types + rawkv.go read-only GGUF KV walker, array support + safetensors.go safetensors header and tensor bytes + hfconfig.go config and tokenizer parsing, validation + convert.go metadata reproduction, tensor mapping, F32 to F16 + verify.go byte-level comparison + convert_test.go tests ``` -### Why a raw KV walker +`go-gguf` returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV +section directly to keep the tokenizer arrays (`tokens`, `merges`, `token_type`) +as raw bytes. Writing still goes through the `go-gguf` writer. GGUF +serialization is not hand-rolled. -`go-gguf`'s `MetadataEntry.Value()` intentionally returns an empty value for -`BTypeArray` (see `readLazyValue` in the library). The tokenizer arrays -(`tokens`, `scores`→`token_type`, `merges`) are exactly the values we must copy -verbatim, so `rawkv.go` reads the KV section directly and preserves each array's -`elem_type`, element count, and **raw element bytes**. Those raw bytes are fed -straight into go-gguf's writer `Value{Raw: ...}`, so serialization is still done -by the library (nothing hand-rolled on the write path). - -### Subcommands +Commands: ```bash -# 2a. dump — emit the reproduction contract as JSON -go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json - -# 2b/c. convert — reproduce metadata + map safetensors→GGUF (fp32→f16) -go run . convert \ - -ref models/open-1b-sft-f16.gguf \ - -safetensors models/open-1b-sft/model.safetensors \ - -config models/open-1b-sft/config.json \ - -tokenizer-config models/open-1b-sft/tokenizer_config.json \ - -tokenizer models/open-1b-sft/tokenizer.json \ - -o models/open-1b-sft-f16-go.gguf - -# 3. verify — byte-level equality against the reference -go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf +go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json +go run . convert -ref models/open-1b-sft-f16.gguf \ + -safetensors models/open-1b-sft/model.safetensors \ + -config models/open-1b-sft/config.json \ + -tokenizer-config models/open-1b-sft/tokenizer_config.json \ + -tokenizer models/open-1b-sft/tokenizer.json \ + -o models/open-1b-sft-f16-go.gguf +go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf ``` -`dump` output (the contract) is at -`models/open-1b-sft-f16.manifest.json` (39 metadata entries + 220 tensors, -with per-array `raw_sha256`). +## Tensor mapping -### Tensor mapping (reference names are the contract) +The reference names are the contract. -Shapes are the reference's; ggml `ne0` is the innermost (= HF input) dim, so HF -bytes are copied unpermuted with the dims reversed. +| GGUF tensor | HF source | Shape | Type | +| --- | --- | --- | --- | +| `token_embd.weight` | `embedding.tok_embeddings.weight` | 2048 x 128256 | F16 | +| `token_embd_norm.weight` | `emb_norm.weight` | 2048 | F32 | +| `output.weight` | `embedding.output.weight` | 2048 x 128256 | F16 | +| `output_norm.weight` | `norm_out.weight` | 2048 | F32 | +| `blk.i.attn_q.weight` | `blocks.i.attn.wq.weight` | 2048 x 2048 | F16 | +| `blk.i.attn_k.weight` | `blocks.i.attn.wk.weight` | 2048 x 512 | F16 | +| `blk.i.attn_v.weight` | `blocks.i.attn.wv.weight` | 2048 x 512 | F16 | +| `blk.i.attn_output.weight` | `blocks.i.attn.wo.weight` | 2048 x 2048 | F16 | +| `blk.i.ffn_gate.weight` | `blocks.i.ffn.w_gate_up.weight`, first half | 2048 x 5632 | F16 | +| `blk.i.ffn_up.weight` | `blocks.i.ffn.w_gate_up.weight`, second half | 2048 x 5632 | F16 | +| `blk.i.ffn_down.weight` | `blocks.i.ffn.w_down.weight` | 5632 x 2048 | F16 | +| `blk.i.attn_norm.weight` | `blocks.i.norm1.weight` | 2048 | F32 | +| `blk.i.ffn_norm.weight` | `blocks.i.norm2.weight` | 2048 | F32 | -| Reference tensor | HF source | Shape | Type | Count | -|---|---|---|---|---| -| `token_embd.weight` | `embedding.tok_embeddings.weight` | [2048,128256] | F16 | 1 | -| `token_embd_norm.weight` | `emb_norm.weight` | [2048] | F32 | 1 | -| `output.weight` | `embedding.output.weight` | [2048,128256] | F16 | 1 | -| `output_norm.weight` | `norm_out.weight` | [2048] | F32 | 1 | -| `blk.{i}.attn_q.weight` | `blocks.{i}.attn.wq.weight` | [2048,2048] | F16 | 24 | -| `blk.{i}.attn_k.weight` | `blocks.{i}.attn.wk.weight` | [2048,512] | F16 | 24 | -| `blk.{i}.attn_v.weight` | `blocks.{i}.attn.wv.weight` | [2048,512] | F16 | 24 | -| `blk.{i}.attn_output.weight` | `blocks.{i}.attn.wo.weight` | [2048,2048] | F16 | 24 | -| `blk.{i}.ffn_gate.weight` | `blocks.{i}.ffn.w_gate_up.weight` (first half) | [2048,5632] | F16 | 24 | -| `blk.{i}.ffn_up.weight` | `blocks.{i}.ffn.w_gate_up.weight` (second half) | [2048,5632] | F16 | 24 | -| `blk.{i}.ffn_down.weight` | `blocks.{i}.ffn.w_down.weight` | [5632,2048] | F16 | 24 | -| `blk.{i}.attn_norm.weight` | `blocks.{i}.norm1.weight` | [2048] | F32 | 24 | -| `blk.{i}.ffn_norm.weight` | `blocks.{i}.norm2.weight` | [2048] | F32 | 24 | +The fused `w_gate_up` is split at the row boundary. Per-channel `*.weight_scale` +tensors in the checkpoint are dropped; the reference has none. F32 to F16 uses +round-to-nearest, ties to even, matching the numpy and torch casts. -- The fused `w_gate_up` is split at the row boundary (`chunk(2,-1)` → first half - `gate`, second half `up`), matching the reference. -- Quantized per-channel `*.weight_scale` tensors in the checkpoint are **dropped** - (the reference has none). -- F32→F16 uses IEEE round-to-nearest, ties-to-even (`f32bitsToF16`), matching - numpy/torch casts; the reference tensors are byte-identical to ours, which - empirically proves the rounding matched. -- HF config/tokenizer are **parsed and validated** against the reference, but the - reference remains authoritative for every emitted value. - ---- - -## 3. Validation - -`go run . verify` streams both files and compares on raw wire bytes: +## Validation ``` metadata: 39 checked, 0 mismatches tensors: 220 checked, 220 byte-identical, 0 mismatches -verification PASSED: 39 metadata keys and 220 tensors byte-identical +verification PASSED ``` -Whole-file check: - ``` $ cmp models/open-1b-sft-f16.gguf models/open-1b-sft-f16-go.gguf && echo IDENTICAL IDENTICAL -$ sha256sum models/open-1b-sft-f16.gguf models/open-1b-sft-f16-go.gguf -a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae open-1b-sft-f16.gguf -a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae open-1b-sft-f16-go.gguf ``` -The per-tensor diff is empty (0 mismatches) — every tensor's payload is -byte-identical, in the same order, with the same shape and ggml type. +Both files hash to +`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`. -### Metadata contract (all 39 keys, all byte-identical) +Notable metadata, all byte-identical: `general.architecture=open1b`, +`open1b.block_count=24`, `open1b.context_length=4096`, +`open1b.attention.head_count=16`, `open1b.attention.head_count_kv=4`, +`open1b.rope.freq_base=500000`, `open1b.attention.sliding_window=512`, +`open1b.attention.sliding_window_pattern=5`, `tokenizer.ggml.model=gpt2`, +`tokenizer.ggml.pre=gpt-2`, `tokenizer.ggml.eos_token_id=4`. The full list is in +the manifest JSON. -| Key | Type | Value | -|---|---|---| -| `general.architecture` | STRING | `open1b` | -| `general.type` | STRING | `model` | -| `general.name` | STRING | `Open 1b Sft` | -| `general.finetune` | STRING | `sft` | -| `general.basename` | STRING | `open` | -| `general.size_label` | STRING | `1B` | -| `general.license` | STRING | `apache-2.0` | -| `general.base_model.count` | UINT32 | 1 | -| `general.base_model.0.name` | STRING | `Open 1b Midtrained 93B` | -| `general.base_model.0.organization` | STRING | `Gensyn` | -| `general.base_model.0.repo_url` | STRING | `https://huggingface.co/Gensyn/open-1b-midtrained-93B` | -| `general.dataset.count` | UINT32 | 1 | -| `general.dataset.0.name` | STRING | `Tulu 3 Sft Olmo 2 Mixture 0225` | -| `general.dataset.0.version` | STRING | `0225` | -| `general.dataset.0.organization` | STRING | `Allenai` | -| `general.dataset.0.repo_url` | STRING | `https://huggingface.co/allenai/tulu-3-sft-olmo-2-mixture-0225` | -| `general.tags` | ARRAY[STRING] | 5 (`gensyn`, `verifiable-training`, `reproducible-training`, `repops`, `text-generation`) | -| `general.languages` | ARRAY[STRING] | 1 (`en`) | -| `open1b.block_count` | UINT32 | 24 | -| `open1b.context_length` | UINT32 | 4096 | -| `open1b.embedding_length` | UINT32 | 2048 | -| `open1b.feed_forward_length` | UINT32 | 5632 | -| `open1b.attention.head_count` | UINT32 | 16 | -| `open1b.attention.head_count_kv` | UINT32 | 4 | -| `open1b.rope.freq_base` | FLOAT32 | 500000 | -| `open1b.attention.layer_norm_rms_epsilon` | FLOAT32 | 1e-05 | -| `open1b.attention.key_length` | UINT32 | 128 | -| `open1b.attention.value_length` | UINT32 | 128 | -| `general.file_type` | UINT32 | 1 (F16) | -| `open1b.attention.sliding_window` | UINT32 | 512 | -| `open1b.attention.sliding_window_pattern` | UINT32 | 5 | -| `general.quantization_version` | UINT32 | 2 | -| `tokenizer.ggml.model` | STRING | `gpt2` | -| `tokenizer.ggml.pre` | STRING | `gpt-2` | -| `tokenizer.ggml.tokens` | ARRAY[STRING] | 128256 (sha256 `30bab437…`) | -| `tokenizer.ggml.token_type` | ARRAY[INT32] | 128256 (sha256 `5ca8c567…`) | -| `tokenizer.ggml.merges` | ARRAY[STRING] | 127994 (sha256 `9e3a7dd4…`) | -| `tokenizer.ggml.eos_token_id` | UINT32 | 4 | -| `tokenizer.chat_template` | STRING | (501-byte Jinja template) | +`go build`, `go vet`, and `go test -race` are clean. -Array `raw_sha256` values are in `models/open-1b-sft-f16.manifest.json`. +## Serve -### HF cross-checks performed by `convert` - -``` -config.json OK: arch=open1b layers=24 hidden=2048 heads=16/4 head_dim=128 ffn=5632 vocab=128256 swa=512/5 tie=false qk_norm=true(gain=false) emb_norm=true -tokenizer_config.json OK: eos=<|eot|> chat_template=501 bytes -tokenizer.json OK: type=BPE vocab=128256 +added=6 -> 128256 tokens, 127994 merges -``` - -### Go build / vet / test - -``` -$ go build ./... # OK -$ go vet ./... # OK -$ go test -race -v ./... ---- PASS: TestF32BitsToF16 (0.00s) ---- PASS: TestF32ToF16Bytes (0.00s) ---- PASS: TestRawKVRoundTrip (0.00s) ---- PASS: TestVerifyRealFiles (6.59s) -PASS -ok hf2gguf 7.602s -``` - -`TestVerifyRealFiles` re-runs the full 3.2 GB byte comparison (skipped on -`-short` or when the artifacts are absent). - ---- - -## 4. Serve on one Arc (Vulkan2) - -``` -setsid --fork /root/projects/inference/build-vulkan/bin/llama-server \ - -m /root/projects/gensyn/inference/models/open-1b-sft-f16-go.gguf \ +```bash +setsid --fork ./build-vulkan/bin/llama-server \ + -m models/open-1b-sft-f16-go.gguf \ --device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \ - --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1 --alias open-1b \ + --spec-type ngram-map-k,ngram-cache \ + -t 16 --parallel 1 --alias open-1b \ --host 127.0.0.1 --port 8085 >/tmp/open1b.log 2>&1 n_ctx_train (4096) -- possible training context overflow -load_model: the slot context (32768) exceeds the training context of the model (4096) - capping -load_model: initializing, n_slots = 1, n_ctx_slot = 4096, kv_unified = 'false' -model loaded -listening on http://127.0.0.1:8085 -``` - -Device confirmation: PID `1378706` maps `/dev/dri/renderD129` with large -`rw-s` regions; `renderD129 → /sys/devices/.../0000:07:00.0` = Arc #2 = Vulkan2. -`/props` reports `"speculative.types":"none,ngram-map-k,ngram-cache"` and -`"model_ftype":"F16"`. - -### Performance (single Arc, 1 slot, `-fa off`, ngram spec) - | Metric | Value | -|---|---| -| Decode | **~111.7–117.2 tok/s** (e.g. 256 tok @ 111.65; 96 tok @ 117.20) | -| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) | -| VRAM on Vulkan2 | free 10971 → 7258 MiB ⇒ **≈ 3.6 GiB** used; fully released after `kill` | +| --- | --- | +| Decode | 111.7 to 117.2 tok/s | +| Prefill | about 892 tok/s | +| VRAM | about 3.6 GiB, released after stop | -### Sample outputs +Sample outputs, greedy: ``` -$ curl ... /completion {"prompt":"The three primary colors are", greedy} - red, blue, and yellow. +prompt: The three primary colors are +out: red, blue, and yellow. -$ curl ... /completion {"prompt":"Q: What is the capital of France?\nA:", greedy, 96} - Paris - -$ curl ... /completion (seeded chat turn, see caveat) 113.9 tok/s -" Venus and Mars. Venus is the hottest planet in our Solar System, with - temperatures reaching 465°C (1,470°F). Mars, on the other hand, is a - terrestrial planet, meaning it has a solid surface and liquid water at its poles." +prompt: Q: What is the capital of France? A: +out: Paris ``` -### Stopping +Stopped with `kill` after reading the pid from `ss -ltnp`. -``` -$ ss -ltnp | grep :8085 # -> pid=1378706 -$ kill 1378706 # never pkill -f -``` +## Caveats -Both the test server (:8085) and a diagnostic server (:8086) were stopped; -`--list-devices` returned to 10971 MiB free on Vulkan2/3. +1. Chat template. With the vendor tail `...assistant<|end_header|>` followed by a + blank line, this checkpoint emits an end-of-turn token first and returns an + empty reply. Reproduced with and without the speculative decoder, so it is not + the decoder. A single newline, an extra newline, or a seed token produces a + coherent answer. Since our GGUF is byte-identical to the reference, this is a + property of the released checkpoint. +2. Context cap. `open1b.context_length` is 4096, so the requested 32768 is capped. +3. The load warning `special_eos_id is not in special_eog_ids` also appears for + the reference file, so it is not introduced by the conversion. +4. Long greedy generation without a repetition penalty loops on this small model. +5. Process note. During one early check the response was piped through `python3` + to pretty-print JSON. That was a mistake and a breach of the no-Python rule. + The tool itself is entirely Go, and nothing in the pipeline depends on Python. + No Python files or virtualenv were created for the tool. ---- - -## 5. Caveats - -1. **Chat template newline quirk (model-side, not conversion-side).** With the - exact vendor template tail `<|end_header|>\n\n` for the assistant turn, this - checkpoint emits `<|eot|>` as the first sampled token → empty reply / instant - EOS (reproduced on both :8085 with ngram spec and a :8086 run with - `--spec-type none`, so it is **not** the speculative decoder). A single `\n`, - `\n\n\n`, or a seed token makes it answer coherently: - - | tail after `assistant<|end_header|>` | result | - |---|---| - | `\n\n` (vendor template) | immediate EOS | - | `\n` | `\n1. **Earth**: …` (coherent) | - | `\n\n\n` | `\n1. **Earth**: …` (coherent) | - | ` ` | `\nSure, here are two names…` (coherent) | - - Because our GGUF is byte-identical to the reference GGUF, this is a - property of the released checkpoint/GGUF (likely SFT formatting), not of our - conversion. Workarounds for consumers: use `ignore_eos` + trim, seed the - assistant turn, or fix the template's trailing newline. -2. **Context capped to 4096.** `open1b.context_length = 4096`, so the requested - `-c 32768` is capped to `n_ctx_slot = 4096`. KV is f16 (`-fa off` mandatory on - Arc per `brainstation-gpu`). -3. **`special_eos_id is not in special_eog_ids`** warning on load — present for - the reference file too (byte-identical), so not introduced by us. -4. **1B greedy degeneration.** Long greedy generation without repetition penalty - loops on this small model; use sampling / `repeat_penalty` for long outputs. -5. **Process note (disclosure).** During an early diagnostic I piped one - `/completion` response through `python3 -c` to pretty-print JSON — a breach of - the "no Python" rule (the tool itself is 100% Go, and nothing in the pipeline - depends on Python). All subsequent parsing was done with Go/raw JSON. No `.py` - file or venv was created; the only Python files in the tree are the vendor's - read-only `models/open-1b-sft/*.py`. - ---- - -## 6. Artifacts +## Files | Path | What | -|---|---| -| `hf2gguf/` | Go module (`dump` / `convert` / `verify`) + tests | -| `hf2gguf/README.md` | tool usage | -| `models/open-1b-sft-f16-go.gguf` | our output (byte-identical to reference) | -| `models/open-1b-sft-f16.manifest.json` | dumped contract (39 KV + 220 tensors) | -| `models/open-1b-sft-f16.gguf` | reference (untouched) | +| --- | --- | +| `hf2gguf/` | Go module with tests | +| `hf2gguf/README.md` | usage | +| `models/open-1b-sft-f16-go.gguf` | our output | +| `models/open-1b-sft-f16.manifest.json` | dumped contract | +| `models/open-1b-sft-f16.gguf` | reference | | `/tmp/open1b.log` | server log | - -No changes were committed; everything lives under gitignored `inference/`. diff --git a/tools/hf2gguf/README.md b/tools/hf2gguf/README.md index 54aacd5..de3e995 100644 --- a/tools/hf2gguf/README.md +++ b/tools/hf2gguf/README.md @@ -1,82 +1,78 @@ # hf2gguf -Pure-Go (Go 1.27) tool to convert a HuggingFace `open-1b` safetensors checkpoint -into GGUF **byte-identically** to a supplied reference GGUF, plus a byte-level -verifier. Built for epic `vado/1b#1` (Gensyn open-1b local bring-up). +A Go 1.27 tool that converts a Hugging Face `open-1b` safetensors checkpoint to +GGUF, byte-identical to a supplied reference file, with a byte-level verifier. -Library: [`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf) -(reader + writer). No Python anywhere in the tool. +Library: [go-gguf](https://github.com/cymertek/go-gguf), reader and writer. No +Python anywhere. ## Build ```bash go build ./... go vet ./... -go test -race ./... # full test includes the 3.2 GB byte comparison +go test -race ./... # the full test includes the 3.2 GB byte comparison ``` -## Subcommands +## Commands -### `dump` — the reproduction contract +### dump -Reads the reference GGUF and emits a JSON manifest of **all** metadata KV pairs -(key, type, value / array elem-type + count + `raw_sha256`) and every tensor -(name, shape, ggml type, nbytes). +Reads the reference GGUF and writes a JSON manifest of all metadata pairs (key, +type, value, or array element type, count, and `raw_sha256`) and every tensor +(name, shape, ggml type, byte count). ```bash go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json ``` -`go-gguf` exposes scalar/string metadata and tensors; its `MetadataEntry.Value()` -returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV section -directly to preserve tokenizer arrays (`tokens`, `merges`, `token_type`) as raw -bytes. Writing still goes through `go-gguf`'s writer — nothing hand-rolls GGUF -serialization. +`go-gguf` returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV +section directly to keep the tokenizer arrays as raw bytes. Writing still goes +through the `go-gguf` writer. -### `convert` — emit our GGUF +### convert ```bash go run . convert \ - -ref models/open-1b-sft-f16.gguf \ - -safetensors models/open-1b-sft/model.safetensors \ - -config models/open-1b-sft/config.json \ + -ref models/open-1b-sft-f16.gguf \ + -safetensors models/open-1b-sft/model.safetensors \ + -config models/open-1b-sft/config.json \ -tokenizer-config models/open-1b-sft/tokenizer_config.json \ - -tokenizer models/open-1b-sft/tokenizer.json \ - -o models/open-1b-sft-f16-go.gguf + -tokenizer models/open-1b-sft/tokenizer.json \ + -o models/open-1b-sft-f16-go.gguf ``` -- Metadata is copied verbatim from the reference KV section. -- Tensors are mapped HF FQN → GGUF name (see the module docs in `convert.go`), - with `blocks.{i}.ffn.w_gate_up.weight` split into `ffn_gate`/`ffn_up`, and - `*.weight_scale` dropped (the reference has none). -- Payloads stream from safetensors and are converted F32→F16 with IEEE - round-to-nearest-even on the fly (O(1) memory per tensor). -- `-config` / `-tokenizer-config` / `-tokenizer` are optional; when given they - are parsed and **validated** against the reference (mismatch is fatal). +- Metadata is copied verbatim from the reference. +- Tensors are mapped by name, with `blocks.i.ffn.w_gate_up.weight` split into + `ffn_gate` and `ffn_up`, and `*.weight_scale` dropped. +- Payloads stream from safetensors and convert F32 to F16 on the fly, using + round-to-nearest, ties to even. +- The config and tokenizer flags are optional. When given, they are parsed and + validated against the reference; a mismatch is fatal. -### `verify` — byte-level equality +### verify ```bash go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf ``` -Compares metadata on raw wire bytes and tensors by shape, type and full payload -equality (streaming, first-diff offset reported on failure). +Compares metadata on raw wire bytes and tensors by shape, type, and full payload. +Reports the first differing offset on failure. ``` metadata: 39 checked, 0 mismatches tensors: 220 checked, 220 byte-identical, 0 mismatches -verification PASSED: 39 metadata keys and 220 tensors byte-identical +verification PASSED ``` ## Layout | File | Purpose | -|---|---| -| `main.go` | dispatch, manifest types, `dump` | -| `rawkv.go` | read-only GGUF KV walker (array support) | -| `safetensors.go` | safetensors header/byte access | -| `hfconfig.go` | HF config/tokenizer parsing + validation | -| `convert.go` | metadata reproduction, tensor mapping, F32→F16 | -| `verify.go` | `verify` command + comparison engine | +| --- | --- | +| `main.go` | dispatch, manifest types, dump | +| `rawkv.go` | read-only GGUF KV walker with array support | +| `safetensors.go` | safetensors header and byte access | +| `hfconfig.go` | HF config and tokenizer parsing, validation | +| `convert.go` | metadata reproduction, tensor mapping, F32 to F16 | +| `verify.go` | verify command and comparison engine | | `convert_test.go` | F16 rounding, KV round-trip, full-file verification | diff --git a/tools/hf2gguf/convert.go b/tools/hf2gguf/convert.go index 16dfc87..70f502d 100644 --- a/tools/hf2gguf/convert.go +++ b/tools/hf2gguf/convert.go @@ -1,6 +1,6 @@ package main -// convert.go — emit models/open-1b-sft-f16-go.gguf from the HF safetensors, +// convert.go converts the HF safetensors into models/open-1b-sft-f16-go.gguf, // reproducing the reference GGUF's metadata (verbatim) and tensor set/order. // // Tensor mapping (HF FQN -> GGUF name), shapes are the reference's (ggml ne0 is @@ -117,7 +117,7 @@ func cmdConvert(args []string) error { defer st.Close() // Validate HF config against the reference contract (no values are taken from - // it — the reference is authoritative — but mismatches are fatal). + // it (the reference is authoritative), but mismatches are fatal). if *hfConfig != "" { if err := validateConfig(*hfConfig, kvs); err != nil { return fmt.Errorf("config validation: %w", err) diff --git a/tools/hf2gguf/hfconfig.go b/tools/hf2gguf/hfconfig.go index d0436dc..c7d4e20 100644 --- a/tools/hf2gguf/hfconfig.go +++ b/tools/hf2gguf/hfconfig.go @@ -1,6 +1,6 @@ package main -// hfconfig.go — parse and validate the HF config/tokenizer against the +// hfconfig.go parses and validates the HF config/tokenizer against the // reference GGUF metadata. The reference remains authoritative for values; this // only proves the reference is consistent with the published HF files and that // we parsed them (per the task), it does not source any GGUF value from them. diff --git a/tools/hf2gguf/main.go b/tools/hf2gguf/main.go index f0db669..1480e1c 100644 --- a/tools/hf2gguf/main.go +++ b/tools/hf2gguf/main.go @@ -35,7 +35,7 @@ func main() { } // --------------------------------------------------------------------------- -// Manifest types — the reproduction contract. +// Manifest types for the reproduction contract. // --------------------------------------------------------------------------- type manifest struct { diff --git a/tools/hf2gguf/safetensors.go b/tools/hf2gguf/safetensors.go index 4317565..23fdcf1 100644 --- a/tools/hf2gguf/safetensors.go +++ b/tools/hf2gguf/safetensors.go @@ -1,6 +1,6 @@ package main -// safetensors.go — minimal read-only HuggingFace safetensors index. +// safetensors.go reads a minimal, read-only HuggingFace safetensors index. // // Format: [u64 LE header_len][header_len bytes of JSON][tensor data ...]. // The JSON header maps tensor name -> {dtype, shape, data_offsets:[start,end]} diff --git a/tools/hf2gguf/verify.go b/tools/hf2gguf/verify.go index b8046bb..27b12f3 100644 --- a/tools/hf2gguf/verify.go +++ b/tools/hf2gguf/verify.go @@ -1,6 +1,6 @@ package main -// verify.go — prove the emitted GGUF equals the reference: identical metadata +// verify.go proves the emitted GGUF equals the reference: identical metadata // (keys, types, values; verified on raw wire bytes) and identical tensor set, // shapes, ggml types and per-tensor payload bytes.