Merge pull request 'docs: rewrite in plain style (no em dashes, no emoji)' (#9) from docs/clean-style into main
This commit is contained in:
@@ -1,54 +1,49 @@
|
||||
# open-1b — independent verification & local inference
|
||||
# open-1b: an independent check, and a local run
|
||||
|
||||
An independent, reproducible check of Gensyn's **open-1b** "auditable training"
|
||||
claim — plus a **pure-Go** path to convert and run the model locally.
|
||||
We looked at Gensyn's open-1b and its claim of auditable training, then ran the
|
||||
model on our own hardware. The audit uses Gensyn's official harness. Everything
|
||||
we wrote is Go.
|
||||
|
||||
> Reference: [Introducing open-1b](https://www.gensyn.ai/news/introducing-open-1b-auditable-training)
|
||||
> · harness: [gensyn-ai/open-transformers](https://github.com/gensyn-ai/open-transformers)
|
||||
> · audit app: [open1b.gensyn.ai](https://open1b.gensyn.ai/)
|
||||
|
||||
Everything we wrote is **Go** (no Python in our code or pipeline). For the audit we
|
||||
used Gensyn's official harness, which is Python-only and the sole vendor-supplied
|
||||
path; for inference we wrote our own Go GGUF converter.
|
||||
Links: [announcement](https://www.gensyn.ai/news/introducing-open-1b-auditable-training),
|
||||
[harness](https://github.com/gensyn-ai/open-transformers),
|
||||
[audit app](https://open1b.gensyn.ai/).
|
||||
|
||||
## Results
|
||||
|
||||
| Item | Result |
|
||||
|------|--------|
|
||||
| **Init-unit audit** (Linux x86-64 CPU, no NVIDIA) | ✅ **MATCH** — published `state_hash 16554a11…` reproduced bit-for-bit |
|
||||
| **Training-interval audit** | ⛔ blocked — no interval unit is published; replay needs ~24 GB RAM / ≥60 GB disk |
|
||||
| **GGUF conversion** (Go, `tools/hf2gguf`) | ✅ **byte-identical** to the vendor GGUF (same SHA-256) |
|
||||
| **Local inference** (1× Intel Arc B580, Vulkan) | ✅ ~114 tok/s decode · ~892 tok/s prefill · ~3.6 GiB VRAM |
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Init-unit audit, x86-64 CPU, no NVIDIA | Match. The published state hash `16554a11...` is reproduced exactly. |
|
||||
| Training-interval audit | Not run. No interval unit is published, and a replay needs about 24 GB RAM and 60 GB disk. |
|
||||
| GGUF conversion (Go) | Byte-identical to the reference conversion, same SHA-256. |
|
||||
| Local inference, one Intel Arc B580 | 114 tok/s decode, 892 tok/s prefill, about 3.6 GiB VRAM. |
|
||||
|
||||
Details: [`docs/audit-open1b-init.md`](docs/audit-open1b-init.md),
|
||||
[`docs/open1b-local-inference.md`](docs/open1b-local-inference.md),
|
||||
[`docs/benchmarks.md`](docs/benchmarks.md),
|
||||
[`docs/inference-stack.md`](docs/inference-stack.md).
|
||||
Reports: [audit](docs/audit-open1b-init.md), [local run](docs/open1b-local-inference.md),
|
||||
[benchmarks](docs/benchmarks.md), [inference stack](docs/inference-stack.md).
|
||||
|
||||
## How it works
|
||||
|
||||
**Audit — replay one published commitment and compare.**
|
||||
Audit. Replay one published commitment and compare hashes.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
KIT["Gensyn audit kit"] --> SHA{"sha256\nvs kit.json"}
|
||||
SHA --> WHL["repop + pretrain wheels"]
|
||||
WHL --> REPLAY["pretrain-audit-replay\n--from-init --device cpu"]
|
||||
REPLAY --> H1["regenerated\ninit state hash"]
|
||||
KIT --> H2["published\ncommitment"]
|
||||
H1 --> CMP{"equal?"}
|
||||
KIT["Gensyn audit kit"] --> SHA{"sha256<br/>against kit.json"}
|
||||
SHA --> WHL["repop and pretrain wheels"]
|
||||
WHL --> REPLAY["pretrain-audit-replay<br/>--from-init --device cpu"]
|
||||
REPLAY --> H1["regenerated init hash"]
|
||||
KIT --> H2["published commitment"]
|
||||
H1 --> CMP{"equal"}
|
||||
H2 --> CMP
|
||||
CMP -->|yes| OK["MATCH"]
|
||||
CMP -->|no| BAD["MISMATCH"]
|
||||
CMP -->|yes| OK["match"]
|
||||
CMP -->|no| BAD["mismatch"]
|
||||
```
|
||||
|
||||
**Inference — convert in Go, serve on one Arc (the engine is a patched llama.cpp).**
|
||||
Inference. Convert in Go, serve on one Arc. The engine is a patched llama.cpp.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
HF["HF checkpoint\nmodel.safetensors (fp32)"] --> GO["tools/hf2gguf\n(pure Go)"]
|
||||
GO --> GGUF["GGUF (f16)\nbyte-identical to reference"]
|
||||
GGUF --> LC["llama.cpp Vulkan\nupstream + open1b.patch"]
|
||||
HF["HF checkpoint<br/>model.safetensors, fp32"] --> GO["tools/hf2gguf<br/>Go"]
|
||||
GO --> GGUF["GGUF, f16<br/>identical to reference"]
|
||||
GGUF --> LC["llama.cpp Vulkan<br/>upstream plus open1b patch"]
|
||||
LC --> GPU["Intel Arc B580"]
|
||||
GPU --> API["llama-server HTTP"]
|
||||
```
|
||||
@@ -57,43 +52,44 @@ flowchart LR
|
||||
|
||||
```
|
||||
docs/
|
||||
audit-open1b-init.md # init-unit audit report (raw evidence)
|
||||
open1b-local-inference.md # local bring-up report
|
||||
benchmarks.md # speed benchmarks + reference comparison
|
||||
inference-stack.md # how llama.cpp was built/launched (NOT stock)
|
||||
llama-cpp-open1b.patch # local Open1B patch applied to llama.cpp
|
||||
tools/hf2gguf/ # pure-Go safetensors → GGUF converter (dump/convert/verify)
|
||||
audit-open1b-init.md
|
||||
open1b-local-inference.md
|
||||
benchmarks.md
|
||||
inference-stack.md
|
||||
llama-cpp-open1b.patch
|
||||
tools/hf2gguf/
|
||||
```
|
||||
|
||||
## Audit — what it proves, and what it does not
|
||||
## The audit, and its limits
|
||||
|
||||
**Proves.** The installed artifacts are the ones the kit's `kit.json` published
|
||||
(all SHA-256s match); the `repop` build commit matches the manifest; and running
|
||||
the vendor harness on a plain CPU regenerates the `1b_repop_v2` initial state hash
|
||||
**bit-for-bit** — an independent third party reproduced the run's init commitment.
|
||||
The init unit regenerates the published initial state hash exactly on a plain
|
||||
CPU. The installed wheels match the kit manifest, and the repop build commit
|
||||
matches the manifest.
|
||||
|
||||
**Does not prove.**
|
||||
It does not cover a training step. An init-only unit says nothing about the
|
||||
forward pass, the backward pass, the optimizer, or the data stream, so the
|
||||
per-step claim stays untested here. Three more reasons to be careful:
|
||||
|
||||
- **No training interval.** An init-only unit says nothing about any forward/backward/AdamW/data-stream step. The claim about bitwise-reproducible *optimizer steps* is not exercised here.
|
||||
- **`repop` is closed source.** Only compiled wheels ship, under the "Gensyn Reproducibility License v1.0". The kernels that make training reproducible cannot be read or independently re-derived.
|
||||
- **The v3 hash omits** RNG, the data-stream cursor, spike state, and `meta.json` descriptor keys.
|
||||
- **No interval unit is published**, so "pick any step, replay it" is not self-serve checkable yet.
|
||||
- Provenance of the *published run* is not established — only that the pinned artifacts reproduce the published commitment on our machine.
|
||||
- repop ships as compiled wheels only, under a custom licence. The kernels that
|
||||
make training reproducible cannot be read or checked.
|
||||
- The v3 state hash omits the RNG, the data-stream cursor, spike state, and the
|
||||
meta.json descriptor keys.
|
||||
- No interval unit is published, so "replay any step" is not yet something an
|
||||
outsider can do without permission.
|
||||
|
||||
## GGUF conversion (pure Go)
|
||||
A match shows that the pinned artifacts reproduce the published commitment on
|
||||
our machine. It does not show how the original run was carried out.
|
||||
|
||||
`tools/hf2gguf` reads the HF checkpoint (`config.json`, `model.safetensors`,
|
||||
`tokenizer.json`) and writes a GGUF file, using
|
||||
[`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf) for GGUF
|
||||
read/write. A reference GGUF produced by llama.cpp's `convert_hf_to_gguf.py`
|
||||
**with our [Open1B patch](docs/llama-cpp-open1b.patch)** is used **read-only as
|
||||
the schema oracle**, so the output is validated byte-for-byte.
|
||||
## Conversion in Go
|
||||
|
||||
`tools/hf2gguf` reads the Hugging Face checkpoint (config.json,
|
||||
model.safetensors, tokenizer.json) and writes a GGUF file, using
|
||||
[go-gguf](https://github.com/cymertek/go-gguf). A reference GGUF from llama.cpp's
|
||||
converter, built with our open1b patch, is read only, as the schema to match.
|
||||
|
||||
```bash
|
||||
cd tools/hf2gguf
|
||||
go build -o hf2gguf .
|
||||
|
||||
./hf2gguf dump -gguf models/open-1b-sft-f16.gguf -o manifest.json
|
||||
./hf2gguf convert -ref models/open-1b-sft-f16.gguf \
|
||||
-safetensors models/open-1b-sft/model.safetensors \
|
||||
-config models/open-1b-sft/config.json \
|
||||
@@ -103,48 +99,40 @@ go build -o hf2gguf .
|
||||
./hf2gguf verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
`verify` reports `39/39 metadata keys` and `220/220 tensors` byte-identical; the
|
||||
two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
|
||||
`verify` reports 39 of 39 metadata keys and 220 of 220 tensors identical. The two
|
||||
files share SHA-256
|
||||
`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
|
||||
|
||||
## Running the model locally
|
||||
## Running the model
|
||||
|
||||
> **Engine caveat:** open-1b needs an `open1b` architecture that upstream llama.cpp
|
||||
> does not have. We run a build with a local patch — see
|
||||
> [`docs/inference-stack.md`](docs/inference-stack.md).
|
||||
|
||||
Serve the (or our Go-built) GGUF on one Arc B580 with llama.cpp's Vulkan backend:
|
||||
Upstream llama.cpp has no open1b architecture, so the model needs a patched
|
||||
build. Details in the [inference stack](docs/inference-stack.md).
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
-m models/open-1b-sft-f16-go.gguf \
|
||||
llama-server -m models/open-1b-sft-f16-go.gguf \
|
||||
--device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \
|
||||
--spec-type ngram-map-k,ngram-cache \
|
||||
-t 16 --parallel 1 --alias open-1b --host 127.0.0.1 --port 8085
|
||||
```
|
||||
|
||||
Notes: `-fa off` is required on these Arc/Vulkan builds (so KV stays f16); the
|
||||
requested `-c 32768` is capped to the model's **4096**-token context.
|
||||
`-fa off` is required on these Arc/Vulkan builds, which keeps the KV cache in
|
||||
f16. The requested 32k context is capped to the model's 4096.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Chat-template quirk.** With the exact vendor tail `…assistant<|end_header|>\n\n`,
|
||||
the released SFT checkpoint emits `<|eot|>` as its first token (empty reply).
|
||||
Our GGUF is byte-identical to the vendor one, so this is a property of the
|
||||
released checkpoint, not of the conversion. Workarounds: seed the assistant
|
||||
turn, use `ignore_eos`, or fix the trailing newline.
|
||||
- **Not a strong model.** open-1b scores 25.4 on the OLMo2 suite (vs 31.9 for
|
||||
OLMo2-1B). Its value is **verifiability**, not quality — see
|
||||
[`docs/benchmarks.md`](docs/benchmarks.md).
|
||||
- **Scope.** We only run/inspect the released weights and replay one published
|
||||
commitment; we do not train or fine-tune anything.
|
||||
- Chat template. With the vendor tail `...assistant<|end_header|>` followed by a
|
||||
blank line, the released SFT checkpoint emits an end-of-turn token first and
|
||||
returns an empty reply. Our GGUF is identical to the reference, so this is a
|
||||
property of the released weights. Seed the assistant turn, set ignore_eos, or
|
||||
adjust the trailing newline.
|
||||
- Model quality. open-1b scores 25.4 on the OLMo2 suite, against 31.9 for
|
||||
OLMo2-1B. It is a test of verifiability, not a strong model.
|
||||
- Scope. We run and inspect released weights and replay one published
|
||||
commitment. We do not train or fine-tune anything.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [`LICENSE`](LICENSE). The vendor harness (`open-transformers`) is
|
||||
Apache-2.0; `repop`, the checkpoints and the corpus are distributed separately by
|
||||
Gensyn under their own terms.
|
||||
MIT, see [LICENSE](LICENSE). The harness is Apache-2.0. repop, the checkpoints,
|
||||
and the corpus are distributed by Gensyn under their own terms.
|
||||
|
||||
## Credits
|
||||
|
||||
Built on Gensyn's open-1b release and audit tooling. Independent work; not
|
||||
affiliated with Gensyn.
|
||||
Built on Gensyn's release. Not affiliated with Gensyn.
|
||||
|
||||
+84
-166
@@ -1,214 +1,132 @@
|
||||
# Gensyn OPEN-1B audit — independent replay (CPU, Linux x86-64)
|
||||
# Audit: open-1b init unit on CPU
|
||||
|
||||
- **Date:** 2026-09-28
|
||||
- **Host:** `brainstation`, Linux x86-64, kernel `7.3.0-070300rc1-generic` (Ubuntu), **no NVIDIA GPU**
|
||||
- **Auditor:** SE sub-agent (independent)
|
||||
- **Workdir:** `/root/projects/gensyn/audit`
|
||||
- **Harness:** `gensyn-ai/open-transformers` @ `d7b7b674cd851f7a16815229f060658180cb772b`
|
||||
("Initial commit: the OPEN-1B pretraining and audit-replay harness", 2026-09-15), cloned via git.
|
||||
- **Verdict:** ✅ **MATCH** on the `1b_repop_v2` init unit. ❌ No training-interval
|
||||
replay run (not defined by this kit and not feasible on this host — see §7).
|
||||
- Date: 2026-09-28
|
||||
- Host: brainstation, Linux x86-64, kernel 7.3.0, no NVIDIA GPU
|
||||
- Harness: `gensyn-ai/open-transformers` at `d7b7b674`
|
||||
- Workdir: `audit/`
|
||||
- Verdict: match on the `1b_repop_v2` init unit. No training interval was replayed.
|
||||
|
||||
---
|
||||
## Kit
|
||||
|
||||
## 1. Kit selection
|
||||
The audit app publishes the mapping at `https://open1b.gensyn.ai/manifest.json`:
|
||||
|
||||
Four kits were listed. All four `trajectory.json` files declare the **same**
|
||||
`1b_repop_v2` init commitment
|
||||
`16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed`, so the choice
|
||||
of kit does not change the init hash under test. The authoritative mapping is
|
||||
published by the audit app itself at `https://open1b.gensyn.ai/manifest.json`:
|
||||
|
||||
```json
|
||||
"run": { "id": "20260722-213626-ad3276b", "model": "1b_repop_v2",
|
||||
"initStateHash": "16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed" },
|
||||
"audit": { "kit": { "id": "pt-f91110a0d0c8_rp-c9ca6e71e673",
|
||||
"urlBase": "https://storage.googleapis.com/gensyn-audit-public/audit-kit/pt-f91110a0d0c8_rp-c9ca6e71e673" } }
|
||||
```
|
||||
run: 1b_repop_v2
|
||||
initStateHash 16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed
|
||||
audit: kit pt-f91110a0d0c8_rp-c9ca6e71e673
|
||||
```
|
||||
|
||||
**Kit under test: `pt-f91110a0d0c8_rp-c9ca6e71e673`.**
|
||||
All four published kits declare the same init commitment, so the choice of kit
|
||||
does not change the hash under test.
|
||||
|
||||
- `pretrain_commit`: `f91110a0d0c84fa3bad808f365567d9c655d10b2`
|
||||
- `repop_commit`: `c9ca6e71e673660afc6c87de2592fa6f62c87b15`
|
||||
- `torch_requirement`: `torch==2.10.0`
|
||||
- pretrain commit: `f91110a0d0c84fa3bad808f365567d9c655d10b2`
|
||||
- repop commit: `c9ca6e71e673660afc6c87de2592fa6f62c87b15`
|
||||
- torch: `2.10.0`
|
||||
|
||||
(The other three kits — `pt-cb4ac29438bc_rp-…`, `pt-f6bc086dfebd_rp-…`,
|
||||
`pt-f5eece09abe6_rp-…` — carry the same init commitment but are older
|
||||
pretrain/repop pins. Two share the `c9ca6e71e673` repop but a different pretrain
|
||||
commit; the app manifest picks `f91110a0d0c8`.)
|
||||
## Toolchain
|
||||
|
||||
## 2. Toolchain
|
||||
| Component | Version |
|
||||
| --- | --- |
|
||||
| Python | 3.11.16, via uv. System Python untouched. |
|
||||
| uv | 0.12.17 |
|
||||
| PyTorch | 2.10.0+cpu |
|
||||
| repop | 0.1.5, build commit `c9ca6e71e673`, matches kit.json |
|
||||
| pretrain | 0.1.0, kit wheel |
|
||||
| Go | 1.27, for all glue and checks |
|
||||
|
||||
| Component | Version / identity |
|
||||
|---|---|
|
||||
| Python | 3.11.16 (managed by `uv`; system Python 3.14 untouched) |
|
||||
| `uv` | 0.12.17 |
|
||||
| PyTorch | `2.10.0+cpu` (`torch.version.cuda == None`) |
|
||||
| `repop` | 0.1.5, `build_info().commit == c9ca6e71e673660afc6c87de2592fa6f62c87b15` ✅ (== kit.json `repop_commit`); backends `["cpu","cuda"]` |
|
||||
| `pretrain` | 0.1.0 (kit wheel) |
|
||||
| Go (glue tools) | go1.27.0 (fabric rule: all glue/parsing in Go) |
|
||||
The runbook targets Linux with an NVIDIA GPU and points at the CUDA 12.9 index.
|
||||
This host has no GPU. The README names CPU as the reference device, so the pinned
|
||||
torch version was installed from the CPU index. Init is device-independent by
|
||||
construction, so the result is unaffected.
|
||||
|
||||
**Deviation, stated plainly:** the RUNBOOK targets Linux hosts with an NVIDIA GPU
|
||||
and points them at the CUDA 12.9 PyTorch index. This host has no GPU, and the
|
||||
README names CPU as the *reference* device, so the exact pin `torch==2.10.0` was
|
||||
installed from PyTorch's official **CPU** index (`.../whl/cpu`, wheel
|
||||
`2.10.0+cpu`). The pinned version is unchanged; only the CPU build variant was
|
||||
chosen deliberately, not by an accidental PyPI fallback. Init is device-independent
|
||||
by construction (see §6), so this does not weaken the init result.
|
||||
|
||||
Python (`uv`-managed) is used **only** because the vendor harness is Python and
|
||||
ships as wheels; all download/verification/listing glue is Go under `tools/`
|
||||
(`kitinspect`, `trajinspect`, `kitfetch`, `gslist`).
|
||||
|
||||
## 3. Commands run
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# workdir + docs
|
||||
mkdir -p /root/projects/gensyn/audit && cd /root/projects/gensyn/audit
|
||||
git clone --depth 1 https://github.com/gensyn-ai/open-transformers
|
||||
|
||||
# kit manifests + trajectories (Go inspector)
|
||||
go run ./tools/kitinspect kits/*/
|
||||
go run ./tools/trajinspect kits/*/trajectory.json
|
||||
go run ./tools/kitfetch kits/pt-f91110a0d0c8_rp-c9ca6e71e673 kit-pt-f91110a0d0c8_rp-c9ca6e71e673
|
||||
|
||||
# download + per-file sha256 verification against kit.json (Go tool,
|
||||
# downloads to .part, verifies, removes on mismatch)
|
||||
go run ./tools/kitfetch kits/pt-f91110a0d0c8_rp-c9ca6e71e673 \
|
||||
kit-pt-f91110a0d0c8_rp-c9ca6e71e673
|
||||
|
||||
# Python 3.11 env
|
||||
uv venv --python 3.11 .venv
|
||||
uv pip install --python .venv --index-url https://download.pytorch.org/whl/cpu "torch==2.10.0"
|
||||
uv pip install --python .venv \
|
||||
./kit-.../repop-0.1.5-cp311-cp311-linux_x86_64.whl \
|
||||
./kit-.../pretrain-0.1.0-cp311-cp311-linux_x86_64.whl
|
||||
./kit-pt-f91110a0d0c8_rp-c9ca6e71e673/repop-0.1.5-cp311-cp311-linux_x86_64.whl \
|
||||
./kit-pt-f91110a0d0c8_rp-c9ca6e71e673/pretrain-0.1.0-cp311-cp311-linux_x86_64.whl
|
||||
|
||||
# provenance assertions
|
||||
.venv/bin/python -c "import repop,json; print(repop.build_info())"
|
||||
|
||||
# THE AUDIT (init unit)
|
||||
PRETRAIN_AUDIT_MEMLOG=1 /usr/bin/time -v .venv/bin/pretrain-audit-replay \
|
||||
--from-init --until-step 0 \
|
||||
--config-name 1b_repop_v2 \
|
||||
--device cpu \
|
||||
.venv/bin/pretrain-audit-replay --from-init --until-step 0 \
|
||||
--config-name 1b_repop_v2 --device cpu \
|
||||
--expect-hash 16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed
|
||||
```
|
||||
|
||||
## 4. SHA-256 verification table (all 5 kit files)
|
||||
## Checksums
|
||||
|
||||
Every file was hashed **before** install; the Go fetcher adopts a file only on a
|
||||
full 64-hex match and deletes rejected bytes. Expected values are read from the
|
||||
kit's own `kit.json`.
|
||||
All five kit files were hashed before install. The Go fetcher accepts a file only
|
||||
on a full match and deletes rejected bytes. Expected values come from kit.json.
|
||||
|
||||
| File | expected (`kit.json`) | computed | result |
|
||||
|---|---|---|---|
|
||||
| `pretrain-0.1.0-cp311-cp311-linux_x86_64.whl` | `ecc49d1c05c0e2876a59d98d877795769d6e9620eb66e4b4c866b16f695fb174` | same | ✅ OK |
|
||||
| `pretrain-0.1.0-cp311-cp311-macosx_14_0_arm64.whl` | `fbd23a54aa0bf87775e86493d67f759d1c1f4cc1dcd63ad991fba59dcb92491c` | same | ✅ OK |
|
||||
| `repop-0.1.5-cp311-cp311-linux_x86_64.whl` | `15f171643871211b2a772ee83c94833c0be2eef0495790cedc33813c95147ff0` | same | ✅ OK |
|
||||
| `repop-0.1.5-cp311-cp311-macosx_14_0_arm64.whl` | `d8658e0327c7278812ecb6dd455cd4cf8723a974a78788adba41d5f747289c69` | same | ✅ OK |
|
||||
| `trajectory.json` | `9435515c37d8956d0c81d5bc78c0a383bb6bf27c57b5dbaf1f8be564df0b324a` | same | ✅ OK |
|
||||
| File | SHA-256 | Result |
|
||||
| --- | --- | --- |
|
||||
| `pretrain-0.1.0-cp311-cp311-linux_x86_64.whl` | `ecc49d1c05c0e2876a59d98d877795769d6e9620eb66e4b4c866b16f695fb174` | match |
|
||||
| `pretrain-0.1.0-cp311-cp311-macosx_14_0_arm64.whl` | `fbd23a54aa0bf87775e86493d67f759d1c1f4cc1dcd63ad991fba59dcb92491c` | match |
|
||||
| `repop-0.1.5-cp311-cp311-linux_x86_64.whl` | `15f171643871211b2a772ee83c94833c0be2eef0495790cedc33813c95147ff0` | match |
|
||||
| `repop-0.1.5-cp311-cp311-macosx_14_0_arm64.whl` | `d8658e0327c7278812ecb6dd455cd4cf8723a974a78788adba41d5f747289c69` | match |
|
||||
| `trajectory.json` | `9435515c37d8956d0c81d5bc78c0a383bb6bf27c57b5dbaf1f8be564df0b324a` | match |
|
||||
|
||||
Byte sizes also matched `kit.json`. Only the two **linux_x86_64** wheels were
|
||||
installed; the macOS arm64 wheels were downloaded and verified but not installed.
|
||||
Only the two linux_x86_64 wheels were installed.
|
||||
|
||||
## 5. Init-unit result — MATCH
|
||||
|
||||
Raw result object printed by the harness (stdout, exit status 0):
|
||||
## Result
|
||||
|
||||
```json
|
||||
{
|
||||
"step": 0,
|
||||
"consumed_tokens": 0,
|
||||
"state_hash": "16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed",
|
||||
"mode": "init",
|
||||
"repop": { "commit": "c9ca6e71e673660afc6c87de2592fa6f62c87b15",
|
||||
"backends": ["cpu", "cuda"], "cuda_arch_list": "7.5 8.0 9.0" },
|
||||
"device": "cpu",
|
||||
"expected": "16554a1119745f1b9bf4ac60b03226e5e9ca1247a2bd56169cd4c822fd457aed",
|
||||
"match": true,
|
||||
"cpu_threads": { "requested": "auto", "source": "runtime-default",
|
||||
"torch_num_threads": 8, "omp_num_threads": null, "mkl_num_threads": null }
|
||||
"match": true
|
||||
}
|
||||
```
|
||||
|
||||
Log line: `from-init: regenerated init state_hash=16554a1119745f1b expected=16554a1119745f1b MATCH=True`.
|
||||
Config `1b_repop_v2`, seed `42`, loss path `fused CE+z-loss`. The only warning is
|
||||
the expected `not distributed + no CUDA — parallelize applies AC only (FSDP
|
||||
skipped; dev mode)`, i.e. the documented CPU reference path.
|
||||
|
||||
**Resource notes:** wall **58.3 s**, user 78.5 s / sys 61.1 s, **peak RSS
|
||||
16.80 GB** (16,804,956 KB), **0 swaps**, exit 0. (README/RUNBOOK size this unit at
|
||||
~21.9 GB peak on CPU; we measured 16.8 GB. Free RAM was ~11 GB and the unit still
|
||||
completed without swapping, so the doc figure is conservative.)
|
||||
Resources: 58.3 s wall, peak RSS 16.8 GB, no swap, exit 0.
|
||||
|
||||
## 6. What this proves — and what it does not
|
||||
## Limits
|
||||
|
||||
**Proves.** The bytes installed are the bytes `kit.json` published (all five
|
||||
SHA-256s match). The `repop` kernel build is the one the kit claims
|
||||
(`build_info().commit == repop_commit`). Running the vendor harness on this
|
||||
CPU, from seed, the `1b_repop_v2` **initial state** — seeded device-independent
|
||||
trunc-normal init plus zeroed optimizer moments — reproduces the published
|
||||
canonical hash `16554a11…` **bit-for-bit**. Independent third-party
|
||||
reproduction of the run's init commitment: ✅.
|
||||
1. No training interval. An init-only unit says nothing about the forward pass,
|
||||
the backward pass, the optimizer, or the data stream. The per-step claim is
|
||||
not exercised.
|
||||
2. repop is closed source. Only compiled wheels ship, under the Gensyn
|
||||
Reproducibility Licence. The kernels that make training reproducible cannot be
|
||||
read or checked. The init unit does not even run the kernels changed by the
|
||||
recent repop repin.
|
||||
3. Hash coverage. The v3 hash covers weights, optimizer state, gradients, and the
|
||||
batch digest. It omits the RNG, the data-stream cursor, spike state, and the
|
||||
meta.json descriptor keys.
|
||||
4. Init is the easy case. init.py samples Philox on CPU and byte-copies, so it is
|
||||
device-independent by design. The hard cross-device claim lives in the
|
||||
training kernels, which this unit does not touch.
|
||||
5. Provenance of the published run is not established. We only show that the
|
||||
pinned artifacts reproduce the published commitment on this machine.
|
||||
|
||||
**Does not prove.**
|
||||
## Interval replay was not attempted
|
||||
|
||||
1. **No training interval.** Per the README: "An initialization-only unit does
|
||||
not verify a training interval." A match here says nothing about any forward,
|
||||
backward, AdamW or data-stream step. The claimed bitwise reproducibility of
|
||||
*optimizer steps* is **not** tested by this run.
|
||||
2. **`repop` is closed source.** Only compiled wheels are shipped; the kernel
|
||||
arithmetic that makes training bitwise reproducible cannot be read or
|
||||
independently re-derived from this kit. The init unit does not even exercise
|
||||
the kernels that the recent `c9ca6e71e673` repin changed (the trajectory notes
|
||||
say exactly this), so a match is *not* evidence for the kernel-level BFR
|
||||
contract at that commit.
|
||||
3. **v3-hash coverage.** The README states the v3 hash covers weights, optimizer
|
||||
state, gradients and the batch digest, but **not** RNG, the data-stream
|
||||
cursor, spike state, or the `meta.json` descriptor keys. Even a matching
|
||||
interval hash would leave those unverified.
|
||||
4. **Init is the easy case by design.** `init.py` samples Philox on CPU and
|
||||
byte-copies, so it is device-independent; it is expected to match across
|
||||
devices. The hard cross-device claim lives in the training kernels, which
|
||||
this unit does not touch.
|
||||
5. Provenance of the *published run* is not established; only that the pinned
|
||||
artifacts reproduce the published commitment on this machine.
|
||||
This is a decision, not a stall.
|
||||
|
||||
## 7. Step-interval replay — deliberately not attempted
|
||||
- Not defined by the kit. This kit's trajectory.json contains only three init
|
||||
units (`100m_smoke_repop`, `1b_repop_run3`, `1b_repop_v2`). No interval unit
|
||||
with a canonical target hash is published, so there is no commitment to check.
|
||||
- Memory. A step replay needs about 24 GB. This host had about 11 GB free, and
|
||||
the init unit already peaked at 16.8 GB.
|
||||
- Disk. One checkpoint `ckpt/step_000000100/` is 19.31 GB; the full `ckpt/` tree
|
||||
is 15.64 TB. With 39 GB free, a single checkpoint is tight.
|
||||
|
||||
Stopping after the init unit is a considered decision, not a stall:
|
||||
Re-attempt on a host with 32 GB or more RAM and 60 GB or more disk, ideally with
|
||||
a CUDA GPU.
|
||||
|
||||
- **Not defined by the kit.** This kit's `trajectory.json` (`init-units-v1`)
|
||||
contains **only 3 init units** — `100m_smoke_repop`, `1b_repop_run3`,
|
||||
`1b_repop_v2`. Its own note says: "Step-interval units are added per run once
|
||||
its checkpoints and state_hash files are published." **No interval unit with a
|
||||
canonical target hash is published here**, so the kit defines no
|
||||
step-interval commitment to check. (The run's per-step log exists at
|
||||
`gs://gensyn-open-1b/logs/state_hashes.jsonl`, but the harness has no automatic
|
||||
lookup and the audit app gates submissions.)
|
||||
- **Memory.** A step-interval replay needs ~24 GB (README), and the audit's
|
||||
auto-offload only reaches "every published unit" at that figure. This host had
|
||||
~11 GB RAM available (29 GB total, 18 GB in use); an init unit already peaked at
|
||||
16.8 GB. The heavy replay would swap-thrash and would not fit honestly.
|
||||
- **Disk/size.** One checkpoint `ckpt/step_000000100/` is **19.31 GB**
|
||||
(150 objects); the full `ckpt/` prefix is **15.64 TB** (810 checkpoint dirs).
|
||||
With 39 GB free, a single checkpoint is tight against free space and the
|
||||
documented `--from-init + --gcs-root` combination is broken (wrong shard
|
||||
anchor), so a from-init interval would first require staging shards via
|
||||
`--data-root`.
|
||||
## Files
|
||||
|
||||
Re-attempting an interval audit should be done on a host with ≥32 GB free RAM
|
||||
(NVIDIA GPU preferred: `--device cuda`) and ≥60 GB free disk.
|
||||
|
||||
## 8. Files produced
|
||||
|
||||
- `open-transformers/` — harness clone (docs read: `README.md`,
|
||||
`docs/audit-replay-usage.md`, `scripts/audit_volunteer/RUNBOOK.md`)
|
||||
- `kits/<kit>/kit.json`, `kits/<kit>/trajectory.json` — manifests
|
||||
- `kit-pt-f91110a0d0c8_rp-c9ca6e71e673/` — verified wheels + trajectory
|
||||
- `.venv/` — Python 3.11 audit env
|
||||
- `init-replay.log` — full captured replay output + `/usr/bin/time -v`
|
||||
- `tools/{kitinspect,trajinspect,kitfetch,gslist}` — Go glue
|
||||
- `REPORT.md` — this file
|
||||
|
||||
No secrets were written. Nothing outside the workdir was modified or deleted.
|
||||
- `open-transformers/`: harness clone
|
||||
- `kits/`: manifests
|
||||
- `kit-pt-f91110a0d0c8_rp-c9ca6e71e673/`: verified wheels
|
||||
- `.venv/`: Python 3.11 audit environment
|
||||
- `tools/`: Go glue (`kitinspect`, `trajinspect`, `kitfetch`, `gslist`)
|
||||
- `init-replay.log`: full replay output
|
||||
|
||||
+27
-34
@@ -1,48 +1,41 @@
|
||||
# Benchmarks
|
||||
|
||||
Speed of **open-1b** on our hardware, with a reference comparison against the
|
||||
model we run in production (`gemma-4-E2B`). All numbers are single-stream
|
||||
(one request at a time) on one GPU.
|
||||
open-1b on our hardware, with the production model, gemma-4-E2B, as reference.
|
||||
Single stream, one GPU.
|
||||
|
||||
## Hardware / setup
|
||||
## Setup
|
||||
|
||||
- **GPU:** 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp `build-vulkan` (b10893).
|
||||
- **CPU:** AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM.
|
||||
- **Serving flags:** `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`.
|
||||
- `-fa off` is mandatory on these Arc/Vulkan builds → KV cache is f16 (no `-ctk q8_0`).
|
||||
- GPU: one Intel Arc B580 (BMG G21, 12 GB), Vulkan, llama.cpp build-vulkan (b10893).
|
||||
- CPU: AMD Ryzen 7 7700X, 29 GB RAM.
|
||||
- Flags: `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`.
|
||||
- `-fa off` is required on these Arc/Vulkan builds, so the KV cache stays in f16.
|
||||
|
||||
## open-1b (our Go-built GGUF, f16)
|
||||
## open-1b, Go-built GGUF, f16
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Decode | **~112–117 tok/s** (e.g. 256 tok @ 111.7; 96 tok @ 117.2) |
|
||||
| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) |
|
||||
| VRAM | **~3.6 GiB** (10971 → 7258 MiB free) |
|
||||
| Context | 4096 (model limit; the requested 32k is capped) |
|
||||
| Weights | f16, ~3.2 GB |
|
||||
| --- | --- |
|
||||
| Decode | 112 to 117 tok/s |
|
||||
| Prefill | about 892 tok/s |
|
||||
| VRAM | about 3.6 GiB |
|
||||
| Context | 4096, the model limit |
|
||||
| Weights | f16, about 3.2 GB |
|
||||
|
||||
## Reference comparison (1× Arc B580)
|
||||
## Reference, one Arc B580
|
||||
|
||||
| Model | Weights | Context | Decode | Prefill |
|
||||
|---|---|---|---|---|
|
||||
| **open-1b-sft** (this repo) | f16 ~3.2 GB | 4096 | ~114 tok/s | ~892 tok/s |
|
||||
| **gemma-4-E2B** (prod) | Q4 ~1.5 GiB | 64k | **~214 tok/s** | **~1659 tok/s** |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| open-1b-sft | f16, about 3.2 GB | 4096 | about 114 tok/s | about 892 tok/s |
|
||||
| gemma-4-E2B | Q4, about 1.5 GiB | 64k | about 214 tok/s | about 1659 tok/s |
|
||||
|
||||
*gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok /
|
||||
prefill 1698 tok) on the same card and build.*
|
||||
The gemma figures come from our 2026-09-20 benchmark on the same card and build.
|
||||
|
||||
**Reading:** gemma is roughly **2× faster**, ~**2.4× smaller**, and has **16×
|
||||
the context** of open-1b. For any practical workload (classification, extraction,
|
||||
translation, tool-calling) gemma wins outright. open-1b's reason to exist is not
|
||||
throughput or quality but **auditable training**.
|
||||
gemma is about twice as fast, about 2.4 times smaller, and has sixteen times the
|
||||
context. For classification, extraction, translation, or tool calls, gemma is the
|
||||
better choice. open-1b is interesting for its training audit, not for its speed.
|
||||
|
||||
## Caveats
|
||||
## Notes
|
||||
|
||||
- This is **not** a controlled A/B: the open-1b figures are a short first-run
|
||||
measurement with different prompts; the gemma figures come from a separate
|
||||
controlled benchmark. Use them as an order-of-magnitude orientation.
|
||||
- The `ngram` speculative decoder helps task-dependent amounts; open-1b's
|
||||
baseline (no-spec) number was not measured here.
|
||||
- Quality: open-1b scores **25.4** on the OLMo2 suite vs **31.9** for OLMo2-1B,
|
||||
i.e. below an ordinary 1B model. It is a proof of *verifiability*, not a strong
|
||||
assistant.
|
||||
- Not a controlled comparison. The open-1b numbers are a short first run with
|
||||
different prompts; the gemma numbers come from a separate controlled benchmark.
|
||||
- No speculative-decoder-off measurement for open-1b.
|
||||
- Quality: open-1b scores 25.4 on the OLMo2 suite, against 31.9 for OLMo2-1B.
|
||||
|
||||
+44
-46
@@ -1,38 +1,38 @@
|
||||
# Inference stack — how llama.cpp was built and launched
|
||||
# Inference stack: how llama.cpp was built and launched
|
||||
|
||||
This documents the exact engine used to run **open-1b** locally, and answers the
|
||||
fair question: *is the llama.cpp build stock?* **No.**
|
||||
This records the engine behind the local run. Is the llama.cpp build stock? No.
|
||||
|
||||
## Engine provenance — not stock
|
||||
## Provenance
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Upstream base** | `ggerganov/llama.cpp` @ `c32d1dabe` — "tests : increase tolerance for Add fusion tests (#28691)", 2026-09-10 |
|
||||
| **Local changes** | uncommitted patches adding the **`open1b`** architecture (9 files, +299 lines) |
|
||||
| **Exported patch** | [`llama-cpp-open1b.patch`](llama-cpp-open1b.patch) |
|
||||
| **Build** | `-DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release` |
|
||||
| **Binary** | `build-vulkan/bin/llama-server` — `0.4.0-dev (build 10893, commit c32d1dabe)` |
|
||||
| --- | --- |
|
||||
| Upstream base | `ggerganov/llama.cpp` at `c32d1dabe`, 2026-09-10 |
|
||||
| Local changes | uncommitted patches adding the `open1b` architecture, 9 files, +299 lines |
|
||||
| Patch | [llama-cpp-open1b.patch](llama-cpp-open1b.patch) |
|
||||
| Build | `-DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release` |
|
||||
| Binary | `build-vulkan/bin/llama-server`, `0.4.0-dev`, build 10893 |
|
||||
|
||||
**Why the patch is required.** Upstream llama.cpp has no `open1b` architecture. open-1b
|
||||
is Llama-3-derived but differs in four ways that the loader/graph must know about:
|
||||
Upstream has no `open1b` architecture, so the patch is required. open-1b is
|
||||
Llama-3-derived, with four differences the loader and graph must know about:
|
||||
|
||||
- gain-free per-head RMSNorm on Q and K before RoPE (`qk_norm`, no learnable gain);
|
||||
- RMSNorm on the **token-embedding output** (`embedding_norm`);
|
||||
- **hybrid sliding-window attention** (512-token window, full causal on the last layer of each group of 5 and on the final layer);
|
||||
- **untied** input embedding / LM head, and interleaved-pair RoPE (`LLAMA_ROPE_TYPE_NORM`).
|
||||
- gain-free per-head RMSNorm on Q and K before RoPE,
|
||||
- RMSNorm on the token-embedding output,
|
||||
- hybrid sliding-window attention, 512 tokens, full causal on the last layer of
|
||||
each group of five and on the final layer,
|
||||
- untied embedding and LM head, with interleaved-pair RoPE.
|
||||
|
||||
## Patch inventory
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `src/models/open1b.cpp` | **new** — `llama_model_open1b`: hparams, tensor mapping, graph |
|
||||
| `src/models/models.h` | declare `llama_model_open1b` |
|
||||
| `src/llama-arch.h` / `.cpp` | add `LLM_ARCH_OPEN1B` (enum + name) |
|
||||
| `src/llama-model.cpp` | dispatch `LLM_ARCH_OPEN1B`; RoPE type `NORM` |
|
||||
| `gguf-py/gguf/constants.py` | `MODEL_ARCH.OPEN1B`, arch name, tensor list |
|
||||
| `gguf-py/gguf/tensor_mapping.py` | HF → GGUF tensor name map |
|
||||
| --- | --- |
|
||||
| `src/models/open1b.cpp` | new: `llama_model_open1b`, hparams, tensors, graph |
|
||||
| `src/models/models.h` | declare the model |
|
||||
| `src/llama-arch.h`, `.cpp` | add `LLM_ARCH_OPEN1B` |
|
||||
| `src/llama-model.cpp` | dispatch the arch, set the RoPE type |
|
||||
| `gguf-py/gguf/constants.py` | arch enum, name, tensor list |
|
||||
| `gguf-py/gguf/tensor_mapping.py` | HF to GGUF tensor names |
|
||||
| `conversion/__init__.py` | register the converter |
|
||||
| `conversion/open1b.py` | **new** — `Open1BModel` HF→GGUF converter |
|
||||
| `conversion/open1b.py` | new: `Open1BModel` converter |
|
||||
|
||||
## Build
|
||||
|
||||
@@ -57,35 +57,33 @@ setsid --fork ./build-vulkan/bin/llama-server \
|
||||
--host 127.0.0.1 --port 8085 >/tmp/open1b.log 2>&1 </dev/null &
|
||||
```
|
||||
|
||||
| Flag | Why |
|
||||
|---|---|
|
||||
| `--device Vulkan2` | a single Intel Arc B580 (Vulkan1/2/3 = the three Arc; Vulkan0 = AMD iGPU) |
|
||||
| Flag | Reason |
|
||||
| --- | --- |
|
||||
| `--device Vulkan2` | one Intel Arc B580. Vulkan1 to 3 are the three Arc cards; Vulkan0 is the AMD iGPU. |
|
||||
| `-ngl 99` | offload all layers |
|
||||
| `-fa off` | Flash-Attention crashes on these Arc/Vulkan builds; KV thus stays f16 |
|
||||
| `-c 32768` | requested, but capped to the model's **4096**-token context |
|
||||
| `--spec-type ngram-map-k,ngram-cache` | ~×2.4 decode on small dense models |
|
||||
| `-fa off` | Flash-Attention crashes on these Arc/Vulkan builds, so the KV cache stays in f16 |
|
||||
| `-c 32768` | requested, capped to the model's 4096-token context |
|
||||
| `--spec-type ngram-map-k,ngram-cache` | about 2.4x decode on small dense models |
|
||||
| `--jinja` | use the model's chat template |
|
||||
|
||||
## Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
HF["HF checkpoint\nmodel.safetensors (fp32)"] --> GO["tools/hf2gguf\n(pure Go)"]
|
||||
GO --> GGUF["open-1b-sft-f16-go.gguf\n(f16, byte-identical to reference)"]
|
||||
GGUF --> LC["llama.cpp Vulkan\nupstream c32d1dabe\n+ open1b.patch"]
|
||||
LC --> GPU["Intel Arc B580\n(Vulkan2)"]
|
||||
GPU --> API["llama-server\nHTTP /completion"]
|
||||
HF["HF checkpoint<br/>model.safetensors, fp32"] --> GO["tools/hf2gguf<br/>Go"]
|
||||
GO --> GGUF["open-1b-sft-f16-go.gguf<br/>f16"]
|
||||
GGUF --> LC["llama.cpp Vulkan<br/>upstream plus open1b patch"]
|
||||
LC --> GPU["Intel Arc B580"]
|
||||
GPU --> API["llama-server<br/>HTTP /completion"]
|
||||
```
|
||||
|
||||
## Caveats
|
||||
## Notes
|
||||
|
||||
- **The patch is ours and uncommitted.** It lives only in the working tree of the
|
||||
build source; `llama-cpp-open1b.patch` is its export. For reproducibility it should
|
||||
be committed and, ideally, proposed upstream.
|
||||
- **The "reference" GGUF is not an independent vendor artifact.** It was produced by
|
||||
llama.cpp's `convert_hf_to_gguf.py` **with this same Open1B patch**. Our Go converter
|
||||
reproduces it byte-for-byte — a genuine independent reimplementation of the GGUF
|
||||
serialization — but "byte-identical to the reference" means *identical to our patched
|
||||
Python path*, not to a Gensyn-published file.
|
||||
- Patch quality: written for bring-up and validated by a working inference run; not
|
||||
reviewed upstream.
|
||||
- The patch is ours and uncommitted. The exported patch file makes the build
|
||||
reproducible. Committing it, or proposing it upstream, would be better.
|
||||
- The reference GGUF is not an independent artifact. It came from llama.cpp's
|
||||
`convert_hf_to_gguf.py` with this same patch. The Go converter reproduces it
|
||||
byte for byte, which is a real reimplementation of the GGUF writer, but the
|
||||
comparison is against our own patched Python output, not a file published by
|
||||
Gensyn.
|
||||
- The patch was written for bring-up. It is not upstream-reviewed.
|
||||
|
||||
+111
-266
@@ -1,332 +1,177 @@
|
||||
# Gensyn open-1b (SFT) — local bring-up & proof of run
|
||||
# Local run: open-1b on one Arc B580
|
||||
|
||||
Epic: `vado/1b#1` / task #4 (SE). Workdir: `/root/projects/gensyn/inference/`.
|
||||
Engine: existing shared llama.cpp **Vulkan** build (`/root/projects/inference/build-vulkan/bin/llama-server`).
|
||||
Language: **Go 1.27 only** (see "Process note" re one accidental `python3` call).
|
||||
- Workdir: `inference/`
|
||||
- Engine: the shared llama.cpp Vulkan build, with the open1b patch
|
||||
- Language: Go 1.27
|
||||
|
||||
## TL;DR
|
||||
## Summary
|
||||
|
||||
- Built a pure-Go tool `hf2gguf` (module `hf2gguf`) that **dumps** the reference GGUF
|
||||
(go-gguf + a minimal raw-KV walker for arrays), **converts** `model.safetensors`
|
||||
→ `models/open-1b-sft-f16-go.gguf`, and **verifies** the two files.
|
||||
- **Validation: PASS — 39/39 metadata KV byte-identical, 220/220 tensors byte-identical.**
|
||||
Whole files are byte-for-byte identical (`cmp` + SHA-256 match).
|
||||
- **Served** on one Arc (Vulkan2 / `07:00.0`) with the existing Vulkan build; coherent
|
||||
output, **~114 tok/s decode**, **~892 tok/s prefill**, ~**3.6 GiB VRAM**.
|
||||
- No commit made (all under gitignored `inference/`).
|
||||
- A Go tool, `hf2gguf`, with three commands: `dump`, `convert`, `verify`.
|
||||
- Validation: 39 of 39 metadata keys and 220 of 220 tensors are byte-identical.
|
||||
The two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
|
||||
- Served on one Arc (Vulkan2, PCI `07:00.0`): about 114 tok/s decode, about 892
|
||||
tok/s prefill, about 3.6 GiB VRAM.
|
||||
|
||||
Reference SHA-256 (and our output — identical):
|
||||
`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`
|
||||
|
||||
---
|
||||
|
||||
## 1. Inputs & topology
|
||||
## Inputs
|
||||
|
||||
| Item | Path | Notes |
|
||||
|------|------|-------|
|
||||
| HF checkpoint (fp32) | `models/open-1b-sft/model.safetensors` | 6,432,081,672 B; dtype **F32**; header 34560 B |
|
||||
| Reference GGUF (f16) | `models/open-1b-sft-f16.gguf` | 3,220,153,856 B; produced by llama.cpp `convert_hf_to_gguf.py` + our local Open1B patch; **read-only oracle** |
|
||||
| HF config | `models/open-1b-sft/config.json` | `model_type=open1b`, 24 layers, hidden 2048, heads 16/4, head_dim 128, ffn 5632, vocab 128256, rope_theta 5e5, rms_eps 1e-5, swa 512 every 5, qk_norm gain-free, embedding_norm, untied |
|
||||
| Vendor Python | `models/open-1b-sft/modeling_open1b.py` | read only, never executed |
|
||||
| Engine | `/root/projects/inference/build-vulkan/bin/llama-server` | symlink → `/mnt/2TB/projects/build-vulkan`; shared prod tree, untouched |
|
||||
| --- | --- | --- |
|
||||
| HF checkpoint | `models/open-1b-sft/model.safetensors` | 6.43 GB, F32 |
|
||||
| Reference GGUF | `models/open-1b-sft-f16.gguf` | 3.22 GB, f16, read-only |
|
||||
| HF config | `models/open-1b-sft/config.json` | see below |
|
||||
| Engine | `build-vulkan/bin/llama-server` | shared prod tree, not modified |
|
||||
|
||||
GPU devices as reported by the engine (`--list-devices`):
|
||||
The config: `model_type=open1b`, 24 layers, hidden 2048, heads 16 and key/value
|
||||
heads 4, head_dim 128, feed-forward 5632, vocab 128256, rope_theta 500000,
|
||||
rms_eps 1e-5, hybrid sliding window 512 every 5 layers, gain-free qk_norm,
|
||||
embedding_norm, untied embeddings.
|
||||
|
||||
Devices as the engine lists them:
|
||||
|
||||
```
|
||||
Vulkan0: AMD Ryzen 7 7700X (RADV RAPHAEL_MENDOCINO) (15835 MiB)
|
||||
Vulkan1: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB)
|
||||
Vulkan2: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB)
|
||||
Vulkan3: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB)
|
||||
Vulkan0: AMD Ryzen 7 7700X (RADV RAPHAEL_MENDOCINO), 15835 MiB
|
||||
Vulkan1: Intel Arc B580 (BMG G21), 12216 MiB
|
||||
Vulkan2: Intel Arc B580 (BMG G21), 12216 MiB
|
||||
Vulkan3: Intel Arc B580 (BMG G21), 12216 MiB
|
||||
```
|
||||
|
||||
`Vulkan2` was confirmed to map to PCI `0000:07:00.0` (Arc #2) via
|
||||
Vulkan2 maps to PCI `0000:07:00.0`, confirmed through
|
||||
`/sys/class/drm/renderD129/device`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tool: `hf2gguf` (Go 1.27, `github.com/cymertek/go-gguf`)
|
||||
## Tool
|
||||
|
||||
```
|
||||
inference/hf2gguf/
|
||||
go.mod, go.sum
|
||||
main.go # subcommand dispatch + JSON manifest types
|
||||
rawkv.go # minimal read-only GGUF KV walker (fills go-gguf's array gap)
|
||||
safetensors.go # safetensors header + tensor byte access
|
||||
hfconfig.go # config.json / tokenizer_config.json / tokenizer.json validation
|
||||
convert.go # metadata reproduction + HF→GGUF tensor mapping + F32→F16
|
||||
verify.go # byte-level metadata & tensor comparison
|
||||
convert_test.go
|
||||
main.go dispatch, manifest types
|
||||
rawkv.go read-only GGUF KV walker, array support
|
||||
safetensors.go safetensors header and tensor bytes
|
||||
hfconfig.go config and tokenizer parsing, validation
|
||||
convert.go metadata reproduction, tensor mapping, F32 to F16
|
||||
verify.go byte-level comparison
|
||||
convert_test.go tests
|
||||
```
|
||||
|
||||
### Why a raw KV walker
|
||||
`go-gguf` returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV
|
||||
section directly to keep the tokenizer arrays (`tokens`, `merges`, `token_type`)
|
||||
as raw bytes. Writing still goes through the `go-gguf` writer. GGUF
|
||||
serialization is not hand-rolled.
|
||||
|
||||
`go-gguf`'s `MetadataEntry.Value()` intentionally returns an empty value for
|
||||
`BTypeArray` (see `readLazyValue` in the library). The tokenizer arrays
|
||||
(`tokens`, `scores`→`token_type`, `merges`) are exactly the values we must copy
|
||||
verbatim, so `rawkv.go` reads the KV section directly and preserves each array's
|
||||
`elem_type`, element count, and **raw element bytes**. Those raw bytes are fed
|
||||
straight into go-gguf's writer `Value{Raw: ...}`, so serialization is still done
|
||||
by the library (nothing hand-rolled on the write path).
|
||||
|
||||
### Subcommands
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
# 2a. dump — emit the reproduction contract as JSON
|
||||
go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json
|
||||
|
||||
# 2b/c. convert — reproduce metadata + map safetensors→GGUF (fp32→f16)
|
||||
go run . convert \
|
||||
-ref models/open-1b-sft-f16.gguf \
|
||||
go run . convert -ref models/open-1b-sft-f16.gguf \
|
||||
-safetensors models/open-1b-sft/model.safetensors \
|
||||
-config models/open-1b-sft/config.json \
|
||||
-tokenizer-config models/open-1b-sft/tokenizer_config.json \
|
||||
-tokenizer models/open-1b-sft/tokenizer.json \
|
||||
-o models/open-1b-sft-f16-go.gguf
|
||||
|
||||
# 3. verify — byte-level equality against the reference
|
||||
go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
`dump` output (the contract) is at
|
||||
`models/open-1b-sft-f16.manifest.json` (39 metadata entries + 220 tensors,
|
||||
with per-array `raw_sha256`).
|
||||
## Tensor mapping
|
||||
|
||||
### Tensor mapping (reference names are the contract)
|
||||
The reference names are the contract.
|
||||
|
||||
Shapes are the reference's; ggml `ne0` is the innermost (= HF input) dim, so HF
|
||||
bytes are copied unpermuted with the dims reversed.
|
||||
| GGUF tensor | HF source | Shape | Type |
|
||||
| --- | --- | --- | --- |
|
||||
| `token_embd.weight` | `embedding.tok_embeddings.weight` | 2048 x 128256 | F16 |
|
||||
| `token_embd_norm.weight` | `emb_norm.weight` | 2048 | F32 |
|
||||
| `output.weight` | `embedding.output.weight` | 2048 x 128256 | F16 |
|
||||
| `output_norm.weight` | `norm_out.weight` | 2048 | F32 |
|
||||
| `blk.i.attn_q.weight` | `blocks.i.attn.wq.weight` | 2048 x 2048 | F16 |
|
||||
| `blk.i.attn_k.weight` | `blocks.i.attn.wk.weight` | 2048 x 512 | F16 |
|
||||
| `blk.i.attn_v.weight` | `blocks.i.attn.wv.weight` | 2048 x 512 | F16 |
|
||||
| `blk.i.attn_output.weight` | `blocks.i.attn.wo.weight` | 2048 x 2048 | F16 |
|
||||
| `blk.i.ffn_gate.weight` | `blocks.i.ffn.w_gate_up.weight`, first half | 2048 x 5632 | F16 |
|
||||
| `blk.i.ffn_up.weight` | `blocks.i.ffn.w_gate_up.weight`, second half | 2048 x 5632 | F16 |
|
||||
| `blk.i.ffn_down.weight` | `blocks.i.ffn.w_down.weight` | 5632 x 2048 | F16 |
|
||||
| `blk.i.attn_norm.weight` | `blocks.i.norm1.weight` | 2048 | F32 |
|
||||
| `blk.i.ffn_norm.weight` | `blocks.i.norm2.weight` | 2048 | F32 |
|
||||
|
||||
| Reference tensor | HF source | Shape | Type | Count |
|
||||
|---|---|---|---|---|
|
||||
| `token_embd.weight` | `embedding.tok_embeddings.weight` | [2048,128256] | F16 | 1 |
|
||||
| `token_embd_norm.weight` | `emb_norm.weight` | [2048] | F32 | 1 |
|
||||
| `output.weight` | `embedding.output.weight` | [2048,128256] | F16 | 1 |
|
||||
| `output_norm.weight` | `norm_out.weight` | [2048] | F32 | 1 |
|
||||
| `blk.{i}.attn_q.weight` | `blocks.{i}.attn.wq.weight` | [2048,2048] | F16 | 24 |
|
||||
| `blk.{i}.attn_k.weight` | `blocks.{i}.attn.wk.weight` | [2048,512] | F16 | 24 |
|
||||
| `blk.{i}.attn_v.weight` | `blocks.{i}.attn.wv.weight` | [2048,512] | F16 | 24 |
|
||||
| `blk.{i}.attn_output.weight` | `blocks.{i}.attn.wo.weight` | [2048,2048] | F16 | 24 |
|
||||
| `blk.{i}.ffn_gate.weight` | `blocks.{i}.ffn.w_gate_up.weight` (first half) | [2048,5632] | F16 | 24 |
|
||||
| `blk.{i}.ffn_up.weight` | `blocks.{i}.ffn.w_gate_up.weight` (second half) | [2048,5632] | F16 | 24 |
|
||||
| `blk.{i}.ffn_down.weight` | `blocks.{i}.ffn.w_down.weight` | [5632,2048] | F16 | 24 |
|
||||
| `blk.{i}.attn_norm.weight` | `blocks.{i}.norm1.weight` | [2048] | F32 | 24 |
|
||||
| `blk.{i}.ffn_norm.weight` | `blocks.{i}.norm2.weight` | [2048] | F32 | 24 |
|
||||
The fused `w_gate_up` is split at the row boundary. Per-channel `*.weight_scale`
|
||||
tensors in the checkpoint are dropped; the reference has none. F32 to F16 uses
|
||||
round-to-nearest, ties to even, matching the numpy and torch casts.
|
||||
|
||||
- The fused `w_gate_up` is split at the row boundary (`chunk(2,-1)` → first half
|
||||
`gate`, second half `up`), matching the reference.
|
||||
- Quantized per-channel `*.weight_scale` tensors in the checkpoint are **dropped**
|
||||
(the reference has none).
|
||||
- F32→F16 uses IEEE round-to-nearest, ties-to-even (`f32bitsToF16`), matching
|
||||
numpy/torch casts; the reference tensors are byte-identical to ours, which
|
||||
empirically proves the rounding matched.
|
||||
- HF config/tokenizer are **parsed and validated** against the reference, but the
|
||||
reference remains authoritative for every emitted value.
|
||||
|
||||
---
|
||||
|
||||
## 3. Validation
|
||||
|
||||
`go run . verify` streams both files and compares on raw wire bytes:
|
||||
## Validation
|
||||
|
||||
```
|
||||
metadata: 39 checked, 0 mismatches
|
||||
tensors: 220 checked, 220 byte-identical, 0 mismatches
|
||||
verification PASSED: 39 metadata keys and 220 tensors byte-identical
|
||||
verification PASSED
|
||||
```
|
||||
|
||||
Whole-file check:
|
||||
|
||||
```
|
||||
$ cmp models/open-1b-sft-f16.gguf models/open-1b-sft-f16-go.gguf && echo IDENTICAL
|
||||
IDENTICAL
|
||||
$ sha256sum models/open-1b-sft-f16.gguf models/open-1b-sft-f16-go.gguf
|
||||
a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae open-1b-sft-f16.gguf
|
||||
a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
The per-tensor diff is empty (0 mismatches) — every tensor's payload is
|
||||
byte-identical, in the same order, with the same shape and ggml type.
|
||||
Both files hash to
|
||||
`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
|
||||
|
||||
### Metadata contract (all 39 keys, all byte-identical)
|
||||
Notable metadata, all byte-identical: `general.architecture=open1b`,
|
||||
`open1b.block_count=24`, `open1b.context_length=4096`,
|
||||
`open1b.attention.head_count=16`, `open1b.attention.head_count_kv=4`,
|
||||
`open1b.rope.freq_base=500000`, `open1b.attention.sliding_window=512`,
|
||||
`open1b.attention.sliding_window_pattern=5`, `tokenizer.ggml.model=gpt2`,
|
||||
`tokenizer.ggml.pre=gpt-2`, `tokenizer.ggml.eos_token_id=4`. The full list is in
|
||||
the manifest JSON.
|
||||
|
||||
| Key | Type | Value |
|
||||
|---|---|---|
|
||||
| `general.architecture` | STRING | `open1b` |
|
||||
| `general.type` | STRING | `model` |
|
||||
| `general.name` | STRING | `Open 1b Sft` |
|
||||
| `general.finetune` | STRING | `sft` |
|
||||
| `general.basename` | STRING | `open` |
|
||||
| `general.size_label` | STRING | `1B` |
|
||||
| `general.license` | STRING | `apache-2.0` |
|
||||
| `general.base_model.count` | UINT32 | 1 |
|
||||
| `general.base_model.0.name` | STRING | `Open 1b Midtrained 93B` |
|
||||
| `general.base_model.0.organization` | STRING | `Gensyn` |
|
||||
| `general.base_model.0.repo_url` | STRING | `https://huggingface.co/Gensyn/open-1b-midtrained-93B` |
|
||||
| `general.dataset.count` | UINT32 | 1 |
|
||||
| `general.dataset.0.name` | STRING | `Tulu 3 Sft Olmo 2 Mixture 0225` |
|
||||
| `general.dataset.0.version` | STRING | `0225` |
|
||||
| `general.dataset.0.organization` | STRING | `Allenai` |
|
||||
| `general.dataset.0.repo_url` | STRING | `https://huggingface.co/allenai/tulu-3-sft-olmo-2-mixture-0225` |
|
||||
| `general.tags` | ARRAY[STRING] | 5 (`gensyn`, `verifiable-training`, `reproducible-training`, `repops`, `text-generation`) |
|
||||
| `general.languages` | ARRAY[STRING] | 1 (`en`) |
|
||||
| `open1b.block_count` | UINT32 | 24 |
|
||||
| `open1b.context_length` | UINT32 | 4096 |
|
||||
| `open1b.embedding_length` | UINT32 | 2048 |
|
||||
| `open1b.feed_forward_length` | UINT32 | 5632 |
|
||||
| `open1b.attention.head_count` | UINT32 | 16 |
|
||||
| `open1b.attention.head_count_kv` | UINT32 | 4 |
|
||||
| `open1b.rope.freq_base` | FLOAT32 | 500000 |
|
||||
| `open1b.attention.layer_norm_rms_epsilon` | FLOAT32 | 1e-05 |
|
||||
| `open1b.attention.key_length` | UINT32 | 128 |
|
||||
| `open1b.attention.value_length` | UINT32 | 128 |
|
||||
| `general.file_type` | UINT32 | 1 (F16) |
|
||||
| `open1b.attention.sliding_window` | UINT32 | 512 |
|
||||
| `open1b.attention.sliding_window_pattern` | UINT32 | 5 |
|
||||
| `general.quantization_version` | UINT32 | 2 |
|
||||
| `tokenizer.ggml.model` | STRING | `gpt2` |
|
||||
| `tokenizer.ggml.pre` | STRING | `gpt-2` |
|
||||
| `tokenizer.ggml.tokens` | ARRAY[STRING] | 128256 (sha256 `30bab437…`) |
|
||||
| `tokenizer.ggml.token_type` | ARRAY[INT32] | 128256 (sha256 `5ca8c567…`) |
|
||||
| `tokenizer.ggml.merges` | ARRAY[STRING] | 127994 (sha256 `9e3a7dd4…`) |
|
||||
| `tokenizer.ggml.eos_token_id` | UINT32 | 4 |
|
||||
| `tokenizer.chat_template` | STRING | (501-byte Jinja template) |
|
||||
`go build`, `go vet`, and `go test -race` are clean.
|
||||
|
||||
Array `raw_sha256` values are in `models/open-1b-sft-f16.manifest.json`.
|
||||
## Serve
|
||||
|
||||
### HF cross-checks performed by `convert`
|
||||
|
||||
```
|
||||
config.json OK: arch=open1b layers=24 hidden=2048 heads=16/4 head_dim=128 ffn=5632 vocab=128256 swa=512/5 tie=false qk_norm=true(gain=false) emb_norm=true
|
||||
tokenizer_config.json OK: eos=<|eot|> chat_template=501 bytes
|
||||
tokenizer.json OK: type=BPE vocab=128256 +added=6 -> 128256 tokens, 127994 merges
|
||||
```
|
||||
|
||||
### Go build / vet / test
|
||||
|
||||
```
|
||||
$ go build ./... # OK
|
||||
$ go vet ./... # OK
|
||||
$ go test -race -v ./...
|
||||
--- PASS: TestF32BitsToF16 (0.00s)
|
||||
--- PASS: TestF32ToF16Bytes (0.00s)
|
||||
--- PASS: TestRawKVRoundTrip (0.00s)
|
||||
--- PASS: TestVerifyRealFiles (6.59s)
|
||||
PASS
|
||||
ok hf2gguf 7.602s
|
||||
```
|
||||
|
||||
`TestVerifyRealFiles` re-runs the full 3.2 GB byte comparison (skipped on
|
||||
`-short` or when the artifacts are absent).
|
||||
|
||||
---
|
||||
|
||||
## 4. Serve on one Arc (Vulkan2)
|
||||
|
||||
```
|
||||
setsid --fork /root/projects/inference/build-vulkan/bin/llama-server \
|
||||
-m /root/projects/gensyn/inference/models/open-1b-sft-f16-go.gguf \
|
||||
```bash
|
||||
setsid --fork ./build-vulkan/bin/llama-server \
|
||||
-m models/open-1b-sft-f16-go.gguf \
|
||||
--device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \
|
||||
--spec-type ngram-map-k,ngram-cache -t 16 --parallel 1 --alias open-1b \
|
||||
--spec-type ngram-map-k,ngram-cache \
|
||||
-t 16 --parallel 1 --alias open-1b \
|
||||
--host 127.0.0.1 --port 8085 >/tmp/open1b.log 2>&1 </dev/null &
|
||||
```
|
||||
|
||||
Startup log:
|
||||
|
||||
```
|
||||
load_model: loading model '.../open-1b-sft-f16-go.gguf'
|
||||
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
|
||||
llama_context: n_ctx_seq (32768) > n_ctx_train (4096) -- possible training context overflow
|
||||
load_model: the slot context (32768) exceeds the training context of the model (4096) - capping
|
||||
load_model: initializing, n_slots = 1, n_ctx_slot = 4096, kv_unified = 'false'
|
||||
model loaded
|
||||
listening on http://127.0.0.1:8085
|
||||
```
|
||||
|
||||
Device confirmation: PID `1378706` maps `/dev/dri/renderD129` with large
|
||||
`rw-s` regions; `renderD129 → /sys/devices/.../0000:07:00.0` = Arc #2 = Vulkan2.
|
||||
`/props` reports `"speculative.types":"none,ngram-map-k,ngram-cache"` and
|
||||
`"model_ftype":"F16"`.
|
||||
|
||||
### Performance (single Arc, 1 slot, `-fa off`, ngram spec)
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Decode | **~111.7–117.2 tok/s** (e.g. 256 tok @ 111.65; 96 tok @ 117.20) |
|
||||
| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) |
|
||||
| VRAM on Vulkan2 | free 10971 → 7258 MiB ⇒ **≈ 3.6 GiB** used; fully released after `kill` |
|
||||
| --- | --- |
|
||||
| Decode | 111.7 to 117.2 tok/s |
|
||||
| Prefill | about 892 tok/s |
|
||||
| VRAM | about 3.6 GiB, released after stop |
|
||||
|
||||
### Sample outputs
|
||||
Sample outputs, greedy:
|
||||
|
||||
```
|
||||
$ curl ... /completion {"prompt":"The three primary colors are", greedy}
|
||||
red, blue, and yellow.
|
||||
prompt: The three primary colors are
|
||||
out: red, blue, and yellow.
|
||||
|
||||
$ curl ... /completion {"prompt":"Q: What is the capital of France?\nA:", greedy, 96}
|
||||
Paris
|
||||
|
||||
$ curl ... /completion (seeded chat turn, see caveat) 113.9 tok/s
|
||||
" Venus and Mars. Venus is the hottest planet in our Solar System, with
|
||||
temperatures reaching 465°C (1,470°F). Mars, on the other hand, is a
|
||||
terrestrial planet, meaning it has a solid surface and liquid water at its poles."
|
||||
prompt: Q: What is the capital of France? A:
|
||||
out: Paris
|
||||
```
|
||||
|
||||
### Stopping
|
||||
Stopped with `kill` after reading the pid from `ss -ltnp`.
|
||||
|
||||
```
|
||||
$ ss -ltnp | grep :8085 # -> pid=1378706
|
||||
$ kill 1378706 # never pkill -f
|
||||
```
|
||||
## Caveats
|
||||
|
||||
Both the test server (:8085) and a diagnostic server (:8086) were stopped;
|
||||
`--list-devices` returned to 10971 MiB free on Vulkan2/3.
|
||||
1. Chat template. With the vendor tail `...assistant<|end_header|>` followed by a
|
||||
blank line, this checkpoint emits an end-of-turn token first and returns an
|
||||
empty reply. Reproduced with and without the speculative decoder, so it is not
|
||||
the decoder. A single newline, an extra newline, or a seed token produces a
|
||||
coherent answer. Since our GGUF is byte-identical to the reference, this is a
|
||||
property of the released checkpoint.
|
||||
2. Context cap. `open1b.context_length` is 4096, so the requested 32768 is capped.
|
||||
3. The load warning `special_eos_id is not in special_eog_ids` also appears for
|
||||
the reference file, so it is not introduced by the conversion.
|
||||
4. Long greedy generation without a repetition penalty loops on this small model.
|
||||
5. Process note. During one early check the response was piped through `python3`
|
||||
to pretty-print JSON. That was a mistake and a breach of the no-Python rule.
|
||||
The tool itself is entirely Go, and nothing in the pipeline depends on Python.
|
||||
No Python files or virtualenv were created for the tool.
|
||||
|
||||
---
|
||||
|
||||
## 5. Caveats
|
||||
|
||||
1. **Chat template newline quirk (model-side, not conversion-side).** With the
|
||||
exact vendor template tail `<|end_header|>\n\n` for the assistant turn, this
|
||||
checkpoint emits `<|eot|>` as the first sampled token → empty reply / instant
|
||||
EOS (reproduced on both :8085 with ngram spec and a :8086 run with
|
||||
`--spec-type none`, so it is **not** the speculative decoder). A single `\n`,
|
||||
`\n\n\n`, or a seed token makes it answer coherently:
|
||||
|
||||
| tail after `assistant<|end_header|>` | result |
|
||||
|---|---|
|
||||
| `\n\n` (vendor template) | immediate EOS |
|
||||
| `\n` | `\n1. **Earth**: …` (coherent) |
|
||||
| `\n\n\n` | `\n1. **Earth**: …` (coherent) |
|
||||
| ` ` | `\nSure, here are two names…` (coherent) |
|
||||
|
||||
Because our GGUF is byte-identical to the reference GGUF, this is a
|
||||
property of the released checkpoint/GGUF (likely SFT formatting), not of our
|
||||
conversion. Workarounds for consumers: use `ignore_eos` + trim, seed the
|
||||
assistant turn, or fix the template's trailing newline.
|
||||
2. **Context capped to 4096.** `open1b.context_length = 4096`, so the requested
|
||||
`-c 32768` is capped to `n_ctx_slot = 4096`. KV is f16 (`-fa off` mandatory on
|
||||
Arc per `brainstation-gpu`).
|
||||
3. **`special_eos_id is not in special_eog_ids`** warning on load — present for
|
||||
the reference file too (byte-identical), so not introduced by us.
|
||||
4. **1B greedy degeneration.** Long greedy generation without repetition penalty
|
||||
loops on this small model; use sampling / `repeat_penalty` for long outputs.
|
||||
5. **Process note (disclosure).** During an early diagnostic I piped one
|
||||
`/completion` response through `python3 -c` to pretty-print JSON — a breach of
|
||||
the "no Python" rule (the tool itself is 100% Go, and nothing in the pipeline
|
||||
depends on Python). All subsequent parsing was done with Go/raw JSON. No `.py`
|
||||
file or venv was created; the only Python files in the tree are the vendor's
|
||||
read-only `models/open-1b-sft/*.py`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Artifacts
|
||||
## Files
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `hf2gguf/` | Go module (`dump` / `convert` / `verify`) + tests |
|
||||
| `hf2gguf/README.md` | tool usage |
|
||||
| `models/open-1b-sft-f16-go.gguf` | our output (byte-identical to reference) |
|
||||
| `models/open-1b-sft-f16.manifest.json` | dumped contract (39 KV + 220 tensors) |
|
||||
| `models/open-1b-sft-f16.gguf` | reference (untouched) |
|
||||
| --- | --- |
|
||||
| `hf2gguf/` | Go module with tests |
|
||||
| `hf2gguf/README.md` | usage |
|
||||
| `models/open-1b-sft-f16-go.gguf` | our output |
|
||||
| `models/open-1b-sft-f16.manifest.json` | dumped contract |
|
||||
| `models/open-1b-sft-f16.gguf` | reference |
|
||||
| `/tmp/open1b.log` | server log |
|
||||
|
||||
No changes were committed; everything lives under gitignored `inference/`.
|
||||
|
||||
+32
-36
@@ -1,39 +1,36 @@
|
||||
# hf2gguf
|
||||
|
||||
Pure-Go (Go 1.27) tool to convert a HuggingFace `open-1b` safetensors checkpoint
|
||||
into GGUF **byte-identically** to a supplied reference GGUF, plus a byte-level
|
||||
verifier. Built for epic `vado/1b#1` (Gensyn open-1b local bring-up).
|
||||
A Go 1.27 tool that converts a Hugging Face `open-1b` safetensors checkpoint to
|
||||
GGUF, byte-identical to a supplied reference file, with a byte-level verifier.
|
||||
|
||||
Library: [`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf)
|
||||
(reader + writer). No Python anywhere in the tool.
|
||||
Library: [go-gguf](https://github.com/cymertek/go-gguf), reader and writer. No
|
||||
Python anywhere.
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test -race ./... # full test includes the 3.2 GB byte comparison
|
||||
go test -race ./... # the full test includes the 3.2 GB byte comparison
|
||||
```
|
||||
|
||||
## Subcommands
|
||||
## Commands
|
||||
|
||||
### `dump` — the reproduction contract
|
||||
### dump
|
||||
|
||||
Reads the reference GGUF and emits a JSON manifest of **all** metadata KV pairs
|
||||
(key, type, value / array elem-type + count + `raw_sha256`) and every tensor
|
||||
(name, shape, ggml type, nbytes).
|
||||
Reads the reference GGUF and writes a JSON manifest of all metadata pairs (key,
|
||||
type, value, or array element type, count, and `raw_sha256`) and every tensor
|
||||
(name, shape, ggml type, byte count).
|
||||
|
||||
```bash
|
||||
go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json
|
||||
```
|
||||
|
||||
`go-gguf` exposes scalar/string metadata and tensors; its `MetadataEntry.Value()`
|
||||
returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV section
|
||||
directly to preserve tokenizer arrays (`tokens`, `merges`, `token_type`) as raw
|
||||
bytes. Writing still goes through `go-gguf`'s writer — nothing hand-rolls GGUF
|
||||
serialization.
|
||||
`go-gguf` returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV
|
||||
section directly to keep the tokenizer arrays as raw bytes. Writing still goes
|
||||
through the `go-gguf` writer.
|
||||
|
||||
### `convert` — emit our GGUF
|
||||
### convert
|
||||
|
||||
```bash
|
||||
go run . convert \
|
||||
@@ -45,38 +42,37 @@ go run . convert \
|
||||
-o models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
- Metadata is copied verbatim from the reference KV section.
|
||||
- Tensors are mapped HF FQN → GGUF name (see the module docs in `convert.go`),
|
||||
with `blocks.{i}.ffn.w_gate_up.weight` split into `ffn_gate`/`ffn_up`, and
|
||||
`*.weight_scale` dropped (the reference has none).
|
||||
- Payloads stream from safetensors and are converted F32→F16 with IEEE
|
||||
round-to-nearest-even on the fly (O(1) memory per tensor).
|
||||
- `-config` / `-tokenizer-config` / `-tokenizer` are optional; when given they
|
||||
are parsed and **validated** against the reference (mismatch is fatal).
|
||||
- Metadata is copied verbatim from the reference.
|
||||
- Tensors are mapped by name, with `blocks.i.ffn.w_gate_up.weight` split into
|
||||
`ffn_gate` and `ffn_up`, and `*.weight_scale` dropped.
|
||||
- Payloads stream from safetensors and convert F32 to F16 on the fly, using
|
||||
round-to-nearest, ties to even.
|
||||
- The config and tokenizer flags are optional. When given, they are parsed and
|
||||
validated against the reference; a mismatch is fatal.
|
||||
|
||||
### `verify` — byte-level equality
|
||||
### verify
|
||||
|
||||
```bash
|
||||
go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
Compares metadata on raw wire bytes and tensors by shape, type and full payload
|
||||
equality (streaming, first-diff offset reported on failure).
|
||||
Compares metadata on raw wire bytes and tensors by shape, type, and full payload.
|
||||
Reports the first differing offset on failure.
|
||||
|
||||
```
|
||||
metadata: 39 checked, 0 mismatches
|
||||
tensors: 220 checked, 220 byte-identical, 0 mismatches
|
||||
verification PASSED: 39 metadata keys and 220 tensors byte-identical
|
||||
verification PASSED
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `main.go` | dispatch, manifest types, `dump` |
|
||||
| `rawkv.go` | read-only GGUF KV walker (array support) |
|
||||
| `safetensors.go` | safetensors header/byte access |
|
||||
| `hfconfig.go` | HF config/tokenizer parsing + validation |
|
||||
| `convert.go` | metadata reproduction, tensor mapping, F32→F16 |
|
||||
| `verify.go` | `verify` command + comparison engine |
|
||||
| --- | --- |
|
||||
| `main.go` | dispatch, manifest types, dump |
|
||||
| `rawkv.go` | read-only GGUF KV walker with array support |
|
||||
| `safetensors.go` | safetensors header and byte access |
|
||||
| `hfconfig.go` | HF config and tokenizer parsing, validation |
|
||||
| `convert.go` | metadata reproduction, tensor mapping, F32 to F16 |
|
||||
| `verify.go` | verify command and comparison engine |
|
||||
| `convert_test.go` | F16 rounding, KV round-trip, full-file verification |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package main
|
||||
|
||||
// convert.go — emit models/open-1b-sft-f16-go.gguf from the HF safetensors,
|
||||
// convert.go converts the HF safetensors into models/open-1b-sft-f16-go.gguf,
|
||||
// reproducing the reference GGUF's metadata (verbatim) and tensor set/order.
|
||||
//
|
||||
// Tensor mapping (HF FQN -> GGUF name), shapes are the reference's (ggml ne0 is
|
||||
@@ -117,7 +117,7 @@ func cmdConvert(args []string) error {
|
||||
defer st.Close()
|
||||
|
||||
// Validate HF config against the reference contract (no values are taken from
|
||||
// it — the reference is authoritative — but mismatches are fatal).
|
||||
// it (the reference is authoritative), but mismatches are fatal).
|
||||
if *hfConfig != "" {
|
||||
if err := validateConfig(*hfConfig, kvs); err != nil {
|
||||
return fmt.Errorf("config validation: %w", err)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package main
|
||||
|
||||
// hfconfig.go — parse and validate the HF config/tokenizer against the
|
||||
// hfconfig.go parses and validates the HF config/tokenizer against the
|
||||
// reference GGUF metadata. The reference remains authoritative for values; this
|
||||
// only proves the reference is consistent with the published HF files and that
|
||||
// we parsed them (per the task), it does not source any GGUF value from them.
|
||||
|
||||
@@ -35,7 +35,7 @@ func main() {
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Manifest types — the reproduction contract.
|
||||
// Manifest types for the reproduction contract.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type manifest struct {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package main
|
||||
|
||||
// safetensors.go — minimal read-only HuggingFace safetensors index.
|
||||
// safetensors.go reads a minimal, read-only HuggingFace safetensors index.
|
||||
//
|
||||
// Format: [u64 LE header_len][header_len bytes of JSON][tensor data ...].
|
||||
// The JSON header maps tensor name -> {dtype, shape, data_offsets:[start,end]}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package main
|
||||
|
||||
// verify.go — prove the emitted GGUF equals the reference: identical metadata
|
||||
// verify.go proves the emitted GGUF equals the reference: identical metadata
|
||||
// (keys, types, values; verified on raw wire bytes) and identical tensor set,
|
||||
// shapes, ggml types and per-tensor payload bytes.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user