open-1b: an independent check, a local run, and a Go GGUF converter

We checked Gensyn's open-1b and its claim of auditable training, then ran the model on our own hardware. The audit uses Gensyn's official harness. The conversion and tooling are Go.

Links: announcement, harness, audit app.

hf2gguf

tools/hf2gguf is the tool at the center of this repo. It converts a Hugging Face open-1b checkpoint to GGUF and checks the result byte for byte, with no Python, no torch, and no pip. For open-1b it reproduces llama.cpp's converter output exactly, down to the same SHA-256.

Why build it this way:

	hf2gguf	llama.cpp `convert_hf_to_gguf.py`
Runtime	one Go binary	Python 3 with torch, numpy, transformers, gguf-py
Setup	`go build`	pip or uv, wheels, pinned versions
Startup	immediate	imports torch and its chain
Memory	streams one tensor at a time	loads tensors through torch
Output check	built-in byte-level `verify`	none
Output	byte-identical to a reference file	the reference
Scope	open-1b only	many architectures
Quantization	none, F16 only	many quant types

It is deliberately narrow. It maps one Llama-derived architecture and copies the GGUF metadata from a reference file, so it is not a general converter. Its value is that it is small, readable, dependency-free, and able to prove the file it writes is identical to the reference.

cd tools/hf2gguf
go build -o hf2gguf .
./hf2gguf convert -ref models/open-1b-sft-f16.gguf \
    -safetensors models/open-1b-sft/model.safetensors \
    -config models/open-1b-sft/config.json \
    -tokenizer-config models/open-1b-sft/tokenizer_config.json \
    -tokenizer models/open-1b-sft/tokenizer.json \
    -o models/open-1b-sft-f16-go.gguf
./hf2gguf verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf

verify reports 39 of 39 metadata keys and 220 of 220 tensors identical. The two files share SHA-256 a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae. See tools/hf2gguf/README.md.

Results

Check	Result
Init-unit audit, x86-64 CPU, no NVIDIA	Match. The published state hash `16554a11...` is reproduced exactly.
Training-interval audit	Not run. No interval unit is published, and a replay needs about 24 GB RAM and 60 GB disk.
GGUF conversion (Go)	Byte-identical to the reference conversion, same SHA-256.
Local inference, one Intel Arc B580	114 tok/s decode, 892 tok/s prefill, about 3.6 GiB VRAM.

Reports: audit, local run, benchmarks, inference stack.

How it works

Audit. Replay one published commitment and compare hashes.

flowchart LR
  KIT["Gensyn audit kit"] --> SHA{"sha256<br/>against kit.json"}
  SHA --> WHL["repop and pretrain wheels"]
  WHL --> REPLAY["pretrain-audit-replay<br/>--from-init --device cpu"]
  REPLAY --> H1["regenerated init hash"]
  KIT --> H2["published commitment"]
  H1 --> CMP{"equal"}
  H2 --> CMP
  CMP -->|yes| OK["match"]
  CMP -->|no| BAD["mismatch"]

Inference. Convert in Go, serve on one Arc. The engine is a patched llama.cpp.

flowchart LR
  HF["HF checkpoint<br/>model.safetensors, fp32"] --> GO["tools/hf2gguf<br/>Go"]
  GO --> GGUF["GGUF, f16<br/>identical to reference"]
  GGUF --> LC["llama.cpp Vulkan<br/>upstream plus open1b patch"]
  LC --> GPU["Intel Arc B580"]
  GPU --> API["llama-server HTTP"]

Repository layout

tools/hf2gguf/          # the Go converter: dump, convert, verify
docs/
  audit-open1b-init.md
  open1b-local-inference.md
  benchmarks.md
  inference-stack.md
  llama-cpp-open1b.patch

The audit, and its limits

The init unit regenerates the published initial state hash exactly on a plain CPU. The installed wheels match the kit manifest, and the repop build commit matches the manifest.

It does not cover a training step. An init-only unit says nothing about the forward pass, the backward pass, the optimizer, or the data stream, so the per-step claim stays untested here. Three more reasons to be careful:

repop ships as compiled wheels only, under a custom licence. The kernels that make training reproducible cannot be read or checked.
The v3 state hash omits the RNG, the data-stream cursor, spike state, and the meta.json descriptor keys.
No interval unit is published, so "replay any step" is not yet something an outsider can do without permission.

A match shows that the pinned artifacts reproduce the published commitment on our machine. It does not show how the original run was carried out.

Running the model

Upstream llama.cpp has no open1b architecture, so the model needs a patched build. Details in the inference stack.

llama-server -m models/open-1b-sft-f16-go.gguf \
  --device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \
  --spec-type ngram-map-k,ngram-cache \
  -t 16 --parallel 1 --alias open-1b --host 127.0.0.1 --port 8085

-fa off is required on these Arc/Vulkan builds, which keeps the KV cache in f16. The requested 32k context is capped to the model's 4096.

Caveats

Chat template. With the vendor tail ...assistant<|end_header|> followed by a blank line, the released SFT checkpoint emits an end-of-turn token first and returns an empty reply. Our GGUF is identical to the reference, so this is a property of the released weights. Seed the assistant turn, set ignore_eos, or adjust the trailing newline.
Model quality. open-1b scores 25.4 on the OLMo2 suite, against 31.9 for OLMo2-1B. It is a test of verifiability, not a strong model.
Scope. We run and inspect released weights and replay one published commitment. We do not train or fine-tune anything.

License

MIT, see LICENSE. The harness is Apache-2.0. repop, the checkpoints, and the corpus are distributed by Gensyn under their own terms.

Built on Gensyn's release. Not affiliated with Gensyn.

6.1 KiB Raw Permalink Blame History