open-1b-verify/tools/hf2gguf/README.md

hf2gguf

Pure-Go (Go 1.27) tool to convert a HuggingFace open-1b safetensors checkpoint into GGUF byte-identically to a supplied reference GGUF, plus a byte-level verifier. Built for epic vado/1b#1 (Gensyn open-1b local bring-up).

Library: github.com/cymertek/go-gguf (reader + writer). No Python anywhere in the tool.

Build

go build ./...
go vet ./...
go test -race ./...        # full test includes the 3.2 GB byte comparison

Subcommands

dump — the reproduction contract

Reads the reference GGUF and emits a JSON manifest of all metadata KV pairs (key, type, value / array elem-type + count + raw_sha256) and every tensor (name, shape, ggml type, nbytes).

go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json

go-gguf exposes scalar/string metadata and tensors; its MetadataEntry.Value() returns an empty value for BTypeArray, so rawkv.go walks the KV section directly to preserve tokenizer arrays (tokens, merges, token_type) as raw bytes. Writing still goes through go-gguf's writer — nothing hand-rolls GGUF serialization.

convert — emit our GGUF

go run . convert \
  -ref          models/open-1b-sft-f16.gguf \
  -safetensors  models/open-1b-sft/model.safetensors \
  -config       models/open-1b-sft/config.json \
  -tokenizer-config models/open-1b-sft/tokenizer_config.json \
  -tokenizer    models/open-1b-sft/tokenizer.json \
  -o            models/open-1b-sft-f16-go.gguf

• Metadata is copied verbatim from the reference KV section.
• Tensors are mapped HF FQN → GGUF name (see the module docs in convert.go), with blocks.{i}.ffn.w_gate_up.weight split into ffn_gate/ffn_up, and *.weight_scale dropped (the reference has none).
• Payloads stream from safetensors and are converted F32→F16 with IEEE round-to-nearest-even on the fly (O(1) memory per tensor).
• -config / -tokenizer-config / -tokenizer are optional; when given they are parsed and validated against the reference (mismatch is fatal).

verify — byte-level equality

go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf

Compares metadata on raw wire bytes and tensors by shape, type and full payload equality (streaming, first-diff offset reported on failure).

metadata: 39 checked, 0 mismatches
tensors:  220 checked, 220 byte-identical, 0 mismatches
verification PASSED: 39 metadata keys and 220 tensors byte-identical

Layout

File	Purpose
main.go	dispatch, manifest types, dump
rawkv.go	read-only GGUF KV walker (array support)
safetensors.go	safetensors header/byte access
hfconfig.go	HF config/tokenizer parsing + validation
convert.go	metadata reproduction, tensor mapping, F32→F16
verify.go	verify command + comparison engine
convert_test.go	F16 rounding, KV round-trip, full-file verification