docs: rewrite in plain style (no em dashes, no emoji), tighten reports

This commit is contained in:
vad
2026-09-29 07:39:12 +00:00
parent 4fc62ef8d1
commit 6d1f79359a
11 changed files with 392 additions and 654 deletions
+37 -41
View File
@@ -1,82 +1,78 @@
# hf2gguf
Pure-Go (Go 1.27) tool to convert a HuggingFace `open-1b` safetensors checkpoint
into GGUF **byte-identically** to a supplied reference GGUF, plus a byte-level
verifier. Built for epic `vado/1b#1` (Gensyn open-1b local bring-up).
A Go 1.27 tool that converts a Hugging Face `open-1b` safetensors checkpoint to
GGUF, byte-identical to a supplied reference file, with a byte-level verifier.
Library: [`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf)
(reader + writer). No Python anywhere in the tool.
Library: [go-gguf](https://github.com/cymertek/go-gguf), reader and writer. No
Python anywhere.
## Build
```bash
go build ./...
go vet ./...
go test -race ./... # full test includes the 3.2 GB byte comparison
go test -race ./... # the full test includes the 3.2 GB byte comparison
```
## Subcommands
## Commands
### `dump` — the reproduction contract
### dump
Reads the reference GGUF and emits a JSON manifest of **all** metadata KV pairs
(key, type, value / array elem-type + count + `raw_sha256`) and every tensor
(name, shape, ggml type, nbytes).
Reads the reference GGUF and writes a JSON manifest of all metadata pairs (key,
type, value, or array element type, count, and `raw_sha256`) and every tensor
(name, shape, ggml type, byte count).
```bash
go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json
```
`go-gguf` exposes scalar/string metadata and tensors; its `MetadataEntry.Value()`
returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV section
directly to preserve tokenizer arrays (`tokens`, `merges`, `token_type`) as raw
bytes. Writing still goes through `go-gguf`'s writer — nothing hand-rolls GGUF
serialization.
`go-gguf` returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV
section directly to keep the tokenizer arrays as raw bytes. Writing still goes
through the `go-gguf` writer.
### `convert` — emit our GGUF
### convert
```bash
go run . convert \
-ref models/open-1b-sft-f16.gguf \
-safetensors models/open-1b-sft/model.safetensors \
-config models/open-1b-sft/config.json \
-ref models/open-1b-sft-f16.gguf \
-safetensors models/open-1b-sft/model.safetensors \
-config models/open-1b-sft/config.json \
-tokenizer-config models/open-1b-sft/tokenizer_config.json \
-tokenizer models/open-1b-sft/tokenizer.json \
-o models/open-1b-sft-f16-go.gguf
-tokenizer models/open-1b-sft/tokenizer.json \
-o models/open-1b-sft-f16-go.gguf
```
- Metadata is copied verbatim from the reference KV section.
- Tensors are mapped HF FQN → GGUF name (see the module docs in `convert.go`),
with `blocks.{i}.ffn.w_gate_up.weight` split into `ffn_gate`/`ffn_up`, and
`*.weight_scale` dropped (the reference has none).
- Payloads stream from safetensors and are converted F32→F16 with IEEE
round-to-nearest-even on the fly (O(1) memory per tensor).
- `-config` / `-tokenizer-config` / `-tokenizer` are optional; when given they
are parsed and **validated** against the reference (mismatch is fatal).
- Metadata is copied verbatim from the reference.
- Tensors are mapped by name, with `blocks.i.ffn.w_gate_up.weight` split into
`ffn_gate` and `ffn_up`, and `*.weight_scale` dropped.
- Payloads stream from safetensors and convert F32 to F16 on the fly, using
round-to-nearest, ties to even.
- The config and tokenizer flags are optional. When given, they are parsed and
validated against the reference; a mismatch is fatal.
### `verify` — byte-level equality
### verify
```bash
go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
```
Compares metadata on raw wire bytes and tensors by shape, type and full payload
equality (streaming, first-diff offset reported on failure).
Compares metadata on raw wire bytes and tensors by shape, type, and full payload.
Reports the first differing offset on failure.
```
metadata: 39 checked, 0 mismatches
tensors: 220 checked, 220 byte-identical, 0 mismatches
verification PASSED: 39 metadata keys and 220 tensors byte-identical
verification PASSED
```
## Layout
| File | Purpose |
|---|---|
| `main.go` | dispatch, manifest types, `dump` |
| `rawkv.go` | read-only GGUF KV walker (array support) |
| `safetensors.go` | safetensors header/byte access |
| `hfconfig.go` | HF config/tokenizer parsing + validation |
| `convert.go` | metadata reproduction, tensor mapping, F32→F16 |
| `verify.go` | `verify` command + comparison engine |
| --- | --- |
| `main.go` | dispatch, manifest types, dump |
| `rawkv.go` | read-only GGUF KV walker with array support |
| `safetensors.go` | safetensors header and byte access |
| `hfconfig.go` | HF config and tokenizer parsing, validation |
| `convert.go` | metadata reproduction, tensor mapping, F32 to F16 |
| `verify.go` | verify command and comparison engine |
| `convert_test.go` | F16 rounding, KV round-trip, full-file verification |
+2 -2
View File
@@ -1,6 +1,6 @@
package main
// convert.go — emit models/open-1b-sft-f16-go.gguf from the HF safetensors,
// convert.go converts the HF safetensors into models/open-1b-sft-f16-go.gguf,
// reproducing the reference GGUF's metadata (verbatim) and tensor set/order.
//
// Tensor mapping (HF FQN -> GGUF name), shapes are the reference's (ggml ne0 is
@@ -117,7 +117,7 @@ func cmdConvert(args []string) error {
defer st.Close()
// Validate HF config against the reference contract (no values are taken from
// it — the reference is authoritative — but mismatches are fatal).
// it (the reference is authoritative), but mismatches are fatal).
if *hfConfig != "" {
if err := validateConfig(*hfConfig, kvs); err != nil {
return fmt.Errorf("config validation: %w", err)
+1 -1
View File
@@ -1,6 +1,6 @@
package main
// hfconfig.go — parse and validate the HF config/tokenizer against the
// hfconfig.go parses and validates the HF config/tokenizer against the
// reference GGUF metadata. The reference remains authoritative for values; this
// only proves the reference is consistent with the published HF files and that
// we parsed them (per the task), it does not source any GGUF value from them.
+1 -1
View File
@@ -35,7 +35,7 @@ func main() {
}
// ---------------------------------------------------------------------------
// Manifest types — the reproduction contract.
// Manifest types for the reproduction contract.
// ---------------------------------------------------------------------------
type manifest struct {
+1 -1
View File
@@ -1,6 +1,6 @@
package main
// safetensors.go — minimal read-only HuggingFace safetensors index.
// safetensors.go reads a minimal, read-only HuggingFace safetensors index.
//
// Format: [u64 LE header_len][header_len bytes of JSON][tensor data ...].
// The JSON header maps tensor name -> {dtype, shape, data_offsets:[start,end]}
+1 -1
View File
@@ -1,6 +1,6 @@
package main
// verify.go — prove the emitted GGUF equals the reference: identical metadata
// verify.go proves the emitted GGUF equals the reference: identical metadata
// (keys, types, values; verified on raw wire bytes) and identical tensor set,
// shapes, ggml types and per-tensor payload bytes.