83 lines
3.0 KiB
Markdown
83 lines
3.0 KiB
Markdown
# hf2gguf
|
|
|
|
Pure-Go (Go 1.27) tool to convert a HuggingFace `open-1b` safetensors checkpoint
|
|
into GGUF **byte-identically** to a supplied reference GGUF, plus a byte-level
|
|
verifier. Built for epic `vado/1b#1` (Gensyn open-1b local bring-up).
|
|
|
|
Library: [`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf)
|
|
(reader + writer). No Python anywhere in the tool.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
go build ./...
|
|
go vet ./...
|
|
go test -race ./... # full test includes the 3.2 GB byte comparison
|
|
```
|
|
|
|
## Subcommands
|
|
|
|
### `dump` — the reproduction contract
|
|
|
|
Reads the reference GGUF and emits a JSON manifest of **all** metadata KV pairs
|
|
(key, type, value / array elem-type + count + `raw_sha256`) and every tensor
|
|
(name, shape, ggml type, nbytes).
|
|
|
|
```bash
|
|
go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json
|
|
```
|
|
|
|
`go-gguf` exposes scalar/string metadata and tensors; its `MetadataEntry.Value()`
|
|
returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV section
|
|
directly to preserve tokenizer arrays (`tokens`, `merges`, `token_type`) as raw
|
|
bytes. Writing still goes through `go-gguf`'s writer — nothing hand-rolls GGUF
|
|
serialization.
|
|
|
|
### `convert` — emit our GGUF
|
|
|
|
```bash
|
|
go run . convert \
|
|
-ref models/open-1b-sft-f16.gguf \
|
|
-safetensors models/open-1b-sft/model.safetensors \
|
|
-config models/open-1b-sft/config.json \
|
|
-tokenizer-config models/open-1b-sft/tokenizer_config.json \
|
|
-tokenizer models/open-1b-sft/tokenizer.json \
|
|
-o models/open-1b-sft-f16-go.gguf
|
|
```
|
|
|
|
- Metadata is copied verbatim from the reference KV section.
|
|
- Tensors are mapped HF FQN → GGUF name (see the module docs in `convert.go`),
|
|
with `blocks.{i}.ffn.w_gate_up.weight` split into `ffn_gate`/`ffn_up`, and
|
|
`*.weight_scale` dropped (the reference has none).
|
|
- Payloads stream from safetensors and are converted F32→F16 with IEEE
|
|
round-to-nearest-even on the fly (O(1) memory per tensor).
|
|
- `-config` / `-tokenizer-config` / `-tokenizer` are optional; when given they
|
|
are parsed and **validated** against the reference (mismatch is fatal).
|
|
|
|
### `verify` — byte-level equality
|
|
|
|
```bash
|
|
go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
|
|
```
|
|
|
|
Compares metadata on raw wire bytes and tensors by shape, type and full payload
|
|
equality (streaming, first-diff offset reported on failure).
|
|
|
|
```
|
|
metadata: 39 checked, 0 mismatches
|
|
tensors: 220 checked, 220 byte-identical, 0 mismatches
|
|
verification PASSED: 39 metadata keys and 220 tensors byte-identical
|
|
```
|
|
|
|
## Layout
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `main.go` | dispatch, manifest types, `dump` |
|
|
| `rawkv.go` | read-only GGUF KV walker (array support) |
|
|
| `safetensors.go` | safetensors header/byte access |
|
|
| `hfconfig.go` | HF config/tokenizer parsing + validation |
|
|
| `convert.go` | metadata reproduction, tensor mapping, F32→F16 |
|
|
| `verify.go` | `verify` command + comparison engine |
|
|
| `convert_test.go` | F16 rounding, KV round-trip, full-file verification |
|