Merge pull request 'feat(inference): Go hf2gguf converter + open-1b local run (#4)' (#6) from docs/4-open1b-go-gguf into main
This commit is contained in:
@@ -8,3 +8,6 @@ gitea/
|
||||
*.whl
|
||||
*.gguf
|
||||
__pycache__/
|
||||
|
||||
# compiled tool binaries
|
||||
/tools/hf2gguf/hf2gguf
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
|--------|--------|
|
||||
| Аудит init-unit open-1b (CPU) | ✅ **MATCH** — `state_hash 16554a11…`; см. [`reports/audit-open1b-init.md`](reports/audit-open1b-init.md) |
|
||||
| Аудит тренировочного интервала | ⛔ заблокирован (нет опубликованного interval-unit + RAM/диск) |
|
||||
| Локальный inference на Arc B580 | 🔄 в работе |
|
||||
| Локальный inference на Arc B580 | ✅ **работает** — Go-конвертер, GGUF байт-идентичен; **~114 tok/s** decode, ~892 prefill, ~3.6 GiB VRAM; см. [`reports/open1b-local-inference.md`](reports/open1b-local-inference.md) |
|
||||
|
||||
## Ссылки
|
||||
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
# Gensyn open-1b (SFT) — local bring-up & proof of run
|
||||
|
||||
Epic: `vado/1b#1` / task #4 (SE). Workdir: `/root/projects/gensyn/inference/`.
|
||||
Engine: existing shared llama.cpp **Vulkan** build (`/root/projects/inference/build-vulkan/bin/llama-server`).
|
||||
Language: **Go 1.27 only** (see "Process note" re one accidental `python3` call).
|
||||
|
||||
## TL;DR
|
||||
|
||||
- Built a pure-Go tool `hf2gguf` (module `hf2gguf`) that **dumps** the reference GGUF
|
||||
(go-gguf + a minimal raw-KV walker for arrays), **converts** `model.safetensors`
|
||||
→ `models/open-1b-sft-f16-go.gguf`, and **verifies** the two files.
|
||||
- **Validation: PASS — 39/39 metadata KV byte-identical, 220/220 tensors byte-identical.**
|
||||
Whole files are byte-for-byte identical (`cmp` + SHA-256 match).
|
||||
- **Served** on one Arc (Vulkan2 / `07:00.0`) with the existing Vulkan build; coherent
|
||||
output, **~114 tok/s decode**, **~892 tok/s prefill**, ~**3.6 GiB VRAM**.
|
||||
- No commit made (all under gitignored `inference/`).
|
||||
|
||||
Reference SHA-256 (and our output — identical):
|
||||
`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`
|
||||
|
||||
---
|
||||
|
||||
## 1. Inputs & topology
|
||||
|
||||
| Item | Path | Notes |
|
||||
|------|------|-------|
|
||||
| HF checkpoint (fp32) | `models/open-1b-sft/model.safetensors` | 6,432,081,672 B; dtype **F32**; header 34560 B |
|
||||
| Reference GGUF (f16) | `models/open-1b-sft-f16.gguf` | 3,220,153,856 B; vendor converter; **read-only oracle** |
|
||||
| HF config | `models/open-1b-sft/config.json` | `model_type=open1b`, 24 layers, hidden 2048, heads 16/4, head_dim 128, ffn 5632, vocab 128256, rope_theta 5e5, rms_eps 1e-5, swa 512 every 5, qk_norm gain-free, embedding_norm, untied |
|
||||
| Vendor Python | `models/open-1b-sft/modeling_open1b.py` | read only, never executed |
|
||||
| Engine | `/root/projects/inference/build-vulkan/bin/llama-server` | symlink → `/mnt/2TB/projects/build-vulkan`; shared prod tree, untouched |
|
||||
|
||||
GPU devices as reported by the engine (`--list-devices`):
|
||||
|
||||
```
|
||||
Vulkan0: AMD Ryzen 7 7700X (RADV RAPHAEL_MENDOCINO) (15835 MiB)
|
||||
Vulkan1: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB)
|
||||
Vulkan2: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB)
|
||||
Vulkan3: Intel(R) Arc(tm) B580 Graphics (BMG G21) (12216 MiB)
|
||||
```
|
||||
|
||||
`Vulkan2` was confirmed to map to PCI `0000:07:00.0` (Arc #2) via
|
||||
`/sys/class/drm/renderD129/device`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tool: `hf2gguf` (Go 1.27, `github.com/cymertek/go-gguf`)
|
||||
|
||||
```
|
||||
inference/hf2gguf/
|
||||
go.mod, go.sum
|
||||
main.go # subcommand dispatch + JSON manifest types
|
||||
rawkv.go # minimal read-only GGUF KV walker (fills go-gguf's array gap)
|
||||
safetensors.go # safetensors header + tensor byte access
|
||||
hfconfig.go # config.json / tokenizer_config.json / tokenizer.json validation
|
||||
convert.go # metadata reproduction + HF→GGUF tensor mapping + F32→F16
|
||||
verify.go # byte-level metadata & tensor comparison
|
||||
convert_test.go
|
||||
```
|
||||
|
||||
### Why a raw KV walker
|
||||
|
||||
`go-gguf`'s `MetadataEntry.Value()` intentionally returns an empty value for
|
||||
`BTypeArray` (see `readLazyValue` in the library). The tokenizer arrays
|
||||
(`tokens`, `scores`→`token_type`, `merges`) are exactly the values we must copy
|
||||
verbatim, so `rawkv.go` reads the KV section directly and preserves each array's
|
||||
`elem_type`, element count, and **raw element bytes**. Those raw bytes are fed
|
||||
straight into go-gguf's writer `Value{Raw: ...}`, so serialization is still done
|
||||
by the library (nothing hand-rolled on the write path).
|
||||
|
||||
### Subcommands
|
||||
|
||||
```bash
|
||||
# 2a. dump — emit the reproduction contract as JSON
|
||||
go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json
|
||||
|
||||
# 2b/c. convert — reproduce metadata + map safetensors→GGUF (fp32→f16)
|
||||
go run . convert \
|
||||
-ref models/open-1b-sft-f16.gguf \
|
||||
-safetensors models/open-1b-sft/model.safetensors \
|
||||
-config models/open-1b-sft/config.json \
|
||||
-tokenizer-config models/open-1b-sft/tokenizer_config.json \
|
||||
-tokenizer models/open-1b-sft/tokenizer.json \
|
||||
-o models/open-1b-sft-f16-go.gguf
|
||||
|
||||
# 3. verify — byte-level equality against the reference
|
||||
go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
`dump` output (the contract) is at
|
||||
`models/open-1b-sft-f16.manifest.json` (39 metadata entries + 220 tensors,
|
||||
with per-array `raw_sha256`).
|
||||
|
||||
### Tensor mapping (reference names are the contract)
|
||||
|
||||
Shapes are the reference's; ggml `ne0` is the innermost (= HF input) dim, so HF
|
||||
bytes are copied unpermuted with the dims reversed.
|
||||
|
||||
| Reference tensor | HF source | Shape | Type | Count |
|
||||
|---|---|---|---|---|
|
||||
| `token_embd.weight` | `embedding.tok_embeddings.weight` | [2048,128256] | F16 | 1 |
|
||||
| `token_embd_norm.weight` | `emb_norm.weight` | [2048] | F32 | 1 |
|
||||
| `output.weight` | `embedding.output.weight` | [2048,128256] | F16 | 1 |
|
||||
| `output_norm.weight` | `norm_out.weight` | [2048] | F32 | 1 |
|
||||
| `blk.{i}.attn_q.weight` | `blocks.{i}.attn.wq.weight` | [2048,2048] | F16 | 24 |
|
||||
| `blk.{i}.attn_k.weight` | `blocks.{i}.attn.wk.weight` | [2048,512] | F16 | 24 |
|
||||
| `blk.{i}.attn_v.weight` | `blocks.{i}.attn.wv.weight` | [2048,512] | F16 | 24 |
|
||||
| `blk.{i}.attn_output.weight` | `blocks.{i}.attn.wo.weight` | [2048,2048] | F16 | 24 |
|
||||
| `blk.{i}.ffn_gate.weight` | `blocks.{i}.ffn.w_gate_up.weight` (first half) | [2048,5632] | F16 | 24 |
|
||||
| `blk.{i}.ffn_up.weight` | `blocks.{i}.ffn.w_gate_up.weight` (second half) | [2048,5632] | F16 | 24 |
|
||||
| `blk.{i}.ffn_down.weight` | `blocks.{i}.ffn.w_down.weight` | [5632,2048] | F16 | 24 |
|
||||
| `blk.{i}.attn_norm.weight` | `blocks.{i}.norm1.weight` | [2048] | F32 | 24 |
|
||||
| `blk.{i}.ffn_norm.weight` | `blocks.{i}.norm2.weight` | [2048] | F32 | 24 |
|
||||
|
||||
- The fused `w_gate_up` is split at the row boundary (`chunk(2,-1)` → first half
|
||||
`gate`, second half `up`), matching the reference.
|
||||
- Quantized per-channel `*.weight_scale` tensors in the checkpoint are **dropped**
|
||||
(the reference has none).
|
||||
- F32→F16 uses IEEE round-to-nearest, ties-to-even (`f32bitsToF16`), matching
|
||||
numpy/torch casts; the reference tensors are byte-identical to ours, which
|
||||
empirically proves the rounding matched.
|
||||
- HF config/tokenizer are **parsed and validated** against the reference, but the
|
||||
reference remains authoritative for every emitted value.
|
||||
|
||||
---
|
||||
|
||||
## 3. Validation
|
||||
|
||||
`go run . verify` streams both files and compares on raw wire bytes:
|
||||
|
||||
```
|
||||
metadata: 39 checked, 0 mismatches
|
||||
tensors: 220 checked, 220 byte-identical, 0 mismatches
|
||||
verification PASSED: 39 metadata keys and 220 tensors byte-identical
|
||||
```
|
||||
|
||||
Whole-file check:
|
||||
|
||||
```
|
||||
$ cmp models/open-1b-sft-f16.gguf models/open-1b-sft-f16-go.gguf && echo IDENTICAL
|
||||
IDENTICAL
|
||||
$ sha256sum models/open-1b-sft-f16.gguf models/open-1b-sft-f16-go.gguf
|
||||
a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae open-1b-sft-f16.gguf
|
||||
a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
The per-tensor diff is empty (0 mismatches) — every tensor's payload is
|
||||
byte-identical, in the same order, with the same shape and ggml type.
|
||||
|
||||
### Metadata contract (all 39 keys, all byte-identical)
|
||||
|
||||
| Key | Type | Value |
|
||||
|---|---|---|
|
||||
| `general.architecture` | STRING | `open1b` |
|
||||
| `general.type` | STRING | `model` |
|
||||
| `general.name` | STRING | `Open 1b Sft` |
|
||||
| `general.finetune` | STRING | `sft` |
|
||||
| `general.basename` | STRING | `open` |
|
||||
| `general.size_label` | STRING | `1B` |
|
||||
| `general.license` | STRING | `apache-2.0` |
|
||||
| `general.base_model.count` | UINT32 | 1 |
|
||||
| `general.base_model.0.name` | STRING | `Open 1b Midtrained 93B` |
|
||||
| `general.base_model.0.organization` | STRING | `Gensyn` |
|
||||
| `general.base_model.0.repo_url` | STRING | `https://huggingface.co/Gensyn/open-1b-midtrained-93B` |
|
||||
| `general.dataset.count` | UINT32 | 1 |
|
||||
| `general.dataset.0.name` | STRING | `Tulu 3 Sft Olmo 2 Mixture 0225` |
|
||||
| `general.dataset.0.version` | STRING | `0225` |
|
||||
| `general.dataset.0.organization` | STRING | `Allenai` |
|
||||
| `general.dataset.0.repo_url` | STRING | `https://huggingface.co/allenai/tulu-3-sft-olmo-2-mixture-0225` |
|
||||
| `general.tags` | ARRAY[STRING] | 5 (`gensyn`, `verifiable-training`, `reproducible-training`, `repops`, `text-generation`) |
|
||||
| `general.languages` | ARRAY[STRING] | 1 (`en`) |
|
||||
| `open1b.block_count` | UINT32 | 24 |
|
||||
| `open1b.context_length` | UINT32 | 4096 |
|
||||
| `open1b.embedding_length` | UINT32 | 2048 |
|
||||
| `open1b.feed_forward_length` | UINT32 | 5632 |
|
||||
| `open1b.attention.head_count` | UINT32 | 16 |
|
||||
| `open1b.attention.head_count_kv` | UINT32 | 4 |
|
||||
| `open1b.rope.freq_base` | FLOAT32 | 500000 |
|
||||
| `open1b.attention.layer_norm_rms_epsilon` | FLOAT32 | 1e-05 |
|
||||
| `open1b.attention.key_length` | UINT32 | 128 |
|
||||
| `open1b.attention.value_length` | UINT32 | 128 |
|
||||
| `general.file_type` | UINT32 | 1 (F16) |
|
||||
| `open1b.attention.sliding_window` | UINT32 | 512 |
|
||||
| `open1b.attention.sliding_window_pattern` | UINT32 | 5 |
|
||||
| `general.quantization_version` | UINT32 | 2 |
|
||||
| `tokenizer.ggml.model` | STRING | `gpt2` |
|
||||
| `tokenizer.ggml.pre` | STRING | `gpt-2` |
|
||||
| `tokenizer.ggml.tokens` | ARRAY[STRING] | 128256 (sha256 `30bab437…`) |
|
||||
| `tokenizer.ggml.token_type` | ARRAY[INT32] | 128256 (sha256 `5ca8c567…`) |
|
||||
| `tokenizer.ggml.merges` | ARRAY[STRING] | 127994 (sha256 `9e3a7dd4…`) |
|
||||
| `tokenizer.ggml.eos_token_id` | UINT32 | 4 |
|
||||
| `tokenizer.chat_template` | STRING | (501-byte Jinja template) |
|
||||
|
||||
Array `raw_sha256` values are in `models/open-1b-sft-f16.manifest.json`.
|
||||
|
||||
### HF cross-checks performed by `convert`
|
||||
|
||||
```
|
||||
config.json OK: arch=open1b layers=24 hidden=2048 heads=16/4 head_dim=128 ffn=5632 vocab=128256 swa=512/5 tie=false qk_norm=true(gain=false) emb_norm=true
|
||||
tokenizer_config.json OK: eos=<|eot|> chat_template=501 bytes
|
||||
tokenizer.json OK: type=BPE vocab=128256 +added=6 -> 128256 tokens, 127994 merges
|
||||
```
|
||||
|
||||
### Go build / vet / test
|
||||
|
||||
```
|
||||
$ go build ./... # OK
|
||||
$ go vet ./... # OK
|
||||
$ go test -race -v ./...
|
||||
--- PASS: TestF32BitsToF16 (0.00s)
|
||||
--- PASS: TestF32ToF16Bytes (0.00s)
|
||||
--- PASS: TestRawKVRoundTrip (0.00s)
|
||||
--- PASS: TestVerifyRealFiles (6.59s)
|
||||
PASS
|
||||
ok hf2gguf 7.602s
|
||||
```
|
||||
|
||||
`TestVerifyRealFiles` re-runs the full 3.2 GB byte comparison (skipped on
|
||||
`-short` or when the artifacts are absent).
|
||||
|
||||
---
|
||||
|
||||
## 4. Serve on one Arc (Vulkan2)
|
||||
|
||||
```
|
||||
setsid --fork /root/projects/inference/build-vulkan/bin/llama-server \
|
||||
-m /root/projects/gensyn/inference/models/open-1b-sft-f16-go.gguf \
|
||||
--device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \
|
||||
--spec-type ngram-map-k,ngram-cache -t 16 --parallel 1 --alias open-1b \
|
||||
--host 127.0.0.1 --port 8085 >/tmp/open1b.log 2>&1 </dev/null &
|
||||
```
|
||||
|
||||
Startup log:
|
||||
|
||||
```
|
||||
load_model: loading model '.../open-1b-sft-f16-go.gguf'
|
||||
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
|
||||
llama_context: n_ctx_seq (32768) > n_ctx_train (4096) -- possible training context overflow
|
||||
load_model: the slot context (32768) exceeds the training context of the model (4096) - capping
|
||||
load_model: initializing, n_slots = 1, n_ctx_slot = 4096, kv_unified = 'false'
|
||||
model loaded
|
||||
listening on http://127.0.0.1:8085
|
||||
```
|
||||
|
||||
Device confirmation: PID `1378706` maps `/dev/dri/renderD129` with large
|
||||
`rw-s` regions; `renderD129 → /sys/devices/.../0000:07:00.0` = Arc #2 = Vulkan2.
|
||||
`/props` reports `"speculative.types":"none,ngram-map-k,ngram-cache"` and
|
||||
`"model_ftype":"F16"`.
|
||||
|
||||
### Performance (single Arc, 1 slot, `-fa off`, ngram spec)
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Decode | **~111.7–117.2 tok/s** (e.g. 256 tok @ 111.65; 96 tok @ 117.20) |
|
||||
| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) |
|
||||
| VRAM on Vulkan2 | free 10971 → 7258 MiB ⇒ **≈ 3.6 GiB** used; fully released after `kill` |
|
||||
|
||||
### Sample outputs
|
||||
|
||||
```
|
||||
$ curl ... /completion {"prompt":"The three primary colors are", greedy}
|
||||
red, blue, and yellow.
|
||||
|
||||
$ curl ... /completion {"prompt":"Q: What is the capital of France?\nA:", greedy, 96}
|
||||
Paris
|
||||
|
||||
$ curl ... /completion (seeded chat turn, see caveat) 113.9 tok/s
|
||||
" Venus and Mars. Venus is the hottest planet in our Solar System, with
|
||||
temperatures reaching 465°C (1,470°F). Mars, on the other hand, is a
|
||||
terrestrial planet, meaning it has a solid surface and liquid water at its poles."
|
||||
```
|
||||
|
||||
### Stopping
|
||||
|
||||
```
|
||||
$ ss -ltnp | grep :8085 # -> pid=1378706
|
||||
$ kill 1378706 # never pkill -f
|
||||
```
|
||||
|
||||
Both the test server (:8085) and a diagnostic server (:8086) were stopped;
|
||||
`--list-devices` returned to 10971 MiB free on Vulkan2/3.
|
||||
|
||||
---
|
||||
|
||||
## 5. Caveats
|
||||
|
||||
1. **Chat template newline quirk (model-side, not conversion-side).** With the
|
||||
exact vendor template tail `<|end_header|>\n\n` for the assistant turn, this
|
||||
checkpoint emits `<|eot|>` as the first sampled token → empty reply / instant
|
||||
EOS (reproduced on both :8085 with ngram spec and a :8086 run with
|
||||
`--spec-type none`, so it is **not** the speculative decoder). A single `\n`,
|
||||
`\n\n\n`, or a seed token makes it answer coherently:
|
||||
|
||||
| tail after `assistant<|end_header|>` | result |
|
||||
|---|---|
|
||||
| `\n\n` (vendor template) | immediate EOS |
|
||||
| `\n` | `\n1. **Earth**: …` (coherent) |
|
||||
| `\n\n\n` | `\n1. **Earth**: …` (coherent) |
|
||||
| ` ` | `\nSure, here are two names…` (coherent) |
|
||||
|
||||
Because our GGUF is byte-identical to the vendor reference, this is a
|
||||
property of the released checkpoint/GGUF (likely SFT formatting), not of our
|
||||
conversion. Workarounds for consumers: use `ignore_eos` + trim, seed the
|
||||
assistant turn, or fix the template's trailing newline.
|
||||
2. **Context capped to 4096.** `open1b.context_length = 4096`, so the requested
|
||||
`-c 32768` is capped to `n_ctx_slot = 4096`. KV is f16 (`-fa off` mandatory on
|
||||
Arc per `brainstation-gpu`).
|
||||
3. **`special_eos_id is not in special_eog_ids`** warning on load — present for
|
||||
the reference file too (byte-identical), so not introduced by us.
|
||||
4. **1B greedy degeneration.** Long greedy generation without repetition penalty
|
||||
loops on this small model; use sampling / `repeat_penalty` for long outputs.
|
||||
5. **Process note (disclosure).** During an early diagnostic I piped one
|
||||
`/completion` response through `python3 -c` to pretty-print JSON — a breach of
|
||||
the "no Python" rule (the tool itself is 100% Go, and nothing in the pipeline
|
||||
depends on Python). All subsequent parsing was done with Go/raw JSON. No `.py`
|
||||
file or venv was created; the only Python files in the tree are the vendor's
|
||||
read-only `models/open-1b-sft/*.py`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Artifacts
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `hf2gguf/` | Go module (`dump` / `convert` / `verify`) + tests |
|
||||
| `hf2gguf/README.md` | tool usage |
|
||||
| `models/open-1b-sft-f16-go.gguf` | our output (byte-identical to reference) |
|
||||
| `models/open-1b-sft-f16.manifest.json` | dumped contract (39 KV + 220 tensors) |
|
||||
| `models/open-1b-sft-f16.gguf` | reference (untouched) |
|
||||
| `/tmp/open1b.log` | server log |
|
||||
|
||||
No changes were committed; everything lives under gitignored `inference/`.
|
||||
@@ -0,0 +1,82 @@
|
||||
# hf2gguf
|
||||
|
||||
Pure-Go (Go 1.27) tool to convert a HuggingFace `open-1b` safetensors checkpoint
|
||||
into GGUF **byte-identically** to a supplied reference GGUF, plus a byte-level
|
||||
verifier. Built for epic `vado/1b#1` (Gensyn open-1b local bring-up).
|
||||
|
||||
Library: [`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf)
|
||||
(reader + writer). No Python anywhere in the tool.
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test -race ./... # full test includes the 3.2 GB byte comparison
|
||||
```
|
||||
|
||||
## Subcommands
|
||||
|
||||
### `dump` — the reproduction contract
|
||||
|
||||
Reads the reference GGUF and emits a JSON manifest of **all** metadata KV pairs
|
||||
(key, type, value / array elem-type + count + `raw_sha256`) and every tensor
|
||||
(name, shape, ggml type, nbytes).
|
||||
|
||||
```bash
|
||||
go run . dump -gguf models/open-1b-sft-f16.gguf -o models/open-1b-sft-f16.manifest.json
|
||||
```
|
||||
|
||||
`go-gguf` exposes scalar/string metadata and tensors; its `MetadataEntry.Value()`
|
||||
returns an empty value for `BTypeArray`, so `rawkv.go` walks the KV section
|
||||
directly to preserve tokenizer arrays (`tokens`, `merges`, `token_type`) as raw
|
||||
bytes. Writing still goes through `go-gguf`'s writer — nothing hand-rolls GGUF
|
||||
serialization.
|
||||
|
||||
### `convert` — emit our GGUF
|
||||
|
||||
```bash
|
||||
go run . convert \
|
||||
-ref models/open-1b-sft-f16.gguf \
|
||||
-safetensors models/open-1b-sft/model.safetensors \
|
||||
-config models/open-1b-sft/config.json \
|
||||
-tokenizer-config models/open-1b-sft/tokenizer_config.json \
|
||||
-tokenizer models/open-1b-sft/tokenizer.json \
|
||||
-o models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
- Metadata is copied verbatim from the reference KV section.
|
||||
- Tensors are mapped HF FQN → GGUF name (see the module docs in `convert.go`),
|
||||
with `blocks.{i}.ffn.w_gate_up.weight` split into `ffn_gate`/`ffn_up`, and
|
||||
`*.weight_scale` dropped (the reference has none).
|
||||
- Payloads stream from safetensors and are converted F32→F16 with IEEE
|
||||
round-to-nearest-even on the fly (O(1) memory per tensor).
|
||||
- `-config` / `-tokenizer-config` / `-tokenizer` are optional; when given they
|
||||
are parsed and **validated** against the reference (mismatch is fatal).
|
||||
|
||||
### `verify` — byte-level equality
|
||||
|
||||
```bash
|
||||
go run . verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
|
||||
```
|
||||
|
||||
Compares metadata on raw wire bytes and tensors by shape, type and full payload
|
||||
equality (streaming, first-diff offset reported on failure).
|
||||
|
||||
```
|
||||
metadata: 39 checked, 0 mismatches
|
||||
tensors: 220 checked, 220 byte-identical, 0 mismatches
|
||||
verification PASSED: 39 metadata keys and 220 tensors byte-identical
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `main.go` | dispatch, manifest types, `dump` |
|
||||
| `rawkv.go` | read-only GGUF KV walker (array support) |
|
||||
| `safetensors.go` | safetensors header/byte access |
|
||||
| `hfconfig.go` | HF config/tokenizer parsing + validation |
|
||||
| `convert.go` | metadata reproduction, tensor mapping, F32→F16 |
|
||||
| `verify.go` | `verify` command + comparison engine |
|
||||
| `convert_test.go` | F16 rounding, KV round-trip, full-file verification |
|
||||
@@ -0,0 +1,396 @@
|
||||
package main
|
||||
|
||||
// convert.go — emit models/open-1b-sft-f16-go.gguf from the HF safetensors,
|
||||
// reproducing the reference GGUF's metadata (verbatim) and tensor set/order.
|
||||
//
|
||||
// Tensor mapping (HF FQN -> GGUF name), shapes are the reference's (ggml ne0 is
|
||||
// the innermost/HF input dim, so the HF bytes are used unpermuted):
|
||||
//
|
||||
// embedding.tok_embeddings.weight -> token_embd.weight
|
||||
// emb_norm.weight -> token_embd_norm.weight
|
||||
// embedding.output.weight -> output.weight
|
||||
// norm_out.weight -> output_norm.weight
|
||||
// blocks.{i}.attn.wq.weight -> blk.{i}.attn_q.weight
|
||||
// blocks.{i}.attn.wk.weight -> blk.{i}.attn_k.weight
|
||||
// blocks.{i}.attn.wv.weight -> blk.{i}.attn_v.weight
|
||||
// blocks.{i}.attn.wo.weight -> blk.{i}.attn_output.weight
|
||||
// blocks.{i}.ffn.w_gate_up.weight -> blk.{i}.ffn_gate.weight (first half)
|
||||
// -> blk.{i}.ffn_up.weight (second half)
|
||||
// blocks.{i}.ffn.w_down.weight -> blk.{i}.ffn_down.weight
|
||||
// blocks.{i}.norm1.weight -> blk.{i}.attn_norm.weight
|
||||
// blocks.{i}.norm2.weight -> blk.{i}.ffn_norm.weight
|
||||
//
|
||||
// The quantized per-channel weight_scale tensors in the checkpoint are dropped,
|
||||
// matching the reference.
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"regexp"
|
||||
|
||||
gguf "github.com/cymertek/go-gguf"
|
||||
)
|
||||
|
||||
// hfRef resolves a GGUF tensor name to a HF safetensors tensor plus an optional
|
||||
// element range [start,end) (for the fused gate+up split). end=-1 means "all".
|
||||
type hfRef struct {
|
||||
Name string
|
||||
Start int64
|
||||
End int64
|
||||
}
|
||||
|
||||
func resolveHF(refName string) (hfRef, error) {
|
||||
switch refName {
|
||||
case "token_embd.weight":
|
||||
return hfRef{"embedding.tok_embeddings.weight", 0, -1}, nil
|
||||
case "token_embd_norm.weight":
|
||||
return hfRef{"emb_norm.weight", 0, -1}, nil
|
||||
case "output.weight":
|
||||
return hfRef{"embedding.output.weight", 0, -1}, nil
|
||||
case "output_norm.weight":
|
||||
return hfRef{"norm_out.weight", 0, -1}, nil
|
||||
}
|
||||
m := blkRe.FindStringSubmatch(refName)
|
||||
if m == nil {
|
||||
return hfRef{}, fmt.Errorf("unmapped tensor %q", refName)
|
||||
}
|
||||
i := m[1]
|
||||
leaf := m[2]
|
||||
switch leaf {
|
||||
case "attn_q":
|
||||
return hfRef{"blocks." + i + ".attn.wq.weight", 0, -1}, nil
|
||||
case "attn_k":
|
||||
return hfRef{"blocks." + i + ".attn.wk.weight", 0, -1}, nil
|
||||
case "attn_v":
|
||||
return hfRef{"blocks." + i + ".attn.wv.weight", 0, -1}, nil
|
||||
case "attn_output":
|
||||
return hfRef{"blocks." + i + ".attn.wo.weight", 0, -1}, nil
|
||||
case "ffn_gate":
|
||||
// fused w_gate_up is [2*ffn, hidden]; chunk(2) => gate = first half
|
||||
return hfRef{"blocks." + i + ".ffn.w_gate_up.weight", 0, 0}, nil // end filled later
|
||||
case "ffn_up":
|
||||
return hfRef{"blocks." + i + ".ffn.w_gate_up.weight", 0, 0}, nil
|
||||
case "ffn_down":
|
||||
return hfRef{"blocks." + i + ".ffn.w_down.weight", 0, -1}, nil
|
||||
case "attn_norm":
|
||||
return hfRef{"blocks." + i + ".norm1.weight", 0, -1}, nil
|
||||
case "ffn_norm":
|
||||
return hfRef{"blocks." + i + ".norm2.weight", 0, -1}, nil
|
||||
}
|
||||
return hfRef{}, fmt.Errorf("unmapped block tensor %q", refName)
|
||||
}
|
||||
|
||||
var blkRe = regexp.MustCompile(`^blk\.(\d+)\.(.+)\.weight$`)
|
||||
|
||||
func cmdConvert(args []string) error {
|
||||
fs := flag.NewFlagSet("convert", flag.ExitOnError)
|
||||
refGGUF := fs.String("ref", "", "reference GGUF (metadata/tensor contract)")
|
||||
stPath := fs.String("safetensors", "", "HF model.safetensors")
|
||||
hfConfig := fs.String("config", "", "HF config.json (validated against ref)")
|
||||
hfTokConfig := fs.String("tokenizer-config", "", "HF tokenizer_config.json (validated against ref)")
|
||||
hfTokenizer := fs.String("tokenizer", "", "HF tokenizer.json (validated against ref)")
|
||||
outPath := fs.String("o", "", "output GGUF path")
|
||||
fs.Parse(args)
|
||||
if *refGGUF == "" || *stPath == "" || *outPath == "" {
|
||||
return fmt.Errorf("convert: -ref, -safetensors and -o are required")
|
||||
}
|
||||
|
||||
hdr, kvs, err := readRawKV(*refGGUF)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read reference metadata: %w", err)
|
||||
}
|
||||
ref, err := gguf.Open(*refGGUF)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open reference: %w", err)
|
||||
}
|
||||
refTensors, err := ref.Tensors()
|
||||
if err != nil {
|
||||
return fmt.Errorf("reference tensors: %w", err)
|
||||
}
|
||||
|
||||
st, err := openSafetensors(*stPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open safetensors: %w", err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
// Validate HF config against the reference contract (no values are taken from
|
||||
// it — the reference is authoritative — but mismatches are fatal).
|
||||
if *hfConfig != "" {
|
||||
if err := validateConfig(*hfConfig, kvs); err != nil {
|
||||
return fmt.Errorf("config validation: %w", err)
|
||||
}
|
||||
}
|
||||
if *hfTokConfig != "" {
|
||||
if err := validateTokenizerConfig(*hfTokConfig, kvs); err != nil {
|
||||
return fmt.Errorf("tokenizer config validation: %w", err)
|
||||
}
|
||||
}
|
||||
if *hfTokenizer != "" {
|
||||
if err := validateTokenizerJSON(*hfTokenizer, kvs); err != nil {
|
||||
return fmt.Errorf("tokenizer.json validation: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
w, err := gguf.OpenForWrite(*outPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create output: %w", err)
|
||||
}
|
||||
// Metadata first: copy the reference KV section verbatim.
|
||||
for _, kv := range kvs {
|
||||
v, err := kvToValue(kv)
|
||||
if err != nil {
|
||||
w.Close()
|
||||
return fmt.Errorf("metadata %q: %w", kv.Key, err)
|
||||
}
|
||||
if err := w.SetKV(kv.Key, v); err != nil {
|
||||
w.Close()
|
||||
return fmt.Errorf("set %q: %w", kv.Key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Tensors in reference order so the data layout matches.
|
||||
byName := map[string]gguf.TensorInfo{}
|
||||
for _, t := range refTensors {
|
||||
byName[t.Info().Name] = t.Info()
|
||||
}
|
||||
for _, t := range refTensors {
|
||||
info := t.Info()
|
||||
r, err := resolveHF(info.Name)
|
||||
if err != nil {
|
||||
w.Close()
|
||||
return err
|
||||
}
|
||||
hfT, ok := st.Tensors[r.Name]
|
||||
if !ok {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: HF source %q missing", info.Name, r.Name)
|
||||
}
|
||||
elemBytes, err := stDtypeBytes(hfT.Dtype)
|
||||
if err != nil {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: %w", info.Name, err)
|
||||
}
|
||||
total := numElems(hfT.Shape)
|
||||
if hfT.Dtype != "F32" && hfT.Dtype != "F16" {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: source dtype %s unsupported", info.Name, hfT.Dtype)
|
||||
}
|
||||
// Resolve the fused gate/up split now that we know the element count.
|
||||
start, end := r.Start, r.End
|
||||
if end == -1 {
|
||||
end = total
|
||||
}
|
||||
if isFusedSplit(info.Name) {
|
||||
half := total / 2
|
||||
switch {
|
||||
case hasSuffix(info.Name, "ffn_gate.weight"):
|
||||
start, end = 0, half
|
||||
case hasSuffix(info.Name, "ffn_up.weight"):
|
||||
start, end = half, total
|
||||
}
|
||||
}
|
||||
if end > total || start < 0 || start >= end {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: bad element range [%d,%d) of %d", info.Name, start, end, total)
|
||||
}
|
||||
wantBytes := uint64(end-start) * uint64(info.GgmlType.BlockBytes()) / uint64(info.GgmlType.ElementsPerBlock())
|
||||
if info.NBytes != wantBytes {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: ref nbytes %d != expected %d", info.Name, info.NBytes, wantBytes)
|
||||
}
|
||||
|
||||
// Shape check: reference shape must be the HF shape reversed (ggml ne0
|
||||
// innermost) and consistent with the slice.
|
||||
if !shapeMatches(info.Shape, hfT.Shape, start, end, total) {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: ref shape %v vs HF %v (range %d:%d)", info.Name, info.Shape, hfT.Shape, start, end)
|
||||
}
|
||||
|
||||
dataOff := st.dataStart + hfT.Start + start*int64(elemBytes)
|
||||
rem := (end - start) * int64(elemBytes)
|
||||
|
||||
idx := w.AddTensor(info.Name, info.Shape, info.GgmlType)
|
||||
var rd io.Reader
|
||||
switch info.GgmlType {
|
||||
case gguf.GgmlF16:
|
||||
rd = &f16StreamReader{f: st.f, pos: dataOff, rem: rem}
|
||||
case gguf.GgmlF32:
|
||||
rd = &f32StreamReader{f: st.f, pos: dataOff, rem: rem}
|
||||
default:
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: unsupported target type %s", info.Name, info.GgmlType.GgmlName())
|
||||
}
|
||||
if err := w.WriteTensorData(idx, rd); err != nil {
|
||||
w.Close()
|
||||
return fmt.Errorf("tensor %s: %w", info.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
n, err := w.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("finalize: %w", err)
|
||||
}
|
||||
fmt.Printf("wrote %s: %d bytes, %d metadata keys, %d tensors (ref nKV=%d nTensors=%d)\n",
|
||||
*outPath, n, len(kvs), len(refTensors), hdr.NKV, hdr.NTensors)
|
||||
return nil
|
||||
}
|
||||
|
||||
func isFusedSplit(name string) bool {
|
||||
return hasSuffix(name, "ffn_gate.weight") || hasSuffix(name, "ffn_up.weight")
|
||||
}
|
||||
|
||||
func hasSuffix(s, suffix string) bool {
|
||||
return len(s) >= len(suffix) && s[len(s)-len(suffix):] == suffix
|
||||
}
|
||||
|
||||
// shapeMatches verifies ref ggml shape against the HF shape for a slice.
|
||||
// Whole tensors: ref == reverse(hf). A gate/up half: ref == [hf[1], hf[0]/2].
|
||||
func shapeMatches(ref []uint64, hf []int64, start, end, total int64) bool {
|
||||
if len(ref) != len(hf) {
|
||||
return false
|
||||
}
|
||||
if end-start != total {
|
||||
// fused split: both ref and hf are 2-D; ref[0]==hf[1],
|
||||
// ref[1]==hf[0]/2 and the split is on the outer (row) dim.
|
||||
if len(ref) != 2 {
|
||||
return false
|
||||
}
|
||||
return ref[0] == uint64(hf[1]) && ref[1] == uint64(hf[0]/2) && hf[0]%2 == 0
|
||||
}
|
||||
for i := range ref {
|
||||
if ref[i] != uint64(hf[len(hf)-1-i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// streaming readers that convert F32 -> F16 on the fly (RNE)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const streamChunk = 1 << 20 // 1 MiB of source bytes per read
|
||||
|
||||
type f16StreamReader struct {
|
||||
f *os.File
|
||||
pos int64
|
||||
rem int64
|
||||
pending []byte
|
||||
}
|
||||
|
||||
func (r *f16StreamReader) Read(p []byte) (int, error) {
|
||||
for len(r.pending) == 0 {
|
||||
if r.rem <= 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
n := r.rem
|
||||
if n > streamChunk {
|
||||
n = streamChunk
|
||||
}
|
||||
n -= n % 4 // whole f32 elements
|
||||
buf := make([]byte, n)
|
||||
if _, err := r.f.ReadAt(buf, r.pos); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
r.pos += n
|
||||
r.rem -= n
|
||||
r.pending = f32ToF16Bytes(buf)
|
||||
}
|
||||
n := copy(p, r.pending)
|
||||
r.pending = r.pending[n:]
|
||||
return n, nil
|
||||
}
|
||||
|
||||
type f32StreamReader struct {
|
||||
f *os.File
|
||||
pos int64
|
||||
rem int64
|
||||
pending []byte
|
||||
}
|
||||
|
||||
func (r *f32StreamReader) Read(p []byte) (int, error) {
|
||||
for len(r.pending) == 0 {
|
||||
if r.rem <= 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
n := r.rem
|
||||
if n > streamChunk {
|
||||
n = streamChunk
|
||||
}
|
||||
buf := make([]byte, n)
|
||||
if _, err := r.f.ReadAt(buf, r.pos); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
r.pos += n
|
||||
r.rem -= n
|
||||
r.pending = buf
|
||||
}
|
||||
n := copy(p, r.pending)
|
||||
r.pending = r.pending[n:]
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// FP32 -> FP16 (IEEE-754 round-to-nearest-even), matching numpy/torch casts.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func f32ToF16Bytes(src []byte) []byte {
|
||||
out := make([]byte, len(src)/4*2)
|
||||
for i := 0; i+4 <= len(src); i += 4 {
|
||||
bits := uint32(src[i]) | uint32(src[i+1])<<8 | uint32(src[i+2])<<16 | uint32(src[i+3])<<24
|
||||
half := f32bitsToF16(bits)
|
||||
j := i / 4 * 2
|
||||
out[j] = byte(half)
|
||||
out[j+1] = byte(half >> 8)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// f32bitsToF16 converts the raw bits of an IEEE-754 binary32 to binary16 using
|
||||
// round-to-nearest, ties-to-even.
|
||||
func f32bitsToF16(b uint32) uint16 {
|
||||
sign := uint16((b >> 16) & 0x8000)
|
||||
exp := int32((b >> 23) & 0xff)
|
||||
mant := b & 0x7fffff
|
||||
|
||||
if exp == 0xff { // Inf or NaN
|
||||
if mant != 0 {
|
||||
return sign | 0x7e00 // quiet NaN
|
||||
}
|
||||
return sign | 0x7c00 // Inf
|
||||
}
|
||||
|
||||
e := exp - 127 // unbiased
|
||||
|
||||
if e > 15 { // overflow -> Inf
|
||||
return sign | 0x7c00
|
||||
}
|
||||
if e >= -14 { // normal
|
||||
hm := uint16(e+15)<<10 | uint16(mant>>13)
|
||||
rem := mant & 0x1fff
|
||||
if rem > 0x1000 || (rem == 0x1000 && hm&1 == 1) {
|
||||
hm++
|
||||
}
|
||||
return sign | hm
|
||||
}
|
||||
if e < -25 { // too small -> signed zero
|
||||
return sign
|
||||
}
|
||||
// subnormal: implicit leading 1, shift into place
|
||||
m := mant | 0x800000
|
||||
shift := uint32(-e - 14) // 1..11
|
||||
hm := uint16(m >> (13 + shift))
|
||||
rem := m & ((1 << (13 + shift)) - 1)
|
||||
halfBit := uint32(1) << (12 + shift)
|
||||
if rem > halfBit || (rem == halfBit && hm&1 == 1) {
|
||||
hm++
|
||||
}
|
||||
return sign | hm
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// HF config / tokenizer config validation (reference is authoritative)
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -0,0 +1,161 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
gguf "github.com/cymertek/go-gguf"
|
||||
)
|
||||
|
||||
func TestF32BitsToF16(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in uint32
|
||||
want uint16
|
||||
}{
|
||||
{"zero", 0x00000000, 0x0000},
|
||||
{"negzero", 0x80000000, 0x8000},
|
||||
{"one", 0x3f800000, 0x3c00},
|
||||
{"neg2", 0xc0000000, 0xc000},
|
||||
{"half", 0x3f000000, 0x3800},
|
||||
{"max_f16", 0x477fe000, 0x7bff},
|
||||
{"overflow_to_inf", 0x477ff000, 0x7c00},
|
||||
{"inf", 0x7f800000, 0x7c00},
|
||||
{"neginf", 0xff800000, 0xfc00},
|
||||
{"qnan", 0x7fc00000, 0x7e00},
|
||||
{"snan", 0x7f800001, 0x7e00},
|
||||
{"min_normal", 0x38800000, 0x0400}, // 2^-14
|
||||
{"min_subnormal", 0x33800000, 0x0001}, // 2^-24
|
||||
{"tie_to_even_zero", 0x33000000, 0x0000}, // 2^-25 (half of min subnormal)
|
||||
{"tie_to_even_two", 0x33c00000, 0x0002}, // 3 * 2^-25
|
||||
{"one_tenth", 0x3dcccccd, 0x2e66},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := f32bitsToF16(c.in); got != c.want {
|
||||
t.Errorf("%s: f32=%08x got %04x want %04x", c.name, c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestF32ToF16Bytes(t *testing.T) {
|
||||
in := make([]byte, 8)
|
||||
putF32 := func(off int, f float32) {
|
||||
b := math.Float32bits(f)
|
||||
in[off] = byte(b)
|
||||
in[off+1] = byte(b >> 8)
|
||||
in[off+2] = byte(b >> 16)
|
||||
in[off+3] = byte(b >> 24)
|
||||
}
|
||||
putF32(0, 1.0)
|
||||
putF32(4, -2.0)
|
||||
out := f32ToF16Bytes(in)
|
||||
want := []byte{0x00, 0x3c, 0x00, 0xc0} // 1.0 -> 0x3c00 LE, -2.0 -> 0xc000 LE
|
||||
for i := range want {
|
||||
if out[i] != want[i] {
|
||||
t.Fatalf("byte %d: got %02x want %02x", i, out[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRawKVRoundTrip creates a small GGUF with go-gguf, reads it with our raw
|
||||
// walker, and re-writes the KVs through kvToValue to confirm arrays (including
|
||||
// string arrays) survive byte-exactly.
|
||||
func TestRawKVRoundTrip(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
src := filepath.Join(dir, "a.gguf")
|
||||
dst := filepath.Join(dir, "b.gguf")
|
||||
|
||||
w, err := gguf.OpenForWrite(src)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
strs := []byte{}
|
||||
for _, s := range []string{"alpha", "béta", ""} {
|
||||
strs = append(strs, leU64(uint64(len(s)))...)
|
||||
strs = append(strs, s...)
|
||||
}
|
||||
kvs := []gguf.KVEntry{
|
||||
{Key: "general.architecture", Value: gguf.Value{BType: gguf.BTypeString, Str: "open1b"}},
|
||||
{Key: "x.count", Value: gguf.Value{BType: gguf.BTypeUint32, Int: 7}},
|
||||
{Key: "x.f", Value: gguf.Value{BType: gguf.BTypeFloat32, Float: float64(float32(1e-5))}},
|
||||
{Key: "x.b", Value: gguf.Value{BType: gguf.BTypeBool, Int: 1}},
|
||||
{Key: "x.tags", Value: gguf.Value{BType: gguf.BTypeArray, ElemType: gguf.BTypeString, Int: 3, Raw: strs}},
|
||||
}
|
||||
for _, kv := range kvs {
|
||||
if err := w.SetKV(kv.Key, kv.Value); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, got, err := readRawKV(src)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != len(kvs) {
|
||||
t.Fatalf("kv count %d want %d", len(got), len(kvs))
|
||||
}
|
||||
|
||||
w2, err := gguf.OpenForWrite(dst)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, kv := range got {
|
||||
v, err := kvToValue(kv)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w2.SetKV(kv.Key, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := w2.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, round, err := readRawKV(dst)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
res, err := compareGGUF(src, dst)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !res.OK() {
|
||||
t.Fatalf("roundtrip not identical: meta=%v tensor=%v", res.MetaMismatches, res.TensorMismatch)
|
||||
}
|
||||
if len(round) != len(got) {
|
||||
t.Fatalf("rewritten kv count %d want %d", len(round), len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifyRealFiles runs the full byte-level verification against the built
|
||||
// artifacts when they are present in the workdir. Skipped when absent (fresh
|
||||
// checkout) or in -short mode.
|
||||
func TestVerifyRealFiles(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping multi-GB comparison in -short mode")
|
||||
}
|
||||
ref := "../models/open-1b-sft-f16.gguf"
|
||||
got := "../models/open-1b-sft-f16-go.gguf"
|
||||
if _, err := os.Stat(ref); err != nil {
|
||||
t.Skipf("reference GGUF not present: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(got); err != nil {
|
||||
t.Skipf("candidate GGUF not present (run convert first): %v", err)
|
||||
}
|
||||
res, err := compareGGUF(ref, got)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !res.OK() {
|
||||
t.Fatalf("verification failed:\n metadata=%v\n tensors=%v", res.MetaMismatches, res.TensorMismatch)
|
||||
}
|
||||
if res.ByteEqual != res.TensorsChecked || res.TensorsChecked == 0 {
|
||||
t.Fatalf("expected all tensors byte-identical: %d/%d", res.ByteEqual, res.TensorsChecked)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module hf2gguf
|
||||
|
||||
go 1.27.0
|
||||
|
||||
require github.com/cymertek/go-gguf v0.0.0-20260818124528-20850052d9e2
|
||||
@@ -0,0 +1,2 @@
|
||||
github.com/cymertek/go-gguf v0.0.0-20260818124528-20850052d9e2 h1:7WUU9/eb/AcBNCyMXIrPuaUaeLJXlYzSxQfNmEGwNV4=
|
||||
github.com/cymertek/go-gguf v0.0.0-20260818124528-20850052d9e2/go.mod h1:F8ePnvWZX6ZD/sc6NAg3Zjm7T2b7f+kDWnCkKzyNVUw=
|
||||
@@ -0,0 +1,223 @@
|
||||
package main
|
||||
|
||||
// hfconfig.go — parse and validate the HF config/tokenizer against the
|
||||
// reference GGUF metadata. The reference remains authoritative for values; this
|
||||
// only proves the reference is consistent with the published HF files and that
|
||||
// we parsed them (per the task), it does not source any GGUF value from them.
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
)
|
||||
|
||||
type hfConfigFile struct {
|
||||
ModelType string `json:"model_type"`
|
||||
HiddenSize int64 `json:"hidden_size"`
|
||||
NumHiddenLayers int64 `json:"num_hidden_layers"`
|
||||
NumAttentionHeads int64 `json:"num_attention_heads"`
|
||||
NumKeyValueHeads int64 `json:"num_key_value_heads"`
|
||||
HeadDim int64 `json:"head_dim"`
|
||||
IntermediateSize int64 `json:"intermediate_size"`
|
||||
VocabSize int64 `json:"vocab_size"`
|
||||
MaxPositionEmbeddings int64 `json:"max_position_embeddings"`
|
||||
RopeTheta float64 `json:"rope_theta"`
|
||||
RMSNormEps float64 `json:"rms_norm_eps"`
|
||||
TieWordEmbeddings bool `json:"tie_word_embeddings"`
|
||||
QKNorm bool `json:"qk_norm"`
|
||||
QKNormGain bool `json:"qk_norm_gain"`
|
||||
EmbeddingNorm bool `json:"embedding_norm"`
|
||||
SlidingWindow int64 `json:"sliding_window"`
|
||||
SWAFullEvery int64 `json:"swa_full_every"`
|
||||
}
|
||||
|
||||
func findKV(kvs []rawKV, key string) (rawKV, bool) {
|
||||
for _, kv := range kvs {
|
||||
if kv.Key == key {
|
||||
return kv, true
|
||||
}
|
||||
}
|
||||
return rawKV{}, false
|
||||
}
|
||||
|
||||
func validateConfig(path string, kvs []rawKV) error {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var c hfConfigFile
|
||||
if err := json.Unmarshal(b, &c); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
intChecks := []struct {
|
||||
name string
|
||||
got int64
|
||||
}{
|
||||
{"open1b.embedding_length", c.HiddenSize},
|
||||
{"open1b.block_count", c.NumHiddenLayers},
|
||||
{"open1b.attention.head_count", c.NumAttentionHeads},
|
||||
{"open1b.attention.head_count_kv", c.NumKeyValueHeads},
|
||||
{"open1b.attention.key_length", c.HeadDim},
|
||||
{"open1b.attention.value_length", c.HeadDim},
|
||||
{"open1b.feed_forward_length", c.IntermediateSize},
|
||||
{"open1b.context_length", c.MaxPositionEmbeddings},
|
||||
{"open1b.attention.sliding_window", c.SlidingWindow},
|
||||
{"open1b.attention.sliding_window_pattern", c.SWAFullEvery},
|
||||
}
|
||||
for _, chk := range intChecks {
|
||||
kv, ok := findKV(kvs, chk.name)
|
||||
if !ok {
|
||||
return fmt.Errorf("reference missing %s", chk.name)
|
||||
}
|
||||
if kv.Int != chk.got {
|
||||
return fmt.Errorf("%s: HF=%d ref=%d", chk.name, chk.got, kv.Int)
|
||||
}
|
||||
}
|
||||
floatChecks := []struct {
|
||||
name string
|
||||
got float64
|
||||
}{
|
||||
{"open1b.rope.freq_base", c.RopeTheta},
|
||||
{"open1b.attention.layer_norm_rms_epsilon", c.RMSNormEps},
|
||||
}
|
||||
for _, chk := range floatChecks {
|
||||
kv, ok := findKV(kvs, chk.name)
|
||||
if !ok {
|
||||
return fmt.Errorf("reference missing %s", chk.name)
|
||||
}
|
||||
if float32(kv.Float) != float32(chk.got) {
|
||||
return fmt.Errorf("%s: HF=%v ref=%v", chk.name, chk.got, kv.Float)
|
||||
}
|
||||
}
|
||||
fmt.Printf("config.json OK: arch=%s layers=%d hidden=%d heads=%d/%d head_dim=%d ffn=%d vocab=%d swa=%d/%d tie=%v qk_norm=%v(gain=%v) emb_norm=%v\n",
|
||||
c.ModelType, c.NumHiddenLayers, c.HiddenSize, c.NumAttentionHeads, c.NumKeyValueHeads,
|
||||
c.HeadDim, c.IntermediateSize, c.VocabSize, c.SlidingWindow, c.SWAFullEvery,
|
||||
c.TieWordEmbeddings, c.QKNorm, c.QKNormGain, c.EmbeddingNorm)
|
||||
return nil
|
||||
}
|
||||
|
||||
type tokConfigFile struct {
|
||||
EOSToken string `json:"eos_token"`
|
||||
ModelMaxLen int64 `json:"model_max_length"`
|
||||
ChatTemplate string `json:"chat_template"`
|
||||
AddedDecoders map[string]struct {
|
||||
Content string `json:"content"`
|
||||
} `json:"added_tokens_decoder"`
|
||||
}
|
||||
|
||||
func validateTokenizerConfig(path string, kvs []rawKV) error {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var tc tokConfigFile
|
||||
if err := json.Unmarshal(b, &tc); err != nil {
|
||||
return err
|
||||
}
|
||||
// chat_template must match the reference verbatim.
|
||||
if kv, ok := findKV(kvs, "tokenizer.chat_template"); ok {
|
||||
if kv.Str != tc.ChatTemplate {
|
||||
return fmt.Errorf("chat_template differs (HF %d bytes, ref %d bytes)", len(tc.ChatTemplate), len(kv.Str))
|
||||
}
|
||||
} else {
|
||||
return fmt.Errorf("reference missing tokenizer.chat_template")
|
||||
}
|
||||
// eos token id from added_tokens_decoder.
|
||||
if kv, ok := findKV(kvs, "tokenizer.ggml.eos_token_id"); ok {
|
||||
var eosID int64 = -1
|
||||
keys := make([]string, 0, len(tc.AddedDecoders))
|
||||
for k := range tc.AddedDecoders {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if tc.AddedDecoders[k].Content == tc.EOSToken {
|
||||
var id int64
|
||||
fmt.Sscanf(k, "%d", &id)
|
||||
eosID = id
|
||||
}
|
||||
}
|
||||
if eosID != kv.Int {
|
||||
return fmt.Errorf("eos_token %q: HF id=%d ref=%d", tc.EOSToken, eosID, kv.Int)
|
||||
}
|
||||
}
|
||||
// context length consistency
|
||||
if kv, ok := findKV(kvs, "open1b.context_length"); ok && tc.ModelMaxLen != 0 && tc.ModelMaxLen != kv.Int {
|
||||
return fmt.Errorf("model_max_length: HF=%d ref=%d", tc.ModelMaxLen, kv.Int)
|
||||
}
|
||||
fmt.Printf("tokenizer_config.json OK: eos=%s chat_template=%d bytes\n", tc.EOSToken, len(tc.ChatTemplate))
|
||||
return nil
|
||||
}
|
||||
|
||||
type tokenizerJSON struct {
|
||||
Model struct {
|
||||
Type string `json:"type"`
|
||||
Vocab map[string]int `json:"vocab"`
|
||||
Merges []json.RawMessage `json:"merges"`
|
||||
} `json:"model"`
|
||||
AddedTokens []struct {
|
||||
ID int64 `json:"id"`
|
||||
Content string `json:"content"`
|
||||
} `json:"added_tokens"`
|
||||
}
|
||||
|
||||
// validateTokenizerJSON proves the tokenizer.json we parsed is the source of the
|
||||
// reference tokenizer arrays: same BPE type, same vocab+added token count, same
|
||||
// merge count, and the added special tokens are the reference's leading tokens.
|
||||
func validateTokenizerJSON(path string, kvs []rawKV) error {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var tj tokenizerJSON
|
||||
if err := json.Unmarshal(b, &tj); err != nil {
|
||||
return err
|
||||
}
|
||||
if kv, ok := findKV(kvs, "tokenizer.ggml.model"); ok && kv.Str != "gpt2" {
|
||||
return fmt.Errorf("ref tokenizer model %q, expected gpt2", kv.Str)
|
||||
}
|
||||
if tj.Model.Type != "BPE" {
|
||||
return fmt.Errorf("tokenizer.json model.type=%q, expected BPE", tj.Model.Type)
|
||||
}
|
||||
tokensKV, ok := findKV(kvs, "tokenizer.ggml.tokens")
|
||||
if !ok {
|
||||
return fmt.Errorf("reference missing tokenizer.ggml.tokens")
|
||||
}
|
||||
// tokenizer.json's vocab sometimes already contains the added specials and
|
||||
// sometimes not; accept either interpretation as long as the count matches.
|
||||
vocabN := uint64(len(tj.Model.Vocab))
|
||||
got := vocabN
|
||||
if got != tokensKV.Count {
|
||||
got = vocabN + uint64(len(tj.AddedTokens))
|
||||
}
|
||||
if got != tokensKV.Count {
|
||||
return fmt.Errorf("vocab=%d (+added=%d =%d) vs ref tokens=%d", vocabN, len(tj.AddedTokens), got, tokensKV.Count)
|
||||
}
|
||||
mergesKV, ok := findKV(kvs, "tokenizer.ggml.merges")
|
||||
if !ok {
|
||||
return fmt.Errorf("reference missing tokenizer.ggml.merges")
|
||||
}
|
||||
if uint64(len(tj.Model.Merges)) != mergesKV.Count {
|
||||
return fmt.Errorf("tokenizer.json merges=%d vs ref=%d", len(tj.Model.Merges), mergesKV.Count)
|
||||
}
|
||||
// Leading tokens (added specials, sorted by id) must match the ref array head.
|
||||
sort.Slice(tj.AddedTokens, func(i, j int) bool { return tj.AddedTokens[i].ID < tj.AddedTokens[j].ID })
|
||||
refHead := decodeStringArray(tokensKV.Raw, uint64(len(tj.AddedTokens)))
|
||||
for i := range tj.AddedTokens {
|
||||
if i >= len(refHead) {
|
||||
return fmt.Errorf("ref tokens shorter than added tokens")
|
||||
}
|
||||
if refHead[i] != tj.AddedTokens[i].Content {
|
||||
return fmt.Errorf("token[%d]: HF %q ref %q", i, tj.AddedTokens[i].Content, refHead[i])
|
||||
}
|
||||
}
|
||||
fmt.Printf("tokenizer.json OK: type=BPE vocab=%d +added=%d -> %d tokens, %d merges\n",
|
||||
vocabN, len(tj.AddedTokens), got, len(tj.Model.Merges))
|
||||
return nil
|
||||
}
|
||||
|
||||
// keep the binary import used (helps future raw comparisons).
|
||||
var _ = binary.LittleEndian
|
||||
@@ -0,0 +1,284 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
|
||||
gguf "github.com/cymertek/go-gguf"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 {
|
||||
fmt.Fprintln(os.Stderr, "usage: hf2gguf <dump|convert|verify> [flags]")
|
||||
os.Exit(2)
|
||||
}
|
||||
var err error
|
||||
switch os.Args[1] {
|
||||
case "dump":
|
||||
err = cmdDump(os.Args[2:])
|
||||
case "convert":
|
||||
err = cmdConvert(os.Args[2:])
|
||||
case "verify":
|
||||
err = cmdVerify(os.Args[2:])
|
||||
default:
|
||||
err = fmt.Errorf("unknown subcommand %q", os.Args[1])
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Manifest types — the reproduction contract.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type manifest struct {
|
||||
Source string `json:"source"`
|
||||
Version uint32 `json:"version"`
|
||||
NTensors uint64 `json:"n_tensors"`
|
||||
NKV uint64 `json:"n_kv"`
|
||||
Metadata []metaKV `json:"metadata"`
|
||||
Tensors []tensorRow `json:"tensors"`
|
||||
}
|
||||
|
||||
type metaKV struct {
|
||||
Key string `json:"key"`
|
||||
Type string `json:"type"`
|
||||
Value any `json:"value,omitempty"`
|
||||
ElemType string `json:"elem_type,omitempty"`
|
||||
Count uint64 `json:"count,omitempty"`
|
||||
RawBytes int `json:"raw_bytes,omitempty"`
|
||||
RawSHA256 string `json:"raw_sha256,omitempty"`
|
||||
Preview []string `json:"preview,omitempty"`
|
||||
Numbers []float64 `json:"numbers,omitempty"`
|
||||
}
|
||||
|
||||
type tensorRow struct {
|
||||
Name string `json:"name"`
|
||||
Shape []uint64 `json:"shape"`
|
||||
Type string `json:"type"`
|
||||
TypeID uint32 `json:"type_id"`
|
||||
NBytes uint64 `json:"nbytes"`
|
||||
}
|
||||
|
||||
func buildManifest(ggufPath string) (*manifest, error) {
|
||||
hdr, kvs, err := readRawKV(ggufPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m := &manifest{
|
||||
Source: ggufPath,
|
||||
Version: hdr.Version,
|
||||
NTensors: hdr.NTensors,
|
||||
NKV: hdr.NKV,
|
||||
}
|
||||
for _, kv := range kvs {
|
||||
m.Metadata = append(m.Metadata, kvToManifest(kv))
|
||||
}
|
||||
|
||||
g, err := gguf.Open(ggufPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("go-gguf open: %w", err)
|
||||
}
|
||||
defer g.Close()
|
||||
tensors, err := g.Tensors()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("go-gguf tensors: %w", err)
|
||||
}
|
||||
for _, t := range tensors {
|
||||
info := t.Info()
|
||||
m.Tensors = append(m.Tensors, tensorRow{
|
||||
Name: info.Name,
|
||||
Shape: info.Shape,
|
||||
Type: info.GgmlType.GgmlName(),
|
||||
TypeID: uint32(info.GgmlType),
|
||||
NBytes: info.NBytes,
|
||||
})
|
||||
}
|
||||
if uint64(len(m.Tensors)) != hdr.NTensors {
|
||||
return nil, fmt.Errorf("tensor count mismatch: header %d, walked %d", hdr.NTensors, len(m.Tensors))
|
||||
}
|
||||
if uint64(len(m.Metadata)) != hdr.NKV {
|
||||
return nil, fmt.Errorf("kv count mismatch: header %d, walked %d", hdr.NKV, len(m.Metadata))
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func kvToManifest(kv rawKV) metaKV {
|
||||
out := metaKV{Key: kv.Key, Type: kv.BType.name()}
|
||||
switch kv.BType {
|
||||
case btBool:
|
||||
out.Value = kv.Int != 0
|
||||
case btUint8, btInt8, btUint16, btInt16, btUint32, btInt32, btUint64, btInt64:
|
||||
out.Value = kv.Int
|
||||
case btFloat32:
|
||||
out.Value = float32(kv.Float)
|
||||
case btFloat64:
|
||||
out.Value = kv.Float
|
||||
case btString:
|
||||
out.Value = kv.Str
|
||||
case btArray:
|
||||
out.ElemType = kv.ElemType.name()
|
||||
out.Count = kv.Count
|
||||
out.RawBytes = len(kv.Raw)
|
||||
sum := sha256.Sum256(kv.Raw)
|
||||
out.RawSHA256 = fmt.Sprintf("%x", sum)
|
||||
decodePreview(kv, &out)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// decodePreview fills Preview/Numbers for small arrays so the manifest stays readable.
|
||||
func decodePreview(kv rawKV, out *metaKV) {
|
||||
if kv.Count > 64 {
|
||||
// give a tiny human preview even for huge token arrays
|
||||
if kv.ElemType == btString {
|
||||
out.Preview = decodeStringArray(kv.Raw, 3)
|
||||
}
|
||||
return
|
||||
}
|
||||
switch kv.ElemType {
|
||||
case btString:
|
||||
out.Preview = decodeStringArray(kv.Raw, kv.Count)
|
||||
case btFloat32:
|
||||
n := int(kv.Count)
|
||||
for i := 0; i < n; i++ {
|
||||
bits := binary.LittleEndian.Uint32(kv.Raw[i*4:])
|
||||
out.Numbers = append(out.Numbers, float64(math.Float32frombits(bits)))
|
||||
}
|
||||
case btInt32:
|
||||
n := int(kv.Count)
|
||||
for i := 0; i < n; i++ {
|
||||
out.Numbers = append(out.Numbers, float64(int32(binary.LittleEndian.Uint32(kv.Raw[i*4:]))))
|
||||
}
|
||||
case btUint32:
|
||||
n := int(kv.Count)
|
||||
for i := 0; i < n; i++ {
|
||||
out.Numbers = append(out.Numbers, float64(binary.LittleEndian.Uint32(kv.Raw[i*4:])))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func decodeStringArray(raw []byte, limit uint64) []string {
|
||||
var out []string
|
||||
off := 0
|
||||
for i := uint64(0); i < limit; i++ {
|
||||
if off+8 > len(raw) {
|
||||
break
|
||||
}
|
||||
n := binary.LittleEndian.Uint64(raw[off:])
|
||||
off += 8
|
||||
if off+int(n) > len(raw) {
|
||||
break
|
||||
}
|
||||
out = append(out, string(raw[off:off+int(n)]))
|
||||
off += int(n)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// dump
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func cmdDump(args []string) error {
|
||||
fs := flag.NewFlagSet("dump", flag.ExitOnError)
|
||||
ggufPath := fs.String("gguf", "", "reference GGUF path")
|
||||
outPath := fs.String("o", "", "output JSON manifest path (default stdout)")
|
||||
fs.Parse(args)
|
||||
if *ggufPath == "" {
|
||||
return fmt.Errorf("dump: -gguf is required")
|
||||
}
|
||||
m, err := buildManifest(*ggufPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.MarshalIndent(m, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b = append(b, '\n')
|
||||
if *outPath == "" {
|
||||
_, err = os.Stdout.Write(b)
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(*outPath, b, 0o644)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// shared helpers for convert/verify
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// btypeFromName maps the manifest type name back to the raw enum.
|
||||
func btypeFromName(s string) (ggufBType, error) {
|
||||
for _, t := range []ggufBType{btUint8, btInt8, btUint16, btInt16, btUint32,
|
||||
btInt32, btFloat32, btBool, btString, btArray, btUint64, btInt64, btFloat64} {
|
||||
if t.name() == s {
|
||||
return t, nil
|
||||
}
|
||||
}
|
||||
return 0, fmt.Errorf("unknown btype %q", s)
|
||||
}
|
||||
|
||||
// kvToValue converts a rawKV into a go-gguf write Value, preserving the wire
|
||||
// representation exactly (arrays keep ElemType + count + raw element bytes).
|
||||
func kvToValue(kv rawKV) (gguf.Value, error) {
|
||||
switch kv.BType {
|
||||
case btBool:
|
||||
return gguf.Value{BType: gguf.BTypeBool, Int: kv.Int}, nil
|
||||
case btUint8:
|
||||
return gguf.Value{BType: gguf.BTypeUint8, Int: kv.Int}, nil
|
||||
case btInt8:
|
||||
return gguf.Value{BType: gguf.BTypeInt8, Int: kv.Int}, nil
|
||||
case btUint16:
|
||||
return gguf.Value{BType: gguf.BTypeUint16, Int: kv.Int}, nil
|
||||
case btInt16:
|
||||
return gguf.Value{BType: gguf.BTypeInt16, Int: kv.Int}, nil
|
||||
case btUint32:
|
||||
return gguf.Value{BType: gguf.BTypeUint32, Int: kv.Int}, nil
|
||||
case btInt32:
|
||||
return gguf.Value{BType: gguf.BTypeInt32, Int: kv.Int}, nil
|
||||
case btUint64:
|
||||
return gguf.Value{BType: gguf.BTypeUint64, Int: kv.Int}, nil
|
||||
case btInt64:
|
||||
return gguf.Value{BType: gguf.BTypeInt64, Int: kv.Int}, nil
|
||||
case btFloat32:
|
||||
return gguf.Value{BType: gguf.BTypeFloat32, Float: kv.Float}, nil
|
||||
case btFloat64:
|
||||
return gguf.Value{BType: gguf.BTypeFloat64, Float: kv.Float}, nil
|
||||
case btString:
|
||||
return gguf.Value{BType: gguf.BTypeString, Str: kv.Str}, nil
|
||||
case btArray:
|
||||
return gguf.Value{
|
||||
BType: gguf.BTypeArray,
|
||||
ElemType: btypeToGguf(kv.ElemType),
|
||||
Int: int64(kv.Count),
|
||||
Raw: kv.Raw,
|
||||
}, nil
|
||||
default:
|
||||
return gguf.Value{}, fmt.Errorf("unsupported btype %d for key %q", kv.BType, kv.Key)
|
||||
}
|
||||
}
|
||||
|
||||
func btypeToGguf(t ggufBType) gguf.BType {
|
||||
return gguf.BType(t)
|
||||
}
|
||||
|
||||
func sha256Hex(b []byte) string {
|
||||
s := sha256.Sum256(b)
|
||||
return fmt.Sprintf("%x", s)
|
||||
}
|
||||
|
||||
func shortHash(b []byte) string {
|
||||
s := sha256Hex(b)
|
||||
if len(s) > 12 {
|
||||
return s[:12]
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
package main
|
||||
|
||||
// rawkv.go implements a minimal read-only GGUF metadata (KV) section walker.
|
||||
//
|
||||
// Why not go-gguf for this: the library's MetadataEntry.Value() deliberately
|
||||
// returns an empty Value for BTypeArray (see readLazyValue in lazy.go), which is
|
||||
// fine for scalar/string consumers but loses the tokenizer arrays
|
||||
// (tokenizer.ggml.tokens / scores / token_type / merges). To reproduce the
|
||||
// reference file's metadata byte-for-byte we need the raw array element bytes.
|
||||
// go-gguf is still used for all tensor reading/writing and for scalar metadata;
|
||||
// this walker only fills the array gap. It never writes GGUF.
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
"math"
|
||||
"os"
|
||||
)
|
||||
|
||||
// ggufBType wire enum (mirrors gguf.BType).
|
||||
type ggufBType uint32
|
||||
|
||||
const (
|
||||
btUint8 ggufBType = 0
|
||||
btInt8 ggufBType = 1
|
||||
btUint16 ggufBType = 2
|
||||
btInt16 ggufBType = 3
|
||||
btUint32 ggufBType = 4
|
||||
btInt32 ggufBType = 5
|
||||
btFloat32 ggufBType = 6
|
||||
btBool ggufBType = 7
|
||||
btString ggufBType = 8
|
||||
btArray ggufBType = 9
|
||||
btUint64 ggufBType = 10
|
||||
btInt64 ggufBType = 11
|
||||
btFloat64 ggufBType = 12
|
||||
)
|
||||
|
||||
func (t ggufBType) name() string {
|
||||
switch t {
|
||||
case btUint8:
|
||||
return "UINT8"
|
||||
case btInt8:
|
||||
return "INT8"
|
||||
case btUint16:
|
||||
return "UINT16"
|
||||
case btInt16:
|
||||
return "INT16"
|
||||
case btUint32:
|
||||
return "UINT32"
|
||||
case btInt32:
|
||||
return "INT32"
|
||||
case btFloat32:
|
||||
return "FLOAT32"
|
||||
case btBool:
|
||||
return "BOOL"
|
||||
case btString:
|
||||
return "STRING"
|
||||
case btArray:
|
||||
return "ARRAY"
|
||||
case btUint64:
|
||||
return "UINT64"
|
||||
case btInt64:
|
||||
return "INT64"
|
||||
case btFloat64:
|
||||
return "FLOAT64"
|
||||
default:
|
||||
return fmt.Sprintf("BTYPE(%d)", uint32(t))
|
||||
}
|
||||
}
|
||||
|
||||
func (t ggufBType) size() int {
|
||||
switch t {
|
||||
case btUint8, btInt8, btBool:
|
||||
return 1
|
||||
case btUint16, btInt16:
|
||||
return 2
|
||||
case btUint32, btInt32, btFloat32:
|
||||
return 4
|
||||
case btUint64, btInt64, btFloat64:
|
||||
return 8
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// rawKV is one metadata entry with its raw wire value preserved.
|
||||
type rawKV struct {
|
||||
Key string
|
||||
BType ggufBType
|
||||
Int int64 // integer/bool scalars
|
||||
Float float64 // float scalars
|
||||
Str string // string scalars
|
||||
ElemType ggufBType // arrays: element type
|
||||
Count uint64 // arrays: element count
|
||||
Raw []byte // arrays: concatenated element bytes (excludes elem_type+count)
|
||||
}
|
||||
|
||||
// rawKVHeader holds the GGUF header fields.
|
||||
type rawKVHeader struct {
|
||||
Version uint32
|
||||
NTensors uint64
|
||||
NKV uint64
|
||||
}
|
||||
|
||||
// readRawKV opens path and returns the header plus every KV entry in file order.
|
||||
func readRawKV(path string) (rawKVHeader, []rawKV, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return rawKVHeader{}, nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var hdr rawKVHeader
|
||||
var b [24]byte
|
||||
if _, err := io.ReadFull(f, b[:]); err != nil {
|
||||
return hdr, nil, fmt.Errorf("read header: %w", err)
|
||||
}
|
||||
if string(b[0:4]) != "GGUF" {
|
||||
return hdr, nil, fmt.Errorf("bad magic %q", b[0:4])
|
||||
}
|
||||
hdr.Version = binary.LittleEndian.Uint32(b[4:8])
|
||||
hdr.NTensors = binary.LittleEndian.Uint64(b[8:16])
|
||||
hdr.NKV = binary.LittleEndian.Uint64(b[16:24])
|
||||
|
||||
br := newBufReader(f)
|
||||
out := make([]rawKV, 0, hdr.NKV)
|
||||
for i := uint64(0); i < hdr.NKV; i++ {
|
||||
kv, err := readOneKV(br)
|
||||
if err != nil {
|
||||
return hdr, nil, fmt.Errorf("kv[%d]: %w", i, err)
|
||||
}
|
||||
out = append(out, kv)
|
||||
}
|
||||
return hdr, out, nil
|
||||
}
|
||||
|
||||
func readOneKV(br *bufReader) (rawKV, error) {
|
||||
var kv rawKV
|
||||
key, err := br.str()
|
||||
if err != nil {
|
||||
return kv, fmt.Errorf("key: %w", err)
|
||||
}
|
||||
kv.Key = key
|
||||
bt, err := br.u32()
|
||||
if err != nil {
|
||||
return kv, fmt.Errorf("btype: %w", err)
|
||||
}
|
||||
kv.BType = ggufBType(bt)
|
||||
|
||||
switch kv.BType {
|
||||
case btBool, btUint8, btInt8:
|
||||
v, err := br.u8()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
kv.Int = int64(v)
|
||||
case btUint16, btInt16:
|
||||
v, err := br.u16()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
if kv.BType == btInt16 {
|
||||
kv.Int = int64(int16(v))
|
||||
} else {
|
||||
kv.Int = int64(v)
|
||||
}
|
||||
case btUint32, btInt32:
|
||||
v, err := br.u32()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
if kv.BType == btInt32 {
|
||||
kv.Int = int64(int32(v))
|
||||
} else {
|
||||
kv.Int = int64(v)
|
||||
}
|
||||
case btFloat32:
|
||||
v, err := br.u32()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
kv.Float = float64(math.Float32frombits(v))
|
||||
case btUint64, btInt64:
|
||||
v, err := br.u64()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
kv.Int = int64(v)
|
||||
case btFloat64:
|
||||
v, err := br.u64()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
kv.Float = math.Float64frombits(v)
|
||||
case btString:
|
||||
s, err := br.str()
|
||||
if err != nil {
|
||||
return kv, err
|
||||
}
|
||||
kv.Str = s
|
||||
case btArray:
|
||||
et, err := br.u32()
|
||||
if err != nil {
|
||||
return kv, fmt.Errorf("array elem_type: %w", err)
|
||||
}
|
||||
kv.ElemType = ggufBType(et)
|
||||
cnt, err := br.u64()
|
||||
if err != nil {
|
||||
return kv, fmt.Errorf("array count: %w", err)
|
||||
}
|
||||
kv.Count = cnt
|
||||
raw, err := readArrayElems(br, kv.ElemType, cnt)
|
||||
if err != nil {
|
||||
return kv, fmt.Errorf("array elems: %w", err)
|
||||
}
|
||||
kv.Raw = raw
|
||||
default:
|
||||
return kv, fmt.Errorf("unsupported btype %d", kv.BType)
|
||||
}
|
||||
return kv, nil
|
||||
}
|
||||
|
||||
// readArrayElems reads `count` elements of elemType, returning their concatenated
|
||||
// wire bytes (exactly the bytes go-gguf's writer expects in Value.Raw).
|
||||
func readArrayElems(br *bufReader, elemType ggufBType, count uint64) ([]byte, error) {
|
||||
if elemType == btArray || elemType == btString {
|
||||
// strings (and nested arrays, unsupported) are variable length
|
||||
if elemType != btString {
|
||||
return nil, fmt.Errorf("nested array element type unsupported")
|
||||
}
|
||||
var out []byte
|
||||
for i := uint64(0); i < count; i++ {
|
||||
slen, err := br.u64()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, leU64(slen)...)
|
||||
buf := make([]byte, slen)
|
||||
if _, err := io.ReadFull(br.r, buf); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, buf...)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
sz := elemType.size()
|
||||
if sz == 0 {
|
||||
return nil, fmt.Errorf("unknown array element type %d", elemType)
|
||||
}
|
||||
total := int(count) * sz
|
||||
// sanity cap: 256 MiB
|
||||
if uint64(total) > 256<<20 {
|
||||
return nil, fmt.Errorf("array too large: %d bytes", total)
|
||||
}
|
||||
buf := make([]byte, total)
|
||||
if _, err := io.ReadFull(br.r, buf); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buf, nil
|
||||
}
|
||||
|
||||
// rawKVHash is a stable fingerprint of a KV entry's full wire value.
|
||||
func (kv rawKV) rawKVHash() string {
|
||||
h := crc32.NewIEEE()
|
||||
h.Write([]byte(kv.Key))
|
||||
var b [8]byte
|
||||
binary.LittleEndian.PutUint32(b[:4], uint32(kv.BType))
|
||||
h.Write(b[:4])
|
||||
switch kv.BType {
|
||||
case btBool, btUint8, btInt8, btUint16, btInt16, btUint32, btInt32, btUint64, btInt64:
|
||||
binary.LittleEndian.PutUint64(b[:], uint64(kv.Int))
|
||||
h.Write(b[:])
|
||||
case btFloat32:
|
||||
binary.LittleEndian.PutUint32(b[:], math.Float32bits(float32(kv.Float)))
|
||||
h.Write(b[:4])
|
||||
case btFloat64:
|
||||
binary.LittleEndian.PutUint64(b[:], math.Float64bits(kv.Float))
|
||||
h.Write(b[:])
|
||||
case btString:
|
||||
h.Write([]byte(kv.Str))
|
||||
case btArray:
|
||||
binary.LittleEndian.PutUint32(b[:4], uint32(kv.ElemType))
|
||||
h.Write(b[:4])
|
||||
binary.LittleEndian.PutUint64(b[:], kv.Count)
|
||||
h.Write(b[:])
|
||||
h.Write(kv.Raw)
|
||||
}
|
||||
return fmt.Sprintf("%08x", h.Sum32())
|
||||
}
|
||||
|
||||
// --- tiny buffered reader helpers ---
|
||||
|
||||
type bufReader struct {
|
||||
r io.Reader
|
||||
}
|
||||
|
||||
func newBufReader(r io.Reader) *bufReader { return &bufReader{r: r} }
|
||||
|
||||
func (b *bufReader) u8() (uint8, error) {
|
||||
var x [1]byte
|
||||
if _, err := io.ReadFull(b.r, x[:]); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return x[0], nil
|
||||
}
|
||||
|
||||
func (b *bufReader) u16() (uint16, error) {
|
||||
var x [2]byte
|
||||
if _, err := io.ReadFull(b.r, x[:]); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return binary.LittleEndian.Uint16(x[:]), nil
|
||||
}
|
||||
|
||||
func (b *bufReader) u32() (uint32, error) {
|
||||
var x [4]byte
|
||||
if _, err := io.ReadFull(b.r, x[:]); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return binary.LittleEndian.Uint32(x[:]), nil
|
||||
}
|
||||
|
||||
func (b *bufReader) u64() (uint64, error) {
|
||||
var x [8]byte
|
||||
if _, err := io.ReadFull(b.r, x[:]); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return binary.LittleEndian.Uint64(x[:]), nil
|
||||
}
|
||||
|
||||
func (b *bufReader) str() (string, error) {
|
||||
n, err := b.u64()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if n > 1<<28 {
|
||||
return "", fmt.Errorf("string too long: %d", n)
|
||||
}
|
||||
buf := make([]byte, n)
|
||||
if _, err := io.ReadFull(b.r, buf); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(buf), nil
|
||||
}
|
||||
|
||||
func leU64(v uint64) []byte {
|
||||
var b [8]byte
|
||||
binary.LittleEndian.PutUint64(b[:], v)
|
||||
return b[:]
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
package main
|
||||
|
||||
// safetensors.go — minimal read-only HuggingFace safetensors index.
|
||||
//
|
||||
// Format: [u64 LE header_len][header_len bytes of JSON][tensor data ...].
|
||||
// The JSON header maps tensor name -> {dtype, shape, data_offsets:[start,end]}
|
||||
// with data_offsets relative to the start of the data section.
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
)
|
||||
|
||||
type stTensor struct {
|
||||
Dtype string `json:"dtype"`
|
||||
Shape []int64 `json:"shape"`
|
||||
Start int64 `json:"-"`
|
||||
End int64 `json:"-"`
|
||||
}
|
||||
|
||||
type stHeader struct {
|
||||
Dtype string `json:"dtype"`
|
||||
Shape []int64 `json:"shape"`
|
||||
DataOffsets [2]int64 `json:"data_offsets"`
|
||||
Extra json.RawMessage `json:"-"`
|
||||
}
|
||||
|
||||
type safetensors struct {
|
||||
path string
|
||||
f *os.File
|
||||
dataStart int64
|
||||
Meta map[string]string
|
||||
Order []string
|
||||
Tensors map[string]stTensor
|
||||
}
|
||||
|
||||
func openSafetensors(path string) (*safetensors, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var lenBuf [8]byte
|
||||
if _, err := io.ReadFull(f, lenBuf[:]); err != nil {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("read header length: %w", err)
|
||||
}
|
||||
hlen := binary.LittleEndian.Uint64(lenBuf[:])
|
||||
if hlen == 0 || hlen > 1<<30 {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("implausible safetensors header length %d", hlen)
|
||||
}
|
||||
raw := make([]byte, hlen)
|
||||
if _, err := io.ReadFull(f, raw); err != nil {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("read header json: %w", err)
|
||||
}
|
||||
|
||||
// Top-level: either "__metadata__" (object of strings) or a tensor descriptor.
|
||||
var top map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &top); err != nil {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("parse header json: %w", err)
|
||||
}
|
||||
|
||||
st := &safetensors{
|
||||
path: path,
|
||||
f: f,
|
||||
dataStart: int64(8 + hlen),
|
||||
Meta: map[string]string{},
|
||||
Tensors: map[string]stTensor{},
|
||||
}
|
||||
if metaRaw, ok := top["__metadata__"]; ok {
|
||||
var m map[string]string
|
||||
if err := json.Unmarshal(metaRaw, &m); err == nil {
|
||||
st.Meta = m
|
||||
}
|
||||
delete(top, "__metadata__")
|
||||
}
|
||||
for name, rawDesc := range top {
|
||||
var h stHeader
|
||||
if err := json.Unmarshal(rawDesc, &h); err != nil {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("tensor %q: %w", name, err)
|
||||
}
|
||||
st.Tensors[name] = stTensor{
|
||||
Dtype: h.Dtype,
|
||||
Shape: h.Shape,
|
||||
Start: h.DataOffsets[0],
|
||||
End: h.DataOffsets[1],
|
||||
}
|
||||
st.Order = append(st.Order, name)
|
||||
}
|
||||
return st, nil
|
||||
}
|
||||
|
||||
func (s *safetensors) Close() error {
|
||||
if s.f != nil {
|
||||
return s.f.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// readTensor returns the raw little-endian bytes of a tensor.
|
||||
func (s *safetensors) readTensor(name string) ([]byte, stTensor, error) {
|
||||
t, ok := s.Tensors[name]
|
||||
if !ok {
|
||||
return nil, stTensor{}, fmt.Errorf("no tensor %q", name)
|
||||
}
|
||||
n := t.End - t.Start
|
||||
if n < 0 {
|
||||
return nil, t, fmt.Errorf("tensor %q: negative size", name)
|
||||
}
|
||||
buf := make([]byte, n)
|
||||
if _, err := s.f.ReadAt(buf, s.dataStart+t.Start); err != nil {
|
||||
return nil, t, fmt.Errorf("read %q: %w", name, err)
|
||||
}
|
||||
return buf, t, nil
|
||||
}
|
||||
|
||||
// numElems returns the product of shape dimensions.
|
||||
func numElems(shape []int64) int64 {
|
||||
n := int64(1)
|
||||
for _, d := range shape {
|
||||
n *= d
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// stDtypeBytes returns bytes per element for the supported safetensors dtypes.
|
||||
func stDtypeBytes(dtype string) (int, error) {
|
||||
switch dtype {
|
||||
case "F32":
|
||||
return 4, nil
|
||||
case "F16":
|
||||
return 2, nil
|
||||
case "BF16":
|
||||
return 2, nil
|
||||
case "I64":
|
||||
return 8, nil
|
||||
case "I32":
|
||||
return 4, nil
|
||||
case "I8":
|
||||
return 1, nil
|
||||
case "U8":
|
||||
return 1, nil
|
||||
case "F64":
|
||||
return 8, nil
|
||||
default:
|
||||
return 0, fmt.Errorf("unsupported safetensors dtype %q", dtype)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
package main
|
||||
|
||||
// verify.go — prove the emitted GGUF equals the reference: identical metadata
|
||||
// (keys, types, values; verified on raw wire bytes) and identical tensor set,
|
||||
// shapes, ggml types and per-tensor payload bytes.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
gguf "github.com/cymertek/go-gguf"
|
||||
)
|
||||
|
||||
type verifyResult struct {
|
||||
MetadataChecked int
|
||||
TensorsChecked int
|
||||
ByteEqual int
|
||||
MetaMismatches []string
|
||||
TensorMismatch []string
|
||||
OrderDiffers bool
|
||||
}
|
||||
|
||||
func (r *verifyResult) OK() bool {
|
||||
return len(r.MetaMismatches) == 0 && len(r.TensorMismatch) == 0 && !r.OrderDiffers
|
||||
}
|
||||
|
||||
func compareGGUF(refPath, gotPath string) (*verifyResult, error) {
|
||||
res := &verifyResult{}
|
||||
|
||||
// --- metadata on raw wire bytes ---
|
||||
_, refKVs, err := readRawKV(refPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read ref kv: %w", err)
|
||||
}
|
||||
_, gotKVs, err := readRawKV(gotPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read got kv: %w", err)
|
||||
}
|
||||
refMap := map[string]rawKV{}
|
||||
for _, kv := range refKVs {
|
||||
refMap[kv.Key] = kv
|
||||
}
|
||||
gotMap := map[string]rawKV{}
|
||||
for _, kv := range gotKVs {
|
||||
gotMap[kv.Key] = kv
|
||||
}
|
||||
for i, rk := range refKVs {
|
||||
gk, ok := gotMap[rk.Key]
|
||||
if !ok {
|
||||
res.MetaMismatches = append(res.MetaMismatches, fmt.Sprintf("missing key %q", rk.Key))
|
||||
continue
|
||||
}
|
||||
if i < len(gotKVs) && gotKVs[i].Key != rk.Key {
|
||||
res.OrderDiffers = true
|
||||
}
|
||||
switch {
|
||||
case gk.BType != rk.BType:
|
||||
res.MetaMismatches = append(res.MetaMismatches, fmt.Sprintf("%s: type %s != %s", rk.Key, gk.BType.name(), rk.BType.name()))
|
||||
case gk.rawKVHash() != rk.rawKVHash():
|
||||
res.MetaMismatches = append(res.MetaMismatches, fmt.Sprintf("%s: value bytes differ", rk.Key))
|
||||
}
|
||||
res.MetadataChecked++
|
||||
}
|
||||
for _, gk := range gotKVs {
|
||||
if _, ok := refMap[gk.Key]; !ok {
|
||||
res.MetaMismatches = append(res.MetaMismatches, fmt.Sprintf("extra key %q", gk.Key))
|
||||
}
|
||||
}
|
||||
|
||||
// --- tensor metadata ---
|
||||
refT, err := loadTensorSet(refPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
gotT, err := loadTensorSet(gotPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(refT.order) != len(gotT.order) {
|
||||
res.TensorMismatch = append(res.TensorMismatch, fmt.Sprintf("tensor count %d != %d", len(gotT.order), len(refT.order)))
|
||||
}
|
||||
for i, name := range refT.order {
|
||||
if i < len(gotT.order) && gotT.order[i] != name {
|
||||
res.OrderDiffers = true
|
||||
}
|
||||
ri := refT.byName[name]
|
||||
gi, ok := gotT.byName[name]
|
||||
if !ok {
|
||||
res.TensorMismatch = append(res.TensorMismatch, fmt.Sprintf("missing tensor %s", name))
|
||||
continue
|
||||
}
|
||||
if !shapesEqual(ri.Shape, gi.Shape) {
|
||||
res.TensorMismatch = append(res.TensorMismatch, fmt.Sprintf("%s: shape %v != %v", name, gi.Shape, ri.Shape))
|
||||
continue
|
||||
}
|
||||
if ri.GgmlType != gi.GgmlType {
|
||||
res.TensorMismatch = append(res.TensorMismatch, fmt.Sprintf("%s: type %s != %s", name, gi.GgmlType.GgmlName(), ri.GgmlType.GgmlName()))
|
||||
continue
|
||||
}
|
||||
res.TensorsChecked++
|
||||
}
|
||||
for _, name := range gotT.order {
|
||||
if _, ok := refT.byName[name]; !ok {
|
||||
res.TensorMismatch = append(res.TensorMismatch, fmt.Sprintf("extra tensor %s", name))
|
||||
}
|
||||
}
|
||||
|
||||
// --- per-tensor payload bytes ---
|
||||
refG, err := gguf.Open(refPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer refG.Close()
|
||||
gotG, err := gguf.Open(gotPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer gotG.Close()
|
||||
|
||||
refHandles := map[string]*gguf.Tensor{}
|
||||
for _, t := range mustTensors(refG) {
|
||||
refHandles[t.Info().Name] = t
|
||||
}
|
||||
gotHandles := map[string]*gguf.Tensor{}
|
||||
for _, t := range mustTensors(gotG) {
|
||||
gotHandles[t.Info().Name] = t
|
||||
}
|
||||
for _, name := range refT.order {
|
||||
rh, ok := refHandles[name]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
gh, ok := gotHandles[name]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
eq, diffOff, err := streamsEqual(rh.Reader(), gh.Reader())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("compare %s: %w", name, err)
|
||||
}
|
||||
if !eq {
|
||||
res.TensorMismatch = append(res.TensorMismatch, fmt.Sprintf("%s: payload differs at byte %d", name, diffOff))
|
||||
continue
|
||||
}
|
||||
res.ByteEqual++
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// tensorSet is an ordered view of a GGUF's tensor metadata.
|
||||
type tensorSet struct {
|
||||
order []string
|
||||
byName map[string]gguf.TensorInfo
|
||||
}
|
||||
|
||||
func loadTensorSet(path string) (*tensorSet, error) {
|
||||
g, err := gguf.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer g.Close()
|
||||
ts, err := g.Tensors()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := &tensorSet{byName: map[string]gguf.TensorInfo{}}
|
||||
for _, t := range ts {
|
||||
info := t.Info()
|
||||
out.byName[info.Name] = info
|
||||
out.order = append(out.order, info.Name)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func mustTensors(g *gguf.GGUF) []*gguf.Tensor {
|
||||
ts, err := g.Tensors()
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return ts
|
||||
}
|
||||
|
||||
func shapesEqual(a, b []uint64) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// streamsEqual compares two readers byte-by-byte, returning the offset of the
|
||||
// first difference.
|
||||
func streamsEqual(a, b io.Reader) (bool, int64, error) {
|
||||
const chunk = 1 << 20
|
||||
ba := make([]byte, chunk)
|
||||
bb := make([]byte, chunk)
|
||||
var off int64
|
||||
for {
|
||||
na, ea := io.ReadFull(a, ba)
|
||||
nb, eb := io.ReadFull(b, bb)
|
||||
if na != nb {
|
||||
return false, off + int64(min(na, nb)), nil
|
||||
}
|
||||
if !bytes.Equal(ba[:na], bb[:nb]) {
|
||||
for i := 0; i < na; i++ {
|
||||
if ba[i] != bb[i] {
|
||||
return false, off + int64(i), nil
|
||||
}
|
||||
}
|
||||
}
|
||||
off += int64(na)
|
||||
if ea == io.EOF || ea == io.ErrUnexpectedEOF {
|
||||
endB := eb == io.EOF || eb == io.ErrUnexpectedEOF
|
||||
return endB, off, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func cmdVerify(args []string) error {
|
||||
fs := flag.NewFlagSet("verify", flag.ExitOnError)
|
||||
ref := fs.String("ref", "", "reference GGUF")
|
||||
got := fs.String("got", "", "candidate GGUF")
|
||||
fs.Parse(args)
|
||||
if *ref == "" || *got == "" {
|
||||
return fmt.Errorf("verify: -ref and -got are required")
|
||||
}
|
||||
res, err := compareGGUF(*ref, *got)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("metadata: %d checked, %d mismatches\n", res.MetadataChecked, len(res.MetaMismatches))
|
||||
fmt.Printf("tensors: %d checked, %d byte-identical, %d mismatches\n", res.TensorsChecked, res.ByteEqual, len(res.TensorMismatch))
|
||||
if res.OrderDiffers {
|
||||
fmt.Println("WARNING: metadata or tensor order differs from reference")
|
||||
}
|
||||
for _, m := range res.MetaMismatches {
|
||||
fmt.Println(" META ", m)
|
||||
}
|
||||
for _, m := range res.TensorMismatch {
|
||||
fmt.Println(" TENSOR", m)
|
||||
}
|
||||
if !res.OK() {
|
||||
return fmt.Errorf("verification FAILED")
|
||||
}
|
||||
fmt.Printf("verification PASSED: %d metadata keys and %d tensors byte-identical\n", res.MetadataChecked, res.ByteEqual)
|
||||
return nil
|
||||
}
|
||||
Reference in New Issue
Block a user