eSlider/open-1b-verify
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite in plain style (no em dashes, no emoji), tighten reports

Browse Source
This commit is contained in:
vad
2026-09-29 07:39:12 +00:00
parent 4fc62ef8d1
commit 6d1f79359a
11 changed files with 392 additions and 654 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+76 -88
View File
@@ -1,54 +1,49 @@
# open-1b — independent verification & local inference
# open-1b: an independent check, and a local run
An independent, reproducible check of Gensyn's **open-1b** "auditable training"
claim — plus a **pure-Go** path to convert and run the model locally.
We looked at Gensyn's open-1b and its claim of auditable training, then ran the
model on our own hardware. The audit uses Gensyn's official harness. Everything
we wrote is Go.
> Reference: [Introducing open-1b](https://www.gensyn.ai/news/introducing-open-1b-auditable-training)
> · harness: [gensyn-ai/open-transformers](https://github.com/gensyn-ai/open-transformers)
> · audit app: [open1b.gensyn.ai](https://open1b.gensyn.ai/)
Everything we wrote is **Go** (no Python in our code or pipeline). For the audit we
used Gensyn's official harness, which is Python-only and the sole vendor-supplied
path; for inference we wrote our own Go GGUF converter.
Links: [announcement](https://www.gensyn.ai/news/introducing-open-1b-auditable-training),
[harness](https://github.com/gensyn-ai/open-transformers),
[audit app](https://open1b.gensyn.ai/).
## Results
| Item | Result |
|------|--------|
| **Init-unit audit** (Linux x86-64 CPU, no NVIDIA) | ✅ **MATCH** — published `state_hash 16554a11…` reproduced bit-for-bit |
| **Training-interval audit** | ⛔ blocked — no interval unit is published; replay needs ~24 GB RAM / ≥60 GB disk |
| **GGUF conversion** (Go, `tools/hf2gguf`) | ✅ **byte-identical** to the vendor GGUF (same SHA-256) |
| **Local inference** (1× Intel Arc B580, Vulkan) | ✅ ~114 tok/s decode · ~892 tok/s prefill · ~3.6 GiB VRAM |
| Check | Result |
| --- | --- |
| Init-unit audit, x86-64 CPU, no NVIDIA | Match. The published state hash `16554a11...` is reproduced exactly. |
| Training-interval audit | Not run. No interval unit is published, and a replay needs about 24 GB RAM and 60 GB disk. |
| GGUF conversion (Go) | Byte-identical to the reference conversion, same SHA-256. |
| Local inference, one Intel Arc B580 | 114 tok/s decode, 892 tok/s prefill, about 3.6 GiB VRAM. |
Details: [`docs/audit-open1b-init.md`](docs/audit-open1b-init.md),
[`docs/open1b-local-inference.md`](docs/open1b-local-inference.md),
[`docs/benchmarks.md`](docs/benchmarks.md),
[`docs/inference-stack.md`](docs/inference-stack.md).
Reports: [audit](docs/audit-open1b-init.md), [local run](docs/open1b-local-inference.md),
[benchmarks](docs/benchmarks.md), [inference stack](docs/inference-stack.md).
## How it works
**Audit — replay one published commitment and compare.**
Audit. Replay one published commitment and compare hashes.
```mermaid
flowchart LR
KIT["Gensyn audit kit"] --> SHA{"sha256\nvs kit.json"}
SHA --> WHL["repop + pretrain wheels"]
WHL --> REPLAY["pretrain-audit-replay\n--from-init --device cpu"]
REPLAY --> H1["regenerated\ninit state hash"]
KIT --> H2["published\ncommitment"]
H1 --> CMP{"equal?"}
KIT["Gensyn audit kit"] --> SHA{"sha256<br/>against kit.json"}
SHA --> WHL["repop and pretrain wheels"]
WHL --> REPLAY["pretrain-audit-replay<br/>--from-init --device cpu"]
REPLAY --> H1["regenerated init hash"]
KIT --> H2["published commitment"]
H1 --> CMP{"equal"}
H2 --> CMP
CMP -->|yes| OK["MATCH"]
CMP -->|no| BAD["MISMATCH"]
CMP -->|yes| OK["match"]
CMP -->|no| BAD["mismatch"]
```
**Inference — convert in Go, serve on one Arc (the engine is a patched llama.cpp).**
Inference. Convert in Go, serve on one Arc. The engine is a patched llama.cpp.
```mermaid
flowchart LR
HF["HF checkpoint\nmodel.safetensors (fp32)"] --> GO["tools/hf2gguf\n(pure Go)"]
GO --> GGUF["GGUF (f16)\nbyte-identical to reference"]
GGUF --> LC["llama.cpp Vulkan\nupstream + open1b.patch"]
HF["HF checkpoint<br/>model.safetensors, fp32"] --> GO["tools/hf2gguf<br/>Go"]
GO --> GGUF["GGUF, f16<br/>identical to reference"]
GGUF --> LC["llama.cpp Vulkan<br/>upstream plus open1b patch"]
LC --> GPU["Intel Arc B580"]
GPU --> API["llama-server HTTP"]
```
@@ -57,43 +52,44 @@ flowchart LR
```
docs/
audit-open1b-init.md # init-unit audit report (raw evidence)
open1b-local-inference.md # local bring-up report
benchmarks.md # speed benchmarks + reference comparison
inference-stack.md # how llama.cpp was built/launched (NOT stock)
llama-cpp-open1b.patch # local Open1B patch applied to llama.cpp
tools/hf2gguf/ # pure-Go safetensors → GGUF converter (dump/convert/verify)
audit-open1b-init.md
open1b-local-inference.md
benchmarks.md
inference-stack.md
llama-cpp-open1b.patch
tools/hf2gguf/
```
## Audit — what it proves, and what it does not
## The audit, and its limits
**Proves.** The installed artifacts are the ones the kit's `kit.json` published
(all SHA-256s match); the `repop` build commit matches the manifest; and running
the vendor harness on a plain CPU regenerates the `1b_repop_v2` initial state hash
**bit-for-bit** — an independent third party reproduced the run's init commitment.
The init unit regenerates the published initial state hash exactly on a plain
CPU. The installed wheels match the kit manifest, and the repop build commit
matches the manifest.
**Does not prove.**
It does not cover a training step. An init-only unit says nothing about the
forward pass, the backward pass, the optimizer, or the data stream, so the
per-step claim stays untested here. Three more reasons to be careful:
- **No training interval.** An init-only unit says nothing about any forward/backward/AdamW/data-stream step. The claim about bitwise-reproducible *optimizer steps* is not exercised here.
- **`repop` is closed source.** Only compiled wheels ship, under the "Gensyn Reproducibility License v1.0". The kernels that make training reproducible cannot be read or independently re-derived.
- **The v3 hash omits** RNG, the data-stream cursor, spike state, and `meta.json` descriptor keys.
- **No interval unit is published**, so "pick any step, replay it" is not self-serve checkable yet.
- Provenance of the *published run* is not established — only that the pinned artifacts reproduce the published commitment on our machine.
- repop ships as compiled wheels only, under a custom licence. The kernels that
make training reproducible cannot be read or checked.
- The v3 state hash omits the RNG, the data-stream cursor, spike state, and the
meta.json descriptor keys.
- No interval unit is published, so "replay any step" is not yet something an
outsider can do without permission.
## GGUF conversion (pure Go)
A match shows that the pinned artifacts reproduce the published commitment on
our machine. It does not show how the original run was carried out.
`tools/hf2gguf` reads the HF checkpoint (`config.json`, `model.safetensors`,
`tokenizer.json`) and writes a GGUF file, using
[`github.com/cymertek/go-gguf`](https://github.com/cymertek/go-gguf) for GGUF
read/write. A reference GGUF produced by llama.cpp's `convert_hf_to_gguf.py`
**with our [Open1B patch](docs/llama-cpp-open1b.patch)** is used **read-only as
the schema oracle**, so the output is validated byte-for-byte.
## Conversion in Go
`tools/hf2gguf` reads the Hugging Face checkpoint (config.json,
model.safetensors, tokenizer.json) and writes a GGUF file, using
[go-gguf](https://github.com/cymertek/go-gguf). A reference GGUF from llama.cpp's
converter, built with our open1b patch, is read only, as the schema to match.
```bash
cd tools/hf2gguf
go build -o hf2gguf .
./hf2gguf dump -gguf models/open-1b-sft-f16.gguf -o manifest.json
./hf2gguf convert -ref models/open-1b-sft-f16.gguf \
-safetensors models/open-1b-sft/model.safetensors \
-config models/open-1b-sft/config.json \
@@ -103,48 +99,40 @@ go build -o hf2gguf .
./hf2gguf verify -ref models/open-1b-sft-f16.gguf -got models/open-1b-sft-f16-go.gguf
```
`verify` reports `39/39 metadata keys` and `220/220 tensors` byte-identical; the
two files share SHA-256 `a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
`verify` reports 39 of 39 metadata keys and 220 of 220 tensors identical. The two
files share SHA-256
`a18530417c732bb115a3501a7e1388ea7a54bd2a96c70be571d1249aa210fbae`.
## Running the model locally
## Running the model
> **Engine caveat:** open-1b needs an `open1b` architecture that upstream llama.cpp
> does not have. We run a build with a local patch — see
> [`docs/inference-stack.md`](docs/inference-stack.md).
Serve the (or our Go-built) GGUF on one Arc B580 with llama.cpp's Vulkan backend:
Upstream llama.cpp has no open1b architecture, so the model needs a patched
build. Details in the [inference stack](docs/inference-stack.md).
```bash
llama-server \
-m models/open-1b-sft-f16-go.gguf \
llama-server -m models/open-1b-sft-f16-go.gguf \
--device Vulkan2 -ngl 99 -c 32768 -fa off --jinja \
--spec-type ngram-map-k,ngram-cache \
-t 16 --parallel 1 --alias open-1b --host 127.0.0.1 --port 8085
```
Notes: `-fa off` is required on these Arc/Vulkan builds (so KV stays f16); the
requested `-c 32768` is capped to the model's **4096**-token context.
`-fa off` is required on these Arc/Vulkan builds, which keeps the KV cache in
f16. The requested 32k context is capped to the model's 4096.
## Caveats
- **Chat-template quirk.** With the exact vendor tail `…assistant<|end_header|>\n\n`,
the released SFT checkpoint emits `<|eot|>` as its first token (empty reply).
Our GGUF is byte-identical to the vendor one, so this is a property of the
released checkpoint, not of the conversion. Workarounds: seed the assistant
turn, use `ignore_eos`, or fix the trailing newline.
- **Not a strong model.** open-1b scores 25.4 on the OLMo2 suite (vs 31.9 for
OLMo2-1B). Its value is **verifiability**, not quality — see
[`docs/benchmarks.md`](docs/benchmarks.md).
- **Scope.** We only run/inspect the released weights and replay one published
commitment; we do not train or fine-tune anything.
- Chat template. With the vendor tail `...assistant<|end_header|>` followed by a
blank line, the released SFT checkpoint emits an end-of-turn token first and
returns an empty reply. Our GGUF is identical to the reference, so this is a
property of the released weights. Seed the assistant turn, set ignore_eos, or
adjust the trailing newline.
- Model quality. open-1b scores 25.4 on the OLMo2 suite, against 31.9 for
OLMo2-1B. It is a test of verifiability, not a strong model.
- Scope. We run and inspect released weights and replay one published
commitment. We do not train or fine-tune anything.
## License
MIT — see [`LICENSE`](LICENSE). The vendor harness (`open-transformers`) is
Apache-2.0; `repop`, the checkpoints and the corpus are distributed separately by
Gensyn under their own terms.
MIT, see [LICENSE](LICENSE). The harness is Apache-2.0. repop, the checkpoints,
and the corpus are distributed by Gensyn under their own terms.
## Credits
Built on Gensyn's open-1b release and audit tooling. Independent work; not
affiliated with Gensyn.
Built on Gensyn's release. Not affiliated with Gensyn.