open-1b-verify/docs/benchmarks.md

Benchmarks

Speed of open-1b on our hardware, with a reference comparison against the model we run in production (gemma-4-E2B). All numbers are single-stream (one request at a time) on one GPU.

Hardware / setup

• GPU: 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp build-vulkan (b10893).
• CPU: AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM.
• Serving flags: -ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1.
• -fa off is mandatory on these Arc/Vulkan builds → KV cache is f16 (no -ctk q8_0).

open-1b (our Go-built GGUF, f16)

Metric	Value
Decode	~112–117 tok/s (e.g. 256 tok @ 111.7; 96 tok @ 117.2)
Prefill	~892 tok/s (961 prompt tok in 1077 ms)
VRAM	~3.6 GiB (10971 → 7258 MiB free)
Context	4096 (model limit; the requested 32k is capped)
Weights	f16, ~3.2 GB

Reference comparison (1× Arc B580)

Model	Weights	Context	Decode	Prefill
open-1b-sft (this repo)	f16 ~3.2 GB	4096	~114 tok/s	~892 tok/s
gemma-4-E2B (prod)	Q4 ~1.5 GiB	64k	~214 tok/s	~1659 tok/s

gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok / prefill 1698 tok) on the same card and build.

Reading: gemma is roughly 2× faster, ~2.4× smaller, and has 16× the context of open-1b. For any practical workload (classification, extraction, translation, tool-calling) gemma wins outright. open-1b's reason to exist is not throughput or quality but auditable training.

Caveats

• This is not a controlled A/B: the open-1b figures are a short first-run measurement with different prompts; the gemma figures come from a separate controlled benchmark. Use them as an order-of-magnitude orientation.
• The ngram speculative decoder helps task-dependent amounts; open-1b's baseline (no-spec) number was not measured here.
• Quality: open-1b scores 25.4 on the OLMo2 suite vs 31.9 for OLMo2-1B, i.e. below an ordinary 1B model. It is a proof of verifiability, not a strong assistant.