49 lines
2.1 KiB
Markdown
49 lines
2.1 KiB
Markdown
# Benchmarks
|
||
|
||
Speed of **open-1b** on our hardware, with a reference comparison against the
|
||
model we run in production (`gemma-4-E2B`). All numbers are single-stream
|
||
(one request at a time) on one GPU.
|
||
|
||
## Hardware / setup
|
||
|
||
- **GPU:** 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp `build-vulkan` (b10893).
|
||
- **CPU:** AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM.
|
||
- **Serving flags:** `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`.
|
||
- `-fa off` is mandatory on these Arc/Vulkan builds → KV cache is f16 (no `-ctk q8_0`).
|
||
|
||
## open-1b (our Go-built GGUF, f16)
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Decode | **~112–117 tok/s** (e.g. 256 tok @ 111.7; 96 tok @ 117.2) |
|
||
| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) |
|
||
| VRAM | **~3.6 GiB** (10971 → 7258 MiB free) |
|
||
| Context | 4096 (model limit; the requested 32k is capped) |
|
||
| Weights | f16, ~3.2 GB |
|
||
|
||
## Reference comparison (1× Arc B580)
|
||
|
||
| Model | Weights | Context | Decode | Prefill |
|
||
|---|---|---|---|---|
|
||
| **open-1b-sft** (this repo) | f16 ~3.2 GB | 4096 | ~114 tok/s | ~892 tok/s |
|
||
| **gemma-4-E2B** (prod) | Q4 ~1.5 GiB | 64k | **~214 tok/s** | **~1659 tok/s** |
|
||
|
||
*gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok /
|
||
prefill 1698 tok) on the same card and build.*
|
||
|
||
**Reading:** gemma is roughly **2× faster**, ~**2.4× smaller**, and has **16×
|
||
the context** of open-1b. For any practical workload (classification, extraction,
|
||
translation, tool-calling) gemma wins outright. open-1b's reason to exist is not
|
||
throughput or quality but **auditable training**.
|
||
|
||
## Caveats
|
||
|
||
- This is **not** a controlled A/B: the open-1b figures are a short first-run
|
||
measurement with different prompts; the gemma figures come from a separate
|
||
controlled benchmark. Use them as an order-of-magnitude orientation.
|
||
- The `ngram` speculative decoder helps task-dependent amounts; open-1b's
|
||
baseline (no-spec) number was not measured here.
|
||
- Quality: open-1b scores **25.4** on the OLMo2 suite vs **31.9** for OLMo2-1B,
|
||
i.e. below an ordinary 1B model. It is a proof of *verifiability*, not a strong
|
||
assistant.
|