eSlider/open-1b-verify
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite in plain style (no em dashes, no emoji), tighten reports

Browse Source
This commit is contained in:
vad
2026-09-29 07:39:12 +00:00
parent 4fc62ef8d1
commit 6d1f79359a
11 changed files with 392 additions and 654 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmarks.md
+27 -34
View File
@@ -1,48 +1,41 @@
# Benchmarks
Speed of **open-1b** on our hardware, with a reference comparison against the
model we run in production (`gemma-4-E2B`). All numbers are single-stream
(one request at a time) on one GPU.
open-1b on our hardware, with the production model, gemma-4-E2B, as reference.
Single stream, one GPU.
## Hardware / setup
## Setup
- **GPU:** 1× Intel Arc B580 (BMG G21, 12 GB), Vulkan (Mesa), llama.cpp `build-vulkan` (b10893).
- **CPU:** AMD Ryzen 7 7700X (16 vCPU), 29 GB RAM.
- **Serving flags:** `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`.
- `-fa off` is mandatory on these Arc/Vulkan builds → KV cache is f16 (no `-ctk q8_0`).
- GPU: one Intel Arc B580 (BMG G21, 12 GB), Vulkan, llama.cpp build-vulkan (b10893).
- CPU: AMD Ryzen 7 7700X, 29 GB RAM.
- Flags: `-ngl 99 -fa off --jinja --spec-type ngram-map-k,ngram-cache -t 16 --parallel 1`.
- `-fa off` is required on these Arc/Vulkan builds, so the KV cache stays in f16.
## open-1b (our Go-built GGUF, f16)
## open-1b, Go-built GGUF, f16
| Metric | Value |
|---|---|
| Decode | **~112–117 tok/s** (e.g. 256 tok @ 111.7; 96 tok @ 117.2) |
| Prefill | **~892 tok/s** (961 prompt tok in 1077 ms) |
| VRAM | **~3.6 GiB** (10971 → 7258 MiB free) |
| Context | 4096 (model limit; the requested 32k is capped) |
| Weights | f16, ~3.2 GB |
| --- | --- |
| Decode | 112 to 117 tok/s |
| Prefill | about 892 tok/s |
| VRAM | about 3.6 GiB |
| Context | 4096, the model limit |
| Weights | f16, about 3.2 GB |
## Reference comparison (1× Arc B580)
## Reference, one Arc B580
| Model | Weights | Context | Decode | Prefill |
|---|---|---|---|---|
| **open-1b-sft** (this repo) | f16 ~3.2 GB | 4096 | ~114 tok/s | ~892 tok/s |
| **gemma-4-E2B** (prod) | Q4 ~1.5 GiB | 64k | **~214 tok/s** | **~1659 tok/s** |
| --- | --- | --- | --- | --- |
| open-1b-sft | f16, about 3.2 GB | 4096 | about 114 tok/s | about 892 tok/s |
| gemma-4-E2B | Q4, about 1.5 GiB | 64k | about 214 tok/s | about 1659 tok/s |
*gemma figures are from our 2026-09-20 controlled benchmark (decode 256 tok /
prefill 1698 tok) on the same card and build.*
The gemma figures come from our 2026-09-20 benchmark on the same card and build.
**Reading:** gemma is roughly **2× faster**, ~**2.4× smaller**, and has **16×
the context** of open-1b. For any practical workload (classification, extraction,
translation, tool-calling) gemma wins outright. open-1b's reason to exist is not
throughput or quality but **auditable training**.
gemma is about twice as fast, about 2.4 times smaller, and has sixteen times the
context. For classification, extraction, translation, or tool calls, gemma is the
better choice. open-1b is interesting for its training audit, not for its speed.
## Caveats
## Notes
- This is **not** a controlled A/B: the open-1b figures are a short first-run
measurement with different prompts; the gemma figures come from a separate
controlled benchmark. Use them as an order-of-magnitude orientation.
- The `ngram` speculative decoder helps task-dependent amounts; open-1b's
baseline (no-spec) number was not measured here.
- Quality: open-1b scores **25.4** on the OLMo2 suite vs **31.9** for OLMo2-1B,
i.e. below an ordinary 1B model. It is a proof of *verifiability*, not a strong
assistant.
- Not a controlled comparison. The open-1b numbers are a short first run with
different prompts; the gemma numbers come from a separate controlled benchmark.
- No speculative-decoder-off measurement for open-1b.
- Quality: open-1b scores 25.4 on the OLMo2 suite, against 31.9 for OLMo2-1B.