Which Small Language Models Actually Run on CPU in 2026: 11 Models Compared
If you are choosing small language models on CPU hardware this month, the benchmark leaderboard is the wrong document to be reading. Nine of the eleven models in this comparison clear the capability bar for ordinary production work — summarisation, extraction, classification, tool-calling, light coding. Capability is no longer the binding constraint in the sub-14B band. What actually decides the choice is a different, unglamorous intersection: whether a first-party quantized build exists, whether the inference binary you can realistically ship will load the architecture at all, and whether the licence is one your legal team will sign. Score the field on that test and the ranking inverts. The model with the best paper numbers becomes the riskiest operational pick, and the model with the least marketing wins.
What this covers: the eleven current candidates with real on-disk sizes, a three-gate decision framework, a decision matrix, the memory-bandwidth arithmetic that turns a file size into a latency budget, and an honest account of what nobody has measured.
Context and Background
The shortlist of small language models on CPU hardware you would have written in late 2025 is now almost entirely stale, and that is the single most important structural fact about this comparison.
Meta has published nothing new in the small band since April 2025. The most recent Llama on the Hub is Llama-4-Scout-17B-16E-Instruct, a 17B-active, 109B-class mixture-of-experts model that has no business on a laptop CPU. There is no current Llama at 1B, 3B or 8B. Microsoft is in a similar position: the newest Phi text model at small scale is Phi-4-mini-instruct, dated 19 February 2025 and now roughly nineteen months old. SmolLM3-3B shipped 8 July 2025 and there is no SmolLM4.
Those three were the default answer to “what runs on a CPU” for most of 2025. All three are still serviceable — Phi-4-mini in particular still carries an MIT licence, the most permissive terms in this entire comparison, and an official ONNX build. But recommending them in September 2026 means recommending a generation-and-a-half-old model against a field that has turned over completely.
The families that are actually current are Qwen3.5 (February 2026), Gemma 4 (March to June 2026), IBM Granite 4.2 (August 2026), MiniCPM5 (August to September 2026), LFM2.5 from Liquid AI (January to August 2026), Ministral 3 from Mistral (October to November 2025, cards refreshed July 2026) and NVIDIA’s Nemotron 3 Nano (March 2026). Seven vendors, none of them the two you would have named a year ago.
That turnover matters because it changed the shape of the decision. The 2025 field was architecturally homogeneous — dense transformers with grouped-query attention, all of which llama.cpp had supported for months. The 2026 field is not. It contains state-space hybrids, gated linear attention, short-convolution blocks and per-layer embedding tables, and each of those is a potential runtime-compatibility failure rather than an interesting design detail. If you want the architectural tour on its own terms, we covered it in the SLM edge architecture breakdown. Here, architecture appears only as risk.
The Three Gates That Actually Decide a CPU Deployment
For CPU-only inference in September 2026, evaluate every candidate against three gates before you look at a single benchmark: does the vendor publish its own GGUF, will a deployable llama.cpp build load the architecture, and is the licence OSI-approved? Models that fail any gate carry operational cost that no benchmark margin repays.
Figure 1: The gate sequence that reorders the CPU-only leaderboard.
Figure 1 runs a candidate under 14 billion parameters through three sequential gates. Gate 1 asks whether the quantized weights come from the model’s own vendor organisation or from a community requantizer. Gate 2 asks whether the architecture uses operators your inference build already supports, or exotic ones that demand a bleeding-edge binary. Gate 3 asks whether the licence is a recognised open-source licence or bespoke vendor terms. Passing all three produces a deployable shortlist; any failure routes the model to legal or engineering review before it can be shipped.
Gate 1: does a first-party GGUF exist?
This is the gate that eliminates the highest-scoring model in the set, and it is the one most shortlists of small language models on CPU skip entirely.
Qwen3.5 publishes no GGUF. Not for the 0.8B, not for the 2B, not for the 4B, not for the 9B. Every quantized Qwen3.5 file you can download today comes from a community repacker — unsloth, bartowski, lmstudio-community. Those are well-regarded operations and the files work. But the provenance chain is: Qwen releases safetensors, a third party converts them with a specific llama.cpp commit and a specific imatrix calibration set, and you deploy the result. Nobody at Qwen has signed that artifact or committed to reproducing it.
Contrast IBM. The ibm-granite/granite-4.2-3b-GGUF and -8b-GGUF repositories carry the full quant ladder from Q2_K through Q8_0 and bf16, published by IBM, with a model.sig signature file and a granite_thinking_parser.py alongside. IBM also ships fp8, mxfp4, nvfp4 and MLX variants. That is a vendor treating quantized CPU deployment as a supported path rather than a community afterthought.
Google, OpenBMB, Liquid AI, Mistral and NVIDIA all publish first-party GGUFs too — with important asymmetries in how much they publish, which the next section quantifies. Qwen is the outlier, and it is the outlier that has the best benchmark numbers.
Gate 2: will your build load it?
Architecture is a compatibility question on CPU, not an aesthetic one.
Qwen3.5’s layer pattern is a hybrid: eight repetitions of three Gated-DeltaNet-plus-FFN blocks followed by one gated-attention-plus-FFN block, across 32 layers. Gated DeltaNet is a state-space-style operator, and supporting it required new llama.cpp operators. Conversion needs a recent build with --fuse-gate-up-exp, as discussed in the GGUF conversion thread. Community GGUFs exist and work today. A llama.cpp binary from six months ago will not load them. If your deployment target is a fleet of machines running a pinned, security-reviewed build, that is a real problem and not a theoretical one.
LFM2.5 is a second case. Its 2.6B model is a dense hybrid of 30 layers — 22 double-gated short-convolution blocks and 8 grouped-query-attention blocks. Those short-convolution tensors are exactly what llama.cpp issue #19184 reports failing: “LFM2(.5) series of models fails with CPU-only inference when built with AMX_INT8 support.” The report is against an Intel Xeon with AMX_INT8 = 1 on llama.cpp version 7875, and the log shows the AMX backend repacking shortconv.in_proj and shortconv.out_proj. The issue was closed as not planned. Whether it still reproduces on current builds is unknown — no fix commit was found, and no one has stated it is resolved.
NVIDIA’s Nemotron 3 Nano 4B is a Mamba-2-transformer hybrid with only four attention layers, compressed from the 9B v2 model via NVIDIA’s Elastic framework. Gemma 4 uses Per-Layer Embeddings, where each decoder layer carries its own small per-token embedding table, plus interleaved local sliding-window and global attention with proportional RoPE on the global layers. Granite 4.2, by contrast, is a plain dense decoder: 40 layers, grouped-query attention with 32 query heads and 8 key-value heads, head size 128, SwiGLU, RMSNorm, RoPE theta of 10,000,000. There is nothing in that description that a two-year-old llama.cpp would choke on.
Gate 2 is therefore an inverse-novelty filter. The more interesting the architecture, the more of your deployment risk sits in the inference binary rather than the weights.
Gate 3: will your lawyer sign it?
Nine of the eleven models are Apache-2.0 or MIT. Two are not, and the difference is decisive for commercial work.
LFM2.5 ships under lfm1.0 — the card declares license: other with license_name: lfm1.0, and each repository carries a 10,574-byte LICENSE file. That licence was not read for this comparison. It is not an OSI-approved licence. This post therefore takes no position on whether it permits your use case, at what revenue threshold, or under what conditions: read the LICENSE file yourself before committing engineering time.
Nemotron 3 Nano carries the NVIDIA Open Model License, again license: other. It is not an OSI-approved licence either. NVIDIA’s card says the model is “ready for commercial use”. The actual terms were not reviewed here, so this post likewise takes no position on what they permit, and the same instruction applies: read them. One detail is worth flagging on its own, because it is the kind of thing that stalls a procurement review — the LICENSE file inside NVIDIA’s own GGUF repository is zero bytes.
On the positive side, one 2026 change deserves a plain statement because it was a genuine surprise. Gemma 4 is Apache-2.0, a substantial departure from the Gemma Terms of Use that governed earlier generations. The model card’s frontmatter reads license: apache-2.0 and links to a Google-hosted page rather than apache.org, which is the sort of indirection that warrants checking. It was checked: that page serves the verbatim, unmodified Apache License 2.0 text — no custom clauses, no acceptable-use rider, no carve-outs. The Google-hosted URL is a hosting choice, nothing more. Gemma 4 is genuinely unencumbered Apache-2.0.
The Eleven Models, Sized in Real Bytes
Every figure in the table below is an exact file size read off the Hugging Face Hub, not a parameter-count estimate. This matters more than it sounds, because marketing parameter counts and on-disk footprints have decoupled.
| Model | Quant | Q4 size (GB) | Q8_0 (GB) | First-party GGUF | Licence |
|---|---|---|---|---|---|
| LFM2.5-230M | Q4_K_M | 0.15 | 0.25 | Yes | lfm1.0 (not OSI) |
| MiniCPM5-1B | Q4_K_M | 0.69 | 1.15 | Yes | Apache-2.0 |
| LFM2.5-1.2B-Instruct | Q4_K_M | 0.73 | 1.25 | Yes | lfm1.0 (not OSI) |
| MiniCPM5-2B | Q4_K_M | 1.56 | 2.68 | Yes | Apache-2.0 |
| LFM2.5-2.6B | Q4_K_M | 1.67 | 2.87 | Yes | lfm1.0 (not OSI) |
| Ministral-3-3B | Q4_K_M | 2.15 | 3.65 | Yes | Apache-2.0 |
| granite-4.2-3b | Q4_K_M | 2.24 | 3.89 | Yes | Apache-2.0 |
| Nemotron-3-Nano-4B | Q4_K_M | 2.84 | not offered | Yes, Q4 only | NVIDIA Open Model |
| Qwen3.5-4B | Q4_K_M | 3.01 | 4.62 | No (bartowski) | Apache-2.0 |
| gemma-4-E2B-it | QAT q4_0 | 3.35 | not offered | Yes, q4_0 only | Apache-2.0 |
| gemma-4-E4B-it | QAT q4_0 | 5.15 | not offered | Yes, q4_0 only | Apache-2.0 |
| LFM2.5-8B-A1B | Q4_K_M | 5.16 | 9.01 | Yes | lfm1.0 (not OSI) |
| Ministral-3-8B | Q4_K_M | 5.20 | 9.03 | Yes | Apache-2.0 |
| granite-4.2-8b | Q4_K_M | 5.35 | 9.35 | Yes | Apache-2.0 |
| Qwen3.5-9B | Q4_K_M | 5.68 | 9.53 | No (unsloth) | Apache-2.0 |
| gemma-4-12B-it | QAT q4_0 | 6.98 | not offered | Yes, q4_0 only | Apache-2.0 |
Three readings of this table are worth making explicit.
Effective parameters are a marketing unit, not a memory unit. Gemma 4 E2B is described as 2.3 billion effective parameters, 5.1 billion with embeddings. Its q4_0 GGUF is 3.35 GB — larger than MiniCPM5-2B’s Q8_0 at 2.68 GB. The Per-Layer Embedding tables inflate the file even though they are consumed as lookups. Size the machine from the file size, never from the parameter count.
Format choice is frequently not yours. Google publishes only q4_0 for Gemma 4 — quantization-aware-trained, which is a real engineering advantage, but there is no Q8_0 and no K-quant. NVIDIA publishes exactly one quant for Nemotron 3 Nano, Q4_K_M. Liquid ships a quantization-aware-distilled QAD-Q4_0 alongside conventional quants. IBM, OpenBMB and Mistral publish full ladders. So the format debate that matters on an NPU — which we covered in the int4 vs int8 vs fp8 comparison — partly evaporates here, replaced by “take what the vendor gave you or requantize it yourself.”
Multimodal projectors are optional weight you probably don’t want. Most of the 2026 small models are multimodal by default, and the vision projector ships as a separate file: Gemma-4-E2B and E4B add 0.99 GB each, Gemma-4-12B adds 0.18 GB, Ministral-3-3B adds 0.84 GB, Ministral-3-8B 0.86 GB, Qwen3.5-4B 0.67 GB, Qwen3.5-9B 0.92 GB. A text-only deployment simply does not download them. On a 2.15 GB Ministral-3-3B, skipping the projector is a 28 percent footprint saving.
The Decision Matrix
Ratings below are qualitative judgements built from the verified facts in this post. They are not scores and they are not derived from a scoring formula. Read them as a shortlisting aid for small language models on CPU, not as a ranking.
| Model | First-party GGUF | Operator risk | Licence | CPU footprint | Benchmark independence | Best fit |
|---|---|---|---|---|---|---|
| granite-4.2-8b | Excellent, signed, full ladder | Very low, dense GQA | Apache-2.0 | 5.35 GB Q4 | Vendor-only, third-party partial | Capable workstation default |
| granite-4.2-3b | Excellent | Very low | Apache-2.0 | 2.24 GB Q4 | Vendor-only, partial cross-check | Mid-range laptop, English-first |
| MiniCPM5-2B | Good, plus MLX/GPTQ/LiteRT-LM | Low, LlamaForCausalLM | Apache-2.0 | 1.56 GB Q4 | Vendor-only, some third-party | Efficiency pick, thin hardware |
| MiniCPM5-1B | Good | Low | Apache-2.0 | 0.69 GB Q4 | Very thin | Classification, extraction |
| gemma-4-E4B-it | Good, QAT, q4_0 only | Medium, PLE and hybrid attention | Apache-2.0 | 5.15 GB Q4 | Vendor-only, disputed | Vendor-backing requirement |
| gemma-4-E2B-it | Good, QAT, q4_0 only | Medium | Apache-2.0 | 3.35 GB Q4 | Vendor-only, disputed | Only if you need audio plus vision |
| gemma-4-12B-it | Good, QAT, q4_0 only | Medium | Apache-2.0 | 6.98 GB Q4 | Vendor-only | Ceiling of the CPU band |
| LFM2.5-2.6B | Excellent, includes QAD | High, see issue #19184 | lfm1.0, not OSI | 1.67 GB Q4 | Vendor plus rival reproduction | CPU-constrained agent |
| Ministral-3-3B | Good, full ladder | Low, dense | Apache-2.0 | 2.15 GB Q4 | Very thin, stale comparators | Apache vision at small size |
| Ministral-3-8B | Good, full ladder | Low, dense | Apache-2.0 | 5.20 GB Q4 | Thin | Apache alternative to Granite |
| Qwen3.5-4B / 9B | None | High, Gated DeltaNet | Apache-2.0 | 3.01 / 5.68 GB | Vendor plus rival reproduction | Teams tracking llama.cpp master |
| Nemotron-3-Nano-4B | Q4_K_M only | High, Mamba-2 hybrid | NVIDIA Open Model | 2.84 GB Q4 | Vendor-only | Not recommended for CPU |
| Phi-4-mini-instruct | Community and ONNX | Very low | MIT | ~1.9 GB Q4 measured | Independently benchmarked | Legacy baseline, permissive licence |
| SmolLM3-3B | Community, ONNX official | Very low | Apache-2.0 | not tabulated | Thin | Superseded 2025 default |
Read the matrix column-wise and the thesis falls out. The “operator risk” and “first-party GGUF” columns are almost perfectly anti-correlated with capability rank: the models with the best paper numbers carry the most deployment risk, and the model with the cleanest operational profile — Granite 4.2 — is the one nobody wrote a launch thread about.
Memory Bandwidth Is the Sizing Lens
The size table means something specific, and it is not disk space. On a CPU, at batch size one, the quantized file size is a latency budget.
Figure 2: Turning a quantized file size into a decode-throughput ceiling.
Figure 2 shows the derivation chain. Peak DRAM bandwidth in gigabytes per second, divided by bytes read per token — approximated by the quantized file size — yields a ceiling in tokens per second. Scaling that ceiling by roughly 0.7 gives a practical estimate on x86 laptop hardware. Three annotations qualify the result: no measured point in the published data exceeds the ceiling; sub-1B models fall well short of it because fixed per-token overheads dominate; and per-layer-embedding and mixture-of-experts models read less than their file size, so their ceilings are understated.
The formal version comes from RooflineBench, a March 2026 preprint on roofline analysis for on-device LLMs. Its derivation: a decoder-only forward pass costs roughly two FLOPs per parameter per token, so compute time is 2·n_params / P_peak, while memory time is approximately n_params · b_prec / BW. Dividing gives an operational intensity during decode of 2 / b_prec — about 1 FLOP per byte at 16-bit precision. The paper states plainly that decoding “is inherently memory-bound” and that its analysis focuses on the sloped region of the roofline where performance is strictly limited by operational intensity times peak bandwidth.
Invert the memory-time equation and you get the practical rule: tokens per second ≤ bandwidth ÷ bytes read per token.
The ceiling table (derived arithmetic, not measurements)
Everything in the next table is arithmetic over the real file sizes above and standard theoretical dual-channel JEDEC bandwidth peaks. Nothing here was measured. Treat every cell as a ceiling that real hardware will sit below.
| Model (Q4) | GB | DDR4-3200 2ch, 51.2 GB/s | DDR5-5600 2ch, 89.6 GB/s | DDR5-6400 2ch, 102.4 GB/s |
|---|---|---|---|---|
| LFM2.5-230M | 0.15 | 334 | 584 | 668 |
| MiniCPM5-1B | 0.69 | 74 | 130 | 149 |
| LFM2.5-1.2B | 0.73 | 70 | 123 | 140 |
| MiniCPM5-2B | 1.56 | 33 | 57 | 66 |
| LFM2.5-2.6B | 1.67 | 31 | 54 | 61 |
| Ministral-3-3B | 2.15 | 24 | 42 | 48 |
| granite-4.2-3b | 2.24 | 23 | 40 | 46 |
| Nemotron-3-Nano-4B | 2.84 | 18 | 32 | 36 |
| Qwen3.5-4B | 3.01 | 17 | 30 | 34 |
| gemma-4-E2B-it | 3.35 | 15 | 27 | 31 |
| gemma-4-E4B-it | 5.15 | 10 | 17 | 20 |
| Ministral-3-8B | 5.20 | 10 | 17 | 20 |
| granite-4.2-8b | 5.35 | 10 | 17 | 19 |
| Qwen3.5-9B | 5.68 | 9 | 16 | 18 |
| gemma-4-12B-it | 6.98 | 7 | 13 | 15 |
Two rows in that table are known to be badly pessimistic and must be read with the caveat attached. Gemma 4’s Per-Layer Embedding tables are lookups, not streamed weights, so its real per-token traffic should be well below its file size — but no measurement or vendor statement exists, so the margin is unknown. LFM2.5-8B-A1B activates roughly 1B of its 8.47B parameters per token, so its true traffic is far below 5.16 GB; router and shared-expert traffic are unquantified. Both models are probably faster on CPU than this table implies. Neither has been measured.
Does the rule hold up?
It does, and the check is worth showing because it is the only cross-validation available.
The one rigorous published CPU benchmark with named hardware is Malakhov’s “Deploying LLMs on CPU-only Environments with llama.cpp Library Set”, CEUR-WS Vol-4164. It measures an Intel Core i7-13700H with 32 GB DDR5, 14 cores, all models at 4-bit GGUF, threads pinned to physical cores, latency excluding prompt processing. Taking that machine’s theoretical dual-channel DDR5-5200 bandwidth of 83.2 GB/s and the paper’s own Q4 memory figures:
| Model | Q4 GB | Derived ceiling | CEUR measured | Fraction of theoretical |
|---|---|---|---|---|
| Phi-4-mini | 1.9 | 43.8 t/s | 40 | 91% |
| Gemma-3-12B | 6.7 | 12.4 t/s | 9 | 73% |
| Qwen2.5-7B | 3.7 | 22.5 t/s | 15 | 67% |
| Mistral-Small-3.1-24B | 13.4 | 6.2 t/s | 4 | 64% |
| Llama-3.2-1B | 0.5 | 166.4 t/s | 80 | 48% |
Every measured point sits under the ceiling. No violations. Real-world lands at roughly 65 to 90 percent of theoretical on a laptop x86 part, clustering near 70, so tok/s ≈ 0.7 × BW / GB is a defensible planning form. And the smallest model falls furthest below: at 0.5 GB the per-token fixed costs — sampling, kernel dispatch, thread synchronisation — start to dominate, so a 230M model will not deliver anything like 584 tokens per second.
Note carefully what this paper does not provide. Every model it benchmarks is 2024–2025 vintage: Phi-4-mini, Gemma 3, Qwen2.5, Mistral-Small-3.1, Llama-3.2, a DeepSeek-R1 distill. None of the eleven models in this comparison appears in it. Use it for the shape of hardware scaling, never for “how fast is Granite 4.2.”
The paper’s other numbers are useful in their own right. Prompt processing for a 256-token prompt on a 7B model took about 6.5 seconds on a 2013 Xeon E5-2695 v2, about 1.5 seconds on a Sapphire Rapids Xeon Platinum 8480+, and about 3 seconds on the i7 laptop. That time-to-first-token is the number most CPU write-ups omit, and for short interactive queries it dominates perceived latency far more than decode rate does. Threading peaked around 12 threads on the 12-core Xeon; using all 24 with hyperthreading “sometimes slightly reduced throughput due to contention for the FPU and memory bandwidth.”
The counterexample that keeps the rule honest
The bandwidth rule is a ceiling, and the gap between ceiling and reality is architecture-dependent in ways that surprise people.
llama.cpp issue #25976 is a well-instrumented field report from a 96-core Arm Neoverse-N2 server with SVE2, i8mm and bf16, measuring about 435 GB/s on STREAM triad. Single-stream decode reached only 30 percent of that bandwidth at roughly 2.7 bits per weight, 54 percent at a Q4_K mix, and 63 percent at classic Q4_K_M. The reporter’s diagnosis is that decode there is compute-bound, not memory-bound: throughput kept scaling past 32 threads where STREAM saturates, and a 25-second system-wide perf capture put over 99 percent of cycles in ggml compute kernels.
The same report claims equivalent x86 setups reach 85 to 95 percent of bandwidth on the same workloads. That claim carries no supporting data and no maintainer confirmation — treat it as the reporter’s assertion, not as established fact. It also sits above the roughly 70 percent that the CEUR cross-check produces on a laptop, which is unsurprising given a many-channel server versus a dual-channel laptop. Both figures are reported here; they are not reconciled.
What the same report measured and ruled out is immediately actionable, because it kills several pieces of folk advice: a KleidiAI build was 18 percent slower at decode; IQ4_NL was slower than Q4_K despite reading 20 percent fewer bytes per token; thread pinning and cpu-strict had no effect; transparent huge pages set to always was 20 percent slower and unstable. Only disabling NUMA balancing helped, by about 3 percent.
What the Benchmarks Say, and Who Is Saying It
Every score below carries its provenance. That is not pedantry — when you are evaluating small language models on CPU, the provenance frequently changes the number more than the model does.
The most useful single artifact is OpenBMB’s head-to-head table on the MiniCPM5-2B card, because it is the only place where nine of these models are run under one protocol. It is also OpenBMB scoring its own model against its competitors, and its own model wins the composite average by 2.8 points over a model twice its size. That structure demands scepticism.
So check it where checking is possible. OpenBMB reproduces Qwen3.5-4B at 78.0 on MMLU-Pro against Qwen’s own 79.1. It reproduces Gemma-4-E4B at 68.3 against Google’s own 69.4. It reproduces granite-4.2-3B at 65.8 against IBM’s 67.84. Within about 1.5 points across three independent vendors — the knowledge-benchmark protocol is broadly sound.
Now look at where it breaks. OpenBMB scores Gemma-4-E2B at 31.4 on IFEval and E4B at 44.4. Those are implausibly low for instruction-tuned Google models, and the same depressed pattern appears on IFBench (25.7 and 28.3) and Multi-IF. The most likely explanation is a chat-template or thinking-tag parsing failure, not a capability gap. Google publishes no IFEval for Gemma 4 at all, so there is no vendor figure to compare against. Do not cite the Gemma IFEval numbers without this caveat, and note that the disagreement is itself the finding — it is not a discrepancy to be resolved by picking whichever vendor you like better.
Composite “average” columns are where this does the most damage. OpenBMB reports Gemma-4-E2B at 24.6 on its composite average, against its own model’s 53.9 — a figure that folds the broken instruction-following rows straight into a headline number. Treat every vendor composite as marketing. Note too that Qwen publishes no benchmark table at all for the Qwen3.5 0.8B and 2B sizes; any number you see quoted for those two is somebody else’s reproduction, and no figures for them appear in this post.
Against that background, here are the scores worth carrying, each labelled:
- Granite-4.2-8b, SWE-bench Verified 47.67 — IBM’s own figure, run with the NVIDIA NeMo Evaluator SDK. No independent reproduction found. Alongside it, MMLU-Pro 74.04, GPQA 64.14, LiveCodeBench v6 73.24, IFBench prompt 79.33, RULER at 128K 71.41. The 3B sibling scores 67.84 MMLU-Pro but only 27.78 on IBM’s MMLU-ProX lite against the 8B’s 61.06 — a cliff that strongly suggests weak multilingual coverage below 8B. IBM marks several 3B agentic cells “NA”, meaning not run, not zero.
- Qwen3.5-9B, GPQA Diamond 81.7 and MMLU-Pro 82.5 — Qwen’s own card, thinking mode implied but not stated. The comparator set is GPT-OSS-120B, GPT-OSS-20B and two Qwen3-Next variants, i.e. comparators Qwen beats. GPQA Diamond 81.7 for a 9B is an extraordinary claim with no third-party confirmation. The 4B’s MMLU-Pro of 79.1 is broadly corroborated by OpenBMB’s 78.0.
- Gemma 4 E4B, MMLU-Pro 69.4, GPQA Diamond 58.6, LiveCodeBench v6 52.0 — Google’s own card. Google publishes no IFEval, no GSM8K, no HumanEval, no SWE-bench and no plain MMLU for any Gemma 4.
- MiniCPM5-2B, SWE-bench Verified 46.4 — OpenBMB’s own card. A 2B model matching an 8B on agentic coding is extraordinary and unreproduced outside OpenBMB. Its GPQA-Diamond of 70.2 and SciCode of 26.3 are marked as coming from an Artificial Analysis release, which is third-party.
- Nemotron-3-Nano-4B, RULER at 128K 91.1 — NVIDIA’s own figure, reasoning-off, via NeMo-Skills. It is the strongest long-context claim in the band, well above Granite-4.2-8b’s 71.41. NVIDIA does not publish MMLU-Pro.
- Ministral 3 8B, GPQA Diamond 0.668 and AIME25 0.787 — Mistral’s own card, reported as fractions. Mistral benchmarks against Qwen3, a generation stale, and publishes no MMLU-Pro, no IFEval and no SWE-bench. The 3B row was not retrievable.
Two omissions are deliberate and worth stating. No LMArena or Chatbot Arena score appears in this post for any model. Every Arena figure that surfaced in research came from SEO-farm blogs, one of which contradicted itself by admitting no E4B scores exist in the launch data. And no HumanEval, GSM8K or plain-MMLU figures appear either, because no 2026 card in this set reports them — they have been retired in favour of LiveCodeBench, OJBench, SciCode, MATH-500, AIME and MMLU-Pro. MATH-500 itself is saturated between 85 and 99 across all nine models in OpenBMB’s table and should be ignored. No card in the set publishes a contamination analysis.
Which Runtime Can Even Load These Models
This is not a runtime performance comparison — we published the llama.cpp versus ONNX Runtime analysis separately. The question here is narrower and, for model selection, more decisive: which runtimes can load these specific models at all?
Figure 3: Runtime availability collapses the candidate set outside llama.cpp.
Figure 3 splits the candidate set by the runtime you already deploy. The llama.cpp and GGUF branch reaches every family: Qwen3.5 through community builds, Gemma 4 through Google’s official QAT q4_0, Granite 4.2 through IBM’s full signed ladder, MiniCPM5 through OpenBMB’s official GGUF, and LFM2.5 and Ministral 3 through their vendors’ own repositories. The ONNX Runtime branch reaches only Ministral-3-3B, LFM2.5-2.6B, Phi-4-mini and SmolLM3 — with no vendor-published build for Qwen3.5, Gemma 4, Granite 4.2 or MiniCPM5.
That asymmetry is the finding, and it is the least-discussed constraint on deploying small language models on CPU this year. If your production stack is ONNX Runtime, your September 2026 model choice is dramatically narrower than if it is llama.cpp, and two of the four models available to you are 2025 vintage. HuggingFaceTB/SmolLM3-3B-ONNX, microsoft/Phi-4-mini-instruct-onnx, mistralai/Ministral-3-3B-Instruct-2512-ONNX and LiquidAI/LFM2.5-2.6B-ONNX exist. Nothing equivalent exists for the four newest families.
ExecuTorch v1.3.1 from May 2026 added export support for Qwen3.5 MoE, Gemma 4 31B and LFM2.5, running on CPU through XNNPACK, with BFloat16 and FP16 activation support in CPU scaled-dot-product attention. OpenVINO 2026.0 added int4 data-aware weight compression for 3D MatMuls aimed at MoE models, which reduces both footprint and bandwidth. But OpenVINO’s named MiniCPM support covers MiniCPM-V-4_5-8B and MiniCPM-o-2.6 — different models from MiniCPM5 — and no evidence was found either way for Granite 4.2. No ONNX Runtime version number is stated here, and no claim is made about which OpenVINO 2026.x point release is current, because neither was verified.
On the ISA side, llama.cpp’s x86 CPU backend uses AVX, AVX2, AVX512 and AMX. A real system_info line from a recent build shows AVX512_VNNI = 1, AVX512_BF16 = 1, AMX_INT8 = 1, REPACK = 1 — this is what you will see on your own machine at startup, and it is the fastest way to know what your binary can use. AMX requires AVX512_VNNI and must be built with clang or gcc; MSVC is not supported. On Arm, KleidiAI microkernels dispatch at runtime across dotprod, i8mm, SVE and SME/SME2, and the docs are refreshingly honest that a CPU_KLEIDIAI model buffer size line at load “does not prove that every operation, or any specific SME-family operation, used a KleidiAI microkernel.”
Two ISA facts change practical advice. First, the official build docs state plainly that BLAS helps prompt processing only at batch sizes above 32, and that “Using BLAS doesn’t affect the generation performance.” Installing OpenBLAS to make chat faster does nothing. Second, on AVX512-VNNI and AMX hardware, 8-bit quantization can be as fast as or faster than 4-bit — which inverts the universal “always use Q4” advice specifically on Sapphire Rapids and later. On such a machine, Granite-4.2-8b’s Q8_0 at 9.35 GB may be the better choice than Q4_K_M at 5.35 GB, if you have the RAM.
Trade-offs, Gotchas, and What Goes Wrong
Figure 4: The two failure points between pulling weights and reaching steady-state decode.
Figure 4 traces the realistic path. The operator pulls quantized weights from the Hub and receives a GGUF plus licence text. The first load attempt against an older build fails with an unknown-architecture error. The operator rebuilds from recent master, which dispatches matmul kernels to the CPU backend — where the wide-integer repack path fails on short-convolution tensors. Disabling that path lets decode proceed, now correctly limited by memory bandwidth rather than by compatibility.
The biggest gotcha is that no vendor publishes a reproducible CPU benchmark for its 2026 small models. Not Qwen, not Google, not IBM, not OpenBMB, not Mistral, not NVIDIA. The single first-party CPU figure that exists anywhere in this field is Liquid AI’s claim of 113 tokens per second on an AMD Ryzen CPU, in under 2.5 GB of memory. The model card that makes that claim names no CPU SKU, no core count, no memory configuration, no quantization, no context length and no runtime. It is unreproducible as written. That negative finding should reshape how you read every CPU-deployment blog post you encounter, including the ones quoting tokens-per-second with confidence.
Speculative decoding claims do not transfer across backends. Liquid’s DSpark drafter claims roughly 2.6× faster decoding with identical outputs, but the card scopes that to SGLang and Apple silicon via Metal — not llama.cpp on x86. Do not budget for it on a Xeon.
Quantization-aware training has no published accuracy delta. Google ships QAT q4_0 weights for Gemma 4 and Liquid ships a QAD-Q4_0 for LFM2.5. Neither publishes an accuracy comparison against the full-precision model, and no third-party measurement was found. The LFM2.5 QAD and plain Q4_0 files differ by 32 bytes in size; whether they differ measurably in quality is unknown.
Long-context behaviour varies enormously and cheaply-reported context windows mislead. Qwen3.5 advertises 262,144 tokens native, extensible to over a million. Ministral 3 and Nemotron 3 Nano advertise 256K and 262K. Granite 4.2 is 128K native with a claimed 512K extension. But RooflineBench’s own insight is that context length is the primary factor determining operational intensity — at short contexts decoding is severely memory-bound, and as context grows the KV cache becomes a second streaming cost on top of weights. A 262K context window on a dual-channel laptop is a specification, not a usable operating point.
Nemotron 3 Nano carries a September 2024 pretraining cutoff, the oldest in the set, and is English-only. For a knowledge-heavy assistant that is disqualifying on its own, before the licence question.
Granite 4.2’s recommended sampling is unusual and easy to get wrong: IBM specifies temperature 1.0 and top_p 0.95 for all tasks, including deterministic extraction work where most teams reflexively drop temperature to zero. Follow the card.
Practical Recommendations
Pick by constraint, not by leaderboard position. For small language models on CPU the constraint is almost always footprint, provenance or licence — rarely capability.
Granite-4.2-8b is the defensible default for a capable CPU workstation. It is the only model combining unambiguous Apache-2.0, a first-party signed GGUF across the full quant ladder, the strongest agentic-coding claim in the band, and a plain dense architecture with essentially zero exotic-operator risk. At 5.35 GB it fits 8 GB of RAM, and its three thinking modes let you trade reasoning depth for latency — which matters enormously when your ceiling is around 17 tokens per second on DDR5.
MiniCPM5-2B is the efficiency pick, with a journalistic asterisk. At 1.56 GB it is the smallest Q4 file with credible capability and roughly a 57 tokens-per-second ceiling on DDR5-5600. It is Apache-2.0 with first-party GGUF, MLX, GPTQ and LiteRT-LM builds. But every headline number comes from OpenBMB’s own card. Pilot it on your own evaluation set before believing any of them.
Gemma 4 E4B is the pick when you need a large vendor behind the model. Apache-2.0 confirmed verbatim, official quantization-aware-trained GGUF, 128K context, and text plus image plus audio in 5.15 GB. Its real CPU speed is probably better than the ceiling table suggests, because of PLE — unmeasured upside.
LFM2.5-2.6B is the pick for a CPU-constrained agent, subject to two caveats that travel with every mention: the lfm1.0 licence is not OSI-approved and was not reviewed here, and issue #19184 remains an open unknown.
Argue against Nemotron-3-Nano-4B for CPU — non-OSI licence, zero-byte LICENSE file in the GGUF repo, English-only, September 2024 cutoff, one quant, and NVIDIA markets it at Jetson and RTX hardware rather than at CPUs. Argue against Qwen3.5 as a CPU default — best numbers on paper, no official GGUF, and an operator that needs a bleeding-edge binary. Recommend it only to teams that track llama.cpp master.
Checklist before you commit:
- Print your llama.cpp
system_infoline and record which ISA flags are actually set. - Confirm the GGUF you plan to ship is published by the model’s vendor, and record the exact repo and revision if it is not.
- Load the model once on the oldest build in your fleet, not just on your laptop.
- Divide the Q4 file size into your machine’s theoretical bandwidth, multiply by 0.7, and check that number against your latency budget before writing any integration code.
- Measure time-to-first-token separately from decode rate. For short queries it dominates.
- On Sapphire Rapids or newer, benchmark Q8_0 against Q4_K_M before assuming Q4 is faster.
- Read the LICENSE file yourself for anything not Apache-2.0 or MIT.
Frequently Asked Questions
What is the best small language model for CPU-only inference in 2026?
IBM Granite-4.2-8b is the most defensible default. It combines genuine Apache-2.0 licensing, first-party signed GGUF files across the full quant ladder, a plain dense grouped-query-attention architecture with no exotic-operator risk in llama.cpp, and the strongest agentic-coding claim in the band at SWE-bench Verified 47.67 by IBM’s own measurement. At 5.35 GB in Q4_K_M it fits comfortably in 8 GB of RAM. If you need a smaller footprint, MiniCPM5-2B at 1.56 GB is the efficiency alternative.
How many tokens per second should I expect from small language models on CPU?
Divide your memory bandwidth by the quantized file size to get a ceiling, then multiply by roughly 0.7. On dual-channel DDR5-5600 at 89.6 GB/s, a 5.35 GB model has a derived ceiling near 17 tokens per second, and the same 0.7 factor puts a planning estimate near 12 — arithmetic, not a measurement or a prediction. Cross-validation against published measurements on an i7-13700H shows real throughput at roughly 65 to 90 percent of theoretical, clustering around 70, with the smallest models falling further below. No vendor publishes reproducible CPU benchmarks for 2026 small models, so every number here is a derivation, not a promise.
Why does Qwen3.5 score highest but rank low for CPU deployment?
Qwen publishes no GGUF for any Qwen3.5 size, so every quantized file comes from a community requantizer with no vendor signature or reproducibility commitment. Separately, Qwen3.5’s Gated DeltaNet operator required new llama.cpp operators, so a stale build simply will not load the weights. Those two facts add operational risk that its benchmark margin does not repay for most teams. If you track llama.cpp master and can tolerate that, the capability is real.
Is Gemma 4 really Apache-2.0, or is there a catch?
It is really Apache-2.0. The model card declares license: apache-2.0 but links to a Google-hosted page rather than apache.org, which is worth checking — and on checking, that page serves the verbatim, unmodified Apache License 2.0 text with no custom clauses, no acceptable-use rider and no carve-outs. The Google hosting is a hosting choice. This is a substantial change from the Gemma Terms of Use that governed earlier generations.
Should I always use Q4 quantization on a CPU?
No. On Intel CPUs with AVX512-VNNI or AMX — Sapphire Rapids and later — 8-bit quantization can match or beat 4-bit, because the integer-dot-product instructions favour the wider format. That inverts the usual advice. Benchmark Q8_0 against Q4_K_M on your actual hardware before assuming. Note also that Google ships only q4_0 for Gemma 4 and NVIDIA only Q4_K_M for Nemotron, so for several models the choice is not yours to make.
Does installing OpenBLAS or MKL make CPU generation faster?
No. llama.cpp’s official build documentation states directly that BLAS improves prompt processing only at batch sizes above 32, and that using BLAS does not affect generation performance. Token generation at batch size one is memory-bound, not compute-bound, so a faster GEMM library has nothing to accelerate. If your workload is long-prompt summarisation, BLAS may help time-to-first-token; if it is interactive chat, it will not.
Further Reading
- Small language models at the edge: the 2026 architecture landscape — the architectural detail deliberately treated here only as runtime risk.
- On-device LLM runtimes compared: llama.cpp and ONNX Runtime — runtime performance, as opposed to the model-availability question above.
- INT4 vs INT8 vs FP8 quantization for edge NPUs — the format comparison on accelerator hardware.
- Small vs large LLMs on agentic tasks: cost and latency — when the sub-14B band is the wrong band entirely.
- RooflineBench: benchmarking on-device LLMs via roofline analysis — the citable derivation behind the bandwidth ceiling.
- Deploying LLMs on CPU-only environments with llama.cpp — the one published CPU benchmark with fully named hardware.
By Riju — about
