LoRA vs QLoRA vs Full Fine-Tuning vs Distillation: Choosing a Method for CPU-Deployed Small Models in 2026

I started this article intending to argue that CPU deployment reorders the method leaderboard — that because llama.cpp forces you to bake the adapter into the weights, and because baking a QLoRA adapter into quantized weights damages it, distillation quietly wins for CPU targets. The first source I opened refuted the first link in that chain, and the corrected answer is more useful. Choosing between LoRA vs QLoRA vs full fine-tuning and distillation barely changes when your endgame is a CPU box, because llama.cpp has served LoRA adapters natively since 2024. The CPU-specific trap is real but tiny: it is one argument in your merge call, and on the only clean published measurement it is worth roughly 14 points of benchmark score.

What this covers: how each method actually consumes memory, which one your GPU can afford, what happens to an adapter on the way into a GGUF file, the two papers that measure the merge question, the widely-repeated numbers that trace back to nothing, and an explicit list of what nobody has measured yet.

Context and Background

The question “which fine-tuning method should I use” used to have a resource-shaped answer: full fine-tuning if you had a cluster, LoRA if you had a workstation, QLoRA if you had a gaming card. That framing survives, but two things changed under it.

First, the small-model target moved. Fine-tuning a 2–9B model that will end its life as a quantized GGUF file on a CPU host is now an ordinary engineering task, not an exotic one — the current crop of CPU-runnable small models is genuinely capable, and this article is the direct sequel to that comparison. Second, the parameter-efficient fine-tuning (PEFT) landscape exploded. Hugging Face’s PEFT library now ships more than forty methods. And yet, in a survey of 20,834 Hub model cards that name exactly one PEFT method, 20,509 — 98.4% — say LoRA, per Hugging Face’s Beyond LoRA analysis published 18 June 2026. from peft import LoraConfig accounts for 71.3% of PEFT imports they scanned on GitHub; LoHa gets 3.7%, AdaLoRA 3.5%.

That gap between forty published methods and one used method is the honest starting condition. It tells you the marginal method paper is not moving practice, and Hugging Face say why in their own words: “researchers are under pressure to provide results that beat the existing benchmark… this can bias the results, e.g. by spending less time tuning the alternative techniques.” They cite a study (arXiv:2602.04998) that found LoRA could match supposedly superior PEFT methods purely by tuning its learning rate. Keep that in your pocket for the rest of this article, and for every method paper you read after it.

Two gates sit upstream of everything here and are not this article’s business. Whether to fine-tune at all — as opposed to retrieval or a longer prompt — is a separate decision with a separate answer. Which training objective to use — supervised fine-tuning, direct preference optimization, reinforcement learning from human feedback — is its own axis entirely. This article stays on the PEFT and compute axis: given that you are fine-tuning, and given the objective, how do you spend your VRAM and what do you ship?

The CPU endgame does not reorder the leaderboard

Short answer: pick your training method by VRAM and dataset size, not by deployment target. llama.cpp loads GGUF LoRA adapters at runtime and evaluates them as a separate low-rank matmul beside the quantized base, which is structurally the same arrangement QLoRA trained under. The only place the CPU target genuinely changes your choice is if you pick a method whose weights have no representation in the GGUF adapter format.

The working thesis I began with had four links. Three of them broke.

Figure 1: How llama.cpp evaluates a linear layer with a runtime LoRA adapter attached.

The hidden state feeds two independent matrix multiplications — the base weight tensor, still in whatever K-quant format the GGUF file stores it in, and the adapter’s lora_a/lora_b pair at 16- or 32-bit — and the results are summed with the adapter’s contribution scaled by alpha over rank. The base weights are never modified; the adapter lives in its own device buffers alongside them.

What the loader actually does

This is worth stating precisely because the folklore says otherwise. The adapter loading path in llama.cpp’s src/llama-adapter.cpp requires the GGUF file to declare general.type = "adapter", adapter.type = "lora", and an adapter.lora.alpha value; its general.architecture must match the base model’s or the load throws "model arch and LoRA arch mismatch". Tensors are paired into an ab_map of lora_a/lora_b, allocated into a device context beside the base model’s buffers, and shape-validated against the base — a mismatch throws "LoRA tensor '<name>' does not exist in base model (hint: maybe wrong base model?)". At graph-build time, llm_graph_context::build_lora_mm() emits the separate low-rank ggml_mul_mat.

On the command line this surfaces as --lora FILE and --lora-scaled FILE S, where negative scales are legal — the documentation demonstrates -5.0, which inverts the adapter’s effect. llama-server accepts multiple --lora flags plus --lora-init-without-apply, and exposes hot-reload through POST /lora-adapters. Conversion from a PEFT checkpoint is a single script, convert_lora_to_gguf.py, which takes the adapter directory and emits the GGUF.

python convert_lora_to_gguf.py ./out/adapter \
  --base ./models/base-hf \
  --outtype f16 \
  --outfile ./out/adapter-f16.gguf

llama-cli -m base-q4_k_m.gguf --lora ./out/adapter-f16.gguf -p "..."

Adapter files are small, but “a few megabytes” understates the range. A server log attached to llama.cpp issue #18466, loading Llama-3.1-8B adapters, reports sizes of 27.66, 28.03, 62.19, 160.00, 320.00 and 640.00 MiB depending on rank and how many module types were targeted. Budget for hundreds of megabytes if you targeted every projection at a high rank, not for tens.

Why this makes QLoRA the quantization-correct choice, not a compromise

Here is the consequence I have not seen stated anywhere, and it inverts the usual intuition.

QLoRA trains by storing the base model in 4-bit NormalFloat and dequantizing it to bf16 on the fly for each forward and backward pass; the adapter itself is trained in bf16 and never quantized. The QLoRA paper puts it plainly: “We dequantize the storage data type to the computation data type to perform the forward and backward pass.” So the adapter is fitted as a correction on top of a quantized base — that is the arrangement it learned under.

The runtime adapter path on llama.cpp is the same arrangement. Quantized base tensor, separate higher-precision low-rank path, summed at each layer. The quantizer differs (llama.cpp K-quants are not bitsandbytes NF4 — different block sizes, different calibration, and I will come back to why that matters), but the structure is identical. Serving a QLoRA adapter at runtime over a quantized GGUF base preserves the relationship the adapter was trained in. Merging is what breaks it.

That single observation kills the commissioning thesis. Nothing about a CPU deployment forces the adapter to be baked in, and the un-baked path is arguably the more faithful one.

The multi-adapter footnote

One adapter is the frame for this whole article — a single fine-tune, a single deployment. The capability does extend: llama-server accepts several --lora flags and can rescale them per request, so if several tenants or tasks share one base, the architecture patterns for multi-LoRA serving apply on CPU too. Check it before designing around it, though. Issue #18466, opened 29 December 2025 against build 7360 on an NVIDIA GB10, reports six --lora flags producing ggml_new_object: not enough space in the context's memory pool — “As far as I can tell, it has never worked to use multiple LoRA adapters.” The issue was closed via PR #18469, but whether that fix has landed in an installable release is something I could not confirm, so verify against the build you actually run before promising anyone multi-adapter CPU serving.

Step 1: pick by VRAM, because that is the binding constraint

The deployment target does not decide this. Your card does.

Figure 2: The four terms of peak training memory, and how each method changes them.

Peak training VRAM has four terms — weights, gradients, optimizer states, activations — and the decisive one is the fourth: activations are essentially unchanged across full fine-tuning, LoRA and QLoRA. PEFT methods shrink three terms and leave the fourth alone.

Hugging Face’s model memory anatomy gives the canonical accounting for mixed-precision training: 6 bytes per parameter for weights (fp16 copy plus fp32 master), 8 bytes per parameter for AdamW’s two moments, 4 bytes per parameter for fp32 gradients — 18 bytes per parameter before activations. Their opening line is a useful calibration: “Training a 4B parameter model in mixed precision on a batch size of 16 requires roughly 85GB of GPU memory.”

A caution on mixing accounting systems, because blog posts do this constantly. torchtune explicitly does not implement mixed precision — “Currently, we don’t support mixed-precision training in torchtune” — so its budget is 2 + 2 + 8 = 12 bytes per parameter. Any article that quotes Hugging Face’s 18 bytes and then cites torchtune’s measured tables is silently splicing two different regimes.

The VRAM table, with the caveats attached

Two published tables are worth more than anyone’s arithmetic, including mine.

Unsloth’s published minimums, from their fine-tuning requirements page, describe themselves as “absolute minimum” — and here is a real flaw in what is probably the most-cited VRAM table on the internet: it states neither sequence length nor batch size. The page separately advises batch size 1–3. Take the numbers as a floor for short sequences, not a specification.

Model size	QLoRA, 4-bit	LoRA, 16-bit
3B	3.5 GB	8 GB
7B	5 GB	19 GB
8B	6 GB	22 GB
9B	6.5 GB	24 GB
14B	8.5 GB	33 GB

torchtune’s instrumented figures for Llama 3.1 8B at batch 2, sequence 2048 packed, compile on: full fine-tuning at 18.9 GiB peak and 1650 tok/s on a single RTX 4090; LoRA at 16.2 GiB and 3083 tok/s; QLoRA at 7.4 GiB and 2413 tok/s. The same full fine-tuning recipe measured 37.4 GiB on an A6000. Two conclusions follow. First, “8B full fine-tuning needs an A100” is false and has been for a while. Second, “how much VRAM does method X need” is not a well-formed question — the same recipe took 18.9 or 37.4 GiB depending only on the card it ran on.

A necessary note on that source: torchtune is discontinued. Its README states “Torchtune is no longer actively maintained: torchtune development wound down in 2025,” with a final release of 0.6.1 in April 2025. Its memory tables remain the best instrumented public numbers in existence and I cite them freely. Do not install it.

From the byte-per-parameter accounting, here are static (non-activation) budgets. These are derived arithmetic, not measurements — label them that way if you repeat them:

Model	Full, mixed (18 B/p)	Full bf16 (12)	+ 8-bit Adam (6)	LoRA (2.06)	QLoRA (0.58)
2B	36 GB	24 GB	12 GB	4.1 GB	1.2 GB
4B	72 GB	48 GB	24 GB	8.2 GB	2.3 GB
8B	144 GB	96 GB	48 GB	16.5 GB	4.6 GB

The derived 8B LoRA static figure of 16.5 GB against Unsloth’s published 22 GB total implies about 5.5 GB of activations and overhead, which is plausible for a short sequence at batch 1–3. Derived QLoRA at 4.6 GB against Unsloth’s 6.0 GB is a similar gap. The three sources are mutually consistent, which is the most you can ask.

One non-obvious result falls out: full fine-tuning an 8B in bf16 with fp32 Adam states needs roughly 96 GB of static budget and therefore does not fit on an 80 GB card without an 8-bit optimizer or offload.

Activations are what actually kill your run

Every method table drops the activation term, and it is usually the one that OOMs you. Hugging Face say it directly: “This is why batch size or sequence length can exhaust GPU memory even if the model itself fits.”

The reference formula is Equation 1 of Korthikanti et al. (NVIDIA, arXiv:2205.05198): activation memory per transformer layer is sbh(34 + 5as/h), for sequence length s, batch b, hidden size h and attention heads a. Two caveats belong with it. It assumes 16-bit activations, and the 5as/h term is the materialized s×s attention matrix, which FlashAttention-style kernels never materialize — so the practical modern term is roughly 34sbh per layer.

Running the full formula for a generic 8B-class model (h=4096, L=32, a=32, batch 1, bf16) gives about 3.6 GB at sequence 512 and about 30.6 GB at sequence 2048 — derived arithmetic on assumed architecture constants, not a measurement. Quadrupling the sequence length costs roughly 8.5× the activation memory. With FlashAttention the growth is linear instead, 3.6 → 9.1 GB. Either way it dwarfs the entire QLoRA weight budget for an 8B.

Independent confirmation from an instrumented source: torchtune’s Llama-3.2-3B ablation shows that packing to sequence 4096 moved peak memory from 25.5 to 60.0 GiB — a 135% increase with everything else held fixed. Their cumulative lever isolation is the best table of its kind anywhere: baseline 25.5 GiB → +packing 60.0 → +compile 51.0 → +chunked cross-entropy 42.9 → +activation checkpointing 24.9 (a 41.9% cut, the single largest lever) → +fused optimizer step 23.1 → +activation offload 21.8 → +8-bit AdamW 17.6 → LoRA 8.5 → QLoRA 4.6. Their summary: “81.9% less memory with a 284.3% increase in tokens per second.”

Lowering LoRA rank to save memory does almost nothing

This is the single most useful paragraph in the QLoRA paper and I have never seen it quoted. Measuring a 7B LLaMA on FLAN v2 at batch 1, the authors report: “the LoRA input gradients have a memory footprint of 567 MB while the LoRA parameters take up only 26 MB. With gradient checkpointing, the input gradients reduce to an average of 18 MB per sequence … the 4-bit base model consumes 5,048 MB. This highlights that gradient checkpointing is important but also that aggressively reducing the amount of LoRA parameter yields only minor memory benefits.”

Read the ratio. Halving your rank halves 26 MB; turning on gradient checkpointing takes 567 MB to 18 MB per sequence. If you are dropping from rank 16 to rank 8 to fit in VRAM, you are optimizing the wrong term by roughly an order of magnitude — and paying for it in capacity, because rank is a quality knob.

While you are in the config, turn on rank-stabilized scaling. In Hugging Face’s own like-for-like benchmark (Llama-3.2-3B, MetaMathQA trained, GSM8K evaluated, same data, code and hardware across arms — vendor-run, not third-party), plain LoRA scored 48.1% at 22.5 GB peak; the same LoRA with rsLoRA scaling scored 53.2% at 22.6 GB. Five points for a boolean. rsLoRA is not a new method and not an initialization scheme — it is a 2023 change to the scaling factor (arXiv:2312.03732), setting it to lora_alpha/√r instead of lora_alpha/r, which PEFT supports and most people leave off.

from peft import LoraConfig

cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    use_rslora=True,                       # ~5 points in HF's benchmark, no VRAM cost
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)

For completeness on the same benchmark: Lily reached 54.9% at 25.6 GB, LoRA-FA sat near plain LoRA’s accuracy at a lower 20.2 GB, and BEFT collapsed to 32.9%. Hugging Face’s own conclusion is blunt: “Normal LoRA only achieves an accuracy of 48.1% at 22.5 GB memory and should thus be avoided in favor of the alternatives.” On image generation, OFT strictly dominated LoRA on their numbers (DINO 0.708 vs 0.697, at 9.01 vs 9.97 GB). The field is not settled; LoRA’s 98.4% share is inertia as much as evidence.

One method that gets mis-scoped constantly: GaLore (arXiv:2403.03507) projects gradients into a low-rank subspace during full-parameter training, and reports “reduces memory usage by up to 65.5% in optimizer states” and the ability to “pre-train a 7B model on consumer GPUs with 24GB memory.” Note both qualifiers. The headline is about pre-training, and the saving is confined to optimizer states. It is not a LoRA substitute for a 2–9B fine-tune.

Step 2: pick by dataset size

Rank and method interact with how much data you have, and the honest state of knowledge here is thinner than the internet implies.

The three organizations that actually sell fine-tuning document minimums two to three orders of magnitude below the folklore. OpenAI’s supervised fine-tuning guide documents a 10-example minimum, improvements typically visible from 50–100, and advises starting with 50. Their most useful line goes unquoted: “If 50 examples have no impact, rethink your task or prompt before adding training data.” Google’s Vertex documentation suggests starting around 100, though I could not read that page verbatim to confirm the wording.

Against that, “you need 10,000 examples” is folklore with no traceable source — a ring of blog posts citing each other with mutually inconsistent numbers. And the famous LIMA result (arXiv:2305.11206, “1,000 examples is enough”) is real but badly scope-abused: it is a 65B model, it is about style alignment rather than narrow-task capability — the Superficial Alignment Hypothesis explicitly locates capability in pre-training — the data was hand-curated, and judging was human and GPT-4 preference. There is no LIMA result at 2–9B. Do not import it.

What does exist at the right scale is vendor research from Baseten on a clinical-scribe task with Qwen3-4B and Qwen3-8B: “Evaluation performance scales log-linearly with dataset size” and “Most LoRA ranks achieve the same performance as full fine-tuning across all dataset sizes, with two exceptions: rank1 and rank4 begin to underperform at 30K examples.” Thinking Machines corroborate the shape, reporting rank-32 on a 7B matching full fine-tuning up to roughly 50k examples. Both are single-task, non-peer-reviewed vendor studies. They are also the closest thing to an answer that exists, on exactly this audience’s model sizes.

The other real measurement is a clean overfitting curve from the Yasuno paper I will lean on heavily in the next section — Japanese civil-infrastructure QA, test negative log-likelihood against training-set size: n=1,000 → 1.342; n=2,000 → 1.276; n=3,000 → 1.215; n=4,000 → 1.127 (optimal); n=5,000 → 1.319, a 17% regression. What makes it valuable is that training loss kept falling while test NLL rose. If you are watching only your training curve, you cannot see that happen.

Practically: prove the task is learnable at 50 examples before building a pipeline, run rank 8–16 from 100 to 30k, raise rank or reconsider full fine-tuning above that — and hold out a real test set, because the U-curve is invisible from the training loss.

Step 3: the merge-target trap, which is the whole CPU-specific story

Now the payload. If you decide to merge — and you may have good reasons — merge into the original 16-bit base you started from, never into the dequantized 4-bit base.

Figure 3: The three things you can do with a QLoRA adapter, and what Playpen measured for each.

The diagram traces an adapter trained against a bf16 view of an NF4 base through its three possible fates. The table below is the measurement behind it.

The one clean measurement

The evidence is Appendix E.2, Table 7 of Playpen (arXiv:2504.08590, Horst et al., EMNLP 2025). The setup: Llama-3.1-8B-Instruct, QLoRA supervised fine-tuning via Unsloth at r=64, α=64, NF4 without nested quantization, evaluated on clembench dialogue games.

Merge strategy	ClemScore	% played	Quality
Unmerged, adapter on 4-bit base	46.82	75.24	62.23
Merged into full-precision base	47.79	74.88	63.82
Merged into 4-bit base, saved in 16-bit	33.52	70.19	47.76
Merged into 4-bit base, saved in 4-bit	30.14	60.00	50.23

The authors’ own summary: “the first strategy outperforms the others, showing comparable results to the unmerged adapter.”

Read that table again, because almost everyone misreads it. The framing that circulates is “merging a QLoRA adapter costs you 10–30%.” That is wrong. Merging into the full-precision base slightly beat the unmerged adapter — 47.79 against 46.82. The 28% collapse to 33.52 comes specifically from merging into the 4-bit-quantized base. The variable is the merge target, not the merge.

The mechanism follows from the training loop described above: the adapter only ever saw the base at bf16. Benjamin Marie’s write-up of the mechanism puts the consequence well — merged into 4-bit weights, the adapter’s parameters sit “among 4-bit parameters that they have never seen before.” (His own numbers are behind a paywall; only the mechanism is citable.)

In code, the correct merge is the boring one — load the adapter against the original base repository, not a 4-bit BitsAndBytesConfig load of it:

from peft import AutoPeftModelForCausalLM
import torch

model = AutoPeftModelForCausalLM.from_pretrained(
    "./out/adapter",
    dtype=torch.bfloat16,         # NOT load_in_4bit=True; `torch_dtype` is deprecated
    device_map="cpu",
)
merged = model.merge_and_unload()
merged.save_pretrained("./out/merged-bf16", safe_serialization=True)

Caveats on the evidence, stated plainly: n=1 paper, one model, one benchmark family, buried in an appendix, and Playpen’s merge experiment was incidental — they were setting up DPO, not studying merging. It is nonetheless the best measurement that exists.

Does requantizing a merged model to Q4_K_M hurt? Measured once, and the answer is no

One paper runs the complete pipeline an engineer actually runs — QLoRA, merged_16bit, convert, llama-quantize to q4_k_m, evaluate. That is Yasuno, “Adapting Methods for Domain-Specific Japanese Small LMs” (arXiv:2603.18037): Japanese civil-infrastructure QA, 4,000 training samples, LLM-as-judge with Qwen2.5-14B over 100 questions scored 0–3.

Model	Format	Avg score	Perfect-score rate	Δ
Swallow-8B	F16	2.820	84%	—
Swallow-8B	Q4_K_M	2.830	86%	+0.010
ELYZA-JP-8B	F16	2.700	73%	—
ELYZA-JP-8B	Q4_K_M	2.730	78%	+0.030
Qwen2.5-7B	F16	2.420	49%	—
Qwen2.5-7B	Q4_K_M	2.140	30%	−0.280

Merge-into-16-bit then requantize was quality-neutral-to-slightly-positive on two of three models and clearly negative on the third. The authors hypothesize — and label as a hypothesis — that quantization acted as a post-training regularizer against overfitting at n=4,000.

Two things must be said about this paper that the paper does not say about itself.

First, its central causal claim is architecturally false. It attributes Qwen2.5’s collapse to grouped-query attention and credits “Llama-3’s standard multi-head attention (MHA) architecture, with independent key/value projections per head.” Llama-3 8B is not multi-head attention — it is grouped-query attention. Primary confirmation is sitting in the llama.cpp load log attached to issue #18466: n_head = 32, n_head_kv = 8, n_gqa = 4. Both models in the comparison are GQA; Qwen2.5-7B simply shares more aggressively. The observation stands. The explanation is built on an error. The defensible restatement is “KV-sharing ratio may matter,” and even that is n=1 per architecture.

Second, a fourth model in their study (Tanuki-8B) could not be converted at all — “GGUF conversion fails due to llm-jp tokenizer compatibility issues with standard GGUF tools.” Tokenizer conversion failure is a real, under-discussed CPU deployment risk, and it is a binary one: you do not get a degraded model, you get no model. Convert your base to GGUF before you spend a week fine-tuning it.

Do not merge inside GGUF

llama.cpp ships llama-export-lora, which merges a GGUF adapter into a base GGUF. I could not land a citable current README for its flags, so I am not going to print a flag list as verified — check --help on your build.

The important point is conceptual and does not depend on the flags. Merging inside GGUF means merging into already-K-quantized weights. That is structurally the exact failure mode Playpen measured at −14 points, in the bitsandbytes analogue. If you merge, do it upstream in bf16 with PEFT’s merge_and_unload(), then convert and quantize. There is also a concrete trap recorded in llama.cpp’s own converter source, whose modify_tensors comment warns that for architectures with tied word embeddings, “adapters targeting lm_head will fail when using llama-export-lora” — so the converter refuses them outright with "lm_head is present in adapter, but is ignored in base model".

Step 4: when distillation is actually worth it

Distillation is not made more attractive by a CPU target. It is made attractive by teacher reuse, which is a workflow property, not a deployment property.

First, split the word in two, because it names two things with utterly different price tags.

Distilling an 8B into a 3B is pre-training-scale work. Minitron ran on 256 H100s. Meta’s Llama 3.2 model card carries a dedicated column for logit generation alone: 86,000 GPU-hours. Ministral 3’s cascade spent 1–3 trillion tokens. NVIDIA’s own Hugging Face example for a 3B→1B distillation specifies “8× RTX 6000 (total ~400GB VRAM)” — the cleanest available rebuttal to “distillation is easy on one GPU.”

Using a big teacher to generate or score data, then fine-tuning a small model, is cheap and is what most people mean. It is also exactly what DeepSeek-R1-Distill was: 800k R1-generated samples, two epochs, token-level cross-entropy, no reinforcement learning and no logits. Calling that logit distillation is simply wrong.

The decisive evidence comes from Lambert & Luccioni, “End-to-End Energy Accounting of Distillation Pipelines” (arXiv:2605.13981, ICML 2026) — NVML telemetry on a single H100 across 2–3 repeats, teacher OLMo-2-32B-SFT into OLMo-2 1B/7B/13B students.

Pipeline	kWh (1B / 7B / 13B)	Quality retention Q (1B / 7B / 13B)
Baseline SFT	7.00 / 19.50 / 34.60	0.69 / 0.90 / 0.99
Logit KD	16.90 / 28.40 / 42.50	0.70 / 0.78 / 0.82
Synthetic SFT	16.65 / 28.25 / 40.70	0.71 / 0.79 / 0.85

Verbatim: “At the 1B scale, KD and synthetic SFT obtain slightly higher aggregate Q than baseline SFT, but require roughly 2.4× more end-to-end energy. At 7B and 13B, baseline SFT strictly dominates both distillation pipelines.” Their break-even teacher-reuse counts are roughly 10 students at 1B, 5–6 at 7B, 4 at 13B. Their framing sentence is the one to keep: “Whether distillation is cost-efficient is therefore primarily a workflow question, not an intrinsic property.” They state a limitation that matters here — no LoRA or quantization arm.

Apple’s “Distillation Scaling Laws” (arXiv:2502.08606, ICML 2025, a 143M–12.6B sweep) arrives at the same condition independently: distillation “can not produce lower model cross-entropies than supervised learning when both … are given enough data or compute,” but is more efficient if the student’s compute stays under a size-dependent threshold and “a teacher already exists, or the teacher to be trained has uses beyond a single distillation.”

Two independent lines, one measured by telemetry and one by scaling law, converge on teacher amortization. Neither mentions the deployment target at all. That is what kills the last link of the commissioning thesis.

On-policy distillation — the student generates rollouts, the teacher scores per-token log-probabilities as a dense reward — is the genuinely new thing in 2025–26, and it belongs here as a compute-efficiency method rather than an alignment one. A provenance correction you must not get wrong: the widely-circulated Thinking Machines table (off-policy 55.0% on AIME’24; +RL 67.6% at 17,920 GPU-hours; +on-policy distillation 74.4% at 1,800) is reproduced from the Qwen3 technical report (arXiv:2505.09388), not measured by Thinking Machines. Coverage reporting “Thinking Machines measured a 30× cost cut” is propagating an error. Their own figures are FLOPs ratios; their line about “cost reduction in GPU hours is closer to 18x” is an inference, not telemetry; the SFT-2M comparator is explicitly an extrapolation; and the “50–100×” number comes from a self-distillation experiment that does not generalize.

The most useful paper for a practitioner is “Rethinking On-Policy Distillation” (arXiv:2604.13016), which reports that “a stronger teacher can completely fail to improve a student, even when a weaker teacher succeeds” — and gives you an early-warning diagnostic. Successful runs show top-k overlap between student and teacher rising from about 72% to 91%, with the shared top-k tokens carrying 97–99% of combined probability mass. Failing runs show overlap stagnant from the outset. You can detect a doomed run in its first hours instead of its last.

Finally, a hygiene list, because “model X was distilled” is asserted far more often than it is sourced. Verified distilled by model card or technical report: Gemma 3 (all sizes, 256 logits per token, teacher unnamed); Llama 3.2 1B/3B (logits from Llama 3.1 8B and 70B, post-pruning recovery); Qwen3 0.6B–14B (strong-to-weak, post-training only — the base models were not distilled, and only the tech report says so); Nemotron-Nano-9B-v2 and Nemotron-3-Nano-4B; Ministral 3 3B/8B/14B. Myths with no primary support: gpt-oss-20b “distilled from 120b or o3” (the word does not appear in the card or paper); Phi-4-mini “distilled from GPT-4”; Gemma 4 “distilled from Gemini” (a full grep of the technical report returns zero matches for “distill”); Gemma 3n E2B “distilled from E4B” (that is MatFormer nesting, a different mechanism entirely).

One more piece of folklore to retire: temperature τ=4 for distillation is inherited from vision knowledge-distillation literature. LLM distillation generally runs at τ=1 or low temperature, because a 30,000-plus token vocabulary already produces structured non-peak probabilities.

The decision matrix

Putting the four steps together. “Best” here means best expected quality per unit of your time and hardware, on the evidence cited above.

Situation	Method	Why	Ship as
8 GB VRAM, any size up to 9B	QLoRA	Only method that fits; ~0.58 B/p static (derived) vs Unsloth’s published 6 GB at 8B	Base GGUF + runtime adapter
16 GB, 2–3B model	LoRA	Fits at bf16; avoids the quantized-base merge question entirely	Either; merge is safe from a bf16 base
16 GB, 8–9B model	QLoRA	LoRA at 8B needs ~22 GB (Unsloth, published)	Base GGUF + runtime adapter
24 GB, 8B model	LoRA, full FT viable	torchtune measured full FT at 18.9 GiB on a 4090	Merged bf16 → convert → quantize
24 GB, 2–4B model	Full fine-tuning	Fits with the lever stack; no adapter artifact to manage	Single merged GGUF
>30k examples, low rank underperforming	Full FT or higher rank	Rank 1–4 degrade past 30k (Baseten, one task)	Single merged GGUF
Need DoRA / OFT / modules_to_save	Reconsider, or plan a merge	No representation in the GGUF adapter format	Merged only
Teacher already exists, ≥4–10 students planned	Distillation	Break-even reuse counts from Lambert & Luccioni	Whatever the student needs
One student, no existing teacher	Not distillation	Baseline SFT strictly dominates at 7B and 13B	—

Note what that table never does: pick a training method because of the CPU target, or recommend merging into a quantized base. The deployment target changes the artifact, not the method — with one exception, which is the next section.

Trade-offs, gotchas, and what goes wrong

Figure 4: Six ways an adapter fails on the road to a CPU deployment, sorted by whether they are loud or silent.

The diagram classifies six known failure modes by how you find out. Four fail loudly — the converter or loader stops with an error and you fix it in minutes. Two fail silently, producing a model that loads, runs, and is quietly worse than the one you evaluated. Those are the expensive ones.

Norm-layer updates are converted and then discarded. This is the sharpest example of a silent failure, and reading both halves of the toolchain makes it precise. The converter explicitly passes norm tensors through — its tensor loop has a branch if "_layernorm" in name or ".norm" in name: yield (base_name, tensor). The loader then throws them away: } else if (str_endswith(name, "_norm.weight")) { // TODO: add support for norm vector … continue;. No warning, no error. If your PEFT config targets norm layers, those updates make it into the GGUF and never reach the graph. I have found no blog post that mentions this.

Only lora_a/lora_b survive, and the converter now enforces it too. The loader rejects anything else with "LoRA tensor '<name>' has unexpected suffix". Reading convert_lora_to_gguf.py on master (19 September 2026) shows the converter fails first and harder: any tensor that is not a lora_A/lora_B pair, a lora_embedding_A/B, or a whitelisted norm triggers logger.error(f"Unexpected name '{name}': Not a lora_A or lora_B tensor") followed by sys.exit(1). The practical consequences: DoRA’s magnitude vector has no slot in the format, and PEFT’s modules_to_save — full replacement copies of embed_tokens or lm_head — has none either. The converter even emits a specific diagnostic for the embeddings case, pointing at PR #9948. I read the source rather than running a DoRA adapter through it, so treat this as a strong reading of the code path rather than an empirical result; but if you plan to use DoRA and ship to llama.cpp, test the conversion on day one, not day thirty.

This is the one place the deployment target genuinely reorders the method ranking. DoRA (arXiv:2402.09353) is not exotic; it landed in PEFT v0.9.0, so do not call it new in 2026, and torchtune’s docs describe it as “shown to improve the performance of LoRA, particularly at low ranks.” It is also barely used: in a Hugging Face sample of 10,000 image-generation PEFT checkpoints, DoRA accounted for 11. If you need it and you need GGUF, PEFT now ships a lora_conversion module for converting other adapter types into plain LoRA — Hugging Face report GraLoRA→LoRA at 0.702 → 0.694, essentially lossless.

The tokenizer may simply refuse to convert, as Yasuno found with Tanuki-8B. Test the conversion before the training run.

Quantization-aware training is not the escape hatch it looks like. Unsloth and TorchAO ship QAT plus LoRA with qat_scheme values of int4, int8-int4, fp8-int4 and fp8-fp8, claiming recovery of “up to 70% of the lost accuracy” and +1.0 point on Gemma3-4B GPQA, +2.1 on Gemma3-12B BBH. But it exports to TorchAO and ExecuTorch, not GGUF K-quants. Whether a QAT→GGUF K-quant path exists at all is unresolved. There is third-party guidance claiming QAT weights are conditioned exclusively for q4_0 and that exporting to other types bypasses the benefit entirely — that comes from a third-party file rather than Google’s documentation, so I am flagging it, not asserting it.

Full fine-tuning’s failure mode is not memory, it is forgetting. Hugging Face cite PEFT’s “greater resistance to catastrophic forgetting” as a headline advantage. Full fine-tuning also gives you no cheap rollback and one artifact per task — which, at 4.9 GB per quantized 8B, is a storage and distribution problem that a 60 MB adapter simply does not have.

And a boundary worth naming: this article never compares quantization formats. Whether your base should be int4, int8 or fp8 in the first place is a different question with a different answer. Here the only question asked of quantization is what merging does to an adapter.

What nobody has measured

This is the honest centre of the article, and I would rather state it than paper over it. Five things bear directly on the recommendations above and have no published measurement behind them.

1. Runtime GGUF adapter versus merge-then-Q4_K_M, on the same model, adapter and task. Zero published head-to-heads. This is the crux question of the entire article and it is unmeasured. Playpen measures merge targets but never touches GGUF or K-quants. Yasuno measures F16 against Q4_K_M but only ever after merging — there is no runtime-adapter arm.

2. Whether the merge damage reproduces with K-quants. Every merge measurement in existence uses bitsandbytes NF4. llama.cpp K-quants are a different quantizer with different block sizes and different calibration. No published result transfers cleanly to the GGUF case. My recommendation to merge into the 16-bit base is an argument from mechanism plus one bitsandbytes measurement, not a GGUF measurement.

3. Distillation versus LoRA/QLoRA on the same task with both cost and quality reported. No such paper exists. The energy-accounting authors name LoRA as an explicitly untested arm. The nearest artifacts are KD-LoRA (arXiv:2410.20777), which is encoder-only BERT and RoBERTa rather than generative small models, and Thinking Machines’ note that “at rank = 32, LoRA trails full finetuning by 13% after SFT, but only 6% after on-policy distillation.”

4. Whether DoRA round-trips to GGUF in practice. I read both the loader and the converter and both reject the tensor names. I did not run an adapter through them.

5. Whether requantize-neutrality generalizes. Yasuno is one domain, three models, 100 LLM-judged questions — and its own causal explanation is architecturally wrong.

If someone runs measurement 1 on a single 8B with a single adapter, they will have produced the most useful artifact in this corner of the field. It is perhaps two days of work.

Practical recommendations

Choose by VRAM and dataset size. Ship the adapter unmerged unless you have a reason not to. If you merge, merge into 16-bit.

For a typical 2–9B fine-tune destined for a CPU host, the default path is: QLoRA if you are under about 16 GB and LoRA above it; rank 8–16 with use_rslora=True; gradient checkpointing always on; a held-out test set watched for the NLL U-curve; then convert_lora_to_gguf.py and ship base.gguf alongside a 30–640 MiB adapter served with --lora. Merge only when a single file is an operational requirement — and then from the original bf16 base.

A note for Unsloth users: save_pretrained_gguf() internally merges LoRA into the base at FP16 before calling llama.cpp’s convert and quantize. That default is the correct merge target. A lot of people have been doing the right thing without knowing why, and would break it the moment they hand-rolled the pipeline.

Checklist before you start the run:

• [ ] Convert the base model to GGUF first. Tokenizer failures are binary and late-discovered.
• [ ] use_rslora=True — roughly 5 points in Hugging Face’s benchmark, no VRAM cost.
• [ ] Gradient checkpointing on: 567 MB → 18 MB per sequence, per the QLoRA authors.
• [ ] Do not lower rank to save memory. It saves 26 MB against a 5 GB base.
• [ ] No norm layers in target_modules; no modules_to_save. Both fail on the GGUF path, one silently.
• [ ] If you want DoRA, test the GGUF conversion on day one.
• [ ] Hold out a test set. Training loss will not show you the overfitting U-curve.
• [ ] If merging: load the original base in bf16, never load_in_4bit=True, then merge_and_unload().
• [ ] Don’t distill unless the teacher is amortized across four or more students.

Frequently Asked Questions

Can llama.cpp run a LoRA adapter without merging it?

Yes, and it has been able to since the September 2024 adapter refactor. Convert the PEFT directory with convert_lora_to_gguf.py, then pass --lora adapter.gguf to llama-cli or llama-server. The base tensors stay quantized and the adapter is evaluated as a separate low-rank matrix multiplication beside them, with --lora-scaled letting you change the strength at load time. llama-server additionally exposes hot-reload through POST /lora-adapters. Adapter files run roughly 30–640 MiB depending on rank and how many module types you targeted.

Is QLoRA worse than LoRA for a model I will quantize anyway?

On structure, arguably the opposite. QLoRA fits the adapter as a correction on top of a quantized base, and serving a quantized GGUF base with a runtime adapter reproduces that arrangement. The caution is specific to merging: a QLoRA adapter has only ever seen the base at bf16, so merging it into 4-bit weights puts it among values it never trained against. Merge into the original 16-bit checkpoint and the problem does not arise — Playpen measured 47.79 that way versus 46.82 unmerged.

Does lowering the LoRA rank save VRAM?

Barely. The QLoRA authors measured a 7B setup where the adapter parameters occupied 26 MB while input gradients took 567 MB and the 4-bit base took 5,048 MB. Their conclusion is explicit: “aggressively reducing the amount of LoRA parameter yields only minor memory benefits.” Gradient checkpointing is the lever that matters — it cut those input gradients to about 18 MB per sequence. Cutting rank costs you capacity and buys you almost nothing, so set rank by task difficulty and dataset size instead.

How many training examples do I actually need?

Far fewer than the folklore claims. OpenAI documents a 10-example minimum with measurable improvement typically appearing from 50–100, and advises that if 50 examples change nothing, the problem is your task definition rather than your data volume. The “10,000 examples” figure has no traceable source. The commonly cited LIMA 1,000-example result is a 65B style-alignment study and does not transfer to a 2–9B narrow-task fine-tune. Watch a held-out test set: one measured study found overfitting at n=5,000 while training loss was still falling.

Should I use distillation for a CPU-deployed small model?

Not because of the CPU. Two independent studies converge on the same condition, and neither mentions deployment. Energy-accounting telemetry found baseline supervised fine-tuning strictly dominates logit knowledge distillation and synthetic-data SFT at 7B and 13B, with break-even at roughly four to ten students reusing one teacher. Apple’s distillation scaling laws reach the same requirement — that a teacher already exists or has uses beyond one distillation. If you are training one student and have no teacher, fine-tune directly.

Can I merge a LoRA adapter inside GGUF instead of upstream?

llama-export-lora exists for that, but it means merging into already-K-quantized weights — structurally the failure mode that cost 14 ClemScore points in the bitsandbytes analogue. Do the merge upstream with PEFT’s merge_and_unload() against the original bf16 base, then convert and quantize. One additional trap is visible in llama.cpp’s own converter source: for architectures with tied word embeddings, adapters targeting lm_head are noted as failing under llama-export-lora.

Further Reading

• Small language models for CPU inference, compared — which base model to fine-tune in the first place, and the memory-bandwidth ceiling that governs CPU throughput. This article is its sequel.
• Fine-tuning vs RAG vs long context — the upstream gate: whether to fine-tune at all.
• DPO vs RLHF vs SFT: an alignment benchmark — the objective axis, orthogonal to everything here.
• Multi-LoRA serving architecture — when one base serves many adapters, on GPU or CPU.
• Hugging Face, “Beyond LoRA: Can you beat the most popular fine-tuning technique?” — the best single artifact on the 2026 PEFT landscape, including the like-for-like benchmark and the warning about under-tuned baselines.
• QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314) — read §3 for the memory measurement that makes rank-cutting pointless.
• Playpen (arXiv:2504.08590) — Appendix E.2, Table 7 is the merge-target evidence.

By Riju — about