On-Device LLM Runtimes in 2026: llama.cpp vs MLC-LLM vs ONNX Runtime

Ship a language model to a phone, a laptop, or a Jetson board and the model weights are the easy part. The hard part is the runtime — the layer that loads the weights, holds them in a quantized form small enough to fit RAM, schedules the matrix multiplies onto whatever accelerator the silicon happens to expose, and keeps a KV cache coherent across hundreds of decode steps. This is where an honest on-device LLM runtime comparison has to start, because the three dominant projects — llama.cpp, MLC-LLM, and ONNX Runtime — make three fundamentally different bets about how that layer should work. One interprets a graph at run time, one compiles the model ahead of time into a device-specific binary, and one dispatches subgraphs to vendor acceleration libraries. Those architectural choices, not raw kernel speed, decide which runtime you should pick for a given SoC, a given quantization budget, and a given team.

What this covers: the execution models behind each runtime, their hardware backends and quantization formats, a head-to-head decision matrix across throughput, memory and portability, the failure modes nobody advertises, and concrete recommendations per platform.

Context and Background

On-device inference is not a fashion; it is a response to four costs that cloud inference cannot escape. Privacy is the first: keystrokes, health data, and enterprise documents never leave the device, which sidesteps entire classes of compliance work. Latency is the second: a local 3B model returns the first token in tens of milliseconds because there is no network round trip, no queue, no cold-start on a shared endpoint. Cost is the third: once the model is on the device, marginal inference is free — no per-token billing, no autoscaling GPU fleet. Offline capability is the fourth: the feature keeps working on a plane, in a warehouse, or on a factory floor with no connectivity. For a broader treatment of when the edge economics beat the cloud, see our edge LLM benchmark on Jetson Orin.

A “runtime” earns its name by doing four jobs that a raw checkpoint cannot. It loads the weights — often memory-mapping a multi-gigabyte file so the OS pages in only what a forward pass touches. It quantizes, or consumes an already-quantized artifact, compressing 16-bit weights to 4 or 5 bits so a 7B model fits in a few gigabytes of unified memory. It schedules the computation graph — deciding which operators fuse, how the KV cache grows, and how work is tiled across cores. And it uses the accelerator: the GPU, the NPU, or the vectorized CPU path that the SoC exposes. Every meaningful difference between llama.cpp, MLC-LLM, and ONNX Runtime traces back to how each one implements those four jobs. The ONNX Runtime execution-provider documentation frames the accelerator problem cleanly, and it is the axis on which the three projects diverge most sharply.

One more fact frames everything that follows: single-stream, batch-one decoding — the on-device case, one user, one token at a time — is memory-bandwidth bound, not compute bound. Each generated token must stream the entire quantized weight matrix out of memory and through the arithmetic units once. The math is straightforward: tokens per second is roughly memory bandwidth divided by model bytes read per token, so a 4-bit 7B model reading about 4 GB per token on a device with 60-70 GB/s of usable bandwidth is capped near the teens of tokens per second no matter how fast the cores are. This is why quantization is not merely a memory trick — halving the bytes per weight nearly doubles the achievable decode rate — and why the choice of runtime, which decides how tightly the dequantization fuses into the matmul and how few bytes cross the memory bus, has an outsized effect on real throughput. Prompt processing (the prefill of the input context) is a different regime, compute bound and parallel, but for interactive on-device use it is the token-by-token decode that users feel.

llama.cpp vs MLC-LLM vs ONNX Runtime: the core distinction

The core distinction is architectural. llama.cpp interprets a fixed GGML compute graph at run time and reaches accelerators through hand-written backends. MLC-LLM compiles the model ahead of time with Apache TVM into a device-specific binary. ONNX Runtime keeps a portable graph and offloads subgraphs to vendor execution providers such as CoreML, QNN, or DirectML. Everything else — quantization formats, mobile packaging, NPU reach — follows from those three stances.

Execution model: interpreted graph vs compiled binary vs pluggable providers

llama.cpp is built on GGML, a small C tensor library. When you load a GGUF file, the runtime constructs a compute graph in memory — a fixed sequence of operators for the transformer’s attention and feed-forward blocks — and then interprets that graph on every forward pass. There is no ahead-of-time compilation step producing a specialized binary; the same prebuilt executable runs any GGUF model whose architecture GGML already implements. The genius of this design is operational simplicity: one binary, one file format, no toolchain. The cost is that operator coverage is whatever the maintainers have hand-written, and adding a novel architecture means writing C. This is why llama.cpp support for a brand-new model family sometimes lands days or weeks after the weights drop — someone has to implement the graph.

MLC-LLM takes the opposite stance. It uses Apache TVM’s Unity flow to compile the model ahead of time. You run a build step that lowers the model’s operators into TVM’s intermediate representation, generates GPU kernels for the specific target — Metal shaders for Apple, WGSL for the browser’s WebGPU, Vulkan or OpenCL for Android — and emits a device-specific binary library plus a weights artifact. TVM Unity performs high-performance code generation without a separate autotuning pass and tracks dynamic shapes by design, which matters for variable sequence lengths. The payoff is that the kernels are specialized for the target and the same source model reaches platforms — notably the web via WebGPU — that no other runtime touches natively. The tax is the compilation step itself: it is a real build, with its own toolchain, and it must be repeated per target and per model. The MLC-LLM documentation describes this compile-and-deploy pipeline in detail.

ONNX Runtime sits in a third position. It loads a portable ONNX graph and, at session-creation time, partitions that graph among registered execution providers (EPs). Each EP is a plug-in backed by a vendor acceleration library: CoreML on Apple, NNAPI on Android, QNN for Qualcomm’s Hexagon NPU, DirectML on Windows GPUs, plus CUDA, TensorRT, and OpenVINO. ONNX Runtime assigns each subgraph to the highest-priority EP that can run it and falls back to CPU for the rest. For generative models, the companion onnxruntime-genai library wraps this engine with the decode loop — KV cache management, greedy and beam search, top-p and top-k sampling, logits processing, and grammar-constrained tool calling — that a bare inference session lacks. The execution model, then, is neither pure interpretation nor whole-model compilation; it is graph partitioning with vendor offload.

It helps to see these three as points on a bind-time spectrum — the moment at which the abstract model gets bound to concrete machine code. llama.cpp binds late, at run time, inside a general interpreter: maximum flexibility, minimum specialization. MLC-LLM binds early, at build time, into a frozen artifact: maximum specialization, minimum flexibility. ONNX Runtime binds in the middle, at session-creation time, when the graph meets the available EPs: a negotiated split that adapts to whatever accelerator is present without a full recompile. Every downside that follows — llama.cpp’s architecture lag, MLC’s compile tax, ONNX Runtime’s silent fallbacks — is the direct, predictable consequence of where each project chose to bind. There is no free lunch on this axis; there is only the trade you are willing to make.

Hardware backends: how each runtime reaches the silicon

llama.cpp reaches accelerators through GGML backends compiled from the same source tree. You build with -DGGML_METAL=ON for Apple Silicon, -DGGML_CUDA=ON for NVIDIA including Jetson, or -DGGML_VULKAN=ON for a cross-vendor GPU path that runs well even on integrated Radeon 780M-class GPUs. A ROCm path exists for AMD, a SYCL path for Intel, and the default CPU build leans on AVX2 or AVX-512 and ARM NEON. Each backend is a hand-maintained set of kernels, which is why coverage is broad but uneven — Metal and CUDA are mature, newer paths less so.

MLC-LLM reaches silicon through whatever backend TVM can emit code for: CUDA, ROCm, Metal, Vulkan, OpenCL, and WebGPU. Because the kernels are generated rather than hand-written, adding a target is, in principle, a compiler problem rather than a hand-coding problem — the same model description compiles to a Metal library for an iPhone and a WGSL bundle for a browser tab. In practice the mobile and web paths are the ones MLC invests in most, and they are its differentiator.

ONNX Runtime reaches silicon through its EPs, and this is the only one of the three with first-class, production NPU paths. The QNN execution provider targets Qualcomm’s Hexagon Tensor Processor directly, and onnxruntime-genai with the QNN EP is what powers local Phi-class models on Snapdragon-based Copilot+ PCs at NPU speed. CoreML lets ONNX Runtime dispatch to Apple’s Neural Engine and GPU; NNAPI does the same for Android accelerators; DirectML covers any DirectX 12 GPU on Windows. This coverage map — which runtime reaches which accelerator on which SoC — is where the three projects visibly separate, as the next figure shows.

A subtlety worth internalizing: NPUs are not general-purpose GPUs, and that constrains what an EP can offload. The Hexagon HTP and Apple’s Neural Engine are fixed-function-leaning matrix engines that prefer static shapes and integer arithmetic, and they do not implement every operator an autoregressive transformer needs. In practice this means the LLM must often be exported with a fixed maximum sequence length, quantized to the integer scheme the NPU expects, and structured so the attention and sampling steps that the NPU cannot run cleanly are handled elsewhere. ONNX Runtime’s value is that it manages this split for you; the risk, revisited in the failure-modes section, is that the split is not always the one you assumed. Neither llama.cpp nor MLC-LLM attempts this NPU offload today, which is not an oversight — it is a different design center. They optimize the GPU and CPU paths that give them predictable, debuggable behavior across the widest range of devices, and they leave the vendor-specific NPU complexity to the runtime that was built around it.

Quantization formats: GGUF K-quants vs MLC group quantization vs ONNX integer quant

Quantization is where the three runtimes differ most in mechanism, and it drives both memory footprint and accuracy. llama.cpp’s GGUF format carries a rich family of quantization types. The K-quants (Q4_K_M, Q5_K_M, and relatives) use a super-block structure — 256 weights per super-block, subdivided into 16- or 32-element sub-blocks, each with its own scale and minimum so that locally sensitive weight ranges keep more precision. Q4_K_M is the community’s default sweet spot, holding most of the model’s quality at roughly a quarter of the FP16 size — about 4.5 GB for a 7B model. Above that sit the IQ (importance) quants, which use non-linear codebooks and lookup tables and are designed to be paired with an importance matrix (imatrix): a per-tensor table recording the mean squared activation magnitude at each weight position across a calibration set, used to bias scale selection toward the weights that typical inputs exercise most, minimizing a weighted mean-square error. That imatrix machinery is why a well-calibrated IQ3 or IQ4 quant can beat a naive 4-bit scheme at the same bit width.

MLC-LLM uses group quantization tied to its compiler. Weights are quantized — typically 4-bit with a 16-bit scale per group, in formats named like q4f16_1 — and the dequantization is fused directly into the generated matmul kernel. Because quantization and code generation are one pipeline, the dequant path is specialized for the target GPU rather than being a generic operator.

ONNX Runtime quantizes at the graph level. Its tooling supports post-training integer quantization (FP32 to INT8, and increasingly INT4 for LLM weight-only quantization), with the quantized model then executed by an EP — QNN, for example, runs the quantized graph on the Hexagon HTP backend. The unifying idea across all three, and a useful mental model, is the same load-and-dequant pipeline shown below: weights are optionally calibrated, packed into blocks with scales, written to an artifact, then memory-mapped and dequantized inside the compute kernel at run time. If you want the accuracy side of this in depth, our FP8 vs INT8 vs INT4 quantization benchmark measures where each bit width starts to hurt.

Deeper analysis: quantization, throughput, memory, and portability

Put the three side by side and the trade-offs stop being abstract. The matrix below is the compressed form of everything above; the tokens-per-second figures are illustrative order-of-magnitude bands for a 7B-class model at 4-bit on a modern phone or laptop SoC, not measured benchmarks — treat them as shape, not score, and validate on your own hardware. Published, apples-to-apples on-device numbers remain scarce precisely because the runtimes target different accelerators, which is the point.

Dimension	llama.cpp	MLC-LLM	ONNX Runtime
Backends / accelerators	Metal, CUDA, Vulkan, ROCm, SYCL, CPU (AVX/NEON)	Metal, Vulkan, OpenCL, WebGPU, CUDA, ROCm	CoreML, NNAPI, QNN (NPU), DirectML, CUDA, TensorRT, OpenVINO, WebGPU, CPU
Quant formats	GGUF K-quants (Q4_K_M…), IQ + imatrix	Group quant (q4f16_1), fused dequant	INT8 / INT4 weight-only, per-EP
Model coverage	Broad, hand-written per arch	Needs a compile per model + target	Broad via ONNX export; genai loop for LLMs
Mobile packaging	Community bindings, DIY	First-class iOS / Android / web SDKs	Official mobile + genai packages
Throughput profile (illustrative)	~15-40 tok/s CPU/GPU mix	~20-50 tok/s on tuned mobile GPU	~20-60 tok/s on NPU via QNN
Memory footprint	Lowest with K-quants + mmap	Low; weights + compiled lib	Moderate; graph + EP overhead
Dev ergonomics	Easiest: one binary, one file	Hardest: build toolchain per target	Middle: export + EP config
Portability	Source-portable, rebuild per backend	One source, recompile per target	One graph, swap EPs

Read the matrix along its columns and a personality emerges for each runtime. llama.cpp optimizes for the shortest path from “I have a GGUF file” to “it runs,” and its K-quant plus mmap combination gives it the lowest practical memory footprint of the three — the OS pages in only the weights a forward pass touches, so a 4-bit 7B model can start responding before the whole file is resident. That is a genuine mechanism-level advantage on RAM-constrained devices, and it is why llama.cpp dominates the hobbyist and desktop-local scene. Its throughput on a GPU backend is competitive, but on a phone it leans on CPU or Vulkan rather than the NPU, so it leaves the most efficient silicon on the table.

MLC-LLM optimizes for reaching GPUs everywhere from one model description. Its throughput advantage shows up specifically on mobile GPUs, where a TVM-generated Metal or Vulkan kernel, with dequantization fused in, can outrun a generic operator path. The unique capability nobody else matches natively is the browser: compiled to WebGPU, an MLC model runs inside a tab with no server, which is a category of deployment the other two do not address. The price is the compile step — the artifact is specialized, so a new model or a new target means another build.

ONNX Runtime optimizes for hardware reach, and its column is the widest for a reason: it is the only runtime here with a production NPU story. On a Snapdragon device, the QNN EP puts the transformer on the Hexagon Tensor Processor, which is both faster and dramatically more power-efficient than a CPU or GPU path — decisive on a battery. That efficiency, not peak tokens per second, is ONNX Runtime’s real edge on an SoC. The cost is pipeline complexity: you export to ONNX, quantize appropriately for the target EP, and accept that an operator the EP cannot run silently falls back to CPU, which can quietly erase the accelerator’s benefit. For how these on-device profiles compare to server engines, our SGLang vs vLLM vs TensorRT-LLM benchmark covers the datacenter end of the same spectrum, and the on-device SLM inference benchmark on Jetson grounds the small-model numbers.

Portability deserves a precise word because all three claim it and mean different things. llama.cpp is source-portable: the same code compiles for every backend, but you rebuild per backend and per architecture support. MLC-LLM is compile-portable: one model description targets many devices, but each target is a separate compiled artifact. ONNX Runtime is graph-portable: one ONNX file runs anywhere an EP exists, and you swap EPs without changing the model. None of these is “write once, run anywhere” — they are three different points on the portability-versus-specialization curve, and choosing among them is really choosing which kind of portability your project needs.

Dev ergonomics is the axis teams underweight and regret. It is not a soft concern; it sets the pace of every iteration. With llama.cpp, the loop is measured in seconds — download a GGUF, point the binary at it, read output — which is why it wins the prototyping phase decisively and why it is the runtime most engineers reach for first. MLC-LLM’s loop includes a compile, so a change to the model or the target is a build you wait on and occasionally debug, and the SDK, while genuinely first-class on mobile and web, brings a toolchain you must keep current. ONNX Runtime lands in between: exporting a model to ONNX is usually mechanical, but getting a specific EP to accept a quantized graph and actually run on the accelerator — rather than falling back to CPU — is where the real time goes. A useful rule is to let the last mile dictate the runtime and the first mile dictate the prototype: prove the idea in llama.cpp, then pay the ergonomic tax of MLC or ONNX Runtime only for the target that justifies it.

The KV cache is the last mechanism worth naming, because it shapes both memory and how each runtime scales with context. Every generated token appends a key and value vector per layer, and the runtime must store, grow, and index that cache across the whole generation. llama.cpp manages the cache in its own allocator with quantized-cache options that trade a little accuracy for a lot of headroom on small devices. MLC-LLM bakes cache handling into the compiled module, with paged strategies that keep the compiled kernels efficient across sequence lengths. ONNX Runtime’s genai layer manages the cache above the EP, which is precisely why the genai package exists — a bare inference session has no notion of a growing cache. On a memory-tight SoC, how a runtime represents this cache can matter as much as how it stores the weights, and it is the single largest reason a model that loads comfortably can still run out of memory deep into a long conversation.

Trade-offs, gotchas, and what goes wrong

Every one of these runtimes has a failure mode that the marketing pages omit, and knowing them ahead of time saves weeks.

MLC-LLM’s is compilation friction. The build is a real toolchain — TVM Unity, target SDKs, and per-model, per-target artifacts — and when a compile fails on a new architecture or a new OS version, you are debugging a compiler, not editing a config. Teams underestimate this and discover that “recompile for the new phone GPU” is a task with its own maintenance cadence.

llama.cpp’s is architecture lag and NPU absence. Because operators are hand-written in C, a genuinely new model family may not run until a maintainer implements its graph. And on mobile, llama.cpp does not target the NPU — it runs on CPU or, at best, a Vulkan GPU path — so on a Snapdragon phone it leaves the most power-efficient accelerator idle. For a desktop or a Jetson with a real GPU this is a non-issue; for a battery-powered handset it is the whole game.

ONNX Runtime’s is silent EP fallback and quantization fragility. The engine partitions the graph and assigns unsupported operators to CPU without erroring, so a model that “runs on the NPU” may in fact be shuttling tensors back and forth between HTP and CPU, and the promised speedup evaporates. Diagnosing this means inspecting the partition, not trusting that the EP name in your config took effect. Quantization adds a second trap: each EP has its own preferred quantization scheme, and a graph quantized for one may run degraded or not at all on another.

The gotcha common to all three is quantization accuracy loss. Four-bit weight-only quantization is usually safe, but push to 3-bit or 2-bit without an importance matrix or good calibration and you get plausible-looking output that fails on reasoning, code, or long context — the model does not crash, it just gets subtly worse, which is the most expensive kind of regression to catch in the field. Memory blow-ups are the other shared surprise: the weights are only part of the footprint. The KV cache grows with context length, and at long contexts on a small device it can rival the weights themselves, so a model that loads fine can OOM mid-conversation.

Thermal throttling is the failure mode that no benchmark captures, because benchmarks run cold. A phone or a fanless laptop sustains peak clocks for seconds, not minutes; a long generation heats the SoC, the governor drops frequency, and steady-state tokens per second can land well below the burst figure a datasheet or a first-token measurement suggests. This interacts directly with runtime choice: the NPU path that ONNX Runtime unlocks on Snapdragon runs at a fraction of the power of a CPU or GPU decode, so it not only starts faster, it stays fast because it generates less heat — an advantage that only appears once you measure sustained, not peak, throughput. Any on-device LLM runtime comparison that reports a single cold number is measuring the wrong thing for a device that has to run the model for more than a paragraph.

Practical recommendations

There is no single winner in an honest on-device LLM runtime comparison — there is a right tool per target, and the decision flowchart above compresses the logic. Match the runtime to the silicon and the team, not to a benchmark leaderboard.

Apple Silicon (Mac, iPhone, iPad): Start with llama.cpp for desktop and fast iteration — Metal is mature, GGUF is everywhere, and the one-binary workflow is unbeatable for prototyping. Move to MLC-LLM when you need a polished iOS app with tuned on-device GPU throughput and a real mobile SDK. Reach for ONNX Runtime with CoreML only if you specifically need the Neural Engine.
Android + Snapdragon: If you need the NPU — and on a battery you almost always do — use ONNX Runtime with the QNN execution provider; it is the only path that puts the model on the Hexagon HTP in production. If you are content with the mobile GPU, MLC-LLM gives you a clean Android SDK and strong Vulkan/OpenCL throughput.
Server-side CPU or a developer laptop: llama.cpp. Lowest memory footprint via K-quants and mmap, no toolchain, broad model support, and CPU paths tuned for AVX-512 and NEON.
Web browser: MLC-LLM compiled to WebGPU is the only mainstream native option — no server, model runs in the tab.
Windows Copilot+ PCs and DirectX GPUs: ONNX Runtime with DirectML or QNN, which is also the shortest path to NPU acceleration on Windows on Arm.

The meta-recommendation: prototype on llama.cpp because it is the fastest to a working demo, then migrate to MLC-LLM or ONNX Runtime when a specific target’s GPU or NPU becomes the constraint. Do not compile with MLC or fight EP partitioning until you have proven the model is worth shipping.

Frequently Asked Questions

Which on-device LLM runtime is fastest?

There is no single fastest runtime — it depends on the accelerator. On a Snapdragon NPU, ONNX Runtime with the QNN execution provider typically wins on both speed and power efficiency because it uses the Hexagon HTP. On a mobile GPU, MLC-LLM’s compiled Metal or Vulkan kernels often lead. On a desktop CPU or a CUDA GPU, llama.cpp is highly competitive. Always benchmark on your exact SoC, since published cross-runtime numbers are scarce and rarely apples-to-apples.

What is the difference between GGUF and MLC quantization?

GGUF, used by llama.cpp, stores weights in K-quant or IQ formats with a super-block structure — a scale and minimum per sub-block — and optionally an importance matrix for accuracy. MLC quantization is group quantization (for example q4f16_1) whose dequantization is fused directly into TVM-generated GPU kernels. GGUF artifacts are portable across llama.cpp backends; MLC artifacts are compiled per target, so the two are not interchangeable.

Can these runtimes use a phone’s NPU?

Only ONNX Runtime does so in production today, via the QNN execution provider for Qualcomm Hexagon and, on Apple, CoreML for the Neural Engine. llama.cpp targets CPU and GPU (Vulkan/Metal) but not the mobile NPU. MLC-LLM targets mobile GPUs through Metal, Vulkan, and OpenCL rather than the NPU. If NPU acceleration and battery life are hard requirements, ONNX Runtime is the default choice.

Do I need to compile the model for MLC-LLM?

Yes. MLC-LLM’s design compiles the model ahead of time with Apache TVM into a device-specific binary and weights artifact, and you repeat that build per model and per target platform. This is the source of its specialization advantage and its main operational cost. By contrast, llama.cpp runs any supported GGUF with a prebuilt binary, and ONNX Runtime runs any exported ONNX graph without a per-target compile.

How much memory does a 7B model need on-device?

At 4-bit (for example GGUF Q4_K_M), a 7B model’s weights occupy roughly 4 to 4.5 GB, illustratively. Budget additional memory for the KV cache, which grows with context length and can rival the weights at long contexts on a small device. llama.cpp’s memory-mapping keeps the resident footprint low by paging in only touched weights, which helps on RAM-constrained hardware.

Which runtime should I start with?

Start with llama.cpp. It has the shortest path from a downloaded GGUF file to running output, the lowest memory footprint, and no build toolchain, which makes it ideal for validating whether a model is even worth shipping. Migrate to MLC-LLM for tuned mobile-GPU or in-browser deployment, or to ONNX Runtime when you need NPU acceleration on Snapdragon or Windows on Arm.

On-Device LLM Runtimes (2026): llama.cpp vs MLC vs ONNX