ONNX Runtime 1.30 vs 1.29: What Changed for Edge AI in 2026

Last Updated: September 24, 2026

The most important changes in ONNX Runtime 1.30 are not the headline LLM features. They are three quiet default changes that can alter the latency, memory footprint, or binary contents of a model you have not touched in months. An FP16 model that ran on an Arm board’s CPU under 1.29.1 may now silently run in FP32. A custom CUDA build may now be missing the BF16 kernels it used to include. And FP4 mixture-of-experts kernels now compile in by default. Comparing ONNX Runtime 1.30 vs 1.29 is therefore less about what you can now do and more about what now happens without asking. The release, tagged v1.30.0 in September 2026, also brings real edge wins: fused LinearAttention kernels for Arm NEON and SVE, INT4 paged KV caches on CUDA, Go bindings, and Linux AArch64 packaging for the WebGPU plugin execution provider. This guide explains each change at the mechanism level and turns it into a tested upgrade plan.

What this covers: the exact scope of the 1.30 notes, the reference architecture with every change placed on it, the Arm and CPU kernel work, the paged INT4 KV cache mechanism, a per-target decision table, the upgrade decision tree, failure modes, and a practical checklist.

Context and Background

ONNX Runtime (ORT) is Microsoft’s open-source inference engine for models in the Open Neural Network Exchange (ONNX) format. It loads a graph, optimizes it, and hands subgraphs to hardware back ends called execution providers (EPs). On edge devices it competes with TensorFlow Lite (now LiteRT), ExecuTorch, Core ML, OpenVINO and llama.cpp. We compared the classic mobile runtimes in ONNX vs TFLite vs ExecuTorch vs Core ML, and ORT’s niche is clear from that analysis. It is the runtime you choose when one model artifact must run on x86 servers, Arm gateways, NVIDIA Jetson modules, Windows laptops with NPUs and browsers.

The project ships frequently. Through 2026 it has moved roughly monthly, and each release has pushed ORT further toward two ideas. The first is generative AI on constrained hardware: quantized KV caches, paged attention and mixture-of-experts kernels. The second is a plugin architecture in which EPs are packaged and versioned separately from the core runtime.

One scoping note matters for accuracy. The official v1.30.0 release notes state that they cover changes since ONNX Runtime 1.29.1. That is the baseline this article uses. Where we mention earlier releases (1.25 through 1.27) we rely on their published notes; for 1.25 and 1.26 we worked from summaries of those notes rather than the full text, so treat those one-line descriptions as orientation, not a changelog. We do not describe the contents of 1.28 or 1.29 beyond that 1.28 moved the ONNX opset target to ONNX 1.22, because we have not verified the rest. If you are jumping from 1.27 or earlier, read the 1.28 and 1.29 notes as well before upgrading.

Two other caveats apply. First, the upstream notes say the highlights were prepared with AI assistance, so this article leans on the pull-request numbers and flag names rather than the summary prose. Second, the notes publish no benchmark numbers. Any performance figure below is either labelled as an estimate with arithmetic shown or omitted.

The ONNX Runtime 1.30 Reference Architecture: Where Each Change Lands

Direct answer: ONNX Runtime 1.30 changes three layers. The CPU execution provider gains fused LinearAttention kernels for AVX-512, Arm64 NEON and SVE, plus a hardware gate on FP16 Gemm and MatMul. The CUDA EP gains INT4 paged KV caches and new default build kernel sets. Packaging adds Go bindings and a Linux AArch64 WebGPU plugin EP.

Figure 1: ONNX Runtime 1.30 architecture. A session runs graph optimizations, the partitioner assigns nodes to execution providers, and 1.30’s changes land mostly in the CPU EP’s MLAS kernels, the CUDA EP’s paged attention path, and the separately versioned plugin EPs.

The figure traces a model from load to kernel. An InferenceSession parses the ONNX or ORT-format file and runs graph optimizations such as constant folding, operator fusion and layout transforms. The partitioner then asks each registered EP, in priority order, which nodes it can run. Whatever no accelerator claims falls to the CPU EP, which is built on MLAS, ORT’s internal math library. Go bindings sit beside the session because they wrap the same C API that every other language binding uses.

Why the partitioner makes the FP16 change dangerous

Understanding the partitioner explains why the most disruptive 1.30 change is easy to miss. EP assignment happens per node, and the CPU EP is the catch-all. In a typical edge deployment, an accelerator EP claims most of the graph and a handful of nodes fall back to CPU. Those fallback nodes are often the ones nobody profiles.

In 1.30, CPU FP16 Gemm and MatMul are now gated on hardware acceleration (PRs #32301 and #32197). When a CPU-assigned FP16 node has no matching accelerated kernel, it falls back to FP32. The notes do not describe the mechanics, but in practice a fallback like this means casts around the node or an FP32 kernel running on upcast data. Either way, expect the weights and activations for that node to double in size, and any extra casts to cost memory bandwidth.

On a server that is a rounding error. On a 2 GB Arm board running an FP16 vision transformer entirely on CPU, it can be the difference between fitting in memory and swapping. The intent of the change is sensible: an unaccelerated FP16 kernel on a CPU without native FP16 arithmetic is often slower than FP32 anyway, because it emulates half precision in software. But “sensible default” and “no behavior change” are different things, and your latency and RSS numbers will tell you which one you got.

Why plugin EPs change how you version a deployment

The second architectural point is the plugin EP model. Since 1.25, ORT has shipped a CUDA plugin EP that loads as a separate library rather than being compiled into the core package. The 1.30 notes list plugin versions WebGPU 0.4.0 and CUDA 0.2, and the WebGPU plugin EP packaging now supports Linux AArch64 (PRs #32287, #31960, #31970).

This splits your version matrix. Before plugins, “ORT 1.29.1 with CUDA” was one artifact. Now a deployment is a tuple: core runtime version, plugin EP version, and driver or toolkit version. A plugin EP can advance on its own cadence, and a core upgrade does not necessarily imply a plugin upgrade. For fleets of edge devices this is good news, since you can update a GPU back end without re-validating the CPU path. It is also a new way to ship an untested combination. Pin all three components in your manifest, not just the ORT wheel.

The 1.30 notes also add LoRA adapter support with plugin EP allocators (PR #32221), a CUDA plugin device-discovery fix on WSL, and Windows ARM64 packaging improvements. LoRA through plugin allocators matters for edge LLM work where a base model is shared and small adapters are swapped per task. Our reading of the entry is that adapter tensors can now be allocated through the plugin EP’s own allocators instead of only the built-in ones; check PR #32221 for the exact memory path before relying on it.

Why Go bindings matter more at the edge than in the cloud

The new Go bindings for the ORT C API (PR #29615) look like a convenience feature. For IoT architectures they are more than that. A large share of edge gateway software, including protocol bridges, MQTT brokers, and fleet agents, is written in Go because it cross-compiles to static binaries for Arm easily. Until now, putting ORT inference inside such a gateway meant community wrappers or a sidecar process speaking gRPC or HTTP to a Python server.

Official bindings over the C API remove a process boundary. For a gateway that scores sensor windows at 100 Hz, eliminating a local RPC hop removes serialization, a context switch and a failure domain per inference. The trade-off is that the bindings use cgo, so you give up Go’s pure static cross-compilation. They also load the native ORT shared library at run time, so you must ship a matching library for each target architecture. That is manageable, but it changes your build pipeline, which we return to in the gotchas section.

Security hardening that affects model loading

1.30 also hardens model parsing. The notes list a nested graph depth limit and canonicalization of external-data paths (PRs #32344 and #32135), checked rounding in the BFC arena allocator, broad shape and rank validation, an allowlist of data types for LoRA adapters, and an update to Protobuf 33.6.

The external-data change deserves attention. ONNX models larger than 2 GB store weights in side files referenced by relative paths inside the model. Canonicalizing those paths is aimed at stopping a crafted model from pointing outside its own directory. The nested depth limit targets models with deeply nested control-flow subgraphs (If, Loop, Scan) that could exhaust the stack during parsing. If your pipeline loads models from partners or model hubs onto devices, these are real risk reductions. If you have unusual layouts, such as external data stored via symlinks in a different directory, test loading before rollout.

Deeper Analysis: The Kernel-Level Changes, Target by Target

The headline features cluster around two workloads. One is transformer and hybrid language models on CPUs, especially Arm. The other is LLM serving with paged KV caches on CUDA and WebGPU. This section explains the mechanism behind each change and who benefits.

Fused LinearAttention on NEON and SVE

The CPU EP gains fused LinearAttention kernels for AVX-512, Arm64 NEON and Arm SVE (PRs #31674, #32178, #32356). To see why this matters, recall how linear attention differs from softmax attention.

Standard attention computes softmax of Q times K-transpose, then multiplies by V. Its cost grows with the square of sequence length during prefill, and its KV cache grows linearly with every generated token. Linear attention replaces the softmax with a kernel feature map, which lets the computation be reordered into a recurrence. Each head keeps a fixed-size state matrix of shape key-dimension by value-dimension. For each new token, the state is updated with the outer product of that token’s key and value, often with a learned decay or gate. The output is the query multiplied by the current state.

The result is constant memory per head regardless of context length. That property is exactly what edge devices want. Recent hybrid architectures interleave linear-attention layers, such as the gated delta rule from the Gated DeltaNet line of research, with a smaller number of full attention layers. 1.30 also adds a compact GatedDeltaNet op with BF16 support on CUDA (PRs #32282, #32307) and expanded op coverage for Qwen-3.5 (PR #32106), which signals that ORT is tracking these hybrid designs.

Why does fusion matter? An unfused implementation runs the state update, gating and output projection as separate ONNX nodes. Each writes its intermediate tensor to memory and the next reads it back. On a Cortex-A class core with modest cache and a narrow memory bus, those round trips dominate. A fused kernel keeps the per-head state in registers or L1 cache across the update and output steps. NEON gives 128-bit vectors on every Arm64 core. SVE gives vector-length-agnostic code that scales from 128 to 2048 bits, so one kernel can use wider vectors on server-class Arm cores without recompiling.

The notes give no speedup figure, and we will not invent one. The practical point is structural. If your model uses linear-attention layers and runs on Arm CPU, 1.30 is the first release in which ORT has dedicated fused kernels for that path on NEON and SVE.

SVE i8mm INT8 QGEMM and INT4 prepacking

Quantized matrix multiply is the workhorse of edge inference. 1.30 adds Arm SVE INT8 QGEMM kernels using the i8mm extension (PR #31146) and SBGemm fast-math on Darwin Arm64 (PR #32152).

The i8mm extension adds matrix-multiply-accumulate instructions such as SMMLA. Rather than a dot product of two vectors, each instruction multiplies a small block of 8-bit values and accumulates into 32-bit results. A block-oriented instruction does more multiply-accumulates per instruction than dot-product instructions, which improves throughput for INT8 GEMM. On Linux you can check support in /proc/cpuinfo. Look for the i8mm flag, and for sve and svei8mm if you want the SVE variant. Older boards built on Armv8.0 cores do not have it.

INT4 weight prepacking on CPU was also optimized. Prepacking rearranges quantized weights into the layout the kernel wants once, at session creation, so every inference avoids that shuffle. Two related entries matter for correctness. KleidiAI Q4 prepacking is now rejected when scales are dynamic (PR #32068). KleidiAI is Arm’s micro-kernel library that MLAS can use for 4-bit matmul. Our reading of that entry is that the KleidiAI path assumes constant scales it can bake in at prepack time, so models whose scales are not constant initializers now take another kernel instead of a wrong one. There is also an Arm64 SymmQgemm INT16 overflow fix (PR #32057). Overflow bugs in quantized kernels show up as accuracy loss on specific inputs, not as crashes. If you saw unexplained quality drops in symmetric-quantized models on Arm64 under 1.29.1, re-test on 1.30.

Classic vision models are not left out

Not every change is about LLMs. The notes list better NCHWc convolution thread utilization, HardSwish fusion for MobileNetV3, AVX-512 Erf, and an NCHWc reorder for MobileCLIP-S0 (PRs #31660, #31957, #31958). NCHWc is MLAS’s blocked channel layout, where channels are grouped into vector-width chunks so convolutions vectorize cleanly.

HardSwish fusion is a textbook edge optimization. MobileNetV3 uses the HardSwish activation after many convolutions. Unfused, each activation is a separate pass over the feature map. Fused, it is applied while the convolution output is still in registers. The notes also add BF16 LayerNorm and RMSNorm on the CPU EP and AVX2 LayerNorm and RMSNorm kernels (PRs #31973, #31974).

Two platform entries round this out. MLAS AArch64 assembly now supports BTI (Branch Target Identification), an Armv8.5 control-flow-integrity feature. Distributions that build everything with branch protection enabled need hand-written assembly to carry BTI landing pads, otherwise the protection is weakened or the build breaks. RISC-V vector checks were also restored (PR #32406), continuing the RISC-V Vector CPU EP work that arrived in 1.26.

INT4 paged KV cache on CUDA

For GPU edge devices and small servers, the largest change is INT4 paged KV caches with per-channel scales, plus an is_causal attribute on PagedAttention (PRs #32515, #32225). Related CUDA work includes split-KV for paged FlashAttention decode (PR #32102), speculative decoding in paged XQA (PR #32340), and a variable-length causal convolution with state for continuous batching (VarlenCausalConvWithState, PRs #32168, #32290).

Figure 2: Paged attention with an INT4 KV cache. Prefill quantizes keys and values per channel into fixed-size blocks mapped by a block table; each decode step looks up the sequence’s blocks, splits reads across them, dequantizes with the stored scales and merges partial results.

The figure shows the two phases. Paging borrows the idea of virtual memory. Instead of reserving one contiguous KV buffer per sequence sized for the maximum context, the cache is carved into fixed-size blocks. A block table maps each sequence’s logical positions to physical blocks. Memory is allocated as tokens arrive, and fragmentation stays low even with many concurrent sequences of different lengths.

Quantizing that cache to INT4 cuts its size by 4x relative to FP16, before scale overhead. Per-channel scales mean each channel of the key and value vectors gets its own scale factor. This matters because KV activations often have a few outlier channels with much larger magnitudes. A single scale per tensor would waste most of the 16 INT4 levels on those outliers and crush the rest. Per-channel scaling keeps quantization error bounded per channel.

Split-KV decode addresses a different bottleneck. During decode, each step processes one query token per sequence, so there is little parallelism along the query axis. Splitting the KV sequence into chunks, computing partial attention per chunk in parallel, and merging the partial results keeps more of the GPU busy on long contexts. That is why it pairs naturally with paging.

A worked KV cache memory estimate

Here is an illustrative estimate, not a benchmark. Take a hypothetical decoder with 32 layers, 8 KV heads (grouped-query attention) and a head dimension of 128.

FP16 bytes per token = 2 tensors (K and V) × 32 layers × 8 heads × 128 dims × 2 bytes = 131,072 bytes, or 128 KiB.
INT4 bytes per token = the same count × 0.5 bytes = 32 KiB, plus scale storage.
At a 4,096-token context: FP16 needs 4,096 × 128 KiB = 512 MiB per sequence. INT4 needs 4,096 × 32 KiB = 128 MiB per sequence, plus scales.

On a GPU module with 8 GB of shared memory, where the weights of a 4-bit 8B-class model alone take several gigabytes, that 384 MiB saving per sequence is the difference between one concurrent long-context session and several. Scale overhead depends on how the implementation stores scales across blocks, so check PR #32515 for the exact layout before planning to the megabyte.

WebGPU: paged attention and GPT-OSS in the browser path

The WebGPU EP gets improved PagedAttention and metadata handling, support for GPT-OSS, INT8 KV cache block quantization, and convolution optimizations (PRs #31727, #32277, #32284, #32420). With the WebGPU plugin EP now packaged for Linux AArch64, the same WebGPU back end can run natively on Arm Linux boards with a capable GPU driver, not only in browsers.

Note the precision difference. WebGPU uses INT8 block quantization for its KV cache, while CUDA gets INT4 with per-channel scales. If you share one model across both back ends, expect different memory footprints and slightly different numerics. Validate outputs per back end rather than assuming parity.

CUDA build defaults and MoE kernels

The rest of the CUDA work targets larger GPUs but affects anyone who builds ORT from source. FP4 QMoE (quantized mixture-of-experts) kernels are now enabled by default in CUDA builds, with Windows build support added. Opt out with -Donnxruntime_USE_FP4_QMOE=OFF (PRs #32096, #32163).

The fpA-intB GEMM path, which multiplies floating-point activations by integer weights, now defaults to a compact kernel set. That set covers FP16 activations, INT4 and INT8 weights, scale-only quantization and block_size=32. BF16 activations, zero-points, bias, larger blocks and native Hopper kernels now require -Donnxruntime_USE_FPA_INTB_GEMM_FULL=ON (PR #32324). Compact kernel sets shrink binaries and build times, which helps embedded images. But a BF16 or zero-point model built against a compact binary will not get the kernel it used to get.

An opt-in FP8 DeepGEMM MoE decode path for Hopper is available via ORT_QMOE_FP4_DEEPGEMM=1. It is off by default and disabled on Windows (PR #32122). FP4 and FP8 GEMV kernels were also tuned for 48-SM SM121 GPUs (PR #32408). We have not verified which product that compute capability corresponds to, so we will not name one. Build dependencies moved to CUTLASS 4.7 and cuDNN Frontend 1.27, with fixes for CUDA 13 CCCL include paths in plugin builds (PR #32392).

Decision table: what 1.30 means for your target

Deployment target	What changed in 1.30	Risk if you upgrade blindly	Recommended action	Upgrade urgency
Arm Linux board, CPU EP only	NEON/SVE LinearAttention, SVE i8mm QGEMM, INT4 prepack, SymmQgemm fix, FP16 gating	FP16 models silently upcast to FP32 on cores without FP16 kernels	Check cpuinfo flags; profile FP16 models; consider an INT8 or FP32 export	High if you run hybrid linear-attention LLMs or symmetric INT8
NVIDIA GPU edge module, CUDA EP	INT4 paged KV, split-KV decode, speculative paged XQA	Package and CUDA major mismatch; kernel set changes in custom builds	Align CUDA major with the published packages; retest accuracy with INT4 KV	Medium; high for long-context LLM serving
Arm Linux with GPU, WebGPU plugin EP	AArch64 plugin packaging, paged attention, INT8 KV blocks, GPT-OSS	Plugin version 0.4.0 drift versus core	Pin core and plugin versions together	Medium
x86 gateway with AVX-512	AVX-512 LinearAttention and Erf, AVX2 norms	Low	Standard regression test	Low to medium
Windows on Arm device	ARM64 packaging improvements, plugin EP fixes	Low to medium	Test the plugin EP load path	Medium
Go-based gateway software	New official Go bindings over the C API	Build pipeline changes due to cgo	Prototype in one service before fleet rollout	Opportunistic
Custom CUDA source builds	FP4 QMoE on by default; compact fpA-intB kernels	Missing BF16, zero-point or bias kernels	Set build flags explicitly in CI	High

The table compresses the argument of this post. For CPU-only Arm deployments, the upgrade is mostly upside with one measurable risk. For source builds, the upgrade is mostly a configuration exercise. For everyone else, it is a normal release.

The Upgrade Path: From 1.29.1 to 1.30 Without Surprises

The migration is not hard, but it has an order. Resolve toolkit questions first, then precision behavior, then build flags, then validation.

Figure 3: Upgrade decision tree for moving from ONNX Runtime 1.29.1 to 1.30. Resolve CUDA major version first, then check FP16 nodes on the CPU EP, then set the fpA-intB and FP4 QMoE build flags explicitly, and finish with a regression test on accuracy, latency and memory.

The tree reads top to bottom. Each diamond is a question you should be able to answer from your deployment manifest. Each rectangle is an action with a concrete flag or test.

Step 1: settle the CUDA 13 migration

ONNX Runtime CUDA 13 migration is not a 1.30 change, but 1.30 is where postponing it gets uncomfortable. The 1.27 release notes deprecated CUDA 12 packages and told users to move to CUDA 13 as soon as possible. The 1.30 notes include fixes for CUDA 13 CCCL include paths in plugin builds, which tells you where the maintainers’ build attention is.

On embedded NVIDIA platforms the CUDA version is usually tied to the board support package, so you cannot always pick freely. Check which CUDA major your platform ships, then check which ORT packages target it. If they diverge, you are building from source, and the build-flag steps below are mandatory rather than optional.

Step 2: find FP16 nodes that will land on the CPU

Run your model with verbose logging and read the node placement, or enable profiling and inspect which EP ran each node. Any FP16 Gemm or MatMul assigned to the CPU EP is a candidate for the new FP32 fallback. The check is quick:

import onnxruntime as ort

so = ort.SessionOptions()
so.log_severity_level = 0          # verbose: logs node-to-EP assignment
so.enable_profiling = True         # writes a JSON trace with per-node EP and timing

sess = ort.InferenceSession("model_fp16.onnx", so,
                            providers=["CPUExecutionProvider"])
print(ort.__version__, sess.get_providers())
# run a representative batch, then:
print(sess.end_profiling())       # path to the trace file

Then check the hardware. On Arm Linux, grep -o 'asimdhp\|fphp' /proc/cpuinfo | sort -u tells you whether the cores advertise half-precision arithmetic. The asimdhp flag indicates Armv8.2 FP16 SIMD. Cores built on Armv8.0, such as the Cortex-A72 in a Raspberry Pi 4, lack it, so expect the FP32 path there. Even on cores that advertise it, confirm by measuring, since the gate depends on ORT having a matching accelerated kernel for that CPU.

Compare peak resident memory and p95 latency between 1.29.1 and 1.30 on the same device. If memory grows and you are near the limit, you have three options. Export an FP32 model deliberately, so behavior is explicit. Quantize to INT8 or INT4 weights, which on Arm benefits from the new QGEMM and prepack work. Or move the FP16 nodes to an accelerator EP.

Step 3: set CUDA build flags explicitly

If you build ORT yourself, stop relying on defaults. Put both flags in your build script with deliberate values:

./build.sh --config Release --use_cuda --parallel \
  --cmake_extra_defines onnxruntime_USE_FPA_INTB_GEMM_FULL=ON \
                        onnxruntime_USE_FP4_QMOE=OFF

Choose FPA_INTB_GEMM_FULL=ON if any production model uses BF16 activations, zero-point quantization, bias in quantized GEMM, block sizes other than 32, or native Hopper kernels. Choose USE_FP4_QMOE=OFF if you never run FP4 mixture-of-experts models and want smaller binaries and faster builds, which matters for embedded images and CI caches. Writing the flags down means the next default change cannot surprise you.

Step 4: re-validate quantized models on Arm

Because of the SymmQgemm INT16 overflow fix and the KleidiAI dynamic-scale guard, quantized outputs on Arm64 may differ between versions. In most cases that is a correction, not a regression. Still, compare outputs on a fixed evaluation set rather than a handful of smoke inputs. Overflow bugs are input-dependent by nature.

Step 5: pin plugin EP versions and test loading

If you use the WebGPU or CUDA plugin EPs, record their versions next to the core version. Test the plugin registration path on each target OS. The 1.30 notes specifically mention a WSL device-discovery fix and Windows ARM64 packaging changes, which suggests these paths were rough before and could still vary by platform.

Step 6: soak test with realistic concurrency

Paged KV caches and split-KV decode change memory allocation patterns. Allocation-related bugs often appear only under concurrency and long runtimes. Run a soak test at your real concurrency and context length for hours, not minutes, and watch memory growth. The BFC arena checked-rounding change is a hardening fix, but it touches the allocator, so allocation behavior is worth observing.

How 1.30 Fits the 2026 Release Arc

A single release makes more sense when you see where the project has been heading. The timeline below lists only features verified from each release’s own notes.

Figure 4: ONNX Runtime releases from April to September 2026. Each box names one verified headline change. The 1.28 and 1.29 contents are deliberately left out beyond the ONNX 1.22 target, because this article does not verify them.

The arc has three threads, and 1.30 continues each.

Plugin EPs. Version 1.25, released in April 2026, introduced the first CUDA plugin EP and raised the build requirement to C++20 (MSVC 19.29+, GCC 10+, Clang 10+). Version 1.27 added zero-copy I/O for plugin EPs. Version 1.30 adds LoRA support through plugin EP allocators and ships the WebGPU plugin for Linux AArch64. The direction is clear: accelerators become separately shipped components. That mirrors what OpenVINO has done with its device plugins, a pattern we traced in OpenVINO 2026.4 vs 2025.4.

Quantized generative inference. Version 1.27 added a quantized KV cache for CPU grouped-query attention, a quantize_static calibration cache and the FLOAT8E8M0 data type, which is used as a shared exponent in microscaling formats. Version 1.30 extends quantized KV to INT4 paged caches on CUDA and INT8 blocks on WebGPU. The runtime is steadily taking over techniques that used to require dedicated LLM engines.

Edge CPUs beyond x86. Version 1.26 added optional memory-mapped loading of .ort files and a RISC-V Vector CPU EP. Memory mapping lets the OS page weights in on demand and share them across processes, which reduces startup time and RSS on constrained devices. Version 1.30 adds Arm NEON and SVE LinearAttention, SVE i8mm QGEMM, BTI support and restored RISC-V vector checks. Non-x86 CPUs are clearly first-class targets now.

An Original Thesis: ORT Is Becoming a Two-Speed Runtime

Most coverage of a release like this lists features. The more useful observation is structural. ONNX Runtime is splitting into two components that move at different speeds and deserve different upgrade policies.

The first is a slow, conservative core: the session, graph optimizer, CPU EP and MLAS kernels. Its changes in 1.30 are about safety and predictability. Hardware-gated FP16 means the CPU EP refuses to run a slow emulated path. The KleidiAI guard refuses a kernel whose assumptions do not hold. Graph-depth limits and path canonicalization refuse suspicious models. Even the new Arm kernels are the kind of change that makes existing models faster without changing their semantics.

The second is a fast, experimental accelerator layer: the CUDA and WebGPU paths and their plugin packages. Its changes are about chasing the newest model architectures. That means FP4 MoE, speculative decoding, GPT-OSS support, hybrid linear-attention ops and opt-in Hopper DeepGEMM paths behind environment variables.

For edge teams, the implication is a policy, not a feature. Treat core upgrades like OS updates. They are cautious and valuable, and should be adopted within a release or two once you verify the FP16 behavior. Treat accelerator and plugin upgrades like application dependencies, pinned per product and advanced only when a model needs the new capability. The build-default changes in 1.30 are the first time this split has shown up as an explicit compatibility notice. Teams that version the two layers separately will absorb future releases with less pain.

This thesis also explains the FP16 decision. A runtime that wants its CPU core to be a predictable fallback cannot run slow emulated kernels silently. Upcasting to FP32 is the predictable choice, even though it surprises people who assumed FP16 always means smaller and faster.

Trade-offs, Gotchas, and What Goes Wrong

No release is free. These are the specific ways a 1.30 upgrade can go wrong, with the reasoning behind each.

Silent FP32 fallback inflates memory. The FP16 gate is the most likely cause of a post-upgrade incident on CPU-only Arm devices. Symptoms are higher RSS, possible OOM kills on small boards, and changed latency, which may go up or down. Nothing errors. Catch it with a memory regression test, not a functional one.

Compact kernel sets break custom builds quietly. A custom CUDA build with default flags will compile and run. A BF16 or zero-point model may then take a slower path, or fail to find a kernel if no fallback exists. Either failure only shows up in the models that need the dropped kernels. Pin the build flags.

Bigger default binaries. FP4 QMoE on by default adds kernels most edge deployments never use. For flash-constrained images, turn it off explicitly.

INT4 KV is a quality decision, not just a memory decision. Four bits per value is aggressive. Per-channel scales help with outlier channels, but long-context retrieval tasks and exact-copy tasks tend to be the first to degrade under KV quantization. Evaluate on your own long-context tasks. Our INT4 vs INT8 vs FP8 quantization guide covers how to structure that evaluation.

Back-end numerics diverge. CUDA’s INT4 per-channel cache and WebGPU’s INT8 block cache produce different outputs for the same prompt. Fleets that mix back ends need per-back-end golden outputs and tolerance thresholds.

cgo complicates Go builds. The Go bindings wrap the C API through a small cgo shim, so cross-compiling needs a C toolchain for each target, and the ORT shared library they load at run time must be present on the device. Container images must ship the native library at a compatible version. Static single-binary deployment is no longer automatic.

Stricter validation can reject odd models. Shape and rank validation hardening, nested-depth limits and external-data path canonicalization may refuse models that loaded before. That usually signals a malformed or unusual model, but it will surface as a load failure in production if you do not test first.

Plugin version drift. With core and plugin EPs versioned separately, it is possible to deploy a combination nobody has tested. The WebGPU 0.4.0 and CUDA 0.2 plugin version numbers are themselves a reminder that these components are young.

The changelog summary is AI-assisted. The upstream notes say so. Where a highlight line and the PR disagree, trust the PR. Several interpretations in this article, such as the reason for the KleidiAI guard, are labelled as our reading for this reason.

Practical Recommendations

Most edge teams should upgrade to ONNX Runtime 1.30, but in a specific order, with specific tests. If you run hybrid linear-attention models or symmetric-quantized INT8 models on Arm CPUs, the upgrade is worth prioritizing. The fused kernels and the overflow fix are direct wins. If you serve long-context LLMs on NVIDIA edge hardware, INT4 paged KV caching is the feature to evaluate first, with a quality gate. If you only run classic vision models on x86, upgrade on your normal cadence.

For deciding between ORT and llama.cpp or MLC for on-device LLMs specifically, the features in 1.30 narrow the gap on KV cache efficiency but do not remove the trade-offs we laid out in llama.cpp vs MLC vs ONNX for on-device LLMs. ORT’s advantage remains one artifact across many back ends, not peak tokens per second on one device.

The checklist:

[ ] Confirm your current baseline is 1.29.1. If older, read the 1.28 and 1.29 notes first.
[ ] Record core version, plugin EP versions and CUDA major for every target in one manifest.
[ ] Move CUDA deployments to CUDA 13, or document why the board support package blocks it.
[ ] List every FP16 Gemm/MatMul node assigned to the CPU EP, using verbose logs or profiling.
[ ] Check /proc/cpuinfo for asimdhp, i8mm, sve and svei8mm on each Arm target.
[ ] Compare peak RSS and p95 latency between 1.29.1 and 1.30 on real hardware.
[ ] Set onnxruntime_USE_FPA_INTB_GEMM_FULL and onnxruntime_USE_FP4_QMOE explicitly in custom builds.
[ ] Re-run quantized-model accuracy on a fixed evaluation set on Arm64.
[ ] Evaluate INT4 paged KV on long-context tasks before enabling it in production.
[ ] Test model loading for any model with external data or deeply nested control flow.
[ ] Soak test at production concurrency for several hours and watch memory growth.

Frequently Asked Questions

What is new in ONNX Runtime 1.30?

ONNX Runtime 1.30 adds fused LinearAttention kernels for AVX-512, Arm64 NEON and SVE on the CPU execution provider. It adds INT4 paged KV caches with per-channel scales, split-KV decode and speculative decoding on CUDA. WebGPU gains improved paged attention, INT8 KV block quantization and GPT-OSS support, and its plugin EP is now packaged for Linux AArch64. The release also introduces official Go bindings over the C API, Arm SVE i8mm INT8 kernels, and security hardening in model loading. The official notes cover changes since 1.29.1.

Will upgrading to ONNX Runtime 1.30 break my FP16 model?

It will not break it functionally, but its behavior may change. In 1.30, FP16 Gemm and MatMul on the CPU execution provider run in FP16 only when hardware acceleration is available. Otherwise the nodes fall back to FP32. The model still produces results, but memory use for those nodes roughly doubles and latency can change. This mostly affects CPU-only deployments on Arm boards whose cores lack FP16 arithmetic, or FP16 nodes that fall back from an accelerator to the CPU. Profile peak memory and latency on real hardware before and after upgrading.

Does ONNX Runtime 1.30 improve Arm64 performance?

It adds several Arm-specific kernels, though the release notes publish no benchmark numbers. New work includes fused LinearAttention for NEON and SVE, SVE INT8 QGEMM using the i8mm extension, optimized INT4 weight prepacking, SBGemm fast-math on Apple Silicon, and BTI support in MLAS assembly. It also fixes an INT16 overflow in Arm64 symmetric quantized GEMM. Gains depend on whether your model uses these ops and whether your CPU supports SVE or i8mm. Measure on your own device; treat any third-party speedup figure as workload-specific.

What is an ONNX Runtime plugin execution provider?

A plugin execution provider is a hardware back end packaged as a separate library that the core runtime loads at run time, rather than one compiled into the main package. ONNX Runtime introduced its first CUDA plugin EP in 1.25. In 1.30 the plugin versions are WebGPU 0.4.0 and CUDA 0.2, and the WebGPU plugin now supports Linux AArch64. Plugins let accelerator support ship on its own cadence. The trade-off is a wider version matrix, so pin the core version, plugin version and driver version together for every deployment.

Do I need CUDA 13 for ONNX Runtime 1.30?

The CUDA 12 packages were deprecated in ONNX Runtime 1.27, whose notes told users to move to CUDA 13 as soon as possible. Version 1.30 includes fixes for CUDA 13 CCCL include paths in plugin builds, so CUDA 13 is clearly where the project’s build attention is. Check the packages published for 1.30 against your platform’s CUDA version. On embedded NVIDIA modules where the board support package fixes the CUDA version, you may need to build from source, and then you should set the CUDA build flags explicitly.

How much memory does an INT4 KV cache save?

Relative to FP16, INT4 stores each cached key and value in a quarter of the space, plus a small overhead for scale factors. As an illustrative estimate, a model with 32 layers, 8 KV heads and a head dimension of 128 needs about 128 KiB per token in FP16. That is 512 MiB for a 4,096-token context. At INT4 the same context needs about 128 MiB plus scales. Actual overhead depends on how ONNX Runtime stores per-channel scales. Always verify output quality, because long-context tasks degrade first under KV quantization.