OpenVINO 2026.4 vs 2025.4: What Changed for Edge LLMs and NPUs

Last Updated: September 23, 2026

If you last touched Intel’s inference stack in December 2025, the upgrade in front of you is bigger than a version bump suggests. OpenVINO 2026.4 vs 2025.4 is not a performance-tuning release pair. Between them sit four feature releases plus 2026.4 itself, a year boundary that finally deleted long-deprecated APIs, and a change in how OpenVINO makes LLMs fast. The 2025.4 recipe was “quantize to INT4 and pick the right device.” The 2026.4 recipe is “quantize, then choose a speculative decoding strategy, then decide which weights are allowed to stay in memory.” That second and third decision did not exist a year ago for most edge teams.

This matters now because 2026.4.0 landed on 16 September 2026, and Python 3.10 support ends with 2026.5. Anyone still pinned to 2025.x is about to be forced across the 2026.0 breaking-change line anyway. Better to cross it deliberately.

What this covers: a release-by-release diff from 2025.4 to 2026.4, how EAGLE-3, Multi-Token Prediction and DFlash actually work inside OpenVINO GenAI, the mechanics of MoE expert offload, the new NPU compile path, a migration checklist for every removed API, and the failure modes already reported on real hardware.

Context and Background

OpenVINO is Intel’s open-source toolkit for converting, optimising and running neural networks on Intel CPUs, integrated and discrete GPUs, and the neural processing units (NPUs) built into Core Ultra processors. Three layers matter for LLM work. The core runtime (openvino) loads a model in Intermediate Representation (IR) form and compiles it for a device plugin. The Neural Network Compression Framework (NNCF) handles quantization and weight compression. OpenVINO GenAI sits on top, providing LLMPipeline, VLMPipeline, WhisperPipeline and friends, with continuous batching, paged KV cache and speculative decoding built in. OpenVINO Model Server (OVMS) wraps all of it behind REST and gRPC endpoints.

In the on-device LLM runtime landscape, OpenVINO occupies a specific niche. It is not the most portable option; llama.cpp and MLC-LLM run on far more silicon, as covered in our comparison of llama.cpp, MLC and ONNX Runtime for on-device LLMs. What OpenVINO offers is the deepest access to Intel’s NPU and Xe graphics, including kernel fusions and compiler passes that generic runtimes cannot reach. For a fleet of Core Ultra industrial PCs, edge gateways or retail kiosks, that is the relevant trade.

The 2025.4.0 release (1 December 2025, with a 2025.4.1 patch on 18 December) was the last of the 2025 line. Its headline items were a preview of Mixture-of-Experts (MoE) optimisation on CPU and GPU, validated on Qwen3-30B-A3B; Gold support for Windows ML; NNCF’s ONNX backend gaining INT8 static post-training quantization, INT8/INT4 weight-only compression and SmoothQuant; an encrypted blob format for GenAI deployments; NPU batch support (models are reshaped to batch size 1 before compilation); and announced support for Core Ultra Series 3. Python 3.9 was dropped with 2025.4 and NNCF 2.19.0.

That baseline was already capable. What changed through 2026 is the architecture of the fast path, and the removal of the compatibility shims that let 2023-era code keep running. The primary source for every item below is the OpenVINO GitHub releases page and the official release notes.

The Release-by-Release Diff: 2025.4 to 2026.4

In short: between 2025.4 and 2026.4, OpenVINO removed the openvino.runtime namespace and several GenAI and NNCF APIs, required AVX2 on CPU, and added four speculative decoding paths (NPU draft models, EAGLE-3, MTP and a DFlash preview), MoE expert offload, lazy weight loading, an in-plugin NPU compiler preview, and NNCF FP8 for ONNX models.

The table below is the compressed view. Each row is anchored to the GitHub release body for that version.

Release	Date	Headline LLM/NPU changes	Breaking or removal items
2025.4.0	1 Dec 2025	MoE preview on CPU/GPU, Windows ML Gold, NNCF ONNX INT8 PTQ and INT4/INT8 weight compression, NPU batch support, encrypted GenAI blobs	Python 3.9 dropped
2026.0.0	23 Feb 2026	Speculative decoding on NPU, EAGLE-3 in GenAI (per Intel’s blog, not the release body), preview in-plugin NPU compiler, NNCF INT4 data-aware compression for 3D MatMuls (MoE), preview FP8-4BLUT, Whisper word timestamps, Phi-3-mini FastDraft	openvino.runtime removed, Type.undefined removed, StreamerBase put removed, NNCF TF backend removed, AVX2 required, RHEL 9 builds, manylinux_2_28, macOS x86 dropped
2026.1.0	7 Apr 2026	Preview OpenVINO backend for llama.cpp, preview release-weights API for NPU compile memory, Core Series 3 (Wildcat Lake) and Arc Pro B70, Prompt Lookup Decoding for VLMs, ICU dependency removed	start_chat()/finish_chat() deprecated
2026.2.0	28 May 2026	Gemma 4 E2B/E4B, Qwen3.5 and Qwen3.6 on CPU/GPU, INT4 KV-cache compression on GPU, GenAI custom-op extensions, Transformers v5.0	none new
2026.3.0	early Aug 2026	EAGLE-3 extended to LLMs and VLMs with Top-K on CPU/GPU/NPU, MoE offload to disk, lazy weight loading, NNCF FP8 for ONNX, Omni/ASR/Embedding pipelines, Xeon 6+	OVMS drops TensorFlow Serving API and stateful models
2026.4.0	16 Sep 2026	MTP for Gemma 4 and Qwen3.5/3.6, preview DFlash for Qwen on GPU, EAGLE-3 Tree Drafting, ITT/VTune on NPU, preview bounded dynamic shapes on NPU, OVMS idle-model unloading	Python 3.10 ends in 2026.5 (announced)

A few details deserve emphasis before we go deeper. First, the date for 2026.3.0 is genuinely ambiguous across sources; mirror sites record early August, while Intel’s community announcement came later. Second, there were patch releases (2026.2.1 on 17 June, and a 2026.3.1) that matter for bug workarounds discussed below. Third, the big removals all happened at 2026.0. Every later release restates the “Discontinued in 2026.0” list, which is why it is easy to misattribute a removal to the latest version.

Model coverage moved with the decoding work

The model lists are not trivia; they tell you which speculative paths are usable. Gemma 4 E2B and E4B arrived in 2026.2, alongside Qwen3.5 and Qwen3.6 on CPU/GPU and Gemma 4 31B and 26B-A4B on GPU only. Those are exactly the families that 2026.4’s Multi-Token Prediction targets. 2026.3 added SmolLM3-3B, LFM2-1.2B and LFM2.5-1.2B across CPU, GPU and NPU, and extended YOLO26 to GPU and NPU. 2026.4 added Gemma-3n on CPU; Kokoro-82M, Qwen3-VL-4B with EAGLE-3, Qwen3-ASR, Muse Glimmer 30B, Qwen3.8 27B, Gemma 4 12B, Hy-MT2-1.8B, DeepSeek OCR-2 and Granite 4.0 H Micro on CPU and GPU; and FLUX.2-Klein 4B plus Kokoro-82M on NPU.

Read that NPU column carefully. In 2025.4, the NPU additions were Gemma-3-4b-it and Qwen2.5-VL-3B-Instruct. A year later, the NPU gets small LLMs, a text-to-speech model and a 4B image generator. The NPU is increasingly a general GenAI device, but model validation still lags CPU and GPU by a release or two.

How the 2026 Fast Path Works: Speculative Decoding in OpenVINO GenAI

Speculative decoding is the most important conceptual change between the two versions, and OpenVINO now ships four distinct flavours of it. Choosing the wrong one wastes the gain entirely, and in one documented case produces a net slowdown.

Figure 1: Which OpenVINO GenAI speculative decoding path to use in 2026.4, by target device and model family.

The tree starts from the device because the NPU has different rules. On the NPU, the draft and target should both run on the NPU. On CPU or GPU, the model family decides: Gemma 4, Qwen3.5 and Qwen3.6 get native Multi-Token Prediction; models with a published EAGLE-3 head get EAGLE-3 in chain or tree mode; Qwen models on GPU can try the DFlash preview; everything else falls back to a classic draft model or Prompt Lookup Decoding.

The shared mechanism: draft cheaply, verify in one pass

Autoregressive decoding is memory-bandwidth bound. For each new token, the device streams essentially all the model’s weights through its compute units to produce a single output position. On an edge device with a 4-bit 8B model, that means several gigabytes of reads per token, while the arithmetic units sit mostly idle. Speculative decoding exploits that idle compute.

A cheap drafter proposes k candidate tokens. The expensive target model then scores all k positions in one forward pass, which costs roughly the same bandwidth as generating one token because the weights are read once. The target accepts the longest prefix of candidates that match its own choice, plus one corrected token. If the drafter is right three times out of five, you get four tokens for the price of one target pass plus the drafting cost. The output distribution is unchanged in principle, which is why Intel describes these methods as lossless. In practice, as one community benchmark noted, batched verification uses a different floating-point reduction order than sequential decoding, so a near-tie argmax can occasionally flip under greedy decoding.

The economics reduce to one inequality: drafting k tokens must cost much less than the target pass it amortises. Every design choice below is a way of making drafting cheaper or acceptance higher.

Classic draft models and FastDraft: what 2026.0 brought to the NPU

The oldest path pairs two separate models that share a tokenizer, for example Qwen3-14B as target and Qwen3-0.6B as draft. OpenVINO GenAI exposes this through openvino_genai.draft_model(path, device) passed to LLMPipeline, with num_assistant_tokens in GenerationConfig controlling how many tokens the drafter proposes per round, or assistant_confidence_threshold to stop drafting when the drafter’s confidence drops.

2026.0 extended this to NPUs, and Intel published a Phi-3-mini FastDraft model on Hugging Face, a draft model trained specifically for speculative decoding. On the NPU, Intel’s guidance is explicit: the preferred configuration puts both the main and the draft model on the NPU. The reason is physics, not software. A community characterisation on a Core Ultra 7 258V found that a GPU target with an NPU draft was a consistent net slowdown compared with GPU-only decoding, while the same draft on the CPU sped things up. The NPU pays a high fixed dispatch cost per inference. That cost dominates when you call a tiny model repeatedly, and cross-device synchronisation adds more.

Two NPU constraints surfaced in the same thread. Per-group asymmetric INT4 weights are not supported by the NPU compiler, so NPU-bound drafts must be exported with symmetric quantization (--sym). And passing a scheduler_config selects the continuous-batching pipeline, whose paged-attention scheduler the NPU does not support, so NPU drafts go through the stateful speculative pipeline instead. That also means assistant_confidence_threshold, a continuous-batching feature, is unavailable there.

EAGLE-3: a draft head that reads the target’s mind

EAGLE-3 (described in the EAGLE-3 paper on arXiv) replaces the separate draft model with a lightweight draft head that consumes the target model’s own hidden states. Intel’s March 2026 engineering blog on EAGLE-3 in GenAI summarises the two ideas that matter. The head fuses features from low, middle and high layers of the target rather than only the last layer. And it is trained with simulated inference-time noise, which keeps acceptance high across multiple drafting steps.

The mechanism has a practical consequence for OpenVINO. To feed the head, GenAI must modify the target model graph so that it exports extra hidden-state outputs from specific decoder layers. The 2026.3 source does this with a pattern-matching transform that looks for the dense feed-forward residual shape in each layer. That detail will matter in the failure-modes section.

EAGLE-3 first appeared in GenAI with 2026.0, including NPU deployment, according to Intel’s blog. 2026.3 extended the pipeline to both LLMs and VLMs, integrated it with continuous batching, and added Top-K sampling, on CPU, GPU and NPU. 2026.4 then added Tree Drafting (Top-K). In chain drafting (Top-1), the head proposes a single sequence; one early miss discards everything after it. In tree drafting, the head keeps the top K candidates at each step and the target verifies a small token tree in one pass using a tree attention mask. More candidates are verified per pass, so the expected accepted length rises, at the cost of a larger verification batch. The 2026.4 notes claim higher throughput on VLM pipelines versus chain drafting; they publish no numbers.

In code, the entry point is the same draft_model() call. The snippet below is adapted from Intel’s published EAGLE-3 sample; community reports on 2026.3.1 also pass eagle3_mode=True to draft_model, so check the signature in your installed version.

# Adapted from Intel's OpenVINO GenAI EAGLE-3 sample (2026.x). Verify flags against your version.
import openvino_genai as ov_genai

scheduler_config = ov_genai.SchedulerConfig()
scheduler_config.cache_size = 2  # GB of KV cache for continuous batching

draft = ov_genai.draft_model("models/qwen3-8b-eagle3-ov", "GPU")
pipe = ov_genai.LLMPipeline(
    "models/qwen3-8b-int4-ov", "GPU",
    scheduler_config=scheduler_config,
    draft_model=draft,
)

config = ov_genai.GenerationConfig()
config.max_new_tokens = 256
config.num_assistant_tokens = 5   # Intel suggests 3 to 7; keep it at 7 or below on NPU

print(pipe.generate("Explain PID control in two paragraphs.", config))

Intel’s tuning advice is to start at 5, sweep 3 to 7, and stay at or below 7 on NPU.

Multi-Token Prediction: the model drafts for itself

MTP is the 2026.4 headline. Some recent model families are trained with extra prediction heads that forecast tokens beyond the next one. At inference time those heads act as a built-in drafter: no separate draft model, no community-trained EAGLE head, and no tokenizer-matching problem. OpenVINO GenAI 2026.4 enables MTP speculative decoding for Gemma 4, Qwen3.5 and Qwen3.6 on CPU and GPU.

The trade-off is scope. MTP only works where the model vendor trained MTP heads and the exported IR preserves them. It is not listed for the NPU in 2026.4. For supported families, though, it removes the single biggest operational burden of speculative decoding: sourcing, exporting and version-matching a second artefact.

DFlash: a GPU-only preview for Qwen

2026.4 also previews DFlash acceleration for Qwen models on GPU, together with visual-token support aimed at Core Ultra Series 3. The release notes describe it only as reducing latency and speeding up GenAI pipelines; they do not document the mechanism in detail or publish numbers. Treat it as an experiment to benchmark on your own workload rather than a default.

What about Prompt Lookup Decoding?

Prompt Lookup Decoding (PLD) is the cheapest drafter of all. It searches the prompt for n-grams matching the recent output and proposes the continuation it finds there. It costs almost nothing and works well when output copies from input: summarisation, code editing, retrieval-augmented answers, form extraction. 2026.1 extended PLD to vision-language pipelines. For edge document-processing and inspection-report workloads, PLD is often the first thing to try, because it needs no second model at all.

MoE Offload, Lazy Loading and Memory: Running 30B on 16 GB

The second structural change is memory. In 2025.4, MoE support was a CPU/GPU preview, and a Qwen3-30B-A3B model needed its full weight set resident. 2026.3 changed that with three features that work together: MoE offload to disk, lazy weight loading, and (from 2026.1) the release-weights API for NPU compilation.

Intel’s 2026.3 claim is that MoE offload lets 30B MoE models like Qwen3-30B-A3B run on devices with 16 GB of memory at what Intel calls “acceptable TPS”. That phrase is Intel’s own; the release notes publish no throughput figure, and neither do we.

Figure 2: How MoE expert offload splits weights between resident memory and backing storage, and what happens on a cache miss.

The figure shows why MoE models are uniquely suited to this trick. A dense model touches every weight on every token, so offloading any of it puts storage bandwidth on the critical path of every step. An MoE model routes each token to a small subset of experts. Qwen3-30B-A3B has roughly 30B total parameters but only about 3B active per token. Most expert weights are idle most of the time.

The mechanism in the GPU plugin

In the Intel GPU plugin, offload is controlled by an OFFLOAD_RATIO property. The header documentation in the OpenVINO source describes it plainly. At 0, the default, all weights are resident on the GPU. At N, N percent of eligible expert weights are streamed on demand through an LRU (least recently used) cache, reducing GPU memory at the cost of latency. The release notes call this “offload to disk”; the plugin comment says “streamed from host memory”. On a unified-memory laptop, where the iGPU shares system RAM, the practical effect is that cold experts live in the memory-mapped model file on SSD and are paged in when the router selects them.

A pull request under review proposes an automatic mode (-1) that computes the ratio from available memory and model weight size. Reviewers pushed back on exactly the right point: if the ratio is computed at compile time from free memory, a browser open during startup can lock in a pessimistic ratio for the life of the process. Whether that auto mode shipped in 2026.4 is not stated in the release notes; check before relying on it.

The illustrative call looks like this. The property name comes from the GPU plugin and from community reproductions on 2026.3.1, not from a tutorial, so treat it as advanced configuration.

# Illustrative: MoE expert offload on the Intel GPU plugin (2026.3+). Verify property name on your build.
import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline(
    "models/Qwen3-30B-A3B-int4-ov",
    "GPU",
    OFFLOAD_RATIO=50,   # 0 = all experts resident; N = N% of eligible expert weights streamed via LRU
)

Worked example: what actually has to fit

Take Qwen3-30B-A3B in INT4. At roughly half a byte per parameter plus quantization scales, the weights come to around 16 GB in total, the large majority of it in expert tensors. The non-expert portion, meaning embeddings, attention projections, norms and router, is on the order of 1 GB. On a 16 GB machine, the operating system, the application and the KV cache also need room. The arithmetic is simple: you cannot keep the whole model resident, but you can keep all fixed weights plus a working set of hot experts.

Throughput then depends on the hit rate. If router decisions were uniformly random across experts, a 50 percent offload ratio would miss half the time. Real routing is skewed: some experts are popular, and consecutive tokens in the same context tend to reuse experts. The LRU cache exploits that locality. The cost of a miss is SSD read latency plus the copy, multiplied by however many experts miss in that layer. That is why the ratio is a dial, not a switch. Lower ratios keep more experts hot and run faster; higher ratios fit smaller machines and run slower. Community measurements on a Lunar Lake laptop show throughput falling steadily as the ratio rises, as you would expect.

Two inferences follow. Fast NVMe matters more than it did in 2025. And prompts from a single domain, with stable expert routing, will behave better than a mixed workload that thrashes the cache.

Lazy weight loading and release-weights

Offload manages steady-state memory. Initialisation has its own peak. Historically, compiling a model could hold the source weights, intermediate transformed copies and the compiled blob in memory simultaneously. On an NPU laptop, that transient peak, not the steady state, is often what kills the process.

2026.2 extended an optimised IR read mode with independently managed constant buffers to Linux, avoiding needless duplication of weight data. 2026.3 went further with lazy weight loading for IR and ONNX models, which automatically selects a loading and compilation path that minimises peak memory at initialisation. And the 2026.1 release-weights preview lets you mark an ov::Model so the NPU plugin frees source weights during compilation, which Intel says dramatically lowers peak memory for edge and client deployments. You must set the property on the model; it is applied during compilation.

The NPU Compile Path in 2026.x

On the NPU, the most consequential change is not a feature users see. It is where the compiler lives.

Figure 3: The NPU compile-and-run lifecycle in OpenVINO 2026.x, from lazy model read through compilation, weight release, inference and VTune tracing.

Historically, the NPU compiler shipped inside the NPU driver. That coupling meant a new OpenVINO feature could depend on a compiler that your OEM’s driver did not yet include, and many OEM laptop images lag Intel’s reference drivers by months. 2026.0 previewed integrating the NPU compiler into the NPU plugin, enabling ahead-of-time and on-device compilation without relying on OEM driver updates. In 2026.0 this was opt-in; the notes say developers “can enable this feature”. Intel’s own EAGLE-3 NPU sample passes NPU_COMPILER_TYPE="DRIVER", which shows the property that selects between compiler locations exists.

For a fleet operator, the value is control. You can ship one package that compiles consistently across machines with different driver versions, and you can compile ahead of time on a build server and ship the blob. The cost is package size and one more thing to validate: the plugin compiler and the driver runtime still have to agree.

Other NPU changes in the window

2025.4 added NPU batch support by reshaping models to batch size 1 before compilation. 2026.0 made speculative decoding and EAGLE-3 available on NPU. 2026.3 extended EAGLE-3 with Top-K to NPU and added SmolLM3-3B and the LFM2 family on NPU. 2026.4 brought two items for engineers rather than marketing. First, ITT (Instrumentation and Tracing Technology) profiling now covers the NPU, so VTune shows CPU, GPU and NPU execution on one timeline; before this, NPU time was largely a black box inside a single infer call. Second, a preview of bounded dynamic shapes on NPU for vision workloads, validated with the ESPCN super-resolution model. The NPU has historically required static shapes, which forces padding or multiple compiled variants for variable-resolution input. Bounded dynamic shapes let you declare an upper bound and compile once.

The practical LLM limits on NPU remain. Prompt and response budgets are set at compile time via properties such as MAX_PROMPT_LEN and MIN_RESPONSE_LEN, which appear in Intel’s NPU samples, and exceeding them is an error, not a slowdown. Quantization formats are narrower than on GPU; OpenVINO GenAI’s sample documentation says NPU LLMs must be exported with symmetric INT4 (--sym), using channel-wise quantization (--group-size -1) for models above 4B parameters. For a broader look at how Intel’s NPU compares with Jetson and Arm NPUs, see our edge AI inference hardware overview.

NNCF in 2026: FP8, MoE-Aware INT4 and a Smaller API

NNCF changed in two directions: it gained formats and lost frameworks.

On the gain side, 2026.0 added INT4 data-aware weight compression for 3D MatMuls. MoE layers store expert weights as a batched 3D tensor, and earlier data-aware algorithms only handled 2D weights, so MoE models were stuck with data-free INT4, which is less accurate. 2026.0 also previewed per-layer and per-group lookup-table quantization, called FP8-4BLUT, a codebook-based scheme. 2026.3 extended FP8 quantization to ONNX models. That matters if your deployment keeps ONNX as the interchange format across runtimes, a pattern we compared in ONNX vs TFLite vs ExecuTorch vs Core ML.

FP8 is not automatically better than INT8 or INT4 on the edge. It preserves dynamic range better for activations with outliers, but whether it runs natively depends on the device. Our INT4 vs INT8 vs FP8 edge NPU quantization guide covers the hardware side. For OpenVINO specifically, the safe pattern is INT4 weight-only compression for LLM weights, INT8 or FP8 for activations where the target device supports it, and measurement before commitment.

An illustrative NNCF call, labelled as such because enum names have moved between NNCF versions:

# Illustrative NNCF usage for 2026.x. Check enum names against your installed NNCF.
import nncf
import onnx

model = onnx.load("detector.onnx")
calib = nncf.Dataset(calibration_loader, transform_fn)

# INT8 static PTQ (the replacement path for create_compressed_model-style workflows)
int8_model = nncf.quantize(model, calib)

# FP8 activations and weights for ONNX (2026.3+), where the target supports it
fp8_model = nncf.quantize(model, calib, mode=nncf.QuantizationMode.FP8_E4M3)

On the loss side, 2026 removed a lot. create_compressed_model() is gone; use nncf.quantize() for INT8 and nncf.prune() for unstructured pruning. The TensorFlow backend is gone; Intel recommends PyTorch for training-aware methods and IR, PyTorch or ONNX for post-training methods. The experimental methods were removed too: NAS, Structural Pruning, AutoML, Knowledge Distillation, Mixed-Precision Quantization and Movement Sparsity. If your 2024 pipeline used any of those, it will not survive the upgrade unchanged.

Migration Checklist: Every Removed API from 2025.4 to 2026.4

This is where most upgrade time goes. The good news is that the API removals were announced as deprecations during 2025. The bad news is that many teams suppress warnings in production logs.

Figure 4: A migration sequence from OpenVINO 2025.4 to 2026.4, ordered so platform blockers surface before code changes and regression testing gates the release.

The order is deliberate. Platform checks come first because an AVX2 or OS problem invalidates everything else. Then pure code changes, then serving changes, then cache rebuilds and a regression gate.

Step 1: Platform and packaging

• AVX2 is now the CPU floor. The CPU plugin no longer supports SSE-only processors. Older Atom-class and Celeron-class industrial PCs are the risk. On Linux, check with grep -m1 -o avx2 /proc/cpuinfo.
• Builds moved from RHEL 8 to RHEL 9, and wheels moved from manylinux2014 to manylinux_2_28, so CentOS 7 no longer works due to glibc.
• macOS x86 is no longer supported. Debian 10 and Ubuntu 20.04 are discontinued.
• The legacy year-based APT and YUM repository layout (paths like 2025/) is unavailable from 2026.0. Update your repo definitions.
• OpenCV binaries were removed from the Docker images. If your container relied on them for pre-processing, install OpenCV yourself.
• Python 3.10 support ends in 2026.5. Moving to 3.11 or later now avoids a second forced upgrade in a few months.

Step 2: Core runtime API changes

The openvino.runtime namespace is removed. This is the change most likely to break old code, because it appeared in countless tutorials.

# Before (2025.x and earlier) - fails on 2026.x
from openvino.runtime import Core, Type
core = Core()
t = Type.undefined

# After (2026.x)
import openvino as ov
core = ov.Core()
t = ov.Type.dynamic   # Type.undefined was removed; use Type.dynamic

PostponedConstant changed its maker signature from Callable[[Tensor], None], which filled a pre-allocated tensor in place, to Callable[[], Tensor], which returns a new tensor. Illustratively:

# Illustrative only - adapt to your PostponedConstant usage
# Old (removed): def maker(out_tensor): fill(out_tensor)
# New:
def maker() -> ov.Tensor:
    return ov.Tensor(build_weights_array())

Also note three further deprecations, only the first of which has a removal date. Modifying RTMap runtime information through read-only ConstOutput objects in Python now warns and will be removed in 2027.0. Runtime auto shape and auto batch size reshaping are deprecated in favour of dynamic-shape models. And the legacy graph iteration path, re-enabled with ONNX_ITERATOR=0, is deprecated.

Step 3: GenAI API changes

Three GenAI removals and one deprecation affect application code.

• Predefined generation configs are removed. Build a GenerationConfig explicitly.
• The Whisper stateless decoder is removed. Re-export Whisper as a stateful model with a current optimum-intel.
• StreamerBase changes. The put method, the bool return type for streaming callbacks and the ChunkStreamer class are gone. Custom streamers implement the newer interface, and callbacks return a streaming status rather than a boolean.
• start_chat() and finish_chat() are deprecated for removal in a future major release. Pass a ChatHistory object to generate() instead.

An illustrative before-and-after for the chat and streamer changes:

# Illustrative - confirm exact class names in your openvino_genai version
import openvino_genai as ov_genai

def on_token(subword: str):
    print(subword, end="", flush=True)
    return ov_genai.StreamingStatus.RUNNING   # previously: return False

history = ov_genai.ChatHistory()
history.append({"role": "user", "content": "Summarise today's alarm log."})
reply = pipe.generate(history, config, on_token)   # replaces start_chat()/finish_chat()

Step 4: Serving and NNCF

OVMS 2026.3 dropped the TensorFlow Serving API; use the KServe API for classic models and the OpenAI-compatible API for generative ones. Stateful model support, originally for Kaldi audio models, was removed in the same release. The Kubernetes and OpenShift operator is deprecated in favour of the KServe operator. The notes have said since 2026.0 that the DAG scheduler would be removed in 2026.3 in favour of MediaPipe graphs, yet the 2026.4 notes still list it under deprecations; confirm on your OVMS build before assuming either way. For NNCF, replace create_compressed_model() as described above.

Step 5: Rebuild caches and gate

Compiled model cache blobs are version-specific. Delete them and let the new runtime rebuild, or recompile ahead of time for NPU. Then run the same accuracy and latency suite you used to qualify 2025.4. Speculative decoding changes latency distribution, not just the mean, so compare p50 and p95 time-per-output-token and time-to-first-token.

Decision Matrix: Which 2026.4 Feature for Which Edge Workload

Workload	Device	Best 2026.4 lever	Why	Avoid
Chat assistant on Gemma 4 or Qwen3.5/3.6	CPU or iGPU	MTP speculative decoding	No second model to manage	Stacking a separate draft on top
Multimodal inspection Q&A with a VLM	GPU	EAGLE-3 Tree Drafting	Claimed higher VLM throughput than chain	Tree mode on memory-tight devices without testing
Always-on small LLM, battery-sensitive	NPU	Both models on NPU, EAGLE-3 or FastDraft	NPU efficiency, no cross-device sync	NPU draft behind GPU target
30B MoE on a 16 GB laptop	iGPU	MoE offload with moderate ratio	Only way to fit	High ratio on slow storage
Document extraction, RAG answers	Any	Prompt Lookup Decoding	Output copies input; zero extra model	Separate drafters first
Qwen on Arc or Xe3	GPU	DFlash preview, benchmarked	Potential latency win	Treating a preview as default

Trade-offs, Gotchas, and What Goes Wrong

Every one of the new fast paths has already failed in public on someone’s hardware. None of these reports is a reason to avoid the features. All of them are reasons to test your own model pairing before shipping.

EAGLE-3 on MoE targets fails late and cryptically. A community report against 2026.3.1 (openvino.genai #4390) found that attaching an EAGLE-3 head to Qwen3-30B-A3B loads cleanly, then fails on the first generate() with “Port for tensor name last_hidden_state was not found”. The reporter traced it to the hidden-state transform, which matches the dense feed-forward residual shape. An MoE layer’s residual is an expert combine, so zero nodes match, and the zero-match branch returns silently. The issue is open and assigned to Intel engineers. Until fixed, do not assume EAGLE-3 heads work on MoE targets, even though published heads exist.

EAGLE-3 on CPU showed intermittent issues in 2026.3.1. Another report (openvino.genai #4425) describes freshly built pipelines occasionally accepting zero draft tokens, and occasionally crashing at the first generate. Build acceptance-rate monitoring into your pipeline; a silent zero-acceptance run looks correct but gives no speedup.

MoE offload with INT8 can intermittently hang on the second request. Issue #37736 reports that with INT8 Qwen3-30B-A3B and OFFLOAD_RATIO on an Arc 140V, the first generate() completes but the second on the same pipeline can hang; the reporter’s later update found it intermittent (5 of 7 runs hung). The same pattern with INT4 was clean. Per the issue thread, Intel had not reproduced it on its own 16 GB test machine. The reporter also noted that the release notes describe the GPU-plugin offload as reducing memory for INT4 models, so INT8 may simply be outside the tested scope. Use INT4 with offload.

NPU drafts behind non-NPU targets are slower. As covered above, cross-device drafting to the NPU lost to GPU-only decoding in community tests. Put the draft on CPU or GPU, or put both models on the NPU.

Speculative decoding can shift greedy output. Batched verification can change floating-point reduction order enough to flip a near-tie. If you have golden-output regression tests, expect a small number of legitimate diffs, and compare semantically rather than byte-for-byte.

Tree drafting costs memory. Verifying a token tree enlarges the verification batch and KV-cache writes. On memory-tight devices already using offload, that pressure competes with the expert cache.

Preview means preview. DFlash, the in-plugin NPU compiler, release-weights, bounded dynamic NPU shapes and OVMS idle-model unloading are all labelled preview. Preview features can change properties or behaviour between minor releases. Pin exact versions.

Suppressed deprecation warnings hide the migration cost. If your 2025.4 logs filtered DeprecationWarning, run the old code once with warnings enabled before upgrading. It is the cheapest inventory of what will break.

Practical Recommendations

Upgrade, but do it as two changes rather than one. First cross the 2026.0 compatibility line with no new features enabled: fix imports, streamers, chat handling, NNCF calls and platform issues, then prove parity with your 2025.4 baseline. Only then turn on speculative decoding and offload, one at a time, with metrics that can tell you whether they helped.

Choose the speculative path by model family, not by novelty. If you run Gemma 4 or Qwen3.5/3.6 on CPU or GPU, MTP is the obvious first test because it adds nothing to your artefact pipeline. If your model has a credible EAGLE-3 head and is dense, EAGLE-3 is next, with tree drafting for VLMs. If your workload copies from its input, try Prompt Lookup before anything else. On NPU, keep everything on the NPU.

Treat MoE offload as a capacity tool, not a speed tool. It lets a 30B-class MoE model run where it otherwise could not. Pair it with INT4, fast NVMe and a moderate ratio, and measure time-per-output-token under your real prompt mix.

Checklist:

• [ ] Verify AVX2, OS and Python 3.11+ across the whole fleet.
• [ ] Replace every openvino.runtime import and Type.undefined usage.
• [ ] Port custom streamers and start_chat() code to the new interfaces.
• [ ] Replace create_compressed_model() and any TensorFlow NNCF flows.
• [ ] Move OVMS clients off the TensorFlow Serving API.
• [ ] Re-export Whisper as stateful; rebuild all cache blobs.
• [ ] Log speculative acceptance rate in production, not just throughput.
• [ ] Pin exact versions for any preview feature.

Frequently Asked Questions

What is the biggest difference between OpenVINO 2026.4 and 2025.4?

The biggest difference is the LLM fast path. OpenVINO 2025.4 relied mainly on quantization and device choice. 2026.4 adds four speculative decoding options in GenAI: classic and FastDraft draft models including on NPU, EAGLE-3 with chain or tree drafting, native Multi-Token Prediction for Gemma 4 and Qwen3.5/3.6, and a DFlash preview for Qwen on GPU. It also adds MoE expert offload and lazy weight loading, and it removes the deprecated openvino.runtime namespace and several GenAI and NNCF APIs that 2025.4 still accepted.

Is openvino.runtime removed in OpenVINO 2026?

Yes. The deprecated openvino.runtime namespace was removed in 2026.0 and stays removed in 2026.4. Import directly from openvino instead, for example import openvino as ov and then ov.Core(). In the same release, openvino.Type.undefined was removed in favour of openvino.Type.dynamic. Code that still uses the old imports fails at import time, so it is usually the first error teams hit after upgrading. A simple search across your repository for openvino.runtime finds nearly all affected call sites before you upgrade.

Does OpenVINO speculative decoding work on the Intel NPU?

Yes. OpenVINO GenAI added speculative decoding for NPUs in 2026.0, and EAGLE-3 with Top-K sampling runs on NPU from 2026.3. Intel’s guidance is that both the main and the draft model should run on the NPU. A community benchmark found that a GPU target with an NPU draft was slower than GPU-only decoding, because the NPU’s fixed per-inference cost dominates on a small draft model. Export NPU drafts with symmetric quantization and keep num_assistant_tokens at or below 7, as Intel advises.

Can OpenVINO run a 30B MoE model on a 16 GB laptop?

Intel says yes from 2026.3: MoE offload lets 30B MoE models such as Qwen3-30B-A3B run on devices with 16 GB of memory. It works because only a few experts are active per token, so cold experts can be streamed on demand through an LRU cache, controlled by the GPU plugin’s OFFLOAD_RATIO property. Intel describes the speed only as “acceptable TPS” and publishes no figure. Use INT4 weights, fast NVMe storage and a moderate ratio, and avoid INT8 with offload given the open hang report.

Does OpenVINO 2026 still support older CPUs without AVX2?

No. From 2026.0, the CPU plugin requires the AVX2 instruction set as a minimum, and SSE-only processors are no longer supported. This mostly affects older low-power industrial and embedded x86 boards. If part of your fleet lacks AVX2, you have two options: keep those nodes on a pinned 2025.x release with its own support timeline, or move their inference to a GPU or NPU on newer hardware. Check the whole fleet before scheduling the upgrade, not only your development machines.

Should I use MTP or EAGLE-3 in OpenVINO 2026.4?

Use MTP if your model is Gemma 4, Qwen3.5 or Qwen3.6 and you run on CPU or GPU, because it needs no separate draft artefact. Use EAGLE-3 for other dense models that have a published EAGLE-3 head, and prefer Tree Drafting for VLM pipelines, where Intel claims higher throughput than chain drafting. On NPU, EAGLE-3 or a FastDraft-style draft model is the option, with both models on the NPU. Avoid EAGLE-3 on MoE targets until the reported hidden-state issue is fixed.

Further Reading

• On-device LLM runtimes compared: llama.cpp vs MLC vs ONNX Runtime — where OpenVINO fits next to the portable runtimes, including its new llama.cpp backend.
• INT4 vs INT8 vs FP8 quantization on edge NPUs — the hardware side of the NNCF format choices.
• ONNX vs TFLite vs ExecuTorch vs Core ML — why NNCF’s ONNX FP8 support matters for multi-runtime fleets.
• Edge AI inference hardware: NVIDIA Jetson, Intel Movidius and Arm NPUs — the silicon context for Intel’s NPU roadmap.

References

1. OpenVINO 2026.4.0 release, GitHub: https://github.com/openvinotoolkit/openvino/releases/tag/2026.4.0
2. OpenVINO releases index (2025.4.0 through 2026.4.0), GitHub: https://github.com/openvinotoolkit/openvino/releases
3. OpenVINO 2026.0.0 release, GitHub: https://github.com/openvinotoolkit/openvino/releases/tag/2026.0.0
4. OpenVINO 2026.1.0 release, GitHub: https://github.com/openvinotoolkit/openvino/releases/tag/2026.1.0
5. OpenVINO release notes, official documentation: https://docs.openvino.ai/2026/about-openvino/release-notes-openvino.html
6. OpenVINO 2026.3 announcement, Intel Community: https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/OpenVINO-2026-3-Available-Now/m-p/1758325
7. OpenVINO GenAI 2026.4.0.0 release, GitHub: https://github.com/openvinotoolkit/openvino.genai/releases/tag/2026.4.0.0
8. OpenVINO Model Server 2026.4.0 release, GitHub: https://github.com/openvinotoolkit/model_server/releases/tag/v2026.4.0
9. Intel OpenVINO blog, “Speed Up Local Inference with EAGLE-3 in OpenVINO GenAI” (March 2026): https://medium.com/openvino-toolkit/still-watching-your-llm-generate-one-token-at-a-time-94b9b7e9fc46
10. Li et al., EAGLE-3, arXiv: https://arxiv.org/abs/2503.01840
11. openvino.genai issue #4390, EAGLE-3 on MoE targets: https://github.com/openvinotoolkit/openvino.genai/issues/4390
12. openvino.genai issue #4425, EAGLE-3 on CPU in 2026.3.1: https://github.com/openvinotoolkit/openvino.genai/issues/4425
13. openvino issue #37736, INT8 MoE offload second-generate hang: https://github.com/openvinotoolkit/openvino/issues/37736
14. openvino PR #36891, automatic MoE offload ratio (proposed): https://github.com/openvinotoolkit/openvino/pull/36891
15. openvino discussion #36484, GPU target with NPU draft characterisation: https://github.com/openvinotoolkit/openvino/discussions/36484

By Riju — about