Diffusion LLMs: How Text Diffusion Models Work (2026)

Diffusion LLMs are the first architecture in years to seriously challenge the assumption that a language model has to write text one token at a time, left to right. Instead of predicting the next token and then the next, a diffusion LLM starts from a sequence of masked placeholders and iteratively denoises the whole thing in parallel, committing high-confidence tokens over a handful of refinement steps. In 2026 this is no longer a research curiosity: commercial systems like Inception Labs’ Mercury 2 report throughput above 1,000 tokens per second, and open families like LLaDA 2.0 have scaled masked discrete diffusion to 100B parameters. This piece is written for engineers who already run autoregressive models in production and want to understand what actually changes under the hood — the mechanism, the serving implications, and where diffusion LLMs win and lose against the incumbents.

What this covers: the masked discrete diffusion mechanism, confidence-based parallel decoding, throughput and KV-cache serving trade-offs, an honest dLLM-vs-autoregressive comparison, the gotchas, and a practical checklist for when to reach for one.

Context and Background

For roughly eight years, one design has dominated text generation: the autoregressive (AR) transformer that factorizes the probability of a sequence into a product of left-to-right conditionals and samples one token at a time. It is a beautiful, simple recipe, and it powers essentially every model you have used. But it carries a structural cost that no amount of hardware fully erases: generation is inherently sequential. To produce the 500th token you must first produce the 499 before it, because each step conditions on the realized prefix. You can batch across requests, you can speculate ahead, you can cache the attention keys and values, but the critical path for a single response is still one forward pass per output token. Latency scales with length, and the GPU spends most of its time memory-bound rather than compute-bound.

That sequential bottleneck is exactly what makes parallel generation attractive. If you could produce many tokens in a single forward pass, you would convert a latency problem into a throughput problem — trading a few extra passes over the whole sequence for the ability to fill dozens or hundreds of positions at once. Diffusion models, which reshaped image and audio generation by learning to reverse a gradual corruption process, offer precisely that shape of computation. Adapting them to discrete text is non-trivial (you cannot add Gaussian noise to a word the way you can to a pixel), but the payoff is an order-agnostic generator with bidirectional context. If you want the broader inference-optimization landscape this sits in, see our guide to KV-cache optimization for LLM inference, which covers the memory economics that diffusion decoding both helps and complicates.

The intellectual lineage runs through discrete diffusion research: D3PM established discrete corruption processes, later refined by masked/absorbing-state formulations such as MDLM and score-entropy approaches like SEDD. Independent primers such as the Hugging Face documentation on diffusion models trace how the continuous-image machinery was reworked for categorical data. What changed in 2025–2026 is scale and productization: the ideas left the lab and started serving real traffic.

It also helps to name the economic pressure driving interest. Inference, not training, is where most of the lifetime cost of a deployed model accumulates, and inference cost for AR models is dominated by the memory bandwidth of streaming one token at a time. A generation paradigm that keeps accelerators compute-bound instead of memory-bound is therefore attractive on pure unit-economics grounds, independent of any quality argument. That is the lens through which most 2026 teams evaluate diffusion LLMs: not “is this a more elegant model of language” but “does this let me hit a latency SLA or a cost-per-token target that autoregressive serving cannot.” Keeping that framing prevents the common mistake of comparing the two paradigms only on leaderboard scores while ignoring the serving envelope where diffusion actually competes.

How Text Diffusion Works: Masked Discrete Diffusion

A text diffusion model learns to reverse a corruption process. The forward process takes clean text and progressively replaces tokens with a special [MASK] symbol until the sequence is fully masked; the reverse process is a transformer trained to predict the original tokens from the partially masked sequence. At inference you begin from an all-masked sequence and run the reverse denoiser for K steps, unmasking and committing the tokens it is most confident about at each step until nothing is masked. Fewer steps than the sequence length means many tokens resolve in parallel.

Figure 1: The forward masking and reverse denoising loop in a masked diffusion LLM. Left to right: clean tokens are progressively masked by the forward process into a fully masked sequence; the reverse denoiser (a transformer) predicts clean tokens; high-confidence predictions are committed while low-confidence positions are remasked and re-predicted on the next pass, until every position is unmasked and the final text is emitted. Long description: a horizontal flow beginning at a clean token sequence, passing through a corruption stage that yields an all-mask state, then entering a loop where a denoiser predicts tokens, a confidence gate commits the certain ones and sends the uncertain ones back through the denoiser, exiting only when the whole sequence is unmasked.

The forward process: absorbing-state corruption

The forward process is deliberately simple and requires no learning. Under the masked (also called absorbing-state) formulation that dominates 2026 diffusion LLMs, corruption means replacing tokens with [MASK] according to a schedule indexed by a continuous or discrete time variable t. At t = 0 the sequence is clean; at t = 1 (or the final step T) every position is the mask token, an absorbing state from which no further corruption is possible. The probability that any given token is masked increases monotonically with t, governed by a noise schedule — often linear or cosine in the masking probability. Because masking each position is independent given the schedule, you can sample a corrupted training example at any noise level in one shot, which makes training efficient.

This is the crucial difference from adding continuous noise. You are not perturbing an embedding; you are deleting information categorically and asking the model to reconstruct it. That framing turns diffusion training into something very close to a generalized masked-language-modeling objective — BERT-style cloze prediction, but across every masking ratio from near-zero to fully masked, weighted by the diffusion loss. Practitioners who remember masked language modeling will find the training loop familiar; the novelty is the inference procedure built on top of it.

The reverse process: the denoiser and its objective

The reverse process is where the learning lives. A bidirectional transformer — an encoder-style stack, not a causal decoder — takes the partially masked sequence and the noise level and outputs a distribution over the vocabulary for every masked position simultaneously. Because attention is bidirectional, each masked position is conditioned on all visible tokens to its left and its right, which is what gives diffusion LLMs their global-planning and infilling abilities. The training objective is a weighted cross-entropy on the masked positions: reconstruct the clean token, with the per-example weighting derived from the diffusion variational bound. Several 2026 systems, including the LLaDA 2.0 family from Ant Group’s InclusionAI team, convert a pretrained autoregressive checkpoint into a diffusion denoiser rather than training from scratch, using block-level schedules to stabilize the transition — a pragmatic move that reuses the enormous investment already sunk into AR pretraining.

One consequence worth internalizing: a single forward pass of the denoiser predicts every masked position at once. If you accepted all of those predictions blindly you would get one-shot, non-iterative generation — fast but low quality, because the predictions are made independently and can be mutually inconsistent (two positions might each be locally plausible yet jointly ungrammatical). Iteration is what fixes this. By committing only a subset and re-running with more context, the model resolves dependencies gradually.

The time conditioning also deserves a note, because it is where diffusion LLMs diverge most from a plain masked-language model. The denoiser is told, explicitly or implicitly, how corrupted the current sequence is — the noise level t — usually through a time embedding added to the token representations or, in the masked formulation, carried by the fraction of positions still masked. That signal lets a single network behave differently across the denoising trajectory: aggressive, low-commitment reconstruction when almost everything is masked, and careful, context-sensitive refinement near the end when most tokens are already fixed. Some implementations drop the explicit time embedding entirely and let the visible mask ratio serve as the signal, which simplifies the architecture and is one reason converting an AR checkpoint into a diffusion denoiser is even feasible — the backbone barely changes, only the attention mask and objective do.

Remasking and confidence-based decoding

The decoding policy — deciding which predicted tokens to keep at each step and which to send back as masked — is the single most important lever in a diffusion LLM’s quality-speed trade-off. The dominant strategy is confidence-based (or “low-confidence remasking”) decoding. At each denoising step the model produces logits for all masked positions; you take the softmax confidence (or a margin/entropy proxy) for the predicted token at each position, commit the top-scoring positions, and remask the rest for the next step. High-confidence tokens lock in early — often punctuation, function words, and content the context strongly determines — while ambiguous positions get more passes and more surrounding context before they settle.

The step schedule then controls how aggressively you commit. A conservative schedule might unmask a fixed small fraction per step over many steps, approaching high quality at the cost of throughput; an aggressive schedule commits large blocks per step, approaching one-shot speed at the cost of coherence. Recent research on adaptive and learnable parallel decoding tries to make this dynamic — committing more when the model is collectively certain and slowing down when it is not. LLaDA 2.0’s Confidence-Aware Parallel training, for example, adds an auxiliary loss so the model’s confidence estimates are better calibrated, which the authors report unlocks more aggressive parallel commits without the usual quality drop (treat the specific gains as reported, not independently verified). The mental model to keep: number of denoising steps is your quality dial, and confidence gating is how you spend that budget wisely. Block-wise decoding — running diffusion within a sliding window while streaming completed blocks — is a common hybrid that recovers some left-to-right streaming behavior for chat UIs.

Inference, Throughput and Serving

Diffusion LLMs invert the usual latency story. Where an AR model’s single-request latency scales with output length (one pass per token), a diffusion LLM’s cost scales with the number of denoising steps K, which can be far smaller than the sequence length N. Fill 64 positions in, say, 8–16 steps and you have done the work of 64 AR forward passes in a fraction of the passes — the origin of the headline throughput numbers.

Figure 2: A single confidence-based parallel decoding step. The denoiser runs one forward pass over all masked positions, producing per-token logits and confidence scores; positions scoring above a threshold are unmasked and committed, the rest stay masked; the loop advances until the step budget is exhausted or the sequence is fully resolved. Long description: a top-to-bottom flow from masked positions through a denoiser forward pass to per-token logits and confidence scores, into a threshold decision that either commits or keeps positions masked, then loops back for another step until no steps remain.

Parallel decoding and the steps-versus-quality curve

The central serving knob is K, the number of denoising steps. Every practical diffusion LLM exposes it (sometimes indirectly, as a “quality” or “effort” setting). Sweep it and you trace a curve: at very low K you get near-instant but sometimes incoherent output; as K rises, quality climbs and eventually plateaus while latency grows roughly linearly. The engineering goal is to find the knee of that curve for your task. Structured outputs (code, JSON, form-filling) often tolerate low K because context heavily constrains each position; open-ended creative or reasoning-heavy generation usually needs more steps. This is genuinely different from AR serving, where you cannot trade a quality dial for latency on a single request — you can only pick a smaller model. With a diffusion LLM you have a continuous throughput/quality control on a fixed model, which is powerful for latency-tiered products.

Two sampling controls sit alongside K and are worth exposing in any serious deployment. The first is the commit budget per step — how many positions you are willing to unmask each pass — which interacts with K to set the effective parallelism; committing 1/K of the sequence per step is the conservative baseline, while adaptive schemes commit variable amounts based on aggregate confidence. The second is temperature applied to the per-position distributions before you read off confidence; lower temperature sharpens the model and makes confidence gating more decisive, which usually improves structured output but can hurt diversity in open-ended text. A subtle failure appears when temperature and commit budget fight each other: high temperature inflates apparent confidence noise, so an aggressive commit budget locks in tokens the model was not really sure about. Treat these three controls — steps, commit budget, temperature — as one coupled system and tune them together on held-out examples.

A practical middle ground that many 2026 systems adopt is semi-autoregressive or block diffusion. Rather than denoising the entire response at once, the model processes the sequence in blocks: it runs full diffusion within a block, freezes and streams it, then moves to the next block conditioned on everything committed so far. This recovers three things AR users take for granted — bounded memory per step, token-by-token-ish streaming for chat UIs, and a stable, cacheable prefix — while retaining parallelism inside each block. The cost is giving up some of the global bidirectional planning that made pure diffusion attractive. For most production chat and coding workloads the block-wise compromise is the sweet spot, and it is worth checking whether your chosen diffusion LLM exposes a block-size parameter.

Figure 3: Autoregressive versus diffusion generation contrast. From the same prompt, the autoregressive path emits tokens strictly one after another (token 1, then 2, then N), so latency scales with length; the diffusion path masks a whole block and denoises it in parallel, refining over K steps before emitting the block. Long description: a branching diagram where a prompt splits into two paths, an autoregressive chain of sequential single-token steps, and a diffusion path that masks a block, denoises all positions in parallel, and refines over K steps, both converging on the output.

The KV-cache challenge

Here is the biggest serving catch, and the honest counterweight to the throughput hype. Autoregressive inference is fast in practice largely because of the KV cache: each new token attends to cached keys and values from all previous tokens, so a step is O(1) in new work rather than O(N). That cache works precisely because AR attention is causal and the prefix is immutable once generated. Diffusion LLMs break both assumptions. Attention is bidirectional, and tokens that were masked (or remasked) change between steps, so the keys and values for many positions are not stable across denoising steps — they have to be recomputed. Naively, each denoising step is a full-sequence forward pass. That is why a diffusion LLM doing K steps over N tokens is not automatically cheaper in FLOPs than AR doing N steps; its advantage is wall-clock parallelism and fewer sequential dependencies, not fewer total operations.

The field is actively closing this gap. Approaches include caching keys/values for the already-committed (frozen) prefix positions and only recomputing the still-masked region, block-wise schemes that freeze completed blocks so their cache is reusable, and approximate cache-refresh policies that recompute stale entries only periodically. These help, but as of 2026 KV reuse in diffusion serving is less mature and less universally effective than in AR stacks. If you are budgeting a deployment, do not assume AR-grade cache efficiency. Our deep dive on KV-cache optimization explains why the cache is such a load-bearing part of AR economics — and by extension why diffusion has to earn its keep on parallelism instead.

Batching, hardware, and the serving stack

Diffusion LLMs are compute-dense: a denoising step does dense matmuls over the full sequence, which maps well to modern accelerators that are starved for arithmetic intensity in memory-bound AR decoding. That is part of why Mercury 2 reports around 1,009 tokens per second on NVIDIA Blackwell hardware — the model keeps the tensor cores busy instead of waiting on memory. Batching still matters, but the throughput/latency profile differs: you can often serve a single request at high speed without needing a large batch to saturate the GPU, which is attractive for interactive, low-concurrency workloads at the edge or in agent loops.

Figure 4: A representative diffusion LLM serving stack. A client request passes through an API gateway to a scheduler/batcher, into the diffusion engine that runs the denoise step loop with a confidence commit gate; when the sequence converges or the step budget is hit, the output is detokenized and streamed back. Long description: a top-to-bottom pipeline from client request to API gateway to scheduler and batcher to the diffusion engine, which loops between a denoise step and a confidence commit gate, exiting on convergence or step-budget exhaustion to detokenize and stream the response.

The table below compares the two paradigms. Treat the numbers as directional and reported by vendors or papers, not as head-to-head benchmarks on identical hardware and tasks.

Dimension	Autoregressive (AR)	Diffusion LLM (dLLM)
Generation order	Strict left-to-right	Order-agnostic, parallel
Throughput (reported)	~70–90 tok/s for small reasoning models	1,000+ tok/s reported (Mercury 2, Blackwell)
Single-request latency	Scales with output length	Scales with denoising steps K (often << N)
Controllability / infilling	Weak native infilling; needs tricks	Native bidirectional infilling and editing
KV-cache reuse	Mature, highly effective	Harder; partial/approximate, less mature
Quality on hard benchmarks	Still leads in many suites	Approaching parity; strongest at parallel-friendly tasks
Exact likelihood	Tractable, exact	Bound-based, harder to compute exactly
Tooling / ecosystem	Vast, battle-tested	Growing but immature in 2026

The quality row deserves emphasis. LLaDA 2.0-flash (100B, MoE) reportedly reaches an average of about 73.18 across a benchmark suite, close to a strong AR peer’s ~73.60 — genuine parity in that comparison, but “close to” is doing real work, and on several reasoning and knowledge benchmarks the best AR models still hold a lead. The correct 2026 summary is not “diffusion won” but “diffusion became competitive, and dominant on latency-sensitive, infill-heavy, or throughput-bound workloads.” For a contrasting speed technique that keeps the AR backbone, compare with speculative decoding, which accelerates AR generation without changing its left-to-right nature.

Trade-offs, Gotchas, and What Goes Wrong

The failure modes of diffusion LLMs are different enough from AR that they surprise teams. First, the quality-versus-steps trap: it is tempting to crank K down to chase the throughput headline, but under-stepping produces subtle incoherence — locally fluent spans that contradict each other, dropped constraints, or repeated phrasing where two positions independently guessed the same token. Always tune K per task and validate on real outputs, not just latency dashboards.

Second, KV-cache assumptions leak into your cost model. If your capacity planning was built on AR cache reuse, a diffusion deployment can consume more compute than you expect because many positions recompute each step. Measure FLOPs and GPU utilization directly rather than extrapolating from token counts.

Third, tooling immaturity is real friction. Constrained decoding, grammar/JSON enforcement, logit biasing, structured-output libraries, and evaluation harnesses were all built for the AR sampling loop. Some transfer to the diffusion setting cleanly; many do not, and you may need to reimplement guardrails around the denoising loop. Streaming UX also needs thought: pure diffusion resolves the whole block at once, so token-by-token streaming requires block-wise decoding or a progressive-reveal UI.

Fourth, exact likelihoods and calibration are harder. If your application relies on per-token log-probabilities for scoring, reranking, or watermark detection, note that diffusion gives you a variational bound, not the exact sequence likelihood, and confidence semantics differ. Fifth, licensing and provenance vary widely — some 2026 diffusion LLMs are open-weight, others are API-only commercial systems, and the ecosystem’s contracts, safety tooling, and fine-tuning support are less standardized than the AR world you are used to. Budget engineering time for the rough edges.

A sixth, easily missed issue is length control. Autoregressive models decide when to stop by emitting an end-of-sequence token, which falls naturally out of left-to-right decoding. A diffusion LLM typically denoises into a fixed-size canvas of positions, so it must learn to place padding or end tokens and you must decide the canvas length up front or grow it in blocks. Get this wrong and you see either truncated answers when the canvas is too small or wasted compute and trailing padding artifacts when it is too large. Variable-length generation is an active area, and the maturity of a model’s length handling is a good proxy for how production-ready it is. Finally, watch for evaluation mismatch: a diffusion LLM tuned for low K to win a latency demo may quietly underperform on your quality suite, because the two are measured under different step budgets. Always report quality and latency at the same decoding configuration you intend to ship, or the comparison is meaningless.

Practical Recommendations

Reach for a diffusion LLM when your bottleneck is latency or throughput on structured, constrained, or infill-heavy generation — code completion and refactoring, form and JSON filling, data transformation, retrieval-augmented responses with tight context, and agentic subtasks where a fast, cheap pass unblocks a larger pipeline. The Augment Code report of swapping Mercury 2 into a context-compaction subagent for a large latency and cost reduction (reported figures) is exactly this shape: a bounded, well-constrained task where speed compounds across an agent loop. Stay with a strong autoregressive model when you need best-in-class reasoning on hard benchmarks, mature constrained-decoding tooling, exact likelihoods, or a battle-tested safety and evaluation ecosystem — and remember you can accelerate AR separately with techniques like speculative decoding or serve a mixture-of-experts model to cut per-token cost without changing paradigms.

Use this checklist to decide whether a diffusion LLM belongs in your stack:

• Latency-critical, interactive path? Sub-second responses at low concurrency favor diffusion.
• Structured or infill-heavy output? Code, JSON, edits, and templating exploit bidirectional context.
• Can you tolerate tuning K per task? You need a quality/latency knob and the discipline to calibrate it.
• Is AR-grade constrained decoding a hard requirement? If yes, verify tooling support before committing.
• Do you depend on exact log-probs? If yes, diffusion’s bound-based scoring may be a blocker.
• Open-weight vs API constraints? Confirm licensing, fine-tuning, and data-residency needs up front.
• Have you measured real GPU utilization? Validate the compute cost rather than trusting token-rate headlines.

Pilot on one bounded workload, measure end-to-end latency, cost, and quality against your current AR baseline, and expand only where the diffusion LLM clearly wins.

Frequently Asked Questions

Are diffusion LLMs better than autoregressive models?

Not universally, and 2026 evidence does not support a blanket claim. Diffusion LLMs win decisively on throughput and single-request latency, and they offer native infilling and a quality-versus-steps knob that AR lacks. But strong autoregressive models still lead on many hard reasoning and knowledge benchmarks, and their tooling, constrained decoding, and exact-likelihood support are far more mature. The honest framing is task-dependent: diffusion excels on latency-sensitive, structured, or infill-heavy work, while AR remains the safer default for peak reasoning quality and ecosystem depth.

How many denoising steps does a diffusion LLM need?

It depends on the task and your quality tolerance, and it is the main knob you tune. Highly constrained outputs like code or JSON can resolve well in a small number of steps because context pins down most positions; open-ended reasoning or creative text usually needs more. Fewer steps means higher throughput but risks incoherence from under-refinement, while more steps approach the model’s quality ceiling at higher latency. Sweep the step count on representative inputs and pick the knee of the quality-versus-latency curve rather than defaulting to a vendor preset.

Why is KV-caching harder for diffusion LLMs?

Autoregressive caching works because attention is causal and the generated prefix is immutable, so each token’s keys and values are computed once and reused. Diffusion LLMs use bidirectional attention, and masked positions change between denoising steps, so many keys and values become stale and must be recomputed. That makes naive diffusion decoding a full-sequence forward pass per step. Techniques like freezing and caching committed prefixes or completed blocks recover some reuse, but as of 2026 diffusion cache reuse is less mature and less universally effective than in AR serving stacks.

What are the main diffusion LLM models in 2026?

The notable systems include Inception Labs’ Mercury and Mercury 2, a commercial diffusion LLM reporting throughput above 1,000 tokens per second on Blackwell hardware; the open LLaDA family, with LLaDA 2.0 scaling masked discrete diffusion to roughly 100B parameters using a mixture-of-experts design; Dream (a 7B diffusion LLM); and Google DeepMind’s Gemini Diffusion, announced in 2025. Research systems also push decoding accelerators like adaptive parallel decoding and confidence-aware commit gating. The space is moving quickly, so treat specific numbers as reported at time of release.

Can diffusion LLMs run on edge devices?

They are a promising fit for the edge precisely because they convert sequential latency into parallel compute. A smaller diffusion LLM — say 1.7B parameters — filling 64 tokens in parallel over a handful of steps can feel effectively instant, potentially outpacing a larger 7B autoregressive model that must emit tokens one at a time. Diffusion decoding is compute-dense and can saturate modest accelerators without needing large batches, which suits low-concurrency on-device inference. The caveats are memory for the full-sequence passes and less mature edge tooling, so validate on your target hardware.

Is a diffusion LLM the same as an image diffusion model?

They share the core idea of reversing a corruption process through iterative denoising, but the mechanics differ because text is discrete. Image models add and remove continuous Gaussian noise in pixel or latent space; you cannot meaningfully add continuous noise to a word. Text diffusion LLMs instead use discrete corruption, most commonly masked or absorbing-state diffusion, where the forward process replaces tokens with a mask symbol and the reverse process predicts the original tokens. The denoiser is a bidirectional transformer over token sequences rather than a U-Net over pixels, so the training objective looks like generalized masked-language modeling.

Further Reading

• Speculative decoding for LLM inference — accelerating autoregressive generation without changing its order.
• KV-cache optimization for LLM inference — the memory economics diffusion has to work around.
• Mixture-of-experts (MoE) LLM architecture — how sparse experts cut per-token cost, including in LLaDA 2.0.
• Mercury: Ultra-Fast Language Models Based on Diffusion (Inception Labs) — external technical report on a commercial diffusion LLM.
• Dream 7B: Diffusion Large Language Models — external paper on an open diffusion LLM.

By Riju — about