Diffusion Language Models: LLaDA, Masked Diffusion and Parallel Decoding

Every mainstream chat model you use today writes its answer one token at a time, left to right, and each token needs a full pass through billions of weights. That loop is the single biggest reason inference is slow and expensive, and it is why a new family of diffusion language models deserves attention: instead of committing to tokens in order, they start from a fully masked response and refine many positions per forward pass.

This is no longer a curiosity. LLaDA showed an 8B-parameter masked diffusion model trained from scratch can sit near LLaMA3 8B on many benchmarks. Inception’s Mercury reported more than 700 tokens per second on H100 GPUs for coding models, and Google DeepMind demonstrated Gemini Diffusion at a reported 1,479 tokens per second. The catch is real too: no native KV cache, fixed-length canvases, and a quality-versus-steps trade that autoregressive models simply do not have.

You will leave with the math behind masked discrete diffusion, the sampling loop that makes it work, a runnable toy implementation, and a clear sense of where parallel decoding wins and where it quietly loses.

What this covers: why autoregressive decoding is memory-bound, the absorbing-state diffusion objective and its likelihood bound, LLaDA training and low-confidence remasking, block diffusion and KV-cache fixes, throughput arithmetic, the reversal-curse claims, failure modes, and deployment advice.

Context and Background

A decoder-only Transformer factorises the probability of a sequence as a product of next-token conditionals, p(x) = ∏ p(x_i | x_<i). Training is cheap to parallelise because a causal mask lets every position be trained at once (teacher forcing). Generation is the opposite: token i cannot be sampled until token i-1 exists, so the model runs once per output token. A 500-token answer means 500 sequential forward passes, no matter how many GPUs you own.

The industry has spent years attacking that bottleneck from the outside. Speculative decoding uses a small draft model to propose several tokens that the large model verifies in one pass; we compared the main variants in our speculative decoding guide covering EAGLE-3, Medusa and Lookahead. Mixture-of-experts reduces the active parameters per token, as explained in our MoE architecture reference. Both keep the autoregressive contract: one committed token at a time, in order, with the model’s own output fed back in.

Diffusion language models break that contract at the root. Image generators already did the same thing: rather than emitting pixels in raster order, they start from noise and denoise the whole canvas together. The continuous version of that idea powers modern image and video models, and the sampling-side relatives are covered in our post on flow matching and rectified flow versus diffusion. Text is discrete, though, so Gaussian noise does not apply. Early attempts added continuous noise to embeddings and struggled. The approach that worked is discrete diffusion, where “noise” means replacing tokens with something else.

Three papers set the foundation. Score Entropy Discrete Diffusion (SEDD, Lou et al., 2023) generalised score matching to discrete spaces. MDLM (Sahoo et al., 2024, arXiv 2406.07524) showed that simple masked diffusion, with a better recipe and a simplified Rao-Blackwellised objective, reaches perplexity close to autoregressive models and sets a new state of the art among diffusion models. Then LLaDA (arXiv 2502.09992) took the masked recipe to 8B parameters and 2.3 trillion training tokens, which is the point where the field started treating it as a serious alternative rather than a research toy.

Since then the ecosystem has moved quickly. Block Diffusion (Arriola et al., arXiv 2503.09573) interpolates between the two paradigms. Fast-dLLM (arXiv 2505.22618) made LLaDA and Dream practical to serve with caching. Dream 7B (arXiv 2508.15487) initialised a diffusion model from an autoregressive checkpoint. Commercial entrants include Inception’s Mercury family and Google’s experimental Gemini Diffusion. Throughout this article, numbers are quoted as the authors reported them, and I flag where something is undisclosed.

Why Autoregressive Decoding Is Sequential and Memory-Bound

To see what diffusion changes, you need the hardware arithmetic of normal decoding. During generation at small batch size, each step must stream essentially all model weights from high-bandwidth memory (HBM) to the compute units, perform a tiny amount of arithmetic on one token per sequence, and write one token back. The ratio of arithmetic to bytes moved (arithmetic intensity) is roughly 1 FLOP per byte of weight at batch 1 in 16-bit precision, far below the hundreds of FLOPs per byte a modern GPU needs to be compute-bound.

Here is a worked estimate, labelled as my own derivation rather than a measurement. An 8B-parameter model in bf16 occupies about 16 GB. An H100 SXM has roughly 3.35 TB/s of HBM bandwidth. Reading 16 GB at 3.35 TB/s takes about 4.8 ms, so the ceiling at batch 1 is around 209 tokens per second per GPU, before counting KV-cache reads, kernel launch overhead, sampling, or communication. Real systems land below that. Peak bf16 compute on the same chip is on the order of 1 PFLOP/s dense, and the decode step uses about 16 GFLOP (2 FLOPs per parameter), roughly 16 microseconds of math. The GPU is idle for more than 99 percent of the step, waiting for memory.

The standard fix is batching: serve many requests at once so one weight read amortises over many tokens. That raises throughput but not single-request latency, and it needs more KV-cache memory per extra request. Interactive agents, voice assistants and code completion all care about latency for one stream, which is exactly where autoregressive models are weakest.

There are two further costs. First, errors compound: a bad token at position 40 conditions everything after it, and the model cannot go back. Second, the left-to-right factorisation bakes in an order. Bidirectional tasks such as infilling a function body between a signature and a return statement need special training formats (fill-in-the-middle) to work at all.

The question diffusion asks is: can a model emit many tokens per forward pass without a separate draft model, by changing the generative process itself? The answer is yes, with caveats we will quantify below. The key shift is to turn a memory-bound loop into a more compute-bound one. If a single forward pass processes 256 positions and finalises 8 of them, the weight read is shared by 8 tokens, and the GPU does more useful arithmetic per byte. That is the entire throughput story, and every limitation of diffusion LLMs follows from how well it can be made to hold.

Masked Discrete Diffusion: The Reference Architecture

Direct answer: A masked diffusion language model is a bidirectional Transformer trained to recover randomly masked tokens at every masking ratio from 0 to 1. At inference it starts from an all-mask response and, over several steps, predicts all masked positions at once, keeps the most confident predictions, and re-masks the rest. The number of steps, not the number of tokens, sets the cost.

Figure 1: Training a masked diffusion language model. Text is corrupted by independent masking at a random ratio, a Transformer with no causal mask predicts the hidden tokens, and cross entropy on masked positions is weighted by 1/t.

Figure 1 shows the training path. There is no noise network, no score function over continuous space, and no special architecture. The backbone is an ordinary Transformer with the causal mask removed, so each position attends to every other position, including masked ones. What changes is the data corruption and the loss.

The forward process: absorbing-state corruption

Discrete diffusion needs a corruption process on a finite vocabulary. The masked (or absorbing-state) variant is the simplest: each token independently either survives or jumps to a special [MASK] symbol and stays there. With a time variable t in [0, 1], the marginal for a single token is

q(x_t | x_0) = (1 – t) · δ(x_t = x_0) + t · δ(x_t = [MASK]).

At t = 0 the sequence is intact; at t = 1 it is entirely masked. Because masking is independent per token, the number of masked tokens in a length-L sequence is Binomial(L, t). Nothing else about the process depends on the data, which is why the training loop is nearly as simple as BERT’s.

It is worth being precise about the relation to BERT. BERT masks a fixed 15 percent and is not a generative model. A masked diffusion model samples the ratio uniformly, so it trains on everything from nearly clean text to nearly pure mask, and that range is exactly what makes iterative generation from a blank canvas possible.

The reverse process and the training objective

Going backwards from time t to an earlier time s < t, the only thing that can happen is that a masked token becomes a real token. Given a token that is masked at time t, the true posterior probability it is unmasked by time s is (t – s) / t, and if it is, its value follows the clean-data distribution conditioned on what is visible. The network’s job is to estimate that conditional, p_θ(x_0^i | x_t), for every masked position i.

LLaDA’s pre-training loss, as reported in the paper (its Eq. 3), is

L(θ) = -E_{t, x_0, x_t} [ (1/t) · Σ_i 1[x_t^i = M] · log p_θ(x_0^i | x_t) ],

with t drawn uniformly from [0, 1]. Only masked positions contribute, and each is scaled by 1/t. The authors note this factor is the difference from MaskGIT-style objectives, which lack it and therefore have no link to maximum likelihood.

Why 1/t? When t is small, only a few tokens are masked, so the per-sample loss is a sum over very few terms; dividing by t rescales it so every masking level contributes with the right weight in expectation. Formally, the paper states this loss upper-bounds the negative log-likelihood of the model, so it is a variational bound of the same kind as the ELBO in VAEs. MDLM derives the same bound in continuous time and shows how a simplified Rao-Blackwellised form yields lower variance.

Two consequences matter for practitioners. First, the bound is not the exact likelihood, so reported perplexity for diffusion models is an upper bound and is not strictly comparable to autoregressive perplexity; MDLM and Block Diffusion are explicit about this. Second, the training signal per sequence is noisier. An autoregressive model gets a loss on all L tokens every step; a masked diffusion model gets a loss only on the masked fraction, on average half. That is a sample-efficiency penalty paid in compute, and it is part of why diffusion models need long training runs to catch up.

Inference: iterative unmasking with remasking

Figure 2: The reverse sampling loop. Each iteration is one forward pass; the step budget, not the sequence length, determines cost.

Sampling follows Figure 2. Choose a response length L and a number of steps N. Start with the prompt followed by L [MASK] tokens. At each step, run the network on the whole sequence, obtain a predicted token and a probability for each masked slot, and decide which predictions to commit.

The theoretically faithful rule is stochastic: at step from t to s, each masked token independently unmasks with probability (t – s)/t, with its value sampled from the model. The practical rule used by LLaDA is deterministic and confidence-based. It predicts every masked token, then re-masks the lowest-confidence ones so that the right number remains masked for the next time level. The paper describes this low-confidence remasking, which follows the MaskGIT idea, and reports that uniform timesteps are the default.

This is the central design choice, so it is worth seeing why it works. A model is more likely to be right about the easy tokens (punctuation, the closing bracket, a word completed by context) than the hard ones (the choice of an entity, the first word of a plan). Committing easy tokens first gives the model more context for the hard tokens on the next pass. Left-to-right decoding is just one fixed, content-blind schedule; confidence-ordered unmasking is an adaptive schedule that lets the answer’s structure emerge in whatever order the model finds most determined.

Fixed canvas and the end-of-sequence problem

The model must be told how long the response is. You place L mask tokens and decode into that canvas. LLaDA’s SFT recipe treats |EOS| as an ordinary token, pads shorter responses with it, and strips the padding at sampling time, so the model learns to emit EOS tokens where the answer ends. This works but creates a known bias: under low-confidence remasking, EOS positions near the end of the canvas are often very confident early, which can bias the model toward finishing too soon. The LLaDA authors attribute weaker GSM8K and Math numbers under pure diffusion sampling for the Instruct model (69.4 and 31.9) to this effect, and report 78.6 and 42.2 when a block-style sampling schedule is used instead.

Training LLaDA: What Was Actually Disclosed

LLaDA is the cleanest data point because the paper reports its recipe. The reported facts are these: two sizes, 1B and 8B parameters; 8B pre-trained from scratch on 2.3 trillion tokens with a fixed 4,096-token context; roughly 1 percent of pre-training sequences with random lengths between 1 and 4,096; and a pre-training cost of 0.13 million H800 GPU hours. The 8B run was executed once, without hyperparameter tuning. Supervised fine-tuning used 4.5 million prompt-response pairs across code, math and instruction following, for 3 epochs, masking only the response tokens during SFT.

Compare that with LLaMA3 8B, which Meta reports was trained on far more tokens (the LLaDA paper uses LLaMA3 and LLaMA2 8B as autoregressive baselines). The relevant fairness point, which the authors raise themselves, is that training data for the strongest autoregressive baselines is not fully transparent, so head-to-head comparisons carry a caveat.

One architectural detail has deployment consequences. LLaDA uses standard multi-head attention rather than grouped-query attention, because the model cannot use a KV cache in its basic form. To keep total parameters at 8B, the authors enlarged attention parameters and shrank the feed-forward dimension. In plain terms, you cannot take a modern autoregressive architecture, remove the mask, and expect the same efficiency profile.

What the benchmark numbers say

The following are the scores the LLaDA paper reports for the base models (Table 1) on selected tasks. They are the authors’ own evaluations, and I have not re-run them.

Benchmark	LLaDA 8B Base	LLaMA3 8B Base
MMLU (5-shot)	65.9	65.4
BBH (3-shot)	49.7	62.1
Hellaswag	70.5	79.1
GSM8K (4-shot)	70.3	48.7
Math (4-shot)	31.4	16.0
HumanEval	35.4	34.8
MBPP	40.0	48.8

The picture is mixed, which is the honest reading. Diffusion is ahead on math-style tasks and Chinese benchmarks (CMMLU 69.9 versus 50.7), roughly level on MMLU and HumanEval, and behind on BBH, Hellaswag and MBPP. For the instruction-tuned versions (Table 2), LLaDA 8B Instruct trails LLaMA3 8B Instruct on MMLU (65.5 versus 68.4), GSM8K (69.4 versus 78.3), HumanEval (49.4 versus 59.8) and MBPP (41.0 versus 57.6), while leading on ARC-C (88.5 versus 82.4) and slightly on Math and GPQA. The authors attribute part of that gap to the absence of reinforcement-learning alignment and to SFT data quality.

A fair summary: masked diffusion at 8B reaches the same neighbourhood as a comparable autoregressive model, not a decisive win. That is a remarkable result for a first from-scratch run of a new paradigm, and it is also why the efficiency story, not the accuracy story, carries the commercial case.

Deeper Analysis: Block Diffusion, KV Caches, and the Throughput Arithmetic

Why the KV cache breaks, and what block diffusion repairs

In an autoregressive model, the key and value vectors of past tokens never change, because causal attention means earlier tokens cannot see later ones. That immutability is what lets you cache them and only compute the new token’s attention. A bidirectional diffusion model has no such property. When a mask at position 90 turns into a real token, every other position’s hidden state can change on the next pass, since they all attend to it. Naively, every step recomputes keys and values for the entire sequence, prompt included.

That makes the cost of a diffusion step roughly that of a full prefill over L_prompt + L_response tokens. For a 1,000-token prompt and a 256-token response, each of N steps costs about 1,256 tokens of compute. Compare that with autoregressive decoding, where each step costs one token of compute plus attention reads over the cache. The diffusion step is more compute-bound, which is the intended effect, but unless the step count is far smaller than the response length the total work can exceed autoregressive decoding.

Block Diffusion (BD3-LM, arXiv 2503.09573, by Arriola and colleagues) addresses this by splitting the sequence into blocks. Blocks are generated left to right, autoregressively across blocks, while tokens inside a block are denoised in parallel by diffusion. Attention across blocks is causal, so finished blocks are immutable and their keys and values can be cached; inside the current block attention is bidirectional. The authors present it as an interpolation: block size 1 recovers an autoregressive model, and block size equal to the full sequence recovers plain diffusion. They claim it supports flexible-length generation, KV caching and parallel token sampling, and state-of-the-art likelihood among diffusion models. The abstract does not give a single headline perplexity, so I will not quote one.

Figure 3: Block diffusion. Blocks are produced in order and cached once complete; within a block, tokens are denoised in parallel for several steps.

The block size is a dial with a clear trade-off. Small blocks (say 4 to 8 tokens) behave like multi-token prediction: great cache behaviour, long-range planning left to the autoregressive structure, but limited parallelism. Large blocks (say 64 to 256) maximise tokens per pass but regress toward the stale-cache problem and the fixed-length difficulties. The right setting is workload dependent and should be measured, not assumed.

LLaDA itself can be sampled in a block-wise, semi-autoregressive way without retraining. The paper reports that block sampling helped the Instruct model on GSM8K and Math, mainly because it limits the early-EOS effect. Treat that as evidence that sampling schedule is part of the model, not an afterthought.

Approximating the cache for pure diffusion models

Fast-dLLM (arXiv 2505.22618) shows a training-free path that works on existing LLaDA and Dream checkpoints. It makes two moves. The first is a block-wise approximate KV cache: reuse the keys and values of prompt and previously decoded blocks across the steps of the current block, accepting that they are slightly stale because activations drift only a little between adjacent steps. The second is confidence-aware parallel decoding: instead of unmasking a fixed number of tokens per step, unmask every token whose confidence exceeds a threshold, which adapts the tokens-per-step to difficulty.

The reported results are substantial. The paper reports up to 27.6 times higher throughput than the vanilla LLaDA baseline in its best configuration (8-shot prompting, a variant called DualCache, 1,024-token generation), parallel decoding alone reaching 13.3 times in one setting, and combined speedups of 11.0 times for LLaDA on GSM8K at length 512 and 7.8 times for Dream on MBPP at length 512. Accuracy mostly stayed within one to two points of the baselines; for example DualCache on LLaDA at 8-shot scored 76.0 against a 77.3 baseline. Note what the baseline is: vanilla LLaDA with no cache, which is a slow starting point. The speedups are real but are measured against a naive diffusion implementation, not against a well-tuned autoregressive server with continuous batching.

Why does confidence thresholding help? Consider the independence problem. When the model unmasks two tokens in the same step, it samples each from its own marginal distribution, ignoring their joint dependence. If the model is 51 percent sure the answer is “New York” and 49 percent sure it is “Los Angeles”, unmasking “New” and “York” independently in the same pass can produce “New Angeles”. High-confidence tokens are nearly deterministic, so the independent-marginals error is small; low-confidence tokens are where parallel sampling hurts. A threshold makes that logic explicit.

Tokens per forward pass, NFE, and a worked comparison

Diffusion people count cost in network function evaluations (NFE): the number of forward passes. For a response of L tokens decoded in N steps, tokens per pass is L / N. Autoregressive decoding has NFE = L. Setting L = 256:

Setup	NFE	Tokens per pass	Notes
Autoregressive, KV cached	256	1	Each pass touches 1 new token
Diffusion, 256 steps	256	1	Matches AR NFE, but each pass is far heavier
Diffusion, 64 steps	64	4	Moderate parallelism
Diffusion, 32 steps	32	8	Quality risk grows
Block diffusion, 8 blocks of 32, 4 steps each	32	8	Cache-friendly

The first two rows are the trap. A diffusion model run with as many steps as tokens does no less sequential work than an autoregressive one and each step costs much more compute. The speed advantage appears only when N is substantially smaller than L, and this is exactly where quality starts to degrade.

Figure 4: Forward passes needed for a 256-token answer. Autoregressive decoding takes 256 sequential passes; a diffusion schedule with 8 tokens per pass takes 32. Per-pass cost is higher for diffusion.

Use the earlier hardware arithmetic for a rough latency model. These are illustrative estimates for an 8B bf16 model on one H100 SXM, not measurements. Autoregressive: 256 passes at a weight-read floor of about 4.8 ms gives at best about 1.2 seconds, ignoring KV reads and overhead. Diffusion at 32 steps: each pass reads the same 16 GB (4.8 ms) but also performs about 2 × 8×10⁹ × 256 ≈ 4.1 TFLOP of arithmetic, which at roughly 1 PFLOP/s peak is about 4 ms, so the pass remains bandwidth-limited at roughly 5 to 10 ms in practice. Thirty-two such passes cost 0.15 to 0.3 seconds. The ratio, around 4 to 8 times, tracks tokens per pass, which is why speedup is bounded by how aggressively you cut steps. With a 1,000-token prompt and no cache, add the prefill-sized attention over the whole context at every step and the advantage shrinks; that is the cost the KV approximations above recover.

Vendor numbers: Mercury and Gemini Diffusion

Inception’s Mercury Coder paper (arXiv 2506.17298) reports two sizes, Mini and Small, built on Transformers with the autoregressive loss replaced by a denoising diffusion loss. Parameter counts are not disclosed. On H100 GPUs, measured by Artificial Analysis, the paper reports 1,109 tokens per second for Mini and 737 for Small, and claims up to 10 times faster than speed-optimised frontier models on average. Reported pass@1 scores for Mini and Small respectively include HumanEval 88.0 and 90.0, MBPP 77.1 and 76.6, and fill-in-the-middle averages of 82.2 and 84.8. On Copilot Arena the paper has Mini tied for second on Elo (993) with the lowest latency. The training data is described only as “the order of trillions of tokens”, and the inference engine is proprietary. Inception has since announced a Mercury 2 positioned for reasoning, which I have not verified in detail.

Google DeepMind’s Gemini Diffusion, shown at I/O 2025, was reported as generating 1,479 tokens per second excluding overhead, with latency to first output as low as 0.84 seconds, and roughly matching Gemini 2.0 Flash-Lite on coding benchmarks (HumanEval 89.6 versus 90.2 percent; MBPP 76.0 versus 75.8) while trailing on GPQA Diamond (40.4 versus 56.5) and Global MMLU Lite (69.1 versus 79.0). Those figures are from Google’s announcement as relayed by press coverage, and the model was an experimental demo behind a waitlist when reported. Note the pattern: the diffusion models look strongest on code and weaker on knowledge-heavy reasoning, mirroring LLaDA’s results.

Dream 7B and initialising from autoregressive weights

Dream 7B (arXiv 2508.15487) is notable for process rather than scale. The authors describe initialising the diffusion model from an autoregressive LLM and applying context-adaptive, token-level noise rescheduling during training. They report that it outperforms existing diffusion language models on general, math and coding tasks and release both Base and Instruct models for research. The reuse of autoregressive weights is the most practical idea in the paper: pre-training a diffusion model from scratch costs trillions of tokens, while adaptation reuses that investment. The excerpt I could verify does not state the training token count or the source model, so I will not cite them.

A Runnable Toy: Masked Diffusion in 60 Lines of PyTorch

The code below implements the whole recipe on a toy language: sequences are a counting pattern modulo the vocabulary size, so the task is learnable in seconds on a CPU. It implements random-ratio masking, the 1/t-weighted loss on masked positions only, a bidirectional encoder, and a confidence-ordered unmasking loop with k tokens committed per step. I ran it to confirm it executes.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, MASK, L, D = 12, 12, 16, 64          # vocab 0..11, id 12 = [MASK]

class Denoiser(nn.Module):
    def __init__(s):
        super().__init__()
        s.emb = nn.Embedding(V + 1, D); s.pos = nn.Embedding(L, D)
        layer = nn.TransformerEncoderLayer(D, 4, 128, batch_first=True)
        s.enc = nn.TransformerEncoder(layer, 2)   # no causal mask: bidirectional
        s.out = nn.Linear(D, V)
    def forward(s, x):
        h = s.emb(x) + s.pos(torch.arange(x.shape[1]))
        return s.out(s.enc(h))

def make_batch(n):                      # toy language: counting sequence mod V
    start = torch.randint(0, V, (n, 1))
    return (start + torch.arange(L)) % V

def loss_fn(model, x0):
    n = x0.shape[0]
    t = torch.rand(n, 1).clamp(min=1e-3)            # t ~ U(0,1]
    masked = torch.rand(n, L) < t                   # mask each token w.p. t
    xt = torch.where(masked, torch.full_like(x0, MASK), x0)
    ce = F.cross_entropy(model(xt).transpose(1, 2), x0, reduction="none")
    return ((ce * masked) / t).sum() / (n * L)      # 1/t weighted, masked only

@torch.no_grad()
def sample(model, prompt_len=2, steps=4):
    x = torch.full((1, L), MASK)
    x[0, :prompt_len] = make_batch(1)[0, :prompt_len]
    for s in range(steps):
        free = x[0] == MASK
        if not free.any(): break
        probs = F.softmax(model(x)[0], -1)
        conf, tok = probs.max(-1)
        conf = torch.where(free, conf, torch.full_like(conf, -1.0))
        k = int(torch.ceil(free.sum() / (steps - s)))   # unmask k per step
        idx = conf.topk(k).indices                      # most confident only
        x[0, idx] = tok[idx]                            # the rest stay masked
    return x[0]

model = Denoiser(); opt = torch.optim.AdamW(model.parameters(), 3e-4)
for it in range(1500):
    l = loss_fn(model, make_batch(128)); opt.zero_grad(); l.backward(); opt.step()
print("final loss", round(l.item(), 3))
for steps in (1, 4, 16):
    out = sample(model, steps=steps)
    ref = (out[0] + torch.arange(L)) % V
    print(steps, "steps ->", out.tolist(), "correct positions", int((out == ref).sum()), "/", L)

On my run the loss fell to about 0.05 and all three step budgets produced a perfectly consistent counting sequence. That is expected and also instructive: this toy has almost no ambiguity, because two prompt tokens determine everything, so even one step with 14 tokens in parallel is correct. Real text has the joint-dependence problem described earlier. To see degradation, change make_batch so each position is sampled with two equally likely continuations and watch the one-step sample incoherently mix them while the 16-step sample commits to one branch. The code also shows the three ingredients you would scale up: the loss, the bidirectional backbone, and the schedule that converts a step budget into a tokens-per-pass number.

The Reversal Curse and Bidirectional Claims

Autoregressive models trained on “A is B” often fail to answer “B is A”, a failure called the reversal curse (Berglund et al., 2023). The reason is structural: next-token training only ever teaches the conditional in the direction the text was written. A masked diffusion model is trained to predict any subset of tokens given any other subset, so there is no preferred direction.

The LLaDA paper tests this with 496 famous Chinese poem sentence pairs, asking the model to produce either the next line or the previous line without special fine-tuning. The authors claim LLaDA maintains consistent performance in the forward and reversal directions and outperforms GPT-4o on the reversal task. I could not retrieve the specific Table 4 numbers in this run, so I will not quote them. The caveats matter: the evaluation is a single narrow test on Chinese poetry, the reversal curse also depends on how much of the knowledge was seen in both orders, and “outperforms GPT-4o on reversal” does not mean “solves reversal in general”. The defensible conclusion is that the training objective removes a structural reason for directional asymmetry, which is a modelling advantage, not a guarantee of factual reliability.

Trade-offs, Gotchas, and What Goes Wrong

Quality falls as you cut steps. The independent-marginals error grows with tokens per pass. Tasks with tight joint constraints, such as code that must balance brackets and keep variable names consistent, or arithmetic where each digit depends on the others, suffer first. The Fast-dLLM results show that confidence thresholds recover most of the loss at moderate speedups, but there is no free lunch; at the extreme end you are back to a single-step sample that mixes modes.

Fixed length is awkward. You must pick the canvas size before generating. Too short truncates the answer; too long wastes compute on EOS padding and invites the early-EOS bias described above. Block diffusion fixes this by allowing generation to continue block by block until EOS appears, at the price of re-introducing some sequential dependence.

Serving infrastructure assumes autoregression. vLLM-style paged KV caches, continuous batching, speculative decoding and prefix caching were all designed around token-by-token generation. Diffusion workloads have different batching behaviour (large, compute-heavy steps), and mixing them with autoregressive traffic on the same GPUs is non-trivial. Expect immature tooling and engine-level optimisation to be a differentiator, which is consistent with Inception describing its inference engine as proprietary.

Evaluation is harder. Likelihood is a bound, so perplexity is not comparable across paradigms. Benchmark results depend on the sampling schedule (pure diffusion versus block, step count, remasking), so a single number hides a configuration space. LLaDA’s own GSM8K figures change by nearly nine points between schedules.

Alignment and reasoning are less mature. The LLaDA authors state that their instruct model did not undergo reinforcement learning alignment. Chain-of-thought reasoning with long outputs is a harder fit for a fixed canvas, and the training signal for RL on diffusion policies is awkward because the sequence log-likelihood is only a bound. Treat claims of parity with frontier reasoning models with caution unless backed by a published, reproducible evaluation.

Training efficiency. Only masked tokens produce loss, and the 1/t weighting increases gradient variance at small t. Variance-reduction tricks from MDLM and others help, but diffusion training remains less sample-efficient than next-token prediction, which is the reason initialisation from autoregressive checkpoints, as in Dream 7B, is attractive.

Where Diffusion Language Models Fit

Pick diffusion where its structural properties are an asset. Code infilling and editing is the clearest case: the model sees code before and after the gap and can fill it with bidirectional context, and Mercury’s fill-in-the-middle averages (82.2 for Mini, 84.8 for Small) were a headline result. Latency-critical short outputs such as autocomplete, classification-with-explanation, structured extraction and voice-agent replies benefit most because the response fits in a few blocks and the step count can be kept low. Constrained or templated generation, where some tokens are fixed and the model fills the blanks, maps naturally onto a masked canvas, since you simply pre-fill known tokens and mask the rest. Self-correction is also natural, because remasking low-confidence tokens lets the model revise earlier choices, which autoregressive decoding cannot do.

Avoid or defer diffusion for very long chain-of-thought generation, workloads dominated by shared long prefixes (where mature prefix caching in autoregressive servers is hard to beat), and applications that need the best available knowledge and reasoning scores today; both LLaDA and Gemini Diffusion trail on knowledge-heavy benchmarks such as GPQA and MMLU variants. Also consider the hybrid path: block diffusion keeps most of the cache story, and speculative decoding on autoregressive models remains a strong, lower-risk way to buy 2 to 3 times speedup without changing the model family.

Practical Recommendations

Start by measuring, not believing. Decide your latency target for a single stream, your typical output length, and whether the output is a fillable template. If your outputs are under roughly 300 tokens and latency matters more than frontier reasoning quality, a diffusion model is worth a pilot. Otherwise a well-tuned autoregressive server with speculative decoding and prefix caching is the safer bet.

When you pilot, evaluate quality as a curve against NFE, not as a point. Run the same prompts at 1, 2, 4 and 8 tokens per pass and plot task accuracy; the knee of that curve is your operating point. Test both pure diffusion and block sampling, because the early-EOS effect can swing results by several points. Use confidence thresholding rather than a fixed tokens-per-step budget, and set the canvas length from your observed output distribution with a margin.

For deployment, budget engineering time for serving. Open checkpoints such as LLaDA and Dream, plus the Fast-dLLM code, give you a research-grade stack; they are not a drop-in replacement for an autoregressive production server. Treat commercial APIs as an easier route to the speed benefits while you wait for open serving engines to mature.

Compute NFE and tokens per pass for every experiment, and report quality versus NFE.
Compare against autoregressive plus speculative decoding, not against a naive baseline.
Test pure diffusion and block sampling schedules; log the EOS behaviour.
Use confidence-threshold parallel decoding and an approximate block cache.
Pilot on code infilling, extraction or templated output before open-ended chat.
Keep a fallback autoregressive path for long reasoning requests.
Re-verify vendor throughput claims on your own hardware and prompts.

Frequently Asked Questions

What are diffusion language models?

Diffusion language models generate text by iterative refinement instead of left-to-right prediction. In the masked variant used by LLaDA, training randomly hides tokens at a ratio between 0 and 1 and teaches a bidirectional Transformer to recover them. At inference, the model begins with a fully masked response, predicts every position in parallel, keeps the most confident tokens, re-masks the rest, and repeats. The number of forward passes can be far smaller than the number of output tokens.

Are diffusion language models faster than autoregressive LLMs?

They can be, but not automatically. Speed comes from emitting several tokens per forward pass, so an answer of 256 tokens in 32 steps needs 8 times fewer passes. Each pass is heavier, and without a KV cache the cost of attending over the whole sequence erodes the gain. Vendor reports such as Mercury’s 737 to 1,109 tokens per second on H100s and Fast-dLLM’s up to 27.6 times over vanilla LLaDA show real gains, but baselines and quality settings matter.

What is LLaDA?

LLaDA, short for Large Language Diffusion with mAsking, is an 8B-parameter masked diffusion language model trained from scratch on 2.3 trillion tokens, with a 1B variant also reported. It uses a Transformer without a causal mask, a 1/t-weighted masked-token loss, and low-confidence remasking at inference. The paper reports performance competitive with LLaMA3 8B on in-context learning benchmarks and instruction following after supervised fine-tuning, though behind LLaMA3 8B Instruct on several benchmarks.

Why can’t diffusion language models use a normal KV cache?

Autoregressive models can cache keys and values because causal attention means past tokens never change. In a bidirectional diffusion model, unmasking a token changes the context that every other position attends to, so hidden states drift between steps. Block diffusion restores caching by generating blocks in order and freezing completed ones; Fast-dLLM approximates caching for pure diffusion models by reusing slightly stale keys and values within a block.

Do diffusion language models solve the reversal curse?

They remove one structural cause of it. Because masked diffusion is trained to predict any subset of tokens from any other, there is no built-in forward direction. The LLaDA paper reports consistent forward and reversal performance on a 496-pair Chinese poem completion test and outperforming GPT-4o on the reversal task. That is one narrow test, so treat it as encouraging evidence rather than a general guarantee against directional knowledge gaps.

Are diffusion language models better than GPT-style models?

Not overall as of the evidence reviewed here. At 8B scale, LLaDA lands near LLaMA3 8B on many benchmarks, ahead on some math tasks and behind on others, and Gemini Diffusion was reported near Gemini 2.0 Flash-Lite on code but weaker on GPQA and Global MMLU Lite. Their advantage is speed on short, structured or infilling tasks and bidirectional flexibility. For frontier reasoning, autoregressive models still lead.

Diffusion Language Models: LLaDA, Masked Discrete Diffusion and Parallel Decoding vs Autoregressive LLMs (2026)