Speculative Decoding for LLM Inference: Architecture (2026)
Last Updated: November 17, 2026
A 70-billion-parameter model generating one token for one user reads every weight from GPU memory and then does almost no arithmetic with it. That imbalance is the whole opportunity. Speculative decoding turns the idle arithmetic into extra tokens. A cheap proposer guesses several tokens ahead, the large model checks all of them in one forward pass, and a carefully designed acceptance rule guarantees the output distribution is exactly the one the large model would have produced on its own. In 2026 it is no longer a research trick. DeepSeek-V3 ships multi-token prediction modules built for it, EAGLE-3 heads exist for most popular open-weight models, and vLLM, SGLang and TensorRT-LLM all expose it as configuration. Yet many teams enable it and see little gain, or a loss, because the speedup depends on batch size far more than vendor charts suggest. This guide gives you the mechanism, the math with worked numbers, a tested NumPy implementation, and an honest operating envelope.
What this covers: the Leviathan and Chen algorithms and their lossless guarantee, the expected-tokens formula, Medusa, EAGLE through EAGLE-3, lookahead and n-gram drafting, MTP heads, engine support as of November 2026, batch-size collapse, and cost trade-offs.
Context and Background
Autoregressive decoding is sequential by construction. Token t+1 depends on token t, so a 1,000-token answer needs 1,000 forward passes through the full network. At small batch sizes each pass is memory-bandwidth-bound. An H100 SXM moves roughly 3.35 TB/s from HBM, so a 70B model stored in FP8 (about 70 GB of weights) needs around 21 ms just to stream its weights once. The matrix multiplies for a single token take a small fraction of that time. The tensor cores wait on memory.
Two papers attacked this independently and almost simultaneously. Yaniv Leviathan, Matan Kalman and Yossi Matias at Google posted “Fast Inference from Transformers via Speculative Decoding” in November 2022 (published at ICML 2023). They reported a 2X–3X acceleration on T5-XXL compared with the standard T5X implementation, with identical outputs. Charlie Chen, Sebastian Borgeaud and colleagues at DeepMind posted “Accelerating Large Language Model Decoding with Speculative Sampling” in February 2023. They reported a 2–2.5x decoding speedup on the 70B Chinchilla model in a distributed setup, “without compromising the sample quality.”
Both papers rest on the same observation. Scoring a short continuation of k tokens in parallel costs about the same wall-clock time as generating one token, because the weights are loaded once either way. If a cheap proposer can guess what the big model would say, the big model can confirm several tokens per weight load.
The field then split into three lines of work. The first kept a separate draft model. The second, starting with Medusa and EAGLE, attached small prediction heads to the target model itself. The third removed the learned drafter entirely: prompt lookup, suffix decoding and lookahead decoding. Model builders also began training models with multi-token prediction (MTP) objectives, which produce a drafter as a by-product of pre-training.
Speculative decoding sits next to other serving optimisations rather than replacing them. It interacts heavily with continuous batching, which also exploits idle compute, and with the KV cache, which must now hold speculative entries that may be thrown away. The rest of this post is about those interactions as much as about the algorithm itself.
Speculative Decoding Architecture: Draft, Verify, Accept
Speculative decoding is a lossless inference technique in which a cheap proposer drafts γ tokens and the target model scores all of them in one forward pass. A rejection-sampling rule accepts a prefix of the draft and resamples the first rejected token. The output distribution matches the target model’s exactly, while the target is called fewer times per generated token.

Figure 1: The speculative decoding loop. A draft proposer emits γ candidates with their probabilities, the target scores them in one pass, and the sampler commits between 1 and γ+1 tokens.
The figure shows one iteration as a cycle. The draft proposer reads the current prefix and emits γ candidate tokens together with the probability distributions q it sampled them from. The target model receives the prefix plus all γ candidates and, thanks to causal masking, returns a distribution p at every position in one pass. The sampler walks left to right. Each candidate is accepted with probability min(1, p/q). At the first rejection the sampler draws a replacement from a residual distribution and stops. If every candidate survives, the sampler takes one extra “bonus” token from the target’s distribution at position γ+1. The committed tokens extend the KV cache and the loop restarts.
The acceptance rule is what makes it exact
For a draft token x sampled from q, the sampler draws u uniformly from [0, 1] and accepts if u ≤ p(x)/q(x). When q over-proposes a token relative to p, some of those proposals are rejected. When q under-proposes a token, the residual distribution makes up the shortfall. That residual is norm(max(0, p − q)): the target’s probability mass that the draft failed to cover, renormalised.
The two pieces add up exactly. The probability of emitting token x is min(p(x), q(x)) from acceptance, plus (1 − α) × residual(x) from resampling, where α is the total accepted mass. Leviathan et al. show that α = Σ min(p, q), which equals one minus the total variation distance between p and q. Since the residual mass sums to 1 − α, the second term equals max(0, p(x) − q(x)). The sum is p(x). That one line of algebra is the whole lossless guarantee.
In practice “lossless” means lossless up to floating-point behaviour. Chen et al. phrase it as preserving the target distribution “within hardware numerics.” The current vLLM documentation says the same, and adds that batched kernels can shift logprobs slightly as batch size changes. That caveat applies to all batched inference, not only to speculation.
Greedy decoding is a special case. With temperature zero, p and q collapse to one-hot vectors. Acceptance becomes an equality test: keep the draft token if it equals the target’s argmax. Many published speedups, including most EAGLE and Medusa headline numbers, are measured at temperature zero. Speedups at temperature 1.0 are usually lower, because sampling noise lowers agreement between draft and target.
The bonus token and the “never worse than one” property
Every iteration commits at least one token. If the first draft token is rejected, the resampled replacement is still a valid sample from the target. If all γ are accepted, the bonus token comes free from the target’s last position. So each target forward pass yields between 1 and γ+1 tokens.
This matters for reasoning about worst cases. Speculative decoding can never make the target produce fewer tokens per pass than plain decoding. It can still make the system slower, because the draft work and the wider verification pass are not free. The worst case is a wall-clock loss, not a correctness loss.
Why the target pass is “almost free” — and when it is not
At batch size one, the verification pass processes γ+1 positions instead of one. On a memory-bound GPU, multiplying a weight matrix by 5 vectors instead of 1 costs about the same, because the dominant cost is reading the matrix. The attention over the existing KV cache is also read once per pass.
This “almost free” claim depends entirely on slack compute. A decoder layer’s arithmetic intensity grows with the number of tokens processed per weight load. Once that number crosses the GPU’s ridge point, the pass becomes compute-bound and each extra position costs real time. Continuous batching pushes servers toward exactly that point by packing many sequences into each step. We return to this conflict with numbers in the analysis section, because it explains most disappointing deployments.
Speculative Decoding Math: Expected Tokens per Step
The key quantity is the expected number of tokens committed per target call. Leviathan et al. model acceptance as independent per position with a constant rate α, and derive:
E[tokens per step] = (1 − α^(γ+1)) / (1 − α)
The intuition is a geometric series. The first token is always produced. The second requires the first draft token to be accepted (probability α), the third requires two acceptances (α²), and so on up to γ+1 tokens. Summing 1 + α + α² + … + α^γ gives the formula.
The same paper gives the expected wall-clock improvement. Let c be the cost of one draft call relative to one target call. One iteration costs γ draft calls plus one target call, or γc + 1 target-call units. The speedup is therefore:
speedup = (1 − α^(γ+1)) / ((1 − α)(γc + 1))
Worked numbers
The table below evaluates these formulas. These are illustrative calculations from the model, not benchmarks. The i.i.d. acceptance assumption is a simplification; real acceptance varies by position and by content.
| α (acceptance) | γ (draft length) | c (draft/target cost) | E[tokens/step] | Modelled speedup |
|---|---|---|---|---|
| 0.4 | 5 | 0.10 | 1.66 | 1.11x |
| 0.6 | 4 | 0.05 | 2.31 | 1.92x |
| 0.7 | 5 | 0.10 | 2.94 | 1.96x |
| 0.8 | 4 | 0.05 | 3.36 | 2.80x |
| 0.8 | 8 | 0.05 | 4.33 | 3.09x |
| 0.9 | 5 | 0.02 | 4.69 | 4.26x |
Walk through one row. With α = 0.8 and γ = 4, E = (1 − 0.8⁵)/0.2 = (1 − 0.328)/0.2 = 3.36 tokens per target pass. Four draft calls at 5% of target cost add 0.2, so the iteration costs 1.2 units. Dividing gives 2.80x.
Three lessons fall out. First, the curve is convex in α. Moving from 0.6 to 0.8 adds far more than moving from 0.4 to 0.6. Second, γ has diminishing returns, because tokens deep in the chain survive only if everything before them survives. At α = 0.8 and c = 0.05 the modelled optimum is around γ = 8. At α = 0.6 it is around γ = 4. Third, draft cost matters most when α is low. A cheap drafter with mediocre acceptance can beat an expensive one with good acceptance.
Acceptance length is the metric to report
Papers increasingly report average acceptance length τ, the mean number of tokens committed per target call, instead of α. It is directly measurable, and it folds in tree drafting, which the scalar α model cannot describe. The EAGLE-3 paper’s results on Vicuna 13B at temperature zero illustrate the scale. EAGLE reached an average acceptance length of 3.96 with a 3.05x speedup. EAGLE-2 reached 4.83 and 4.22x.
Note that measured speedup is lower than τ. The gap is the draft cost plus the extra verification overhead. If you track only one number in production, track τ per request class, alongside time per output token.
A tested NumPy implementation
The following code implements one verification step of speculative sampling. It then checks two claims empirically: the first emitted token follows the target distribution, and the mean tokens per step matches the closed-form formula when acceptance really is i.i.d.
import numpy as np
def speculative_step(p, q, draft_tokens, rng):
"""One verification step of speculative sampling.
p: (gamma+1, V) target probabilities at each draft position plus the bonus slot
q: (gamma, V) draft probabilities the draft tokens were sampled from
draft_tokens: (gamma,) token ids proposed by the draft
Returns the list of tokens committed this step (1 to gamma+1 tokens).
"""
out = []
for i, x in enumerate(draft_tokens):
if rng.random() < min(1.0, p[i, x] / q[i, x]):
out.append(int(x)) # accept draft token
continue
residual = np.maximum(p[i] - q[i], 0.0) # reject: resample from norm(max(0, p - q))
residual /= residual.sum()
out.append(int(rng.choice(len(residual), p=residual)))
return out
out.append(int(rng.choice(p.shape[1], p=p[-1]))) # all accepted: free bonus token
return out
def expected_tokens(alpha, gamma):
"""Leviathan et al.: expected tokens per target call with i.i.d. acceptance alpha."""
return (1 - alpha ** (gamma + 1)) / (1 - alpha)
def walltime_speedup(alpha, gamma, c):
"""Expected improvement factor, c = draft-call cost / target-call cost."""
return expected_tokens(alpha, gamma) / (gamma * c + 1)
if __name__ == "__main__":
rng = np.random.default_rng(0)
V, gamma, trials = 8, 4, 200_000
# Fixed toy distributions (context-independent) so we can check the marginal exactly.
p_row = rng.dirichlet(np.ones(V)); q_row = rng.dirichlet(np.ones(V))
p = np.tile(p_row, (gamma + 1, 1)); q = np.tile(q_row, (gamma, 1))
first = np.zeros(V); n_tok = 0
for _ in range(trials):
d = rng.choice(V, size=gamma, p=q_row)
toks = speculative_step(p, q, d, rng)
first[toks[0]] += 1; n_tok += len(toks)
alpha = np.minimum(p_row, q_row).sum() # alpha = 1 - TV(p, q)
print("max |empirical - p| for first token:", np.abs(first / trials - p_row).max().round(4))
print(f"alpha={alpha:.3f} mean tokens/step={n_tok/trials:.3f} formula={expected_tokens(alpha, gamma):.3f}")
for a, g, c in [(0.6, 4, 0.05), (0.8, 4, 0.05), (0.8, 8, 0.05), (0.7, 5, 0.1)]:
print(f"alpha={a} gamma={g} c={c}: E={expected_tokens(a, g):.2f} speedup={walltime_speedup(a, g, c):.2f}x")
Running it with NumPy prints a maximum deviation of about 0.001 between the empirical first-token frequencies and p over 200,000 trials. It also prints a mean of 2.101 tokens per step against a formula value of 2.100 for the sampled α of 0.547. The toy uses context-independent distributions so the marginal can be checked exactly. A real engine conditions p and q on the growing prefix, vectorises the loop over the batch on the GPU, and fuses the sampler into a kernel. The logic is the same.
Two implementation traps are worth naming. Never resample from p itself after a rejection; that double-counts mass the draft already covered and biases the output. And always use the exact q the draft sampled from, after temperature, top-p and any logit processors. If the draft samples from a filtered distribution but reports the unfiltered one, the guarantee silently breaks.

Figure 2: One verification step with γ = 4. Two draft tokens are accepted, the third is rejected and resampled from the residual, three tokens are committed and the draft state rolls back.
The sequence diagram makes the bookkeeping visible. The draft proposes x1 to x4 sequentially, which is the cheap but serial part. The target scores the prefix plus all four candidates in one batch and returns five distributions, including the bonus slot. The sampler accepts x1 and x2, rejects x3 and resamples it. Three tokens are committed. The KV entries for x3 and x4 in the target cache, and any draft-side state past x2, must be discarded. That rollback is cheap with paged KV caches, but it is the most common source of bugs in custom implementations.
Deeper Analysis: The Speculative Decoding Drafter Family Tree
Every variant of speculative decoding keeps the verify-and-accept core and changes only where the draft comes from. That choice sets α, c, memory cost and operational burden.

Figure 3: Where the draft tokens come from. Separate draft models, heads on the target, and model-free drafting trade acceptance against cost and operational complexity.
Separate draft models
The original recipe pairs a target with a smaller model from the same family, for example an 8B sibling drafting for a 70B model. Same-family pairs share a tokenizer and similar training data, which keeps α reasonable. The cost is real: an 8B drafter is not negligible against a 70B target, it needs its own KV cache, and it must run γ sequential steps per iteration.
A newer branch drafts all γ tokens in parallel rather than sequentially. The current TensorRT-LLM documentation lists PARD and DFlash, and vLLM documents a dflash method for parallel drafters. Parallel drafting attacks the serial part of the draft cost directly. Treat these as early-stage options and benchmark them on your own traffic.
Tokenizer mismatch used to rule out cross-family drafts entirely. Research on heterogeneous-vocabulary speculation has relaxed that, but engines still mostly expect a shared vocabulary. The practical default remains a same-tokenizer drafter.
Medusa: independent heads on the target
Medusa (Cai et al., 2024) removes the separate draft model. It adds several extra decoding heads to the target’s final hidden state. Head k predicts the token k+1 positions ahead. Each head offers its top candidates, and the combinations form a tree that the target verifies in one pass using a tree attention mask. The paper reports that Medusa-1, which trains only the heads on a frozen backbone, achieves over 2.2x speedup. Medusa-2, which fine-tunes the backbone jointly, improves this to 2.3–3.6x.
Medusa’s weakness is structural. Each head predicts its offset without seeing the tokens the earlier heads chose. Head 3 must guess position +3 without knowing what positions +1 and +2 will be. Accuracy falls quickly with distance. That limitation motivated EAGLE.
Medusa is also fading from engine support. The current vLLM speculative decoding docs no longer list it among documented methods, and TensorRT-LLM’s current feature page focuses on Draft/Target, EAGLE-3, MTP and newer parallel drafters. Older TensorRT-LLM releases did document Medusa. If you have a Medusa deployment, plan a migration.
EAGLE, EAGLE-2 and EAGLE-3: autoregression at the feature level
EAGLE (Li et al., 2024) keeps the draft autoregressive but runs it on the target’s hidden features rather than on tokens alone. A single lightweight transformer layer takes the target’s top-layer feature and the embedding of the sampled next token, and predicts the next feature and token. Because it is autoregressive, each draft position sees its predecessors, which Medusa heads cannot.
EAGLE-2 kept the drafter and changed how the tree is built. Instead of a fixed tree shape, it uses the drafter’s own confidence scores to grow a dynamic draft tree, expanding promising branches and pruning weak ones. The EAGLE-3 paper’s Vicuna 13B table puts EAGLE-2 at 4.22x versus EAGLE’s 3.05x at temperature zero.
EAGLE-3 (Li et al., NeurIPS 2025) made two further changes. It fuses low-, middle- and high-level features from the target instead of only the top layer. And it drops the feature-prediction loss, predicting tokens directly, with a “training-time test” procedure that feeds the drafter its own outputs during training. The authors argue this lets the drafter keep improving as training data scales. The abstract reports “a speedup ratio up to 6.5x, with about 1.4x improvement over EAGLE-2.” It also reports a 1.38x throughput improvement at batch size 64 in SGLang. That second number is worth remembering: it is far below the batch-one headline, which is the batch-size effect in action.
EAGLE-3 is now the default learned drafter in the major engines. SGLang’s documentation reports throughput on Llama 3.1 8B Instruct over MT-Bench on one H100 of 158.34 tokens/s without speculative decoding, 244.10 with EAGLE-2 and 373.25 with EAGLE-3. TensorRT-LLM notes that EAGLE v1/v2 draft checkpoints are incompatible with its EAGLE-3 path, so you need EAGLE-3 heads specifically.
Multi-token prediction heads: the drafter ships with the model
MTP moves the drafter into pre-training. DeepSeek-V3’s technical report trains sequential MTP modules alongside the main model, with depth D = 1, so each position also predicts one additional token. The motivation was partly training quality. The report’s ablations show MTP improving benchmark performance, but the modules can be reused as a drafter at inference.
The report gives the inference numbers directly: “the acceptance rate of the second token prediction ranges between 85% and 90% across various generation topics,” enabling “1.8 times TPS (Tokens Per Second).” That is consistent with the formula. With γ = 1 and α between 0.85 and 0.90, E = 1 + α gives 1.85–1.90 tokens per step. The measured 1.8x sits just below that, reflecting overhead.
MTP is attractive operationally. There is no separate artifact to train, the drafter is aligned with the target by construction, and engines can load it from the same checkpoint. vLLM exposes it as method: "mtp", SGLang runs it through its EAGLE machinery (with NEXTN as an alias), and TensorRT-LLM provides MTPDecodingConfig. Several other open-weight model families have since shipped MTP layers too. Check each model card rather than assuming.
Model-free drafting: n-grams, suffixes and lookahead
Some workloads copy heavily from their input. Code editing, retrieval-augmented answers, summaries that quote sources and agent loops that echo tool output all repeat long spans. Prompt lookup decoding exploits this. It matches the last few generated tokens against the prompt and proposes the continuation that followed the match. vLLM configures it as method: "ngram" with prompt_lookup_min and prompt_lookup_max. The draft cost is essentially zero, so c ≈ 0, and even modest α pays.
Suffix decoding generalises this with suffix trees built over the prompt and over previous outputs, which suits repetitive agentic workloads. vLLM documents a suffix method, and TensorRT-LLM lists a suffix automaton approach. Note that TensorRT-LLM’s docs mark its NGram method as deprecated, with removal planned for January 2027.
Lookahead decoding (Fu et al., ICML 2024) generates its own n-grams. It runs Jacobi-style parallel iterations to produce candidate n-grams, caches them in a pool, and verifies matching ones, all within the target model. The paper reports up to 1.8x on MT-bench and up to 4x “with strong scaling on multiple GPUs” on code completion. It needs no drafter, but it spends substantial extra compute per step, which makes it the most sensitive method to batch size. Engine support for lookahead decoding is thinner in 2026 than for EAGLE-3 or MTP.
Decision matrix
| Drafter | Extra training | Typical draft cost c | Acceptance profile | Best fit |
|---|---|---|---|---|
| Same-family draft model | None if sibling exists | Moderate | Moderate, workload-dependent | Quick start when a small sibling exists |
| Medusa heads | Heads, optionally backbone | Very low | Falls off with offset | Legacy deployments |
| EAGLE-3 head | Drafter training | Low | High, best published | General chat and code at low batch |
| Native MTP | None, ships with model | Low | High at short γ | Models like DeepSeek-V3 |
| Prompt lookup and suffix | None | Near zero | High on copy-heavy text, low elsewhere | RAG, code edits, agents |
| Lookahead decoding | None | High extra compute | Moderate | Spare-compute single-stream use |
The Batch-Size Problem: Why Speedups Collapse
Every batch-one speedup quoted above assumes the GPU has idle compute. Production servers try hard not to. This is the most under-explained part of speculative decoding, and the source of most disappointing rollouts.
A roofline model of the collapse
Consider a decode step that processes N tokens in total across the batch. Its time is roughly the larger of the weight-streaming time and the compute time. Weight streaming is fixed per step. Compute grows linearly with N. The crossover, or ridge point, is the token count where the two are equal.
Take an illustrative 70B model in FP8 on an H100-class GPU. Assume 70 GB of weights, 3.35 TB/s of bandwidth and an achieved 500 TFLOP/s of FP8 compute. Weight streaming costs about 21 ms per step. Each token needs about 140 GFLOP, or about 0.28 ms of compute. The ridge point is around 75 tokens per step.
Without speculative decoding, a batch of B sequences processes B tokens per step. With γ = 4, verification processes 5B tokens. The table below applies the formula with α = 0.8 and c = 0.05. These are illustrative outputs of a simplified model. It ignores KV-cache reads, attention cost, kernel efficiency changes and tree drafting.
| Batch size B | Plain step | Verify step (5B tokens) | Modelled speedup |
|---|---|---|---|
| 1 | 20.9 ms | 20.9 ms | 2.80x |
| 4 | 20.9 ms | 20.9 ms | 2.80x |
| 16 | 20.9 ms | 22.4 ms | 2.64x |
| 32 | 20.9 ms | 44.8 ms | 1.43x |
| 64 | 20.9 ms | 89.6 ms | 0.75x |
| 128 | 35.8 ms | 179.2 ms | 0.65x |
The shape matters more than the exact values. Up to the batch where 5B crosses the ridge, verification is nearly free and the speedup holds. Beyond it, every speculative position costs compute, and rejected positions are pure waste. With α = 0.8 and γ = 4, about one-third of verified positions are discarded on average. At high batch, the server spends compute verifying tokens it will throw away, which tokens from other sequences could have used.
Real systems blur the curve. Attention over long KV caches stays memory-bound longer, MoE models have lower per-token compute, and tree drafting changes the token count. The direction is consistent with published evidence, though. EAGLE-3’s 6.5x headline becomes 1.38x throughput at batch 64 in SGLang. TensorRT-LLM’s current documentation states plainly that there is “currently no way to dynamically disable speculation,” so speedups are seen only at low batch sizes.
Latency versus throughput
The batch effect changes what speculative decoding is for. At low concurrency, it cuts time per output token (TPOT) for each user, often by 2x or more. At high concurrency, the same GPU can usually deliver more total tokens per second without it. Time to first token is mostly a prefill cost and barely improves; speculation helps the decode phase.
That makes speculative decoding a latency tool. It fits interactive chat on dedicated capacity, coding assistants, voice agents and agent loops where tool calls wait on generation. It is a poor fit for offline batch jobs, where throughput per dollar is the target. If you define service levels in terms of TTFT, TPOT and goodput, speculation buys TPOT headroom at low load and can cost goodput at peak.
Dynamic speculative decoding
The fix is to make γ a function of load. When the batch is small, draft deeply. As it grows, shorten γ. Past a threshold, turn speculation off. The current vLLM docs list “Dynamic Speculative Decoding” as a feature “useful for RL or workload with fluctuating QPS.” SGLang defaults its draft-depth and tree-width settings to automatic. TensorRT-LLM, as quoted above, does not yet disable speculation dynamically.
Adaptive approaches can also track acceptance per request. A request producing poetry might see τ near 1.5, while a code edit might see τ above 4. Spending verification compute on the second and not the first is pure gain.
Speculative Decoding Engine Support in November 2026
All three major open-source engines support speculative decoding, but method lists change faster than most blog posts. Here is the state of their documentation this month. Always check the version you deploy. For broader engine trade-offs, see our vLLM vs SGLang vs TensorRT-LLM comparison.
vLLM. The speculative decoding documentation configures methods through a speculative_config with a method key and num_speculative_tokens. Documented methods include draft_model, eagle3, mtp, ngram, ngram_gpu, suffix and dflash, plus dynamic speculation, adaptive verification and per-request acceptance metrics. The docs state that sampling is “theoretically lossless up to the precision limits of hardware numerics.” They also document a “FLy” verification mode that “deliberately relaxes verification and is not distribution preserving.” If you enable that, you are no longer running lossless speculative decoding.
SGLang. The SGLang docs select a method with --speculative-algorithm, with values including EAGLE, EAGLE3, STANDALONE (a separate draft model), NGRAM and DFLASH. MTP models run through the EAGLE path, with NEXTN as an alias. Three knobs shape the tree: --speculative-num-steps (draft depth), --speculative-eagle-topk (branching factor per step) and --speculative-num-draft-tokens (maximum tokens verified per step). SGLang combines this with RadixAttention prefix caching, which helps agentic workloads with long shared system prompts.
TensorRT-LLM. The current TensorRT-LLM feature page lists Draft/Target, EAGLE-3, MTP, PARD, DFlash and a suffix automaton approach. NGram and user-provided drafting are marked deprecated. Configuration uses classes such as Eagle3DecodingConfig, MTPDecodingConfig and DraftTargetDecodingConfig in the LLM API. The docs note that EAGLE v1/v2 checkpoints are incompatible with EAGLE-3 and that speculation cannot yet be disabled dynamically.
A practical note on versioning. Engine docs now change monthly, and method names, defaults and deprecations shift between minor releases. Pin the engine version, record the speculative configuration in your deployment manifest, and re-benchmark on every upgrade.
Memory and scheduling costs
Speculative decoding consumes memory that would otherwise hold KV cache. A separate draft model needs its weights and its own KV cache. EAGLE-3 and MTP heads are small, but they still add weights and per-sequence state. The target cache must also reserve space for γ speculative positions per sequence, some of which will be rolled back.
Less KV cache means fewer concurrent sequences, which feeds back into the batch-size question. For long-context deployments, review your KV cache optimisation strategy before enabling speculative decoding, because the two compete for the same HBM.
CUDA graphs are another hidden cost. Engines capture graphs for fixed shapes, and speculation multiplies the shapes, since each combination of batch size and draft length needs one. Expect longer warm-up and higher graph memory.
Speculative Decoding Cost and Latency Trade-offs
Speculative decoding trades extra compute for lower latency. Whether that is cheap depends on what you pay for.
On dedicated GPUs at low utilisation, the compute was already paid for and idle, so the trade is close to free. A 2x TPOT gain at batch one can let you meet a latency target on fewer, more heavily loaded replicas. On a shared fleet running near the ridge point, the same compute could have served other users. Speculative decoding then raises cost per token.
A simple accounting helps. Measure two curves on your own traffic: tokens per second per GPU versus concurrency, with and without speculative decoding. Then measure p50 and p95 TPOT on both. The crossover concurrency is where speculative decoding stops paying. Run below it with speculation on and above it with speculation off. That one chart replaces most vendor speedup claims.
Draft training is a one-off cost. EAGLE-3 heads for popular open models are often published, so you may not need to train one. For a fine-tuned target, acceptance can drop sharply because the public head was trained against the base model’s features and outputs. Budget time to retrain the head on your fine-tune’s outputs, or fall back to MTP or model-free drafting.
There is also a quality-cost line that teams sometimes cross without noticing. Lossy relaxations such as vLLM’s FLy mode, or “typical acceptance” rules that accept tokens the target merely finds plausible, improve speed by abandoning the guarantee. They may be fine for your use case. Treat them as a model change, though, and run your evaluation suite, because they are no longer the same model in distribution.
Trade-offs, Gotchas, and What Goes Wrong
Silent decay of acceptance. τ depends on the match between drafter and traffic. A prompt-template change, a new language mix or a fine-tuned target can halve it without any error message. Export per-request acceptance metrics (vLLM documents them) and alert on τ, not just latency.
Fine-tuned targets with stale drafters. An EAGLE-3 head trained on a base model and paired with an instruction-tuned or domain-tuned derivative usually keeps working, but with lower acceptance. The output stays correct; only the speed suffers. That makes the regression easy to miss.
Temperature and sampling filters. Acceptance drops as temperature rises, because both distributions spread out and diverge in their tails. Repetition penalties, top-p and structured output grammars must be applied consistently to p and q. An engine that applies a grammar mask to the target but not the drafter wastes drafts. A custom implementation that applies different filters to each breaks exactness.
Batch-size cliffs. As shown above, speculative decoding that wins at 4 concurrent users can lose at 64. Load tests at a single concurrency level hide this. Sweep concurrency.
Reasoning models and long outputs. Long chain-of-thought outputs are often less predictable than chat replies, and they run for thousands of tokens. A small τ drop multiplied across very long outputs has a large cost effect. Measure on reasoning traces specifically.
Rollback bugs. Custom pipelines that forget to discard rejected KV entries, or that desynchronise draft and target caches after preemption, produce subtle corruption. Prefer engine-native implementations and validate with greedy-equality tests: at temperature zero, outputs with and without speculation should match token for token, apart from rare floating-point ties.

Figure 4: A decision path for choosing a drafter. Concurrency decides whether speculation is worth benchmarking; model assets and output style decide which drafter to use.
The flowchart starts with concurrency because that dominates. Below about eight concurrent sequences per GPU, speculative decoding is almost always worth enabling. Between 8 and 32, benchmark with a short draft length. Above 32, leave it off unless the engine can adapt dynamically. These thresholds are rules of thumb for 70B-class dense models. Check your own ridge point. Then the model’s assets decide: native MTP first, an EAGLE-3 head next, prompt lookup for copy-heavy outputs, and a sibling draft model as the fallback.
Practical Recommendations
Start from your traffic, not from the paper with the biggest number. Measure concurrency distribution per GPU over a real week, and identify the fraction of time spent below the ridge point. If most traffic sits at high concurrency, speculative decoding will not help much. Spend the effort on continuous batching and scheduler tuning instead.
If you serve latency-sensitive traffic at low concurrency, pick the drafter by asset availability. Models with native MTP layers should use them first, because alignment is guaranteed and setup is trivial. Otherwise, use a published EAGLE-3 head for your exact base model. Add prompt lookup or suffix decoding for RAG, code-editing and agent workloads, where copying is common.
Tune γ empirically. The formula’s optimum shifts with α, and tree drafting complicates it further. Sweep draft depth and tree width at two or three concurrency levels and pick per-tier settings. Keep a comparison slice of traffic with speculation disabled so you can detect regressions directly instead of inferring them.
Finally, treat lossless and lossy modes as different products. Lossless speculative decoding can ship without re-running quality evaluations. Lossy relaxations cannot.
Checklist before rollout:
- Measure concurrency per GPU and find your ridge point.
- Choose the drafter: MTP, then EAGLE-3, then n-gram or suffix, then a sibling model.
- Sweep γ and tree width at low, medium and peak load.
- Verify greedy outputs match token for token with speculation off.
- Export acceptance length per request class; alert on drops.
- Confirm lossless mode; gate any lossy mode behind evaluation.
- Pin engine version and re-benchmark on every upgrade.
Frequently Asked Questions
What is speculative decoding in LLMs?
Speculative decoding is an inference technique in which a cheap proposer drafts several tokens and the large target model verifies them all in one forward pass. A rejection-sampling rule accepts a prefix of the draft and resamples the first mismatch, so the output distribution is identical to the target’s. It works because small-batch decoding is memory-bound: scoring five tokens costs about the same as generating one. Published speedups range from about 2x to over 4x at batch size one, depending on drafter quality.
Does speculative decoding change model output quality?
Standard speculative decoding does not change output quality. The acceptance rule from Leviathan et al. and Chen et al. provably preserves the target distribution, up to floating-point numerics. At temperature zero, outputs should match plain decoding token for token, apart from rare numerical ties. Quality changes only if you enable a lossy variant, such as relaxed or typical acceptance, or vLLM’s FLy verification mode, which its documentation describes as not distribution preserving. Treat those modes as model changes and re-run your evaluations.
How do I calculate the speedup from speculative decoding?
With per-token acceptance rate α and draft length γ, the expected tokens per target call is (1 − α^(γ+1)) / (1 − α). Divide by (γc + 1), where c is draft-call cost relative to a target call, to get the modelled wall-clock speedup. For α = 0.8, γ = 4 and c = 0.05, that gives 3.36 tokens per step and about 2.8x. The model assumes idle GPU compute and i.i.d. acceptance, so treat it as an upper-bound estimate at low batch.
Why does speculative decoding not help at high batch sizes?
At high batch sizes the GPU is already near its compute limit. Verifying γ extra positions for every sequence then costs real compute, and rejected positions waste it. Continuous batching exploits the same idle compute that speculative decoding needs, so the two compete. The EAGLE-3 paper reports up to 6.5x at low batch but 1.38x throughput at batch 64 in SGLang. TensorRT-LLM’s docs state speedups appear only at low batch sizes. Dynamic modes that shorten or disable drafting under load mitigate this.
Is EAGLE-3 better than Medusa?
On published benchmarks, yes. Medusa’s heads each predict a fixed offset without seeing the tokens before it, so accuracy falls quickly with distance. EAGLE-3 drafts autoregressively over fused target features and reports up to 6.5x, about 1.4x better than EAGLE-2. Medusa’s paper reports 2.2x for Medusa-1 and 2.3–3.6x for Medusa-2. Engine support has also moved: current vLLM and TensorRT-LLM docs centre on EAGLE-3 and MTP rather than Medusa.
What is multi-token prediction and how does it relate to speculative decoding?
Multi-token prediction (MTP) trains a model to predict more than one future token during pre-training, using extra modules. At inference those modules serve as a built-in drafter. DeepSeek-V3 uses one MTP module (depth 1). Its technical report states the second-token acceptance rate is 85–90% across topics, delivering 1.8 times tokens per second. vLLM, SGLang and TensorRT-LLM all support MTP-based speculative decoding for models that ship the weights.
Further Reading
- EAGLE-3 vs Medusa vs lookahead decoding: speedup comparison — a benchmark-oriented companion to this architecture guide.
- Continuous batching for LLM inference — the scheduler technique that competes with speculation for idle compute.
- vLLM vs SGLang vs TensorRT-LLM in 2026 — engine selection beyond speculative decoding.
- KV cache optimisation for LLM inference — managing the memory that speculation also needs.
- LLM inference SLOs: TTFT, TPOT and goodput — how to define the latency targets speculation should serve.
- Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023.
- Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, DeepMind, 2023.
References
- Leviathan, Y., Kalman, M., Matias, Y. “Fast Inference from Transformers via Speculative Decoding.” ICML 2023. arXiv:2211.17192.
- Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., Jumper, J. “Accelerating Large Language Model Decoding with Speculative Sampling.” arXiv:2302.01318, 2023.
- Cai, T., et al. “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.” arXiv:2401.10774, 2024.
- Li, Y., Wei, F., Zhang, C., Zhang, H. “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.” arXiv:2401.15077, 2024.
- Li, Y., et al. “EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.” arXiv:2406.16858, 2024.
- Li, Y., et al. “EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test.” NeurIPS 2025. arXiv:2503.01840.
- Fu, Y., Bailis, P., Stoica, I., Zhang, H. “Break the Sequential Dependency of LLM Inference Using Lookahead Decoding.” ICML 2024. arXiv:2402.02057.
- DeepSeek-AI. “DeepSeek-V3 Technical Report.” arXiv:2412.19437, 2024. Section 5.4.3.
- vLLM documentation, “Speculative Decoding,” accessed November 2026.
- SGLang documentation, “Speculative Decoding,” accessed November 2026.
- NVIDIA TensorRT-LLM documentation, “Speculative Decoding,” accessed November 2026.
By Riju — about

Pingback: FlashAttention Explained: IO-Aware Attention, Tiling, Online