RWKV vs Mamba vs Linear Attention: Recurrent LLM Architectures Compared (2026)

A transformer serving a 128,000-token conversation has to keep roughly 16 GiB of key-value cache per user, even for a modest 8-billion-parameter model with grouped-query attention. A recurrent language model holding the same conversation keeps a few tens of megabytes, and that number does not change when the conversation gets ten times longer. That asymmetry is the whole reason the RWKV vs Mamba question, and the wider fight between linear attention, state space models and softmax attention, matters in 2026.

It matters now because the idea has left the lab. Qwen3-Next, Kimi Linear and NVIDIA’s Nemotron 3 family all ship hybrids that replace most attention layers with a recurrent layer, and the Mamba-3 and RWKV-7 papers show the pure-recurrent branch closing quality gaps that looked structural two years ago. Choosing between these designs is now a serving-cost decision as much as a research one.

This post takes the three families apart at the level of the state update, does the memory and bandwidth arithmetic with explicit assumptions, names where each one fails, and ends with a decision matrix you can use.

What this covers: the shared recurrent view of attention; how RWKV-7, Mamba-2 and Mamba-3, and Gated DeltaNet differ; worked state-size and decode-cost numbers; recall and state-tracking failure modes; hybrid ratios in shipping models; and a practical selection guide.

Context and Background

Standard softmax attention computes, for each new token, a weighted sum over every previous value, with weights from a softmax over query-key dot products. Training cost is quadratic in sequence length, and inference needs the full key-value (KV) cache. The cache is the dominant memory consumer for long-context serving, which is why so much of the serving stack, from paged allocators to offload tiers, exists to manage it. If you want that side of the story, our piece on disaggregated LLM serving with NVIDIA Dynamo and LMCache KV-cache offload covers how production systems cope when the cache will not fit.

There are two broad escapes. One keeps softmax attention but makes it sparse, selecting which tokens each query reads; DeepSeek’s approach is covered in native sparse attention and DeepSeek Sparse Attention. The other, which this post is about, drops the growing cache entirely and compresses history into a fixed-size state that is updated once per token.

The second path has a long pedigree. Katharopoulos et al. showed in 2020 in “Transformers are RNNs” that replacing the softmax with a kernel feature map lets you reorder the matrix products and obtain a linear-time recurrent form, reducing the cost from O(N^2) to O(N) for sequence length N. Their paper reported speedups of up to 4000x on autoregressive prediction of very long sequences, though the “up to” qualifier hides conditions that rarely resemble modern LLM serving. The quality of those early linear transformers lagged softmax attention, and that lag is what the last five years of work has tried to close.

Three lineages emerged. State space models descend from control theory and signal processing, with Mamba as the breakout. RWKV grew from the Receptance Weighted Key Value line, developed in the open by Bo Peng and a large community. And the linear-attention family, including DeltaNet and Gated DeltaNet, grew out of the fast-weight-programmer view of attention. The surprising thing is how much they converged. By 2026 the three share a core idea: treat the state as a matrix of fast weights, update it each token with a decay and a corrective write, and read it with the query.

The convergence is not a coincidence. The Mamba-2 paper by Dao and Gu, “Transformers are SSMs”, formalised a state space duality connecting SSMs and attention variants through structured semiseparable matrices, and reported that Mamba-2 is 2 to 8 times faster than Mamba while staying competitive with transformers on language modelling. Once you accept that duality, “RWKV vs Mamba” stops being a question about two different species and becomes a question about which design choices in a shared template matter. If you want the Mamba-versus-transformer background in a single place, see our earlier explainer on Mamba, state space models and hybrid architectures; here we focus on the comparison across the recurrent family itself.

The Shared Template: One Recurrence, Three Dialects

Every model in this post can be written as a matrix-valued state S that is updated at each token t and read by a query q. The state has shape d_k by d_v per head, and a layer runs many heads. Everything interesting is in how S is decayed and how it is written.

Figure 1: Lineage of recurrent LLM layers. Gating and the delta rule are the two big ideas; RWKV-7, Mamba-3 and Gated DeltaNet each combine them differently.

The figure shows how each family descends from the kernelised form of attention. Adding a forget gate gives the Mamba and RWKV-5/6 branch; adding the delta rule gives DeltaNet; combining them gives Gated DeltaNet and RWKV-7. All end at the same practical property, constant-memory decode.

Plain linear attention: an additive memory

In the simplest form, the state accumulates an outer product of each key and value, and the output is the state multiplied by the query:

S_t = S_{t-1} + v_t k_t^T, o_t = S_t q_t

This is a pure additive associative memory. Every token adds a rank-one term, and nothing is ever removed. After n tokens the state is a sum of n outer products squeezed into a d_k by d_v matrix. If d_k is 128, a matrix of that size can represent at most 128 linearly independent keys exactly; beyond that, stored pairs interfere. Retrieval then returns a blend of every value whose key has non-zero dot product with the query, and the blend gets noisier as the context grows. This interference is the root cause of the weak recall of early linear transformers.

Forget gates: letting the memory decay

The first fix is a decay. Multiply the old state by a gate before writing:

S_t = a_t * S_{t-1} + v_t k_t^T

When the gate a_t is a scalar between 0 and 1, the model can fade old content. When it is input-dependent, the model can choose to forget aggressively on a topic change or hold on to a fact. Mamba’s central contribution was making the SSM parameters depend on the input (selectivity), which is exactly this: the decay becomes a function of the current token. A quick calculation shows why the gate value matters. If a_t is a constant 0.99 per token, a stored item’s contribution halves every ln(2)/-ln(0.99), about 69 tokens. At 0.999 the half-life is about 693 tokens. A model that wants to carry a fact across 50,000 tokens needs gates very close to 1 on at least some channels, and needs the write to avoid disturbing them. Different channels learn different time constants, which is why these models devote many state dimensions to a spectrum of decays.

The delta rule: write what is missing

Decay alone does not fix interference. The delta rule does. Instead of blindly adding v k^T, the layer first asks what the state currently predicts for key k, and writes only the correction:

S_t = S_{t-1} (I – b_t k_t k_t^T) + b_t v_t k_t^T

Here b_t is a learned write strength between 0 and 1 and k_t is normalised. If b_t is 1, the term (I – k k^T) erases whatever the state held along direction k, and the new value replaces it. This is one step of online gradient descent on the loss that asks the state to reconstruct v from k, which is why papers describe it as “test-time training” of fast weights. The DeltaNet line of work, built on this rule, gave far better associative recall than purely additive linear attention.

Gated DeltaNet: both at once

The ICLR 2025 paper “Gated Delta Networks: Improving Mamba2 with Delta Rule” by Songlin Yang, Jan Kautz and Ali Hatamizadeh combines the two. The authors argue the mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates. The resulting recurrence multiplies the decayed state by the delta-rule transition, with a scalar gate alpha_t on the old state:

S_t = alpha_t * S_{t-1} (I – b_t k_t k_t^T) + b_t v_t k_t^T

The paper also provides a hardware-friendly chunkwise parallel training algorithm, and reports that Gated DeltaNet beats Mamba-2 and DeltaNet across language modelling, commonsense reasoning, in-context retrieval, length extrapolation and long-context understanding. The abstract gives no single headline number, so take the direction of the result, not a magnitude. This is the layer that Qwen3-Next and Qwen3.5 use in their linear layers, according to the architecture survey we cite below.

How RWKV-7 generalises the same step

RWKV-7, called “Goose”, is described in the paper “RWKV-7 ‘Goose’ with Expressive Dynamic State Evolution” (arXiv 2503.14456, March 2025, 18 listed authors led by Bo Peng). The abstract describes it as a generalised delta rule with vector-valued gating and in-context learning rates, plus a relaxed value replacement rule. In the template above, the scalar decay alpha_t becomes a per-channel vector, so each key dimension has its own forgetting rate, and the write strength b_t also becomes a per-channel quantity. The state-transition matrix is therefore diagonal plus low-rank rather than a scalar times a rank-one projector.

I will not reproduce the full RWKV-7 equations here, because the author’s explanatory post renders them as images and I could not verify the exact notation in this run. The qualitative point is verified: the state does not store key-value pairs directly, it learns the key-to-value relationship from context by taking gradient-like steps, with a learnable diagonal decay that acts like an LSTM forget gate and can zero out channels.

Mamba-2 and Mamba-3: the SSM dialect

In Mamba-2, the state per head is shaped by a head dimension P and a state dimension N, the transition is a scalar decay per head per token, and the write is an input-dependent outer product. That makes it close to gated linear attention, which is precisely what the duality paper argues. Mamba-3 appeared in March 2026 (arXiv 2603.15569) as “Mamba-3: Improved Sequence Modeling using State Space Principles”. Per its abstract it changes three things: a more expressive recurrence derived from SSM discretisation, a complex-valued state update intended to enable richer state tracking, and a multi-input multi-output (MIMO) formulation that improves quality without raising decode latency. The abstract reports that at 1.5B scale Mamba-3 improves average downstream accuracy by 0.6 percentage points over Gated DeltaNet, the next best model, with the MIMO variant adding 1.2 points for a total of 1.8, and reaches comparable perplexity to Mamba-2 using half the state size. Those are abstract-level figures from the authors, not independent reproductions.

The takeaway from this section is that the three names describe overlapping design spaces. What differs in practice is the shape of the transition, the parameterisation of the gates, and the kernel quality of the open implementations.

Worked Numbers: State Size, Memory and Decode Cost

Architecture talk is cheap; the serving bill is what decides adoption. The arithmetic below uses explicit, illustrative configurations. These are not measurements of any specific checkpoint, and you should substitute your own model’s dimensions. Where a config resembles a public model I say so, and where I am recalling rather than verifying, I flag it.

Figure 2: Per-token decode data flow. The transformer reads a cache that grows with context; the recurrent layer reads and overwrites a fixed state.

The figure contrasts the two memory behaviours. In the transformer path the amount of data touched per generated token rises linearly with n. In the recurrent path it is the same at token 10 and at token 10 million.

The KV cache baseline

Take a transformer with 32 layers, 8 KV heads, head dimension 128, and fp16 storage. Those dimensions match the published Llama 3 8B configuration as I recall it; treat that as a convenient illustration, not a citation. Each token stores a key and a value in every layer:

Per token: 2 (K and V) x 32 layers x 8 heads x 128 dims x 2 bytes = 131,072 bytes, which is 128 KiB.
At 8,192 tokens: 1 GiB.
At 131,072 tokens (128k): 16 GiB.
At 1,048,576 tokens (1M): 128 GiB.

The growth is exactly linear. No clever allocator changes the total number of bytes that must exist; paging, quantisation to 8 or 4 bits, and offload to CPU or NVMe change where they live and how many you need.

The recurrent state for the same model width

Now take a linear-attention-style recurrent model with the same 32 layers, 32 heads per layer, and a 128 by 128 matrix state per head, stored in fp32 because recurrent states accumulate and are numerically sensitive:

Per head: 128 x 128 x 4 bytes = 65,536 bytes, 64 KiB.
Per layer: 32 heads x 64 KiB = 2 MiB.
Whole model: 32 layers x 2 MiB = 64 MiB.

That is the entire memory of the conversation, whether it holds 500 tokens or 5 million. Comparing the two, 64 MiB divided by 128 KiB per token equals 512. The recurrent state occupies the same space as the KV cache of a 512-token prompt. Past that length the recurrent model is smaller, at 128k by a factor of 16 GiB / 64 MiB = 256, and at 1M by a factor of 2,048.

For Mamba-2 style layers with an expanded inner width of 8,192 and a state dimension N of 128, the state is 8,192 x 128 = 1,048,576 elements per layer; over 32 layers that is 33,554,432 elements, or exactly 128 MiB in fp32. For an RWKV-7-style model with head size 64, if the model width is 2,560 and there are 32 layers (my recollection for the 2.9B World model; verify against the model card), the state is 40 heads x 64 x 64 = 163,840 elements per layer, 5,242,880 elements over 32 layers, 20 MiB in fp32. The ratio between these designs is therefore a hyper-parameter choice (head size, number of heads, state dimension), not a fundamental property of “RWKV” or “Mamba”. Mamba-3’s claim of matching Mamba-2’s perplexity at half the state size is exactly a claim about this axis.

Concurrency: the number that sets cost per user

Assume an 80 GB accelerator, 16 GB of fp16 weights for an 8B-parameter model, and say 64 GB left for caches after activations and overhead (a deliberately round assumption). The number of simultaneous 128k-token sessions is:

Transformer: 64e9 / 17.18e9 (16 GiB in bytes) = 3.7, so 3 sessions.
Recurrent at 64 MiB: 64e9 / 67.1e6 = 953 sessions.

Of course 953 is a memory-only ceiling. Compute, batching efficiency and scheduling will bind first, but the point holds: for long-context, many-user workloads the recurrent design moves the bottleneck from capacity to compute.

Decode bandwidth: why batch size one hides the gap

Single-stream decode on a modern GPU is memory-bandwidth bound. An H100 SXM has about 3.35 TB/s of HBM3 bandwidth per NVIDIA’s published specification. Reading 16 GB of fp16 weights per token costs about 16 / 3350 = 4.8 ms, a floor of roughly 209 tokens per second at batch one.

Add a 128k KV cache for that one user and each token must also read 17.18 GB: (16 + 17.18) / 3350 = 9.9 ms, or about 101 tokens per second, half the speed. The recurrent model reads 64 MiB, which adds 0.02 ms, leaving it near 4.8 ms. These are idealised roofline numbers that ignore kernel efficiency, attention being partly compute-bound at prefill, and prefix caching. Real systems land below the roofline, but the ratio tells you where the time goes.

At large batch the weights are amortised across users while each user’s cache is not, which is why KV traffic dominates throughput for long-context serving. The recurrent state is also read once per token per user, but at 64 MiB it stays a rounding error against the weight read until batches reach the hundreds.

What the recurrent model pays instead

There is no free lunch. The cost shifts to three places. First, the fixed state is a lossy compression: with d_k of 128 the per-head memory is limited to about 128 independent items, so exact recall of an arbitrary token from 500k tokens ago is not guaranteed. Second, training and prefill need specialised chunked kernels, discussed next. Third, the state is a single non-decomposable object, which complicates prefix caching and speculative decoding, discussed under trade-offs.

Training and Prefill: How a Sequential Recurrence Runs in Parallel

A recurrence looks sequential, which would make training on GPUs painfully slow. All the current designs avoid this with chunkwise parallelism, the same trick FlashLinearAttention and Mamba-2’s SSD kernels exploit.

Figure 3: Chunkwise prefill splits the prompt into chunks processed with dense matmuls, passes a carried state between chunks, then decode switches to a one-token-at-a-time update.

The sequence diagram shows the handoff. Inside a chunk of, say, 64 tokens, the layer computes outputs with attention-like matrix multiplications that tensor cores handle well. Between chunks, it carries only the state matrix.

The cost model of chunking

Let L be sequence length, C the chunk size and d the head dimension. Intra-chunk work is about L x C x d multiply-adds, because each token interacts with at most C neighbours; inter-chunk work is about L x d^2, because each token reads the d by d carried state. For L = 8,192, C = 64 and d = 128, there are 128 chunks. Intra-chunk work scales with 8,192 x 64 x 128 = 67M and inter-chunk work with 8,192 x 128 x 128 = 134M, per head. Both are linear in L, against 8,192 x 8,192 x 128 = 8.6B for full softmax attention scores, a ratio of about 43x on this one term. At 128k the softmax term grows 256x faster (quadratic in 16x length) while the chunked cost grows 16x, so the gap widens.

The delta rule complicates the parallel form. Because each update depends on the current state through the (I – b k k^T) projector, a chunk cannot just sum independent outer products. The DeltaNet family uses a WY-style representation to pack a product of Householder-like transforms into a compact form that can be computed with batched triangular solves. That is more arithmetic than a plain gated linear attention chunk, which is why delta-rule layers cost more to train per token than Mamba-2’s simpler scalar-decay update, even though their decode cost is similar. I describe this from the general structure of these algorithms; the exact constants depend on the kernel implementation.

Precision and state handoff

States are normally kept in higher precision than weights. A bf16 state accumulating over millions of decayed updates will drift, because bf16 has about three significant decimal digits, and a decay of 0.999 applied a million times is a long chain of rounding. Production kernels usually hold the state in fp32 even when activations are bf16. This is the main reason the fp32 assumption in the arithmetic above is not pessimistic. If you quantise the state to save memory, test long-context recall explicitly; it is where quality quietly erodes.

Hardware fit

The decode update for a matrix-state layer is a rank-one or low-rank modification of a d by d matrix followed by a matrix-vector product. That is memory-bound per head, with arithmetic intensity of roughly 1 FLOP per byte read, so on a GPU it is limited by HBM bandwidth just like attention decode, only with a constant footprint. Keeping many heads’ states in SRAM across a fused kernel is the main optimisation, and it is one reason kernel quality differs so much between projects at the same nominal complexity.

RWKV vs Mamba vs Gated DeltaNet: Where They Actually Differ

With the shared template in hand, the real differences are narrow enough to list. I compare on the axes that change a deployment decision.

State transition and expressivity

Mamba-2 uses a scalar decay per head per token. Gated DeltaNet adds a delta-rule projector on top of a scalar gate. RWKV-7 uses a per-channel (vector) decay and a per-channel write strength, giving a diagonal-plus-low-rank transition. More expressive transitions can implement operations that a scalar-decay linear model cannot, the best-known being state tracking, such as following the permutation of items through a sequence or evaluating nested parentheses.

The RWKV-7 authors make a formal claim here: that the architecture can perform state tracking and recognise all regular languages while retaining parallelisable training, which they argue exceeds what Transformers can do under standard complexity conjectures (the TC0 limit). This is a theoretical statement about expressive power, not a guarantee about what trained models achieve on real data, and it is the authors’ own claim from the abstract. The Mamba-3 authors target the same gap from another direction: their complex-valued update is aimed at state tracking that, they argue, many recent linear models lack.

Public evidence on quality

The RWKV-7 paper reports models from 0.19B to 2.9B parameters trained on an open multilingual corpus of 3.1 trillion tokens, and claims the 2.9B model sets a new state of the art at 3B on multilingual tasks and matches the English state of the art at that size despite using far fewer training tokens than comparable top models. Weights, dataset component listings and code are released under Apache 2.0. The Mamba-3 abstract reports its gains at 1.5B. Notice what is missing: there is no single public benchmark comparing a current RWKV-7, Mamba-3 and Gated DeltaNet at the same size, with the same data and tokenizer, from an independent party. Treat any ranking you see as provisional, and see the benchmarking advice later.

A comparison table

Dimension	Linear attention (plain)	Gated DeltaNet	Mamba-2 / Mamba-3	RWKV-7
State form	Matrix, additive writes	Matrix, gated delta writes	Matrix per head (head dim x state dim)	Matrix per head, vector gates
Forgetting	None or fixed	Scalar gate, input-dependent	Input-dependent decay (SSM discretisation)	Per-channel vector decay
Targeted overwrite	No	Yes (delta rule)	Via recurrence design, Mamba-3 adds complex update	Yes (generalised delta)
Decode memory	Constant	Constant	Constant	Constant
Training kernels	Mature, simple	Chunked WY style, more arithmetic	Mature SSD kernels	Custom CUDA and Triton
Largest public use	Research	Qwen3-Next, Qwen3.5 linear layers	Nemotron 3 Mamba-2 layers; Mamba-3 research	Open community models
Ecosystem tooling	Limited	Growing (flash-linear-attention style libraries)	Largest of the recurrent family	Smaller, active community
License of reference code	Varies	Varies	Check repo	Apache 2.0 (per paper)

The “largest public use” row relies on the architecture survey we cite in the next section; the tooling and kernel-maturity rows are my qualitative assessment and not benchmarked claims.

Why the labels RWKV vs Mamba are less useful than they sound

Both reduce to the same recurrent skeleton, so the choice is usually decided by non-architectural things: which has a maintained checkpoint at the size you need, whether your inference engine has a fused kernel for it, whether you are fine-tuning, and what license you can live with. If you control training, choose by what your evaluations say. If you are consuming open weights, choose by checkpoint quality and runtime support. Anyone arguing that one name is categorically better is, in 2026, selling something.

Hybrids: What Production Models Actually Ship

The most important fact about recurrent LLMs in 2026 is that the largest deployments are not pure. They interleave a minority of attention layers with a majority of recurrent layers, keeping a small KV cache but recovering exact lookup.

According to Sebastian Raschka’s LLM architecture gallery on hybrid attention, the following configurations are in use (I fetched the page for this run; context lengths are not listed there):

Model	Params total / active	Recurrent layer	Recurrent : attention
Qwen3-Next 80B-A3B	80B / 3B	Gated DeltaNet	3:1 (36 of 48 layers recurrent)
Qwen3.5 397B-A17B	397B / 17B	Gated DeltaNet	3:1 (45 of 60)
Kimi Linear 48B-A3B	48B / 3B	Kimi Delta Attention	about 3:1 (20 KDA, 7 MLA)
Ling 2.5 1T	about 1T / not stated	Lightning Attention	7:1
Nemotron 3 Nano 30B-A3B	30B / 3B	Mamba-2	about 4:1 (23 Mamba-2, 6 attention)
Nemotron 3 Super 120B-A12B	120B / 12B	Mamba-2	5:1 (40 Mamba-2, 8 attention)

Two points stand out. First, the ratio clusters between 3:1 and 7:1, which suggests the community converged on “a few attention layers are enough to rescue recall.” Second, the recurrent layers split roughly evenly between delta-rule linear attention (Qwen, Kimi) and Mamba-2 (NVIDIA), so neither wins the hybrid slot outright. I did not find a hybrid among the large releases that uses RWKV-7 layers; that may reflect my search rather than reality.

Cache arithmetic for a hybrid

If only one layer in four keeps a KV cache, the cache shrinks to a quarter of the transformer baseline at the same attention geometry. Using our earlier 128 KiB per token for 32 layers at full attention, an 8-attention-layer hybrid stores 32 KiB per token. At 128k tokens that is 4 GiB rather than 16 GiB, plus a few tens of MiB of recurrent state. At 1M tokens it is 32 GiB. So a 3:1 hybrid cuts the cache 4x, a 7:1 hybrid 8x, and none of them makes the cache disappear. Several of these models also use multi-head latent attention (MLA) or grouped-query attention in the attention slots, which shrink the per-token cost further; those compounding effects are why direct comparisons by ratio alone mislead.

Why attention layers are still needed

Associative recall tasks, where a model must retrieve a specific value by its key from far back in context, put a hard bound on fixed-size memory. A state with 128 key dimensions per head cannot represent more than a few hundred independent associations per head. Full attention layers give a lookup table that grows with the context, and a small number is enough to handle the retrieval-heavy queries while the recurrent layers handle the bulk of local mixing and compression. That architectural split also mirrors a cost split: the attention layers pay the cache and the quadratic compute; the recurrent layers are cheap.

Figure 4: A selection flow. Exact recall of arbitrary old tokens pushes you to a hybrid; state tracking or streaming pushes you to pure recurrent layers.

The flow encodes a simple rule. If product requirements include retrieval over long documents with exact quotes, keep some attention. If you need infinite streaming, edge deployment or very high concurrency with soft memory, pure recurrent models are viable.

Trade-offs, Gotchas, and What Goes Wrong

Recall is the structural weakness. Fixed-size state means finite capacity. Evaluations on synthetic copying and multi-query associative recall repeatedly show recurrent models lagging attention as the amount of information to retain grows; papers such as “Repeat After Me” (Jelassi et al., 2024) and the Zoology work on associative recall made this argument, though I cite them from memory and did not re-fetch them this run. The practical symptom is a model that summarises well and fails on a verbatim “what was the third item in the table on page 40” query. Needle-in-a-haystack tests pass at short lengths and degrade as the haystack grows past what the state can hold.

Prefix caching breaks. In a transformer, a shared system prompt is a prefix of the KV cache, and you can truncate or share it across requests. A recurrent state is the result of folding every token in, so you can only reuse it by snapshotting the state at exact boundaries. Reusing the prefix of a prefix is impossible without a stored snapshot there. Serving stacks that rely on radix-tree prefix sharing need either checkpoints at chosen intervals (which costs the 64 MiB per checkpoint in our example) or a hybrid design where only the attention part is shared.

Speculative decoding needs rollback. When a draft token is rejected, a transformer discards cache entries. A recurrent model must restore the previous state, which means saving state per speculative step or recomputing. At 64 MiB per sequence, saving eight speculative states costs 512 MiB per sequence, undoing part of the memory advantage unless the engine is smart about it.

Numerical drift. Long chains of decayed updates in low precision accumulate error. Keep the state in fp32, validate against a reference at 100k+ tokens, and be cautious about state quantisation.

Length generalisation is not automatic. A model trained at 4k or 8k context with a particular decay spectrum may behave unpredictably past that, because decays that were fine for 8k tokens fade information that 200k tokens would need. The Gated DeltaNet paper reports good length extrapolation, but you should test at your own target length.

Kernel and ecosystem risk. The best recurrent kernels are specialised CUDA or Triton code; a model that lacks a fused kernel in your inference engine may decode slower than a same-size transformer. Check that your runtime (vLLM, SGLang, llama.cpp or the vendor’s engine) supports the exact architecture variant, including the new Mamba-3 layers, before planning capacity.

Benchmarks mislead. Most public comparisons use different tokenizers, data and training budgets. A one-point gain at 1.5B parameters in an abstract is an invitation to run your own test, not a procurement decision.

Practical Recommendations

Start by writing down the question the model must answer at your longest context. If it is “summarise” or “classify” or “keep conversational memory,” pure recurrent models are credible and the memory savings are large. If it is “quote this clause exactly,” keep attention in the design.

If you serve open-weight models, favour a shipped hybrid (Qwen3-Next class or Nemotron 3 class) over a research pure-recurrent checkpoint for general assistants, because someone has already tuned the kernels and the recall behaviour. If you are building a small on-device or streaming model, RWKV-7 and Mamba-based checkpoints are worth a trial, particularly for the Apache 2.0 license on the RWKV reference code and the constant memory footprint.

If you train your own models, use Gated DeltaNet or a Mamba variant inside a hybrid with 3:1 to 7:1 ratios as a starting point, since that is where the large public releases cluster, and ablate the ratio on your recall evaluation rather than copying it.

A short checklist to run before committing:

Compute state bytes and KV bytes for your real config at your real max context, and plot concurrency per GPU for both.
Confirm your inference engine has fused kernels for the exact layer type.
Build a recall suite from your data: verbatim retrieval, multi-key lookup, and counting at 8k, 32k, 128k.
Test fp32 versus lower-precision state at your longest length.
Decide the prefix-caching story (snapshot interval) before launch.
Re-run on your hardware; do not reuse roofline numbers from this post.

For teams that need multi-hour streaming from sensors, a constant-memory recurrent model is attractive for the same reason a model-predictive controller carries a bounded state; the analogy is developed in our piece on model predictive control in digital twins, where a fixed-size state is also what makes the horizon computation tractable.

Frequently Asked Questions

Is RWKV better than Mamba?

Neither is categorically better. Both are recurrent, constant-memory-per-token layers built on the same fast-weight idea, and quality depends on size, data and training budget more than on the label. RWKV-7 offers an Apache 2.0 release and a richer per-channel transition; Mamba has the larger tooling ecosystem and the new Mamba-3 update. Evaluate checkpoints on your own recall and latency tests, since no independent same-data comparison exists.

Is linear attention as good as softmax attention?

Not on exact recall. Plain linear attention loses quality because a fixed-size state interferes as more items are stored. Gated and delta-rule variants narrow the gap substantially, and hybrids that keep a few softmax layers largely close it in practice. Pure linear models still tend to trail on tasks needing verbatim retrieval from far back in context, which is why shipping large models use 3:1 to 7:1 mixes.

What does constant memory inference mean?

It means the memory the model holds for a conversation does not grow with the number of tokens processed. A recurrent layer folds each token into a fixed-size state matrix and discards the raw keys and values. Compute per generated token is also constant. The trade is lossy compression: old information can be forgotten or blurred, unlike a KV cache that keeps everything verbatim.

Do state space models replace transformers?

Not yet, and probably not as pure replacements. The recent pattern is hybridisation: Nemotron 3 interleaves Mamba-2 layers with a few attention layers, and Qwen3-Next uses Gated DeltaNet in three of every four layers. The recurrent layers cut memory and cost; the attention layers preserve exact lookup. Pure SSMs remain attractive for streaming, edge and very high-concurrency workloads.

How much memory does a recurrent LLM save at 128k tokens?

In the illustrative 8B-class configuration here, a transformer KV cache is 16 GiB at 128k tokens, while a fp32 matrix state of 32 layers by 32 heads by 128 by 128 is 64 MiB, a 256-fold difference. Real savings depend on head counts, state dimensions and GQA ratios, and hybrids keep a smaller cache in their attention layers. Compute your own config rather than quoting these numbers.

Why do hybrid models keep some attention layers?

Because fixed-size memory cannot guarantee exact recall of arbitrary earlier tokens. A few softmax attention layers act as an exact lookup, while the recurrent layers do most of the mixing cheaply. Published hybrids use ratios from about 3:1 up to 7:1 recurrent to attention, and the right value depends on how recall-heavy your workload is.