Multi-Head Latent Attention vs GQA vs MQA: KV-Cache Compression Explained

A 72-billion-parameter model with 80 layers and a 128K-token prompt can spend more GPU memory on its attention cache than on any other single runtime structure except the weights themselves. At bf16 precision, a Grouped-Query Attention model of that shape needs roughly 40 GiB of key/value cache for one sequence. A model that stores the same information as a 576-element latent vector per token per layer needs a fraction of that. That gap is what multi-head latent attention (MLA) exists to close.

The question for anyone serving or designing a transformer in 2026 is not whether the KV cache matters, but which compression strategy buys the most batch size and context length for the least quality loss. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) shrink the cache by sharing heads. MLA instead factorises the keys and values through a low-rank latent, which is a different trade entirely.

This post derives the cache size for each scheme from first principles, walks through the DeepSeek MLA algebra including decoupled rotary embeddings and the absorbed-matrix trick, and works the memory arithmetic with real model configs.

What this covers: the KV cache problem, per-token cache formulas for MHA, MQA, GQA and MLA, the low-rank latent mechanism, runnable PyTorch for GQA and a latent projection, serving implications, failure modes, and a decision matrix.

Context and Background

Autoregressive decoding generates one token at a time. To avoid recomputing the keys and values of every earlier token at each step, the model stores them: this is the KV cache. Without it, generating token n would cost a full forward pass over n tokens; with it, each step processes only the new token and reads the stored keys and values. The price is memory that grows linearly with sequence length, with batch size, and with the number of layers.

The cache is a problem because decoding is memory-bandwidth bound, not compute bound. Each decode step must stream the whole cache for every active sequence from high-bandwidth memory (HBM) into the compute units, to do a tiny amount of arithmetic per byte. A larger cache means fewer concurrent sequences fit on the GPU, and slower steps for those that do. We covered the broader serving picture, including paging and quantisation, in our guide to KV cache optimization for LLM inference, and the economics in AI inference cost optimization. This article isolates the architectural lever: changing the attention mechanism itself so there is less to cache.

The original multi-head attention (MHA) from “Attention Is All You Need” gives every query head its own key head and value head. MQA, proposed by Noam Shazeer in “Fast Transformer Decoding: One Write-Head is All You Need” (arXiv 1911.02150), keeps many query heads but shares a single key head and a single value head across all of them. It cuts the cache by the head count but costs quality and can destabilise training.

GQA, from Ainslie et al. (arXiv 2305.13245), sits between the two. Query heads are divided into G groups and each group shares one key head and one value head. The GQA paper also showed how to convert an existing MHA checkpoint: mean-pool the key and value heads inside each group, then continue pre-training for a small fraction of the original steps. Their main setting used 5% of the original pre-training steps. On T5-XXL, GQA with 8 groups scored 47.1 on their average against 47.2 for MHA and 46.6 for MQA, at an inference time of 0.28 s per sample versus 1.51 s for MHA and 0.24 s for MQA.

That result explains why GQA became the default. A public config confirms the pattern in dense open models: Mistral 7B has 32 layers, 32 query heads and 8 key/value heads, and Qwen2.5-72B has 80 layers, 64 query heads and 8 key/value heads (both read from the published config.json files). The cache is cut by 4x and 8x relative to MHA respectively, with little reported quality loss.

MLA is the third approach, introduced in the DeepSeek-V2 technical report (arXiv 2405.04434). Rather than sharing heads, it projects each token’s hidden state down to a small latent vector and reconstructs per-head keys and values from that latent. DeepSeek reported that, for the 236B-parameter, 21B-active DeepSeek-V2 model, this reduced the KV cache by 93.3% relative to their earlier DeepSeek 67B and raised maximum generation throughput to 5.76 times that of the 67B model. We should note carefully what that baseline is: the comparison is against a different, GQA-based model, not against an MHA twin of V2. We return to this below because it is easy to misquote.

The Cache Arithmetic: What Each Scheme Actually Stores

The KV cache per token, per layer, is the number of scalar elements stored for keys and values, multiplied by bytes per element. For MHA this is 2 * n_h * d_h, where n_h is the number of heads and d_h the per-head dimension; the 2 counts keys and values. Multiply by the number of layers l and by the element width to get bytes per token. MQA stores 2 * d_h per layer, GQA stores 2 * n_g * d_h with n_g groups, and MLA stores (d_c + d_r) per layer, where d_c is the latent dimension and d_r the decoupled rotary key dimension.

Figure 1: What each attention variant writes into the KV cache for every token and layer. MHA, GQA and MQA store keys and values directly; multi-head latent attention stores a compressed latent plus a small rotary key.

Figure 1 summarises the four layouts. MHA, GQA and MQA all store literal keys and values, differing only in how many distinct heads exist. MLA stores something that is not a key or a value at all: a compressed latent from which both are recomputed on demand. That conceptual shift, from sharing to compressing, is the whole story.

Direct answer: how much smaller is each cache?

Relative to MHA with n_h heads, MQA shrinks the cache by a factor of n_h, GQA with n_g groups by n_h / n_g, and MLA by 2 * n_h * d_h / (d_c + d_r). For the DeepSeek-V3 geometry (128 heads, d_h = 128, d_c = 512, d_r = 64), MLA is about 57 times smaller than an MHA cache of the same shape, and about 7 times smaller than a GQA model with 8 KV heads of dimension 128.

Worked numbers from real configs

Take the DeepSeek-V3 config from the Hugging Face repository: 61 layers, 128 attention heads, kv_lora_rank 512, qk_rope_head_dim 64, qk_nope_head_dim 128 and v_head_dim 128. The latent plus rotary key is 512 + 64 = 576 elements per layer per token. Across 61 layers that is 35,136 elements, or 70,272 bytes at 2 bytes per element: about 68.6 KiB per token.

Now compute what an MHA cache would be at that same geometry: 2 * 128 * 128 = 32,768 elements per layer, and 1,998,848 elements over 61 layers, about 3.8 MiB per token. The ratio is 1,998,848 / 35,136 = 56.9. This is where “about 57x” comes from, and it is an arithmetic property of the config, not a measured benchmark.

For a comparison with GQA, use Qwen2.5-72B’s shape (80 layers, 8 KV heads, d_h = 128 since 8192 / 64 = 128). GQA stores 2 * 8 * 128 * 80 = 163,840 elements, which is 327,680 bytes or 320 KiB per token at bf16. The same model as MHA would store 64 heads rather than 8, so 2.5 MiB per token; as MQA it would store 40 KiB per token.

Scheme	Shape used	Elements per token	bf16 bytes per token	128K-token sequence
MHA	80 layers, 64 heads, d_h 128	1,310,720	2.5 MiB	320 GiB
GQA, 8 KV heads	80 layers, d_h 128	163,840	320 KiB	40 GiB
MQA	80 layers, d_h 128	20,480	40 KiB	5 GiB
MLA, V3 geometry	61 layers, d_c 512, d_r 64	35,136	68.6 KiB	8.6 GiB

Note that the MLA row uses a different model, so only compare its order of magnitude. MLA lands between MQA and GQA in raw bytes (closer to MQA) while retaining 128 distinct query-side heads and, as we will see, per-head expressivity in keys and values. The DeepSeek-V2 paper describes it as equivalent in cache size to a GQA model with only 2.25 groups, because with d_c = 4 d_h and d_r = d_h / 2 the cache is 4.5 d_h per layer, against 2 d_h per group.

Why the baseline matters

The 93.3% reduction headline compares V2 with DeepSeek 67B, which itself used GQA. Against an MHA-shaped V2, the same arithmetic gives 1 – 1/56.9 = 98.2%. Against a GQA model with 8 KV heads at the same layer count and head dimension, the reduction is 1 – 576/2048 = 71.9% per layer. All three numbers are correct; they answer different questions. When you read a claim about KV cache savings, always ask what the denominator is.

The mechanism: low-rank joint compression

MHA computes keys and values as separate linear maps of the hidden state h_t. MLA replaces them with a two-step factorisation. First, a down-projection W^DKV of shape d_c x d maps h_t to a latent c_t^KV of dimension d_c. Second, two up-projections, W^UK and W^UV, expand the latent to per-head keys and values: k_t^C = W^UK c_t^KV and v_t^C = W^UV c_t^KV.

Only c_t^KV needs caching. The DeepSeek-V2 paper states that during inference MLA only needs to cache the latent, so its KV cache has d_c * l elements, before adding the rotary component. The compression is joint because keys and values share the same latent, in contrast to a design that compresses them separately.

Figure 2: The MLA data path. The 512-dimensional latent and the 64-dimensional shared rotary key are the only tensors written to the cache; per-head keys and values are reconstructed from the latent.

Why does this work at all? A rank argument helps. An MHA layer with 128 heads of dimension 128 produces 16,384-dimensional keys per token, but nothing guarantees those keys span a 16,384-dimensional space. If trained keys and values concentrate in a lower-dimensional subspace, which is plausible given heavy redundancy across heads, a 512-dimensional bottleneck loses little. MLA bets that this is so and trains the projections end to end so the bottleneck is learned rather than imposed post hoc.

Why rotary embeddings force a second cache entry

There is a catch. Rotary position embedding (RoPE) applies a position-dependent rotation to queries and keys. If you rotate the reconstructed key, the rotation matrix sits between W^UK and the query, so W^UK can no longer be merged into the query projection, which destroys the absorption trick described below. The cure in the DeepSeek-V2 paper is decoupled RoPE.

Each head gets an extra rotary query q^R of dimension d_r, computed as RoPE(W^QR c^Q), and the layer gets one shared rotary key k^R = RoPE(W^KR h_t), shared across all heads. The final query and key for head i are concatenations: q_i = [q_i^C ; q_i^R] and k_i = [k_i^C ; k^R]. Content carries no position, the rotary part carries position only, and their dot products add. The shared k^R is the 64 in the 512 + 64 cache entry, and it must be cached, which is why the cache total is (d_c + d_r) per layer rather than d_c.

Inside MLA: Absorption, Query Compression and Quality

The most counter-intuitive property of MLA is that you never need to materialise the full keys and values at inference time. That is the absorbed-matrix trick, and it is what makes MLA a bandwidth win and not merely a storage win.

Consider the content part of one head’s attention score. With a query q^C and a key k^C = W^UK c, the logit is (q^C)^T W^UK c. Since matrix multiplication is associative, you can compute (W^UK)^T q^C first, a vector of dimension d_c, and then dot it with the cached latent c directly. The per-head up-projection is folded into the query side. Likewise the value path: the attention-weighted sum of v = W^UV c equals W^UV applied to the weighted sum of latents, and W^UV can be merged into the output projection W^O.

Figure 3: Absorbed MLA decoding. The key up-projection joins the query side and the value up-projection joins the output projection, so attention runs directly against the 512-dimensional latent cache.

The DeepSeek-V2 paper says exactly this: because W^UK can be absorbed into W^Q and W^UV into W^O, there is no need to compute keys and values out for attention. In practice the merge produces a per-head matrix that is larger than the original factors, so the trick is a trade: more compute per step in exchange for far less memory traffic. During decode, where bandwidth dominates, that trade is favourable. During prefill, where compute dominates, many implementations use the non-absorbed form instead.

Query compression is separate

MLA also compresses queries through a lower-rank bottleneck of dimension d_c’ (1536 in both V2 and V3 per the published hyperparameters). This does not affect the KV cache. The paper presents it as a way to reduce activation memory during training. Beginners often conflate the two compressions, but only the key/value latent shrinks what you store between decode steps.

A minimal GQA implementation

The following PyTorch module implements GQA with a KV cache. The cache holds only n_kv heads; they are expanded to the full query head count just before attention. Setting n_kv = 1 gives MQA, and n_kv = n_heads gives MHA, so this single class covers three of the four variants. I have not run it against a benchmark here; treat it as a reference implementation of the mechanism.

import torch
import torch.nn as nn
import torch.nn.functional as F

class GQA(nn.Module):
    def __init__(self, d_model, n_heads, n_kv, d_h):
        super().__init__()
        assert n_heads % n_kv == 0
        self.H, self.G, self.dh = n_heads, n_kv, d_h
        self.q = nn.Linear(d_model, n_heads * d_h, bias=False)
        self.k = nn.Linear(d_model, n_kv * d_h, bias=False)
        self.v = nn.Linear(d_model, n_kv * d_h, bias=False)
        self.o = nn.Linear(n_heads * d_h, d_model, bias=False)

    def forward(self, x, cache=None):
        B, T, _ = x.shape
        q = self.q(x).view(B, T, self.H, self.dh).transpose(1, 2)
        k = self.k(x).view(B, T, self.G, self.dh).transpose(1, 2)
        v = self.v(x).view(B, T, self.G, self.dh).transpose(1, 2)
        if cache is not None:                      # decode step
            k = torch.cat([cache[0], k], dim=2)
            v = torch.cat([cache[1], v], dim=2)
        new_cache = (k, v)                         # only G heads are stored
        rep = self.H // self.G
        k = k.repeat_interleave(rep, dim=1)        # expand for compute only
        v = v.repeat_interleave(rep, dim=1)
        out = F.scaled_dot_product_attention(q, k, v, is_causal=cache is None)
        out = out.transpose(1, 2).reshape(B, T, self.H * self.dh)
        return self.o(out), new_cache

Note that repeat_interleave materialises a temporary expanded copy. Production kernels avoid it by indexing the shared head directly inside the attention kernel, so the memory traffic stays proportional to G rather than H.

A latent KV projection in PyTorch

Below is a compact MLA layer with the latent cache and decoupled RoPE. It omits query compression for clarity, uses the straightforward (non-absorbed) path for correctness, and then shows the absorbed decode form separately. Again, it is a mechanism sketch, not DeepSeek’s production kernel.

def rope(x, pos, base=10000.0):
    # x: [B, H, T, d] with even d; pos: [T]
    d = x.shape[-1]
    inv = 1.0 / (base ** (torch.arange(0, d, 2, device=x.device) / d))
    ang = pos[:, None] * inv[None, :]
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1).flatten(-2)

class MLA(nn.Module):
    def __init__(self, d_model, n_heads, d_h, d_c, d_r):
        super().__init__()
        self.H, self.dh, self.dc, self.dr = n_heads, d_h, d_c, d_r
        self.dkv = nn.Linear(d_model, d_c, bias=False)      # W_DKV
        self.uk = nn.Linear(d_c, n_heads * d_h, bias=False)  # W_UK
        self.uv = nn.Linear(d_c, n_heads * d_h, bias=False)  # W_UV
        self.q = nn.Linear(d_model, n_heads * d_h, bias=False)
        self.qr = nn.Linear(d_model, n_heads * d_r, bias=False)
        self.kr = nn.Linear(d_model, d_r, bias=False)        # shared rope key
        self.o = nn.Linear(n_heads * d_h, d_model, bias=False)

    def forward(self, x, cache=None, start=0):
        B, T, _ = x.shape
        pos = torch.arange(start, start + T, device=x.device).float()
        c = self.dkv(x)                                       # [B, T, d_c]
        kr = rope(self.kr(x)[:, None], pos)                   # [B, 1, T, d_r]
        if cache is not None:
            c = torch.cat([cache[0], c], dim=1)
            kr = torch.cat([cache[1], kr], dim=2)
        new_cache = (c, kr)            # d_c + d_r numbers per token, not 2*H*d_h
        S = c.shape[1]
        k = self.uk(c).view(B, S, self.H, self.dh).transpose(1, 2)
        v = self.uv(c).view(B, S, self.H, self.dh).transpose(1, 2)
        q = self.q(x).view(B, T, self.H, self.dh).transpose(1, 2)
        qr = rope(self.qr(x).view(B, T, self.H, self.dr).transpose(1, 2), pos)
        qf = torch.cat([q, qr], dim=-1)
        kf = torch.cat([k, kr.expand(-1, self.H, -1, -1)], dim=-1)
        out = F.scaled_dot_product_attention(qf, kf, v, is_causal=cache is None)
        out = out.transpose(1, 2).reshape(B, T, self.H * self.dh)
        return self.o(out), new_cache

def mla_absorbed_decode(m, x, cache, start):
    # One-token decode that never builds per-head K or V.
    B = x.shape[0]
    pos = torch.arange(start, start + 1, device=x.device).float()
    c_new = m.dkv(x)
    kr_new = rope(m.kr(x)[:, None], pos)
    c = torch.cat([cache[0], c_new], dim=1)                   # [B, S, d_c]
    kr = torch.cat([cache[1], kr_new], dim=2)                 # [B, 1, S, d_r]
    WUK = m.uk.weight.view(m.H, m.dh, m.dc)                   # [H, d_h, d_c]
    WUV = m.uv.weight.view(m.H, m.dh, m.dc)
    q = m.q(x).view(B, 1, m.H, m.dh).transpose(1, 2)          # [B, H, 1, d_h]
    qr = rope(m.qr(x).view(B, 1, m.H, m.dr).transpose(1, 2), pos)
    q_lat = torch.einsum('bhtd,hdc->bhtc', q, WUK)            # absorb W_UK
    s = q_lat @ c[:, None].transpose(-1, -2)                  # [B, H, 1, S]
    s = s + qr @ kr.transpose(-1, -2)
    p = (s / (m.dh + m.dr) ** 0.5).softmax(-1)
    ctx = p @ c[:, None]                                      # [B, H, 1, d_c]
    out = torch.einsum('bhtc,hdc->bhtd', ctx, WUV)            # absorb W_UV
    out = out.transpose(1, 2).reshape(B, 1, m.H * m.dh)
    return m.o(out), (c, kr)

The two paths are algebraically identical, which gives you a handy unit test: run both on the same weights and inputs and assert the outputs match to numerical tolerance. The cache tuple returned by both holds a [B, S, d_c] latent and a [B, 1, S, d_r] rotary key, which is the entire MLA cache. For a real deployment the absorbed matrices are precomputed once at load time rather than via einsum each step.

Serving Implications: Batch Size, Context Length and Bandwidth

Cache size is not an aesthetic metric. It sets how many sequences a GPU can hold concurrently, and concurrency is what amortises the cost of streaming model weights. During decoding, each step reads all the weights once regardless of batch size, so a bigger batch gives more tokens per weight read. The KV cache limits how big that batch can get.

Worked example: how many sequences fit

Suppose a node has 100 GiB of HBM left for KV cache after weights and activations. This is an illustrative budget, not a measurement of any specific deployment. Use 32,768-token sequences at bf16.

With the Qwen2.5-72B-shaped GQA cache of 320 KiB per token, one sequence needs 320 KiB * 32,768 = 10 GiB. The budget holds 10 concurrent sequences. With the DeepSeek-V3 MLA cache of 70,272 bytes per token, one sequence needs 70,272 * 32,768 bytes, about 2.14 GiB, and the budget holds 46 sequences. That is a 4.6x gain in concurrency from the cache alone, and it widens in favour of MLA as contexts grow because both costs scale linearly.

Run the same calculation for an MHA-shaped version of the Qwen model at 2.5 MiB per token: 80 GiB per sequence, so the budget does not even hold two. This is why every large dense model since the mid-2020s has moved away from vanilla MHA.

Cache design	Per token	Per 32K sequence	Sequences in 100 GiB
MHA, 80 layers, 64 heads, d_h 128	2.5 MiB	80 GiB	1
GQA, 8 KV heads	320 KiB	10 GiB	10
MQA	40 KiB	1.25 GiB	80
MLA, V3 geometry, 61 layers	68.6 KiB	2.14 GiB	46

The table mixes model shapes deliberately, so that you can see magnitudes; do not read the MQA row as a recommendation. MQA wins on bytes but, as covered below, gives up capacity that matters for quality.

Bandwidth, not just capacity

Memory capacity determines how many sequences fit. Memory bandwidth determines how fast each decode step runs. For each layer and sequence, the attention kernel reads the cached keys and values once per step. Total bytes read per step scale with batch times context times cache-bytes-per-token.

With a smaller cache, fewer bytes cross the HBM bus for the same context. This is why the DeepSeek-V2 paper pairs the cache reduction with a throughput claim rather than only a capacity claim. Their reported 5.76x maximum generation throughput relative to DeepSeek 67B reflects both larger batches and cheaper attention reads, though it also reflects that V2 activates only 21B parameters per token via mixture-of-experts. The throughput gain is therefore not attributable to MLA alone, and the paper does not claim that it is.

Arithmetic intensity changes under absorption

Standard attention at decode has very low arithmetic intensity: every cached byte is used for roughly one multiply-add per query head. GQA raises intensity for the shared KV head, because the same loaded key serves rep = H / G query heads. MQA raises it the most.

MLA’s absorbed form has the same property in a stronger way. Every query head attends against the same 512-dimensional latent, so one loaded latent vector is reused by all 128 heads. That means the attention step does far more floating-point work per byte loaded than MHA does, moving the kernel toward the compute roofline. Specialised kernels exist to exploit this; DeepSeek has published an open-source kernel called FlashMLA, though I have not benchmarked it here and the throughput depends heavily on the GPU generation and batch shape.

Prefill versus decode

Prefill processes the whole prompt in parallel and is compute-bound. Absorbing the up-projections there would multiply the effective head dimension, since each head’s query would be mapped into the 512-dimensional latent space, inflating the attention FLOPs. Many implementations therefore run prefill in the explicit form (reconstruct per-head keys and values) and switch to the absorbed form for decode. The cache format is the same either way, so the switch is a kernel choice, not a model change.

Context length scaling

Cache bytes grow linearly in sequence length, so a compression ratio converts directly into context. A budget that fits 128K tokens with GQA fits roughly 4.6x more with the V3-style MLA cache at the same layer count, ignoring layer count differences. The V3 config advertises a max_position_embeddings of 163,840 and DeepSeek-V2 reports a 128K context. Note that attention compute per decode step is still linear in context regardless of cache layout, so MLA gives you memory headroom, not free long-context speed.

Interaction with quantisation and paging

KV cache compression techniques compose. You can store the MLA latent in FP8 rather than bf16, which would halve its bytes again, and you can page it with the same block allocator used for GQA caches. What changes is the page content: a page holds block_size * 576 elements per layer instead of block_size * 2 * H_kv * d_h.

Prefix caching works unchanged because the latent for a given token depends only on the prefix. Speculative decoding and multi-token prediction heads share the cache in the usual way. For the paging mechanics, see our KV cache optimization guide. For a concrete current model that uses a DeepSeek-style attention stack, see our analysis of DeepSeek V4.1 Flash: architecture, KV cache and benchmarks.

Which models use which

The following is what I could verify directly from published configs and papers, plus what is widely reported. Check each model’s own card before relying on it.

• MHA: the original transformer and many older models, including the first GPT-style open models.
• MQA: Shazeer’s original proposal; PaLM is widely reported to use it, and some Falcon variants as well.
• GQA: verified from configs here for Mistral 7B (8 KV heads, 32 query heads) and Qwen2.5-72B (8 KV heads, 64 query heads). The Llama 2 70B and Llama 3 families are widely reported to use GQA with 8 KV heads, but the Llama 3 config is gated and I did not read it this run.
• MLA: DeepSeek-V2 (paper) and DeepSeek-V3 (config shows kv_lora_rank 512, qk_rope_head_dim 64). Other recent large models are reported to adopt V3-style attention; verify the specific config for any model you plan to deploy.

Quality: Does Compression Cost Accuracy?

The cache formulas tell you nothing about quality, so the answer has to come from experiments. The honest summary is that MQA costs quality, GQA costs little at moderate group counts, and MLA is reported by its authors to match or exceed MHA at far smaller cache.

What the original papers show

The GQA paper’s T5-XXL numbers above show the shape of the trade: MQA 46.6, GQA-8 47.1, MHA 47.2 on an average over summarisation, translation and QA tasks, with speed close to MQA for GQA. Differences of half a point are small but they were consistent enough to motivate the intermediate design. The paper also reports that mean-pooling the heads works better than selecting one head or random initialisation, and that returns diminish after about 10% uptraining.

For MLA, the DeepSeek-V2 paper includes ablations on small mixture-of-experts models comparing MHA, GQA, MQA and MLA at matched settings. The authors report that MLA is stronger than GQA and MQA at much smaller cache and also stronger than MHA on their evaluation sets. I cannot independently reproduce those ablations, and an author-run comparison on a specific architecture family should be treated as encouraging evidence rather than a universal law.

Why MLA might not lose quality

There are two plausible reasons, both mechanism-level rather than proven. First, MLA keeps 128 distinct heads of keys and values on the compute side: each head still has its own up-projection rows, so head-level diversity survives even though the stored bits are shared. GQA, by contrast, forces whole groups to have identical keys and values.

Second, the bottleneck is learned jointly with the rest of the network. A post-hoc low-rank approximation (for example SVD of existing key and value matrices) has no such chance to adapt. This is also why converting a pre-trained GQA or MHA model to MLA is non-trivial: the shared latent has to be discovered, and the position-encoding structure has to be rearranged to separate content from rotary parts. Research papers on converting MHA checkpoints to MLA (for example TransMLA and MHA2MLA, both reported in early 2025) show it is feasible with additional training, though I did not verify their numbers for this post.

The caveat about d_c

The latent dimension is a hyperparameter, and the quality-versus-cache curve depends on it. DeepSeek chose d_c = 4 d_h, i.e. 512 for d_h = 128. A smaller d_c cuts the cache further but eventually starves the model, in the same way that going from GQA-8 to MQA does. No public paper that I verified gives a clean sweep of this curve across scales, so choose d_c by your own ablation.

Trade-offs, Gotchas, and What Goes Wrong

MLA is not a free lunch. Its costs show up in implementation complexity, compute, and ecosystem support rather than in the cache budget.

Figure 4: A decision flow for choosing an attention variant. The key branches are whether you control pre-training and whether your serving stack has MLA kernels.

Positional encoding is entangled. Because RoPE cannot be applied to the reconstructed keys without breaking absorption, MLA needs the decoupled rotary key. Extensions of context length by scaling rotary frequencies must be applied correctly to the rotary sub-dimension only. Bugs here do not crash; they silently degrade long-context retrieval.

Compute increases. MLA adds up-projections and, in the absorbed form, larger per-head matrices. A prefill pass costs more FLOPs than the GQA equivalent. At low concurrency or short contexts, where the cache is not the bottleneck, MLA can be a net loss. The benefit appears when memory bound.

Kernel and framework support lags. GQA is in every serving framework and every attention kernel library. MLA requires specialised kernels, and the choice between absorbed and explicit forms matters for performance. If your framework falls back to naive implementations, you may materialise full keys and values and lose the benefit while keeping the complexity. Verify that your stack caches the latent and not the expanded keys; if it caches the expanded form, the savings vanish.

Tensor parallelism is awkward. GQA shards naturally: assign KV heads to ranks. With a single shared latent, there is nothing to shard across heads, so tensor-parallel deployments typically replicate the latent cache on each rank or use data parallelism for attention. That replication erodes part of the saving at high tensor-parallel degree. The details depend on the framework and I have not measured them, so test your configuration before assuming the headline ratio.

Fine-tuning and LoRA. The factorised projections add more places for adapters to attach and more room for mistakes. Adapter ranks must be set relative to the latent size, and merging adapters into absorbed matrices needs care.

MQA’s own failure modes. Beyond the quality gap, the GQA paper notes that MQA training can be unstable, and that a single shared KV head becomes a capacity bottleneck as models grow. Also, with a single head, tensor parallelism must replicate the KV cache across ranks.

GQA’s failure mode is the group count. Going from 8 groups to 4 or 2 halves the cache each time but eventually behaves like MQA. The optimal G tends to track how much retrieval-heavy, long-context work you expect.

Cache size is not the only bottleneck. Weights, activations, and the MoE expert-parallel communication in DeepSeek-style models can dominate at some operating points. Treat KV-cache compression as one lever among several.

Practical Recommendations

If you are training a model from scratch and expect long contexts at high concurrency, MLA is the strongest option verified in the open literature, provided you can commit to the engineering. You will need decoupled RoPE, a kernel path for absorbed decoding, and a deployment stack that understands the latent cache. If your team lacks that capacity, GQA with 8 KV heads remains the safe default and the one with the most mature tooling.

If you are choosing between existing open models, check what the config actually says. Look for num_key_value_heads to identify GQA or MQA, and for kv_lora_rank to identify MLA. Then compute bytes per token with the formulas above rather than trusting marketing. For hardware planning on local machines, the same arithmetic applies; our note on local LLM inference on AMD Ryzen AI Max workstations shows why unified-memory boxes are cache-limited long before they are compute-limited.

If you already serve a GQA model and the cache is your bottleneck, try cheaper levers first: FP8 or INT8 KV quantisation, paged allocation, prefix caching and sliding-window layers. Conversion to MLA is a research-grade option.

A short checklist:

• Compute bytes per token for your actual config: layers * cache elements * bytes per element.
• Compute concurrency: free HBM divided by bytes per sequence at your target context.
• Measure whether decode is bandwidth bound before investing in cache compression.
• Confirm that your serving stack stores the latent, not expanded keys.
• Check tensor-parallel replication of the latent cache at your parallelism degree.
• Test long-context retrieval after any rotary scaling on the decoupled key.
• Prefer GQA with 8 KV heads when kernel support or team time is the constraint.

Frequently Asked Questions

What is multi-head latent attention?

Multi-head latent attention is an attention variant introduced in the DeepSeek-V2 technical report. Instead of caching full per-head keys and values, it caches one low-dimensional latent vector per token per layer, plus a small shared rotary key. Per-head keys and values are reconstructed from the latent with learned up-projections, or absorbed into the query and output matrices so they never need to be built. The result is a much smaller KV cache while keeping many distinct attention heads.

How is MLA different from GQA?

GQA reduces the cache by making groups of query heads share identical key and value heads, so the number of distinct stored heads drops. MLA keeps per-head key and value capacity but stores a compressed latent from which they are recomputed. In the DeepSeek-V2 paper, MLA’s cache is equivalent to GQA with about 2.25 groups, yet the authors report quality at least equal to MHA in their ablations. GQA is simpler and universally supported; MLA is more complex and cache-efficient.

How much smaller is the KV cache with MLA?

It depends on the baseline. DeepSeek reported a 93.3% reduction for V2 against DeepSeek 67B, which used GQA. With the V3 geometry of 128 heads, head dimension 128, latent 512 and rotary key 64, MLA stores 576 elements per layer against 32,768 for an MHA twin, roughly 57 times fewer, or about 72% fewer than a GQA design with 8 KV heads of dimension 128. These are arithmetic results from published configs.

Why does MLA need decoupled RoPE?

Rotary embeddings rotate keys by a position-dependent matrix. If applied after the up-projection, that matrix sits between the key up-projection and the query, which prevents merging the up-projection into the query side, so keys would have to be materialised for every cached token. DeepSeek separates a content part, without position, from a small rotary part with a shared key. Position lives only in the rotary part, so the content path remains absorbable.

Can I convert an existing GQA or MHA model to MLA?

Research papers reported in early 2025, including TransMLA and MHA2MLA, claim that conversion is possible with extra training, but I did not verify their results for this article. The conversion is not a pure post-hoc weight rewrite: the latent must be learned or factorised from existing keys and values, and rotary components need restructuring. For most teams, quantising the KV cache of the existing GQA model is the lower-risk first step.

Is MQA still used?

MQA still appears in some models and is the most cache-efficient of the sharing approaches, storing a single key and value head per layer. Its drawbacks are lower quality at scale, as the GQA paper’s T5-XXL numbers indicate, and some training instability. Most recent open dense models instead use GQA with around 8 KV heads, and the newest large DeepSeek models use MLA. For new designs MQA is rarely the best choice unless cache size dominates everything else.

Further Reading

• DeepSeek V4.1 Flash explained: architecture, KV cache and benchmarks
• KV cache optimization for LLM inference
• AI inference cost optimization
• Local LLM inference on an AMD Ryzen AI Max Pro workstation
• DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need

By Riju — about