In January 2025, DeepSeek released R1 (and, weeks earlier, V3), competing with o1 and GPT-4o in reasoning — at a fraction of the training and inference cost. The weights were released openly. The trick is not a dataset trick: it is a rewrite of the Transformer's core, Multi-Head Latent Attention (MLA), attacking the bottleneck that stalls GPUs during text generation: the KV cache. This text follows the video How DeepSeek Rewrote the Transformer [MLA], by Stephen Welch (Welch Labs).
The underlying math is the same as in the posts what is a neural network and GPU vs TPU: MatMul. Here the problem is how many bytes that MatMul needs to fetch from HBM for each new token.
Classic attention and the cost of the KV cache
The 2017 paper, Attention Is All You Need, projects the embedding X into three matrices:
Q = X W_Q o que o token atual busca
K = X W_K o rótulo que cada token oferece
V = X W_V o conteúdo que sai se houver match
Attention(Q, K, V) = softmax( (Q K^T) / sqrt(d_k) + M ) V
Q K^T measures affinity. M is the causal mask (no peeking into the future). Multiplying by V mixes what mattered.
GPT-2 Small: 12 heads × 12 layers = 144 patterns. DeepSeek-V3/R1: 128 heads × 61 layers = 7.808.
Generation is autoregressive: one token at a time, append it to the prompt, run again. Recalculating Q, K, and V for the entire past at every step would take O(N²) computation. The shortcut is the KV cache: K and V for old tokens do not change. Store them in VRAM. The new token only computes the new row of Q (and its K/V) and looks up the history.
tokens 1..t-1 K e V já prontos → KV cache (HBM)
token t calcula Q_t → lê o cache inteiro
Computation drops. The memory bill explodes. In MHA:
KV cache = 2 × n_tokens × d_h × n_heads × L × bytes
V3/R1, por token (FP16):
2 × 128 × 128 × 61 × 2 = 3.997.696 bytes (~4 MB)
A window of 100 thousand tokens (a book): ~400 GB read from HBM to emit a word. The world's most expensive GPU becomes memory-bound — the same wall as in the post on graphics cards.
MQA and GQA: shrink the cache, pay in quality
- MQA: the layer's 128 heads share a K/V pair. Cache ÷ 128 (~31 KB/token). Heads lose specialization.
- GQA (Llama 3): groups — 8 heads, 1 K/V. Cache ÷ 8 (~500 KB/token). A compromise. Still not MHA.
MLA: compress the pair, not the heads
DeepSeek's question: what if K and V are projected into a latent space and the network itself learns to compress/decompress, instead of forcing the heads to share the same key?
It is worth recording where the idea came from, because almost everyone credits R1: MLA did not debut in V3. It appears in the paper for DeepSeek-V2, published on May 7, 2024 — a MoE with 236 B parameters, 21 B activated, and 128 K context. The paper's abstract already gives the number that matters: compared with the dense DeepSeek 67B, V2 cut the KV cache by 93,3%, increased maximum generation throughput to 5,76× and also saved 42,5% in training costs. V3 and R1 inherited the finished component and took it to the scale of 671 B.
entrada h_t
│
▼
c_t^{KV} vetor latente (512)
k_t^R chave RoPE (64)
│
└─ os dois vão para o KV cache → 576 números
(~70 KB por token, 61 camadas, FP16)
na inferência as 128 cabeças reconstroem K e V
a partir do latente — sem guardar 128 pares
The cache size no longer depends on n_heads. Only on the latent representation (and the RoPE part the paper decouples, the 64). 576 × 61 × 2 bytes = 70.272 bytes ≈ 70 KB/token. A reduction of ~57× compared with 4 MB MHA.
Decompressing 128 heads at every token looked like extra computation. MatMul associativity:
Q K^T = Q (c^{KV} W^{UK})^T
= (Q (W^{UK})^T) (c^{KV})^T
W^{UK} and W^{UV} are fixed after training. Absorb them into the Q matrix and the output matrix W_O before serving. At inference, attention operates directly on the latent cache — without the intermediate MatMul at every step. It is the same spirit as the systolic array in the TPU post: do not keep loading what can stay folded into the weights.
Per token, side by side
| Architecture | K/V | Cache/token | Factor |
|---|---|---|---|
| MHA | pair per head | ~4 MB | 1× |
| GQA (Llama 3, groups of 8) | pair per group | ~500 KB | 8× |
| MQA | 1 pair per layer | ~31 KB | 128× (quality drops) |
| MLA (V3/R1) | latent 576 + absorption | ~70 KB | 57× |
Welch reports more than 6× faster generation in R1 compared with a “full” Transformer under the same memory constraints: fewer bytes from HBM per millisecond. Contexts of 64k–128k no longer require a cluster just for multiple users' caches. In DeepSeek's tests, MLA does not merely match MHA — on several tasks it outperforms it, with latent compression acting as regularization.
After MLA: sparse, then hybrid
MLA solved the size of the cache. The other side of the equation remained: even compressed, attention still looks at all previous tokens at each step — quadratic cost in context length. DeepSeek tackled that next.
In November 2025 came the release of DeepSeek-V3.2-Exp, with DSA (DeepSeek Sparse Attention). This is fine-grained sparse attention: an indexer module chooses which positions in the history actually deserve to be read, and only those enter the calculation. The announcement promises major gains in long-context training and inference while keeping output quality virtually identical. MLA does not leave the scene — the indexer and MLA coexist in the same model (to the point of requiring a RoPE layout correction between the two modules, the sort of detail that only appears when the two pieces actually run together).
In April 26, 2026 came the paper for DeepSeek-V4, subtitled Towards Highly Efficient Million-Token Context Intelligence. Two MoEs, both with 1 million tokens of context:
DeepSeek-V4-Pro 1,6 T parâmetros 49 B ativados
DeepSeek-V4-Flash 284 B 13 B ativados
Attention is now hybrid, with two sparse variants working together: CSA (Compressed Sparse Attention) and HCA (Heavily Compressed Attention). Added to these are Manifold-Constrained Hyper-Connections (mHC) and the Muon optimizer during training. The final versions are on Hugging Face under the MIT license — DeepSeek-V4-Flash-0731 (July 2026), DeepSeek-V4-Pro-0813 (August 2026), and an experimental multimodal variant, V4-Flash-Vision-Exp. The pattern repeats: open weights, open paper, and the battle fought in architecture rather than cluster size.
What this changes on real-world Linux
Serving V3/R1 locally still requires datacenter VRAM (671 B total, 37 B active). MLA is what makes the long window fit. A distro with recent llama.cpp/vLLM already supports MLA; GGUF quantization and the workflow of llama-model or Ollama cover the distilled models. Four 16 GB V100s in the refurbished server cannot load the full V3 — they load a 27B/32B. MLA is the reason an open 671 B model does not implode the cache in a real cluster.
Timeline
- Jun/2017 — Attention Is All You Need defines the Transformer and, with it, the KV cache problem.
- 2019–2023 — MQA and then GQA shrink the cache by trading quality for memory; GQA becomes the default in Llama.
- Jul/2023 — DeepSeek is founded in Hangzhou, emerging from the quantitative fund High-Flyer.
- 07/05/2024 — the DeepSeek-V2 paper introduces MLA: 93,3% less KV cache and 5,76× throughput compared with DeepSeek 67B.
- Dec/2024 — DeepSeek-V3, a 671 B MoE with 37 B activated, inherits MLA and releases its weights.
- Jan/2025 — R1 reaches o1-level reasoning with open weights, becoming a market story, not just a laboratory one.
- Nov/2025 — DeepSeek-V3.2-Exp debuts DSA, fine-grained sparse attention with an indexer module.
- 26/04/2026 — paper for DeepSeek-V4: hybrid CSA + HCA attention, 1 million tokens of context, Pro with 1,6 T (49 B activated), and Flash with 284 B (13 B activated).
- Jul–Aug/2026 — final versions V4-Flash-0731 and V4-Pro-0813 are released on Hugging Face under the MIT license.
References
- Welch Labs — How DeepSeek Rewrote the Transformer [MLA]
- DeepSeek-V3 Technical Report (MLA §2.1, latent representation + decoupled RoPE)
- DeepSeek-V2 — the paper that introduced MLA (May 2024)
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (April 2026)
- deepseek-ai on Hugging Face — weights for the V3 and V4 series
- What is a neural network?
- GPU vs TPU
- How graphics cards work
- Deep Learning for beginners
- Local AI on Linux with Ollama
R1 did not come from a bigger cluster. It came from not loading 4 MB of K/V per token when 70 KB of latent representation, with MatMul absorbed into the weights, does the same job. The Transformer still had that room for improvement. DeepSeek simply did the math.