{"notes":[{"id":"mamba/kimi-linear","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Kimi Linear","subtitle":"Forgetting, Channel by Channel","author":"Z. Yuan","paperDate":"2025-10-30","publishedOn":"2026-08-06","world":"external/kimi-linear","paperTitle":"Kimi Linear: An Expressive, Efficient Attention Architecture","paperUrl":"https://arxiv.org/abs/2510.26692","bibliography":{"title":"Kimi Linear: An Expressive, Efficient Attention Architecture","authors":["Kimi Team","Yu Zhang","Zongyu Lin","Xingcheng Yao","Jiaxi Hu","Fanqing Meng","Chengyin Liu","Xin Men","Songlin Yang","Zhiyuan Li","Wentao Li","Enzhe Lu","Weizhou Liu","Yanru Chen","Weixin Xu","Longhui Yu","Yejie Wang","Yu Fan","Longguang Zhong","Enming Yuan","Dehao Zhang","Yizhi Zhang","T. Y. Liu","Haiming Wang","Shengjun Fang","Weiran He","Shaowei Liu","Yiwei Li","Jianlin Su","Jiezhong Qiu","Bo Pang","Junjie Yan","Zhejun Jiang","Weixiao Huang","Bohong Yin","Jiacheng You","Chu Wei","Zhengtao Wang","Chao Hong","Yutian Chen","Guanduo Chen","Yucheng Wang","Huabin Zheng","Feng Wang","Yibo Liu","Mengnan Dong","Zheng Zhang","Siyuan Pan","Wenhao Wu","Yuhao Wu","Longyu Guan","Jiawen Tao","Guohong Fu","Xinran Xu","Yuzhi Wang","Guokun Lai","Yuxin Wu","Xinyu Zhou","Zhilin Yang","Yulun Du"],"url":"https://arxiv.org/abs/2510.26692","pdfUrl":"https://arxiv.org/pdf/2510.26692v2","codeUrl":"https://github.com/MoonshotAI/Kimi-Linear","preprint":{"id":"2510.26692","submittedOn":"2025-10-30","url":"https://arxiv.org/abs/2510.26692"},"publication":null,"citation":"@misc{mambakimilinear2025,\n  title = {{Kimi Linear: An Expressive, Efficient Attention Architecture}},\n  author = {Kimi Team and Yu Zhang and Zongyu Lin and Xingcheng Yao and Jiaxi Hu and Fanqing Meng and Chengyin Liu and Xin Men and Songlin Yang and Zhiyuan Li and Wentao Li and Enzhe Lu and Weizhou Liu and Yanru Chen and Weixin Xu and Longhui Yu and Yejie Wang and Yu Fan and Longguang Zhong and Enming Yuan and Dehao Zhang and Yizhi Zhang and T. Y. Liu and Haiming Wang and Shengjun Fang and Weiran He and Shaowei Liu and Yiwei Li and Jianlin Su and Jiezhong Qiu and Bo Pang and Junjie Yan and Zhejun Jiang and Weixiao Huang and Bohong Yin and Jiacheng You and Chu Wei and Zhengtao Wang and Chao Hong and Yutian Chen and Guanduo Chen and Yucheng Wang and Huabin Zheng and Feng Wang and Yibo Liu and Mengnan Dong and Zheng Zhang and Siyuan Pan and Wenhao Wu and Yuhao Wu and Longyu Guan and Jiawen Tao and Guohong Fu and Xinran Xu and Yuzhi Wang and Guokun Lai and Yuxin Wu and Xinyu Zhou and Zhilin Yang and Yulun Du},\n  year = {2025},\n  eprint = {2510.26692},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2510.26692}\n}\n","citationPath":"/content/papers/citations/mamba/kimi-linear.38a5b561e63e.bib"},"thumbnail":{"src":"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png","width":1440,"height":960},"locale":"en","path":"/content/papers/en/mamba/kimi-linear.9f3e6ef90c77.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"kda-splitting-the-decay-across-channels","text":"KDA: Splitting the Decay Across Channels","depth":3},{"id":"other-components","text":"Other Components","depth":3},{"id":"a-31-mix-and-mla-without-positional-encoding","text":"A 3:1 Mix, and MLA Without Positional Encoding","depth":3},{"id":"working-out-the-cache-by-hand","text":"Working Out the Cache by Hand","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"choosing-the-ratio","text":"Choosing the Ratio","depth":3},{"id":"same-recipe-14t-tokens","text":"Same Recipe, 1.4T Tokens","depth":3},{"id":"speed","text":"Speed","depth":3},{"id":"how-to-read-these-results","text":"How to Read These Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/gated-deltanet","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]},{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"Erase, Then Write","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"en","path":"/content/papers/en/mamba/gated-deltanet.76d17f2230a0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"linear-attention-is-a-phone-book-where-entries-collide","text":"Linear Attention Is a Phone Book Where Entries Collide","depth":3},{"id":"mamba-2-everything-gets-discounted-together","text":"Mamba-2: Everything Gets Discounted Together","depth":3},{"id":"deltanet-change-only-that-entry","text":"DeltaNet: Change Only That Entry","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-update-rules-worked-through","text":"Three Update Rules, Worked Through","depth":3},{"id":"combining-the-two","text":"Combining the Two","depth":3},{"id":"making-it-trainable-in-parallel","text":"Making It Trainable in Parallel","depth":3},{"id":"architecture-and-hybrid-variants","text":"Architecture and Hybrid Variants","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"needle-in-a-haystack-how-the-three-rules-divide-the-work","text":"Needle in a Haystack: How the Three Rules Divide the Work","depth":3},{"id":"language-modeling-and-real-world-retrieval","text":"Language Modeling and Real-World Retrieval","depth":3},{"id":"speed","text":"Speed","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"Attention's Distant Cousin","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"en","path":"/content/papers/en/mamba/mamba-2.84790bbee84c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"writing-an-ssm-as-a-matrix","text":"Writing an SSM as a Matrix","depth":3},{"id":"reducing-a-to-a-single-number","text":"Reducing A to a Single Number","depth":3},{"id":"working-through-four-tokens","text":"Working Through Four Tokens","depth":3},{"id":"half-one-way-half-the-other","text":"Half One Way, Half the Other","depth":3},{"id":"the-mamba-2-block","text":"The Mamba-2 Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"a-bigger-state-remembers-more","text":"A Bigger State Remembers More","depth":3},{"id":"language-modeling","text":"Language Modeling","depth":3},{"id":"the-original-authors-recommend-hybrids-too","text":"The Original Authors Recommend Hybrids Too","depth":3},{"id":"not-an-across-the-board-upgrade","text":"Not an Across-the-Board Upgrade","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]}],"text":"On October 29, 2025, MiniMax's head of pretraining published a post explaining why the company's new model, M2, went back to full attention: models that mixed in linear attention looked just as good on small-scale evaluations, but once they were scaled up, multi-hop reasoning exposed their weaknesses. [MiniMax, 2025-10-29](https://www.minimax.io/news/why-did-m2-end-up-as-a-full-attention-model)\n\nThe next day, Moonshot AI uploaded this technical report. The first sentence of its abstract reads: “We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, **outperforms** full attention under fair comparisons across various scenarios—including short-context, long-context, and reinforcement learning (RL) scaling regimes.”\n\nIn the same week, two Chinese teams made opposite choices, each under the conditions of its own models and products.\n\nMoonshot AI makes the Kimi chat assistant. Kimi K2, which it released in July of that year, is a 1T-parameter MoE model whose attention is MLA, proposed by DeepSeek. This report instead trains a model with 48B total parameters and 3B active per token to test a new attention design. The author list includes Songlin Yang, the first author of Gated DeltaNet.\n\n## Defining the Problem\n\nThe authors start from agents and reinforcement learning. At inference time, models have to handle very long trajectories: logs of tool calls, repeated rounds of thinking and revision, and ever-longer outputs. Work like this is dominated by **decoding**, where the size of the KV cache and the time to generate each token directly determine the cost.\n\nSwitch to linear layers and the KV cache problem goes away, but new problems arrive: as tool logs stretch on and on, the fixed-size state has to compress more and more history, and by the time some detail needs to be recovered, it has often blurred. Mamba-2's gating and DeltaNet's delta rule both aim to improve this compression, but linear layers alone still aren't enough, so the industry began keeping a few full-attention layers among them.\n\nThis paper takes two angles: first, making the linear layers' memory update finer-grained; second, checking, at a large enough scale and with thorough enough evaluations, whether such a hybrid can **match or even exceed** full attention in quality while genuinely saving inference cost.\n\n## Solving the Problem\n\n### KDA: Splitting the Decay Across Channels\n\nThe core module is called **KDA** (Kimi Delta Attention), and it takes Gated DeltaNet one step further. First, the Gated DeltaNet update:\n\n$$\nS_t=\\alpha_t\\big(I-\\beta_t k_t k_t^\\top\\big)S_{t-1}+\\beta_t k_t v_t^\\top\n$$\n\n(Here we follow the notation of the Kimi Linear report: the state is $d_k\\times d_v$, so the matrices multiply from the left.)\n\n$\\alpha_t$ is a scalar, so the memory of the whole head gets discounted together. KDA replaces it with a diagonal matrix:\n\n$$\nS_t=\\big(I-\\beta_t k_t k_t^\\top\\big)\\,\\text{Diag}(\\alpha_t)\\,S_{t-1}+\\beta_t k_t v_t^\\top,\\qquad \\alpha_t\\in[0,1]^{d_k}\n$$\n\nEach row of the state corresponds to one direction in key space, and now each direction has its own rate of forgetting. Some channels can remember for a long time, while others can be cleared quickly.\n\nLet's get a feel for it with a tiny two-channel state:\n\n| State before update | Scalar decay $\\alpha=0.5$ | Per-channel decay $\\alpha=(1.0,\\ 0.1)$ |\n| --- | --- | --- |\n| Channel 1: 8 | 4 | 8 |\n| Channel 2: 6 | 3 | 0.6 |\n\n*Table: Illustration for this note. Only the forget-gate step $\\text{Diag}(\\alpha_t)S_{t-1}$ is computed, before the delta-rule correction and the write.*\n\nSuppose channel 1 holds information that will still be needed much later, and channel 2 holds content that has already gone stale. Scalar decay can only take 50% off both; the per-channel version can keep channel 1 and almost empty channel 2.\n\nThis kind of per-channel gate already appeared in GLA (Gated Linear Attention). KDA's work is to join it to the delta rule while keeping it efficient to train in parallel.\n\nThe authors write the transition matrix in a special “diagonal-plus-low-rank” (DPLR) form and design a dedicated chunkwise algorithm for it. Compared with the general DPLR formulation, it needs considerably less computation and stays closer to the original delta rule.\n\n### Other Components\n\nEach head's inputs are produced as follows:\n\n- $q$, $k$: linear projection, short convolution, Swish, then L2 normalization to keep the eigenvalues stable.\n- $v$: linear projection, short convolution, Swish.\n- $\\alpha$: first a low-rank projection (with rank equal to the head dimension), then a decay function similar to those of GDN and Mamba.\n- $\\beta$: linear projection plus sigmoid.\n\nThe output first goes through per-head RMSNorm, is then multiplied by a data-dependent sigmoid gate, and is finally projected back. The authors say this output gate also helps mitigate attention sinks.\n\n### A 3:1 Mix, and MLA Without Positional Encoding\n\nThe model follows the skeleton of Moonlight (Moonshot AI's earlier MoE model): each block is a token-mixing layer plus a feed-forward layer, and all of the feed-forward layers are MoE except the first, which is an ordinary dense one. Among the token-mixing layers, every 3 KDA layers are followed by 1 MLA layer. The authors chose to alternate whole layers rather than mix different heads within one layer, because it keeps the infrastructure simpler and training more stable.\n\nOne design choice stands out: the MLA layers use **no positional encoding at all** (NoPE). Positional information and the bias toward “nearer matters more” are left entirely to the KDA layers. The authors regard KDA's decay itself as a kind of data-dependent positional encoding.\n\n### Working Out the Cache by Hand\n\nAccording to the configuration file of Kimi-Linear-48B-A3B on Hugging Face, the model has 27 layers: 20 KDA and 7 MLA (the last layer is also MLA). For each token, every MLA layer caches a 512-dimensional compressed vector plus a 64-dimensional positional part:\n\n```python\nGiB, MiB = 2**30, 2**20\nmla_per_layer = 512 + 64                    # MLA cache per layer per token: compressed KV (512) + positional part (64)\nfull = 27 * mla_per_layer * 2               # all 27 layers are MLA, bf16\nhybrid = 7 * mla_per_layer * 2              # Kimi Linear: only 7 MLA layers\nprint(f\"All-MLA: {full / 1024:.1f} KiB per token; Kimi Linear: {hybrid / 1024:.1f} KiB ({1 - hybrid / full:.0%} saved)\")\nn = 2**20\nprint(f\"1M tokens: all-MLA {full * n / GiB:.1f} GiB; Kimi Linear {hybrid * n / GiB:.1f} GiB\")\nkda_state = 20 * 32 * 128 * 128             # 20 KDA layers, 32 heads each, a 128×128 matrix state per head\nprint(f\"State of 20 KDA layers: {kda_state:,} values, bf16 {kda_state * 2 / MiB:.0f} MiB (independent of length)\")\n```\n\n```text\nAll-MLA: 30.4 KiB per token; Kimi Linear: 7.9 KiB (74% saved)\n1M tokens: all-MLA 30.4 GiB; Kimi Linear 7.9 GiB\nState of 20 KDA layers: 10,485,760 values, bf16 20 MiB (independent of length)\n```\n\nThe share saved depends only on how many layers still keep a KV cache: 7/27 comes close to a quarter, which is where the paper's “up to 75%” comes from. This assumes that every MLA layer in both models caches the same 576 values, and it compares only the part that grows with length. Whether the 64-dimensional positional part in the config is actually cached under NoPE, and what precision the KDA state is really stored in, both depend on the inference engine's implementation; if neither side stores those 64 dimensions, the share saved stays the same.\n\nSince MLA already compresses KV to a very small size, the absolute numbers here are much smaller than for earlier generations of models. KDA's 20 MiB of state is independent of context length.\n\n## Discussion\n\n### Choosing the Ratio\n\nThe authors first swept several KDA-to-MLA ratios using the smallest model tier from their scaling-law experiments (16 layers, 16 heads), with the same training compute for every version. The validation set was deliberately chosen to be high-quality text far from the pretraining distribution, to see how the models generalize under distribution shift.\n\n![Dot plot: the x-axis is the ratio of KDA layers to MLA layers, from left to right 0:1 (pure MLA), 1:1, 3:1, 7:1, and 15:1; the y-axis is validation perplexity, from 5.60 to 5.85. The five points are 5.77, 5.66, 5.65, 5.70, and 5.82: 3:1 is lowest, and the two ends are highest.](/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png)\n\n*Figure: Redrawn for this note from the validation perplexities in Table 1 of the paper. The paper is licensed under CC BY-NC-ND 4.0, so the original figure is not reproduced here.*\n\nThe full numbers, including two additional component ablations:\n\n| Setting | Training perplexity | Validation perplexity |\n| --- | ---: | ---: |\n| **3:1** | **9.23** | **5.65** |\n| 0:1 (pure MLA) | 9.45 | 5.77 |\n| 1:1 | 9.29 | 5.66 |\n| 7:1 | 9.23 | 5.70 |\n| 15:1 | 9.34 | 5.82 |\n| 3:1, without output gate | 9.25 | 5.67 |\n| 3:1, without convolution | 9.29 | 5.70 |\n\n*Table: Excerpted from Table 1 of the paper.*\n\n7:1 has the same training perplexity as 3:1, but its validation perplexity is noticeably worse; 1:1 is almost as good but costs more at inference. Pure MLA also clearly lags behind. In a 16-layer model, 15:1 leaves just one MLA layer, and its validation perplexity is the worst of the five ratios.\n\nCompared with Jamba's 1:7 and the NVIDIA study's roughly 8% attention, the optimum here leans toward keeping more full attention. With different linear layers and different ways of evaluating, the answers differ.\n\n### Same Recipe, 1.4T Tokens\n\nThe main comparison uses three models: pure MLA, GDN-H (a hybrid in which KDA is replaced by Gated DeltaNet), and Kimi Linear. All three use the same data and training recipe and are trained on 1.4T tokens. A few results:\n\n| Base model | MLA | GDN-H | Kimi Linear |\n| --- | ---: | ---: | ---: |\n| MMLU | 71.6 | 72.2 | 73.8 |\n| MMLU-Pro | 47.2 | 47.9 | 51.0 |\n| BBH | 71.6 | 70.6 | 72.9 |\n| GSM8K | 83.7 | 81.7 | 83.9 |\n| EvalPlus | 59.5 | 63.1 | 60.2 |\n\n*Table: Excerpted from Table 3 of the paper.*\n\nKimi Linear is highest on most items, with exceptions: it loses to GDN-H on EvalPlus and ties MLA on MATH. After instruction tuning (Table 4), the picture is broadly the same; it doesn't win on LiveBench, MATH500, or EvalPlus.\n\nLong-context evaluations at a 128K context, after each model's 1.4T-token pretraining (Table 5):\n\n| Model | RULER | MRCR | LongBench V2 | Frames | Eight-task average |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| MLA | 81.3 | 22.6 | 36.1 | 60.5 | 52.2 |\n| GDN-H | 80.5 | 23.9 | 32.6 | 58.7 | 51.2 |\n| Kimi Linear (RoPE) | 78.8 | 22.0 | 35.4 | 59.9 | 51.8 |\n| Kimi Linear | 84.3 | 29.6 | 35.0 | 58.8 | 54.5 |\n\n*Table: Excerpted from Table 5 of the paper.*\n\nKimi Linear has the highest average, with clear leads on RULER and MRCR; on LongBench V2 and Frames, pure MLA does better. Another interesting comparison is Kimi Linear (RoPE), the same model with positional encoding added back to MLA: its long-context average actually drops to 51.8, which supports the design choice of “leaving position to KDA.”\n\n### Speed\n\nAll efficiency comparisons use the 48B setting, with the same number of layers and heads in all three models:\n\n- **Prefill**: on par with MLA from 4K to 16K; clearly faster from 128K on, 2.3 times faster at 512K and 2.9 times faster at 1M. KDA is almost exactly as fast as GDN-H, so per-channel decay adds no noticeable extra cost.\n- **Decoding**: with the batch size fixed at 1 (Figure 7b), Kimi Linear is about 2.3 times faster at a 1M context. Factoring in the larger batches that the smaller KV cache allows, the authors estimate a theoretical maximum of about 6.3 times (Figure 1b: 1.84 ms vs. 11.48 ms per output token).\n\nThe scaling-law experiments show that, under compute-optimal training, Kimi Linear's compute efficiency is about 1.16 times that of MLA. The authors also mention that the KDA side wasn't given hyperparameter tuning as careful as MLA's.\n\n### How to Read These Results\n\nThe conditions of this report's comparison are more rigorous than in many earlier hybrid-architecture papers: same data, same recipe, same scale, plus ablations. Still, a few things are worth keeping in mind when reading it:\n\n1. **The baselines are self-trained**. MLA and GDN-H are both baselines the authors trained with the same recipe, not external public models. In a commentary the day after the report came out, Benjamin Marie pointed out that a well-tuned full-attention baseline is missing, and that many of the scores also fall below Qwen3-8B, or even Qwen3-4B; his reading is that “better than full attention” here means better than the particular setups the authors tried. [The Kaitchup, 2025-10-31](https://kaitchup.substack.com/p/minimax-m2-and-kimi-linear-why-full)\n2. **The scale is 48B total parameters, 3B active, and 1.4T tokens**. The problem MiniMax described surfaced at larger scales and in more complex multi-hop reasoning; this report doesn't answer questions at that scale.\n3. **The margins vary**. Short-context gaps are mostly 1 to 4 points, and long-context results are a mix of wins and losses.\n\nOn Hacker News, someone asked what a “hybrid linear attention architecture” is, and the reply was succinct: “1/4 of their layers are conventional quadratic attention.” [Hacker News, 2025-10-31](https://news.ycombinator.com/item?id=45766937)\n\n## Conclusion\n\nKimi Linear contributes on three levels:\n\n1. **KDA**: it splits Gated DeltaNet's scalar decay down to each channel and comes with a matching chunkwise algorithm, running at about the same speed as GDN.\n2. **A 3:1 layerwise hybrid with NoPE MLA**: positional information is left entirely to KDA, and the KV cache shrinks by about three quarters.\n3. **In fair comparisons controlled by the authors themselves**, the 48B model beats pure MLA on most short-context, long-context, and reinforcement learning evaluations; decoding at a 1M context is about 2.3 times faster at batch size 1, with a theoretical maximum of about 6.3 times once larger batches are factored in.\n\nWhen the report came out, whether “linear attention can replace most full-attention layers in large models” was still an open question; in the same week, MiniMax chose full attention for its M2. Nine months later, Moonshot AI's flagship Kimi K3 kept the hybrid of KDA and MLA: 2.8T parameters, with 69 KDA layers out of 93. [Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3) This shows that Moonshot AI scaled KDA up to a larger model; how it compares with full attention at that scale still awaits corresponding evaluations.\n\n\n> [!TIP]\n> \n> For KDA's predecessor, and for the decay and delta rule it builds on, see:\n> \n> - **[[24.12] Gated DeltaNet: Erase, Then Write](/blog/papers/mamba/gated-deltanet)**\n> - **[[24.05] Mamba-2: Attention's Distant Cousin](/blog/papers/mamba/mamba-2)**\n","imageSizes":{"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png":{"width":1440,"height":960}}}