{"notes":[{"id":"mamba/kimi-linear","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Kimi Linear","subtitle":"逐通道的遗忘","author":"Z. Yuan","paperDate":"2025-10-30","publishedOn":"2026-08-06","world":"external/kimi-linear","paperTitle":"Kimi Linear: An Expressive, Efficient Attention Architecture","paperUrl":"https://arxiv.org/abs/2510.26692","bibliography":{"title":"Kimi Linear: An Expressive, Efficient Attention Architecture","authors":["Kimi Team","Yu Zhang","Zongyu Lin","Xingcheng Yao","Jiaxi Hu","Fanqing Meng","Chengyin Liu","Xin Men","Songlin Yang","Zhiyuan Li","Wentao Li","Enzhe Lu","Weizhou Liu","Yanru Chen","Weixin Xu","Longhui Yu","Yejie Wang","Yu Fan","Longguang Zhong","Enming Yuan","Dehao Zhang","Yizhi Zhang","T. Y. Liu","Haiming Wang","Shengjun Fang","Weiran He","Shaowei Liu","Yiwei Li","Jianlin Su","Jiezhong Qiu","Bo Pang","Junjie Yan","Zhejun Jiang","Weixiao Huang","Bohong Yin","Jiacheng You","Chu Wei","Zhengtao Wang","Chao Hong","Yutian Chen","Guanduo Chen","Yucheng Wang","Huabin Zheng","Feng Wang","Yibo Liu","Mengnan Dong","Zheng Zhang","Siyuan Pan","Wenhao Wu","Yuhao Wu","Longyu Guan","Jiawen Tao","Guohong Fu","Xinran Xu","Yuzhi Wang","Guokun Lai","Yuxin Wu","Xinyu Zhou","Zhilin Yang","Yulun Du"],"url":"https://arxiv.org/abs/2510.26692","pdfUrl":"https://arxiv.org/pdf/2510.26692v2","codeUrl":"https://github.com/MoonshotAI/Kimi-Linear","preprint":{"id":"2510.26692","submittedOn":"2025-10-30","url":"https://arxiv.org/abs/2510.26692"},"publication":null,"citation":"@misc{mambakimilinear2025,\n  title = {{Kimi Linear: An Expressive, Efficient Attention Architecture}},\n  author = {Kimi Team and Yu Zhang and Zongyu Lin and Xingcheng Yao and Jiaxi Hu and Fanqing Meng and Chengyin Liu and Xin Men and Songlin Yang and Zhiyuan Li and Wentao Li and Enzhe Lu and Weizhou Liu and Yanru Chen and Weixin Xu and Longhui Yu and Yejie Wang and Yu Fan and Longguang Zhong and Enming Yuan and Dehao Zhang and Yizhi Zhang and T. Y. Liu and Haiming Wang and Shengjun Fang and Weiran He and Shaowei Liu and Yiwei Li and Jianlin Su and Jiezhong Qiu and Bo Pang and Junjie Yan and Zhejun Jiang and Weixiao Huang and Bohong Yin and Jiacheng You and Chu Wei and Zhengtao Wang and Chao Hong and Yutian Chen and Guanduo Chen and Yucheng Wang and Huabin Zheng and Feng Wang and Yibo Liu and Mengnan Dong and Zheng Zhang and Siyuan Pan and Wenhao Wu and Yuhao Wu and Longyu Guan and Jiawen Tao and Guohong Fu and Xinran Xu and Yuzhi Wang and Guokun Lai and Yuxin Wu and Xinyu Zhou and Zhilin Yang and Yulun Du},\n  year = {2025},\n  eprint = {2510.26692},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2510.26692}\n}\n","citationPath":"/content/papers/citations/mamba/kimi-linear.38a5b561e63e.bib"},"thumbnail":{"src":"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png","width":1440,"height":960},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/kimi-linear.0bff756e8ead.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"kda把衰减拆到每个通道","text":"KDA：把衰减拆到每个通道","depth":3},{"id":"其他组件","text":"其他组件","depth":3},{"id":"31-的混合以及没有位置编码的-mla","text":"3:1 的混合，以及没有位置编码的 MLA","depth":3},{"id":"手算一次缓存","text":"手算一次缓存","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"比例怎么选","text":"比例怎么选","depth":3},{"id":"同样的配方14t-tokens","text":"同样的配方，1.4T tokens","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"怎么读这些结果","text":"怎么读这些结果","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/gated-deltanet","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]},{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"先擦再写","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/gated-deltanet.76941d1611bb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"线性注意力是一本会撞号的电话簿","text":"线性注意力是一本会撞号的电话簿","depth":3},{"id":"mamba-2全部一起打折","text":"Mamba-2：全部一起打折","depth":3},{"id":"deltanet只改那一条","text":"DeltaNet：只改那一条","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三种写法算一次","text":"三种写法，算一次","depth":3},{"id":"把两者合起来","text":"把两者合起来","depth":3},{"id":"让它能并行训练","text":"让它能并行训练","depth":3},{"id":"架构与混合版本","text":"架构与混合版本","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"大海捞针三种规则的分工","text":"大海捞针：三种规则的分工","depth":3},{"id":"语言建模与真实检索","text":"语言建模与真实检索","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"注意力的远亲","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/mamba-2.16b1612b4608.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"把-ssm-写成一个矩阵","text":"把 SSM 写成一个矩阵","depth":3},{"id":"把-a-简化成一个数","text":"把 A 简化成一个数","depth":3},{"id":"跟着四个-token-算一次","text":"跟着四个 token 算一次","depth":3},{"id":"两条路各走一半","text":"两条路各走一半","depth":3},{"id":"mamba-2-模块","text":"Mamba-2 模块","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"状态大一点记得多一点","text":"状态大一点，记得多一点","depth":3},{"id":"语言建模","text":"语言建模","depth":3},{"id":"原作者也建议混合","text":"原作者也建议混合","depth":3},{"id":"也不是全面升级","text":"也不是全面升级","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]}],"text":"2025 年 10 月 29 日，MiniMax 的预训练负责人发了一篇文章，解释他们的新模型 M2 为什么退回全注意力：混合了线性注意力的模型在小规模评测上看起来不输，一放大，多跳推理就露出了破绽。[MiniMax，2025-10-29](https://www.minimax.io/news/why-did-m2-end-up-as-a-full-attention-model)\n\n第二天，月之暗面（Moonshot AI）上传了这份技术报告。摘要第一句写的是：在公平比较下，这是首个在短上下文、长上下文与强化学习三种场景中都**胜过**全注意力的混合线性注意力架构。\n\n同一周，两家中国团队在各自的模型与产品条件下，做出了方向相反的选择。\n\n月之暗面做的是 Kimi 聊天助手，同年 7 月发布的 Kimi K2 是一个 1T 参数的 MoE 模型，注意力用的是 DeepSeek 提出的 MLA。这份报告则训练了一个总参数 48B、每个 token 激活 3B 的模型，来测试新的注意力设计。作者名单里有 Gated DeltaNet 的第一作者 Songlin Yang。\n\n## 定义问题\n\n作者的出发点是 agent 与强化学习。模型要在推理时处理很长的轨迹：工具调用的记录、反复的思考与修正，输出也越来越长。这类工作以**解码**为主，KV cache 的大小与每个 token 的生成时间直接决定成本。\n\n换成线性层，KV cache 的问题就没了，可是新问题接踵而来：工具记录越拉越长，固定大小的状态就得压缩越来越多的历史，之后要找回某个细节时，往往已经糊了。Mamba-2 的门控与 DeltaNet 的 delta rule，都是在改善这种压缩，但光靠线性层还是不够，所以业界开始在其中保留几层全注意力。\n\n这篇的切入点有两个：一是把线性层的记忆更新做得更精细；二是在足够大的规模、足够完整的评测上，看看这种混合能不能在质量上**追平甚至超过**全注意力，同时真正省下推理成本。\n\n## 解决问题\n\n### KDA：把衰减拆到每个通道\n\n核心模块叫 **KDA**（Kimi Delta Attention），是在 Gated DeltaNet 的基础上再改进一步。先看 Gated DeltaNet 的更新：\n\n$$\nS_t=\\alpha_t\\big(I-\\beta_t k_t k_t^\\top\\big)S_{t-1}+\\beta_t k_t v_t^\\top\n$$\n\n（这里沿用 Kimi Linear 报告的写法，状态是 $d_k\\times d_v$，所以矩阵乘在左边。）\n\n$\\alpha_t$ 是一个标量，整个头的记忆一起打折。KDA 把它换成一个对角矩阵：\n\n$$\nS_t=\\big(I-\\beta_t k_t k_t^\\top\\big)\\,\\text{Diag}(\\alpha_t)\\,S_{t-1}+\\beta_t k_t v_t^\\top,\\qquad \\alpha_t\\in[0,1]^{d_k}\n$$\n\n状态的每一行对应钥匙空间的一个方向，现在每个方向都有自己的遗忘速度。有的通道可以记很久，有的可以很快清掉。\n\n我们用一个只有两个通道的小状态感受一下：\n\n| 更新前的状态 | 标量衰减 $\\alpha=0.5$ | 逐通道衰减 $\\alpha=(1.0,\\ 0.1)$ |\n| --- | --- | --- |\n| 通道 1：8 | 4 | 8 |\n| 通道 2：6 | 3 | 0.6 |\n\n*表：本文示意，只算遗忘门 $\\text{Diag}(\\alpha_t)S_{t-1}$ 这一步，还没做 delta rule 的修正与写入。*\n\n假设通道 1 存的是很久以后还要用的信息，通道 2 存的是已经过时的内容。标量衰减只能两边一起打五折；逐通道的版本可以留住通道 1、几乎清空通道 2。\n\n这种逐通道的门控，此前在 GLA（Gated Linear Attention）里就出现过。KDA 的工作是把它和 delta rule 结合起来，并且让它仍然能高效地并行训练。\n\n作者把转移矩阵写成一种特殊的「对角加低秩」（DPLR）形式，并为它设计了专用的分块算法。和一般的 DPLR 写法相比，计算量少了不少，也更贴近原本的 delta rule。\n\n### 其他组件\n\n每个头的输入是这样产生的：\n\n- $q$、$k$：线性投影、短卷积、Swish，再做 L2 归一化，让特征值保持稳定。\n- $v$：线性投影、短卷积、Swish。\n- $\\alpha$：先经过一个低秩投影（秩等于头维度），再通过与 GDN、Mamba 类似的衰减函数。\n- $\\beta$：线性投影加 sigmoid。\n\n输出先做逐头的 RMSNorm，再乘上一个由数据决定的 sigmoid 门控，最后投影回去。作者说这个输出门控也有助于缓解 attention sink。\n\n### 3:1 的混合，以及没有位置编码的 MLA\n\n整个模型沿用 Moonlight（月之暗面此前的 MoE 模型）的骨架，每个块是一层 token 混合加一层前馈层，除了第一层用普通的 dense 前馈层，其余都是 MoE。token 混合层每 3 层 KDA 接 1 层 MLA。作者选择整层交替，而不是在同一层里混合不同的头，理由是基础设施更简单、训练也更稳定。\n\n一个值得注意的设计是：MLA 层**不用任何位置编码**（NoPE）。位置信息与「越近越重要」的偏好，全部交给 KDA 层负责。作者认为，KDA 的衰减本身就是一种由数据决定的位置编码。\n\n### 手算一次缓存\n\n根据 Hugging Face 上 Kimi-Linear-48B-A3B 的配置文件，这个模型共 27 层，其中 20 层 KDA、7 层 MLA（最后一层也是 MLA）。MLA 每层每个 token 要缓存一个 512 维的压缩向量，外加 64 维的位置部分：\n\n```python\nGiB, MiB = 2**30, 2**20\nmla_per_layer = 512 + 64                    # MLA 每层每 token 缓存：压缩后的 KV（512）＋位置部分（64）\nfull = 27 * mla_per_layer * 2               # 27 层全是 MLA，bf16\nhybrid = 7 * mla_per_layer * 2              # Kimi Linear：只有 7 层 MLA\nprint(f\"全 MLA：每 token {full / 1024:.1f} KiB；Kimi Linear：{hybrid / 1024:.1f} KiB（省 {1 - hybrid / full:.0%}）\")\nn = 2**20\nprint(f\"1M tokens：全 MLA {full * n / GiB:.1f} GiB；Kimi Linear {hybrid * n / GiB:.1f} GiB\")\nkda_state = 20 * 32 * 128 * 128             # 20 层 KDA，每层 32 头，每头 128×128 的矩阵状态\nprint(f\"20 层 KDA 状态：{kda_state:,} 个数，bf16 {kda_state * 2 / MiB:.0f} MiB（与长度无关）\")\n```\n\n```text\n全 MLA：每 token 30.4 KiB；Kimi Linear：7.9 KiB（省 74%）\n1M tokens：全 MLA 30.4 GiB；Kimi Linear 7.9 GiB\n20 层 KDA 状态：10,485,760 个数，bf16 20 MiB（与长度无关）\n```\n\n省下的比例只取决于「还有几层保留 KV cache」，7/27 恰好接近四分之一，这就是论文说「最多省 75%」的由来。这里假设两种模型的每个 MLA 层都缓存同样的 576 个值，只比较会随长度增长的部分。配置文件里的 64 维位置部分在 NoPE 下是否真的存进缓存、KDA 状态实际用什么精度存储，都要看推理引擎的实现；如果两边都不存那 64 维，省下的比例不变。\n\nMLA 本身已经把 KV 压得很小，所以这里的绝对数字比前几代模型小得多。KDA 那 20 MiB 的状态则与上下文长度无关。\n\n## 讨论\n\n### 比例怎么选\n\n作者先用 scaling law 实验里最小一档的模型（16 层、16 头）扫了几种 KDA 与 MLA 的比例，每个版本的训练计算量相同。验证集刻意选了和预训练数据分布差异很大的高质量文本，用来观察模型在分布变化时的泛化能力。\n\n![点图：横轴是 KDA 层对 MLA 层的比例，从左到右为 0:1（纯 MLA）、1:1、3:1、7:1、15:1；纵轴是验证困惑度，从 5.60 到 5.85。五个点依次是 5.77、5.66、5.65、5.70、5.82，3:1 最低，两端最高。](/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png)\n\n*图：本文依据原论文 Table 1 的验证困惑度重绘。原论文采用 CC BY-NC-ND 4.0 许可，这里不转载原图。*\n\n完整的数字，包括另外两组组件消融：\n\n| 设置 | 训练困惑度 | 验证困惑度 |\n| --- | ---: | ---: |\n| **3:1** | **9.23** | **5.65** |\n| 0:1（纯 MLA） | 9.45 | 5.77 |\n| 1:1 | 9.29 | 5.66 |\n| 7:1 | 9.23 | 5.70 |\n| 15:1 | 9.34 | 5.82 |\n| 3:1，去掉输出门控 | 9.25 | 5.67 |\n| 3:1，去掉卷积层 | 9.29 | 5.70 |\n\n*表：原论文 Table 1 节选。*\n\n7:1 的训练困惑度和 3:1 一样，验证困惑度却差了一截；1:1 差不多一样好，但推理成本高。纯 MLA 也明显落后。在 16 层的模型里，15:1 相当于只剩一层 MLA，验证困惑度是五种比例里最差的。\n\n和 Jamba 的 1:7、NVIDIA 研究的约 8% 注意力相比，这里的最佳点偏向保留更多全注意力。线性层不同、评测方式不同，答案也不一样。\n\n### 同样的配方，1.4T tokens\n\n正式比较用了三个模型：纯 MLA、GDN-H（把 KDA 换成 Gated DeltaNet 的混合版本），以及 Kimi Linear。三者使用同样的数据与训练配方，都训练 1.4T tokens。挑几项来看：\n\n| 基础模型 | MLA | GDN-H | Kimi Linear |\n| --- | ---: | ---: | ---: |\n| MMLU | 71.6 | 72.2 | 73.8 |\n| MMLU-Pro | 47.2 | 47.9 | 51.0 |\n| BBH | 71.6 | 70.6 | 72.9 |\n| GSM8K | 83.7 | 81.7 | 83.9 |\n| EvalPlus | 59.5 | 63.1 | 60.2 |\n\n*表：原论文 Table 3 节选。*\n\n多数项目 Kimi Linear 最高，但也有例外：EvalPlus 输给 GDN-H，MATH 和 MLA 同分。指令微调之后（Table 4）情况大致相同，LiveBench、MATH500、EvalPlus 这几项没有赢。\n\n各模型预训练 1.4T tokens 后，在 128K 上下文设置下的长文本评测（Table 5）：\n\n| 模型 | RULER | MRCR | LongBench V2 | Frames | 八项平均 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| MLA | 81.3 | 22.6 | 36.1 | 60.5 | 52.2 |\n| GDN-H | 80.5 | 23.9 | 32.6 | 58.7 | 51.2 |\n| Kimi Linear（RoPE） | 78.8 | 22.0 | 35.4 | 59.9 | 51.8 |\n| Kimi Linear | 84.3 | 29.6 | 35.0 | 58.8 | 54.5 |\n\n*表：原论文 Table 5 节选。*\n\n平均分是 Kimi Linear 最高，RULER、MRCR 领先明显；LongBench V2 和 Frames 则是纯 MLA 更好。另一个有意思的对照是 Kimi Linear（RoPE），也就是同样的模型、但 MLA 加回位置编码，长上下文平均分反而掉到 51.8，这支持了「把位置交给 KDA」的设计。\n\n### 速度\n\n效率比较都在 48B 的设置下进行，三个模型的层数与头数相同：\n\n- **预填充（prefill）**：4K 到 16K 时和 MLA 差不多；从 128K 开始明显变快，512K 时快 2.3 倍，1M 时快 2.9 倍。KDA 和 GDN-H 几乎一样快，逐通道的衰减没有带来明显的额外成本。\n- **解码**：批大小固定为 1 时（Figure 7b），1M 上下文下 Kimi Linear 快约 2.3 倍。再把 KV cache 变小后能开更大批量的收益算进去，作者估计理论上最高约 6.3 倍（Figure 1b：每个输出 token 1.84 ms 对 11.48 ms）。\n\nscaling law 实验则显示，在计算最优的训练下，Kimi Linear 的计算效率约是 MLA 的 1.16 倍。作者也提到，KDA 这一侧没有像 MLA 那样仔细调过超参数。\n\n### 怎么读这些结果\n\n这份报告的比较条件，比过去很多混合架构论文都严谨：同数据、同配方、同规模，还有消融实验。但读的时候仍有几件事要放在心上：\n\n1. **对照组是自己训练的**。MLA 与 GDN-H 都是作者用同一套配方训练的基线，而不是外部的公开模型。Benjamin Marie 在报告发布次日的评论里指出，这里缺少一个调校充分的全注意力基线，不少分数也低于 Qwen3-8B，甚至 Qwen3-4B；他的解读是，「胜过全注意力」在这里指的是胜过作者试过的那几种设置。[The Kaitchup，2025-10-31](https://kaitchup.substack.com/p/minimax-m2-and-kimi-linear-why-full)\n2. **规模是总参数 48B、激活 3B，1.4T tokens**。MiniMax 说的问题，是在更大规模、更复杂的多跳推理上才浮现的；这份报告没有回答那个规模下的问题。\n3. **领先幅度不一**。短上下文大多是 1 到 4 分的差距，长上下文则有输有赢。\n\nHacker News 上，有人问「混合线性注意力架构」是什么，下面的回答很简洁：四分之一的层是传统的平方注意力。[Hacker News，2025-10-31](https://news.ycombinator.com/item?id=45766937)\n\n## 结论\n\nKimi Linear 的贡献有三个层面：\n\n1. **KDA**：把 Gated DeltaNet 的标量衰减细分到每个通道，并设计了相应的分块算法，速度和 GDN 差不多。\n2. **3:1 的层级混合加上 NoPE 的 MLA**：位置信息全部交给 KDA，KV cache 少了约四分之三。\n3. **在作者自己控制的公平比较下**，48B 模型在多数短上下文、长上下文与强化学习评测上胜过纯 MLA；1M 上下文的解码，批大小为 1 时快约 2.3 倍，算上更大批量的理论值最高约 6.3 倍。\n\n报告发布时，「线性注意力能不能在大模型上取代大部分全注意力层」还没有定论；MiniMax 在同一周为自己的 M2 选择了全注意力。九个月后，月之暗面的旗舰模型 Kimi K3 继续采用 KDA 与 MLA 的混合：2.8T 参数，93 层里有 69 层 KDA。[Kimi K3 模型卡](https://huggingface.co/moonshotai/Kimi-K3) 这说明月之暗面把 KDA 扩展到了更大的模型；它在这个规模下和全注意力相比差多少，还需要相应的评测。\n\n\n> [!TIP]\n> \n> KDA 的前身，以及它所延续的衰减与 delta rule，可以参考：\n> \n> - **[[24.12] Gated DeltaNet：先擦再写](/blog/papers/mamba/gated-deltanet)**\n> - **[[24.05] Mamba-2：注意力的远亲](/blog/papers/mamba/mamba-2)**\n","imageSizes":{"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png":{"width":1440,"height":960}}}