{"notes":[{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"先擦再写","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/gated-deltanet.76941d1611bb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"线性注意力是一本会撞号的电话簿","text":"线性注意力是一本会撞号的电话簿","depth":3},{"id":"mamba-2全部一起打折","text":"Mamba-2：全部一起打折","depth":3},{"id":"deltanet只改那一条","text":"DeltaNet：只改那一条","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三种写法算一次","text":"三种写法，算一次","depth":3},{"id":"把两者合起来","text":"把两者合起来","depth":3},{"id":"让它能并行训练","text":"让它能并行训练","depth":3},{"id":"架构与混合版本","text":"架构与混合版本","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"大海捞针三种规则的分工","text":"大海捞针：三种规则的分工","depth":3},{"id":"语言建模与真实检索","text":"语言建模与真实检索","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"注意力的远亲","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/mamba-2.16b1612b4608.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"把-ssm-写成一个矩阵","text":"把 SSM 写成一个矩阵","depth":3},{"id":"把-a-简化成一个数","text":"把 A 简化成一个数","depth":3},{"id":"跟着四个-token-算一次","text":"跟着四个 token 算一次","depth":3},{"id":"两条路各走一半","text":"两条路各走一半","depth":3},{"id":"mamba-2-模块","text":"Mamba-2 模块","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"状态大一点记得多一点","text":"状态大一点，记得多一点","depth":3},{"id":"语言建模","text":"语言建模","depth":3},{"id":"原作者也建议混合","text":"原作者也建议混合","depth":3},{"id":"也不是全面升级","text":"也不是全面升级","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/kimi-linear","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Kimi Linear","subtitle":"逐通道的遗忘","author":"Z. Yuan","paperDate":"2025-10-30","publishedOn":"2026-08-06","world":"external/kimi-linear","paperTitle":"Kimi Linear: An Expressive, Efficient Attention Architecture","paperUrl":"https://arxiv.org/abs/2510.26692","bibliography":{"title":"Kimi Linear: An Expressive, Efficient Attention Architecture","authors":["Kimi Team","Yu Zhang","Zongyu Lin","Xingcheng Yao","Jiaxi Hu","Fanqing Meng","Chengyin Liu","Xin Men","Songlin Yang","Zhiyuan Li","Wentao Li","Enzhe Lu","Weizhou Liu","Yanru Chen","Weixin Xu","Longhui Yu","Yejie Wang","Yu Fan","Longguang Zhong","Enming Yuan","Dehao Zhang","Yizhi Zhang","T. Y. Liu","Haiming Wang","Shengjun Fang","Weiran He","Shaowei Liu","Yiwei Li","Jianlin Su","Jiezhong Qiu","Bo Pang","Junjie Yan","Zhejun Jiang","Weixiao Huang","Bohong Yin","Jiacheng You","Chu Wei","Zhengtao Wang","Chao Hong","Yutian Chen","Guanduo Chen","Yucheng Wang","Huabin Zheng","Feng Wang","Yibo Liu","Mengnan Dong","Zheng Zhang","Siyuan Pan","Wenhao Wu","Yuhao Wu","Longyu Guan","Jiawen Tao","Guohong Fu","Xinran Xu","Yuzhi Wang","Guokun Lai","Yuxin Wu","Xinyu Zhou","Zhilin Yang","Yulun Du"],"url":"https://arxiv.org/abs/2510.26692","pdfUrl":"https://arxiv.org/pdf/2510.26692v2","codeUrl":"https://github.com/MoonshotAI/Kimi-Linear","preprint":{"id":"2510.26692","submittedOn":"2025-10-30","url":"https://arxiv.org/abs/2510.26692"},"publication":null,"citation":"@misc{mambakimilinear2025,\n  title = {{Kimi Linear: An Expressive, Efficient Attention Architecture}},\n  author = {Kimi Team and Yu Zhang and Zongyu Lin and Xingcheng Yao and Jiaxi Hu and Fanqing Meng and Chengyin Liu and Xin Men and Songlin Yang and Zhiyuan Li and Wentao Li and Enzhe Lu and Weizhou Liu and Yanru Chen and Weixin Xu and Longhui Yu and Yejie Wang and Yu Fan and Longguang Zhong and Enming Yuan and Dehao Zhang and Yizhi Zhang and T. Y. Liu and Haiming Wang and Shengjun Fang and Weiran He and Shaowei Liu and Yiwei Li and Jianlin Su and Jiezhong Qiu and Bo Pang and Junjie Yan and Zhejun Jiang and Weixiao Huang and Bohong Yin and Jiacheng You and Chu Wei and Zhengtao Wang and Chao Hong and Yutian Chen and Guanduo Chen and Yucheng Wang and Huabin Zheng and Feng Wang and Yibo Liu and Mengnan Dong and Zheng Zhang and Siyuan Pan and Wenhao Wu and Yuhao Wu and Longyu Guan and Jiawen Tao and Guohong Fu and Xinran Xu and Yuzhi Wang and Guokun Lai and Yuxin Wu and Xinyu Zhou and Zhilin Yang and Yulun Du},\n  year = {2025},\n  eprint = {2510.26692},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2510.26692}\n}\n","citationPath":"/content/papers/citations/mamba/kimi-linear.38a5b561e63e.bib"},"thumbnail":{"src":"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png","width":1440,"height":960},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/kimi-linear.0bff756e8ead.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"kda把衰减拆到每个通道","text":"KDA：把衰减拆到每个通道","depth":3},{"id":"其他组件","text":"其他组件","depth":3},{"id":"31-的混合以及没有位置编码的-mla","text":"3:1 的混合，以及没有位置编码的 MLA","depth":3},{"id":"手算一次缓存","text":"手算一次缓存","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"比例怎么选","text":"比例怎么选","depth":3},{"id":"同样的配方14t-tokens","text":"同样的配方，1.4T tokens","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"怎么读这些结果","text":"怎么读这些结果","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/gated-deltanet","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]}],"text":"副标题说得很清楚：这篇要改进 Mamba-2，用的是 delta rule。\n\ndelta rule 不是什么新东西。它最早可以追溯到 1960 年 Widrow 等人的研究；2021 年 Schlag 等人把它放进线性注意力，称为 DeltaNet；2024 年又有人为它设计了能在 GPU 上并行训练的算法。\n\n作者是 MIT CSAIL 的 Songlin Yang，以及 NVIDIA 的 Jan Kautz 与 Ali Hatamizadeh。他们观察到，Mamba-2 和 DeltaNet 各自解决了线性注意力一半的问题，而且两者恰好互补。这篇发表于 ICLR 2025。\n\n## 定义问题\n\n### 线性注意力是一本会撞号的电话簿\n\n先把线性注意力写成递归形式。去掉 softmax，再省略归一化与 query/key 的特征映射，它每读一个 token，就把一组「钥匙–值」写进一个矩阵状态：\n\n$$\nS_t=S_{t-1}+v_t k_t^\\top,\\qquad o_t=S_t q_t\n$$\n\n查询时，拿 $q_t$ 去乘状态。假设状态从零开始，所有钥匙两两正交、长度都是 1，那么查询 $q_t=k_i$ 时，乘出来的就是 $v_i$。\n\n问题在于，$d_k$ 维空间最多只有 $d_k$ 个相互正交的方向。序列长度一超过 $d_k$，钥匙之间一定会相互干扰，作者称之为**记忆碰撞**（memory collision）：查一把钥匙，拿回来的是好几个值叠在一起的结果。\n\n### Mamba-2：全部一起打折\n\nMamba-2 的做法是加一个由数据决定的衰减：\n\n$$\nS_t=\\alpha_t S_{t-1}+v_t k_t^\\top,\\qquad \\alpha_t\\in(0,1)\n$$\n\n旧的东西会越来越淡，碰撞就没那么严重了。\n\n但这个衰减是**一视同仁**的。想忘掉某一条数据时，其他所有数据也跟着一起变淡。\n\n### DeltaNet：只改那一条\n\nDeltaNet 换了一种写法。写入新的钥匙–值之前，先用这把钥匙把旧值读出来，再把它换成新的：\n\n$$\nv^{\\text{old}}_t=S_{t-1}k_t,\\qquad\nv^{\\text{new}}_t=\\beta_t v_t+(1-\\beta_t)v^{\\text{old}}_t\n$$\n\n$$\nS_t=S_{t-1}-v^{\\text{old}}_t k_t^\\top+v^{\\text{new}}_t k_t^\\top\n=S_{t-1}\\big(I-\\beta_t k_t k_t^\\top\\big)+\\beta_t v_t k_t^\\top\n$$\n\n$\\beta_t$ 是写入强度。钥匙长度为 1、$\\beta_t=1$ 时，用这把钥匙读到的旧值会被整个替换成新值。其他钥匙只有在和它正交时才完全不受影响；不正交的话，读出的值也会跟着被修正一些。\n\n这很精准，但一次只能改一条。遇到情境切换、需要一口气清掉大量旧信息时，DeltaNet 就清不干净。作者引用了先前的观察：DeltaNet 在合成检索任务上很强，放到真实任务上却表现平平。\n\n## 解决问题\n\n### 三种写法，算一次\n\n我们用一个很小的例子比较三种更新方式。钥匙是三维的单位向量，值是一个数。依次写入三条：\n\n1. 钥匙 $k_1$，值 5\n2. 钥匙 $k_2$，值 7\n3. 钥匙 $k_1$，值改成 9\n\n最后查 $k_1$ 应该拿到 9，查 $k_2$ 应该拿到 7：\n\n```python\nimport numpy as np\n\nk1, k2 = np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 0.0])\nevents = [(k1, 5.0), (k2, 7.0), (k1, 9.0)]     # 钥匙 k1 的值后来从 5 改成 9\n\ndef run(update):\n    S = np.zeros(3)                              # 值是 1 维，状态就是 1×3\n    for k, v in events:\n        S = update(S, k, v)\n    return S @ k1, S @ k2\n\nlinear = lambda S, k, v: S + v * k                              # 线性注意力：直接累加\ndecay  = lambda S, k, v: 0.5 * S + v * k                        # Mamba-2：先整体打折再写入\ndelta  = lambda S, k, v: S - (S @ k) * k + 1.0 * v * k          # delta rule（β=1）：先擦掉旧值再写入\nfor name, f in [(\"线性注意力\", linear), (\"整体衰减 α=0.5\", decay), (\"delta rule\", delta)]:\n    a, b = run(f)\n    print(f\"{name:<14} 查 k1 → {a:5.2f}（应为 9）  查 k2 → {b:5.2f}（应为 7）\")\n```\n\n```text\n线性注意力          查 k1 → 14.00（应为 9）  查 k2 →  7.00（应为 7）\n整体衰减 α=0.5     查 k1 → 10.25（应为 9）  查 k2 →  3.50（应为 7）\ndelta rule     查 k1 →  9.00（应为 9）  查 k2 →  7.00（应为 7）\n```\n\n- **线性注意力**把两次写入 $k_1$ 的值加在一起，拿回 14。\n- **整体衰减**让旧的 5 淡了一些，拿回 10.25；可是 $k_2$ 也被打折成 3.5，它根本没做错什么。\n- **delta rule** 先把 $k_1$ 的旧值擦掉，再写入 9；$k_2$ 完全不受影响。\n\n这个例子的钥匙恰好正交，是 delta rule 最舒服的情况。真实模型的钥匙不会这么干净，但方向是一样的：delta rule 擅长精准地改一条，衰减擅长大范围地清除。\n\n### 把两者合起来\n\n作者提出的 **gated delta rule**，只是把衰减乘回 delta rule 前面：\n\n$$\nS_t=S_{t-1}\\Big(\\alpha_t\\big(I-\\beta_t k_t k_t^\\top\\big)\\Big)+\\beta_t v_t k_t^\\top\n$$\n\n两个旋钮各管一件事：\n\n- $\\alpha_t$ 管保留多少旧状态。趋近 0 时，旧信息被清掉，只剩这一步写入的 $\\beta_t v_t k_t^\\top$，适合情境切换；趋近 1 时，就回到纯 delta rule。\n- $\\beta_t$ 管沿着当前这把钥匙修正的力度。\n\n作者还从**在线学习**的角度解释这几种规则：每次更新状态，都像是在解一个小的优化问题。线性注意力与 Mamba-2 对应的目标，是在提高 $S_t k_t$ 与 $v_t$ 内积的同时，惩罚新状态离旧状态（或打折后的旧状态）太远；delta rule 则对应于减小 $S_t k_t$ 与 $v_t$ 的平方误差，并且只走一步梯度下降。后者更像是在做回归，存进去的配对不那么容易糊在一起。\n\n\n> [!NOTE]\n> \n> 论文脚注还提到，如果让 $\\beta_t$ 取 0 到 2 之间的值，$I-\\beta_t k_t k_t^\\top$ 就可能出现负特征值，这和线性 RNN 能不能做「状态追踪」（例如判断奇偶）有关。这篇没有往这个方向深入，后来的 Mamba-3 从另一个角度处理了这个问题。\n\n\n### 让它能并行训练\n\n递归写起来很简单，但要在 GPU 上训练，就必须能并行。Mamba-2 的分块算法可以处理标量衰减；delta rule 的转移矩阵 $I-\\beta_t k_t k_t^\\top$ 每一步都不同，连乘起来比较麻烦。\n\n前人已经用 **WY 表示法**（一种把一串 Householder 型矩阵的乘积压缩成低秩形式的技巧）把 DeltaNet 改写成分块并行的形式。作者把衰减项也放进这套算法，保留了「块内用矩阵乘法、块间传递状态」的结构。\n\n### 架构与混合版本\n\n[![三张图。左边 Gated DeltaNet-H1：每个重复单元由 Gated DeltaNet 加 MLP、再由滑动窗口注意力 SWA 加 MLP 组成，重复 N 次。中间 Gated DeltaNet-H2：依次是 Mamba2 加 MLP、Gated DeltaNet 加 MLP、SWA 加 MLP。右边是 Gated DeltaNet 模块：q、k 经过线性投影、短卷积、SiLU 与 L2 归一化，v 经过线性投影、短卷积与 SiLU，α 与 β 只经过线性投影，一起送进 Gated Delta Rule；输出经过归一化，再与另一支门控相乘，最后做线性投影](/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png)](/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png)\n\n*原论文 Figure 1，取自 [**v3 PDF 第 7 页**](https://arxiv.org/pdf/2412.06464v3#page=7)，CC BY 4.0。*\n\n先看右边的模块设计。整体骨架沿用 Llama：token mixer 与 SwiGLU MLP 交替堆叠，只是把自注意力换成了 gated delta rule。\n\n- $q$、$k$、$v$ 都经过线性投影、短卷积与 SiLU；$q$、$k$ 另外做 L2 归一化，论文给出的理由是让训练稳定。（本文补充一点：钥匙长度固定为 1 时，$\\beta_t=1$ 的 $I-k_tk_t^\\top$ 恰好把这个方向整个投影掉，「擦掉」才擦得干净。）\n- $\\alpha$、$\\beta$ 只用线性投影，$\\alpha$ 沿用 Mamba-2 的参数化方式。\n- 输出先归一化，再乘上一支门控，最后投影回去。\n\n左边和中间是两个混合版本：\n\n- **Gated DeltaNet-H1**：Gated DeltaNet 与滑动窗口注意力（SWA）交替。\n- **Gated DeltaNet-H2**：Mamba2、Gated DeltaNet、SWA 三种轮流。\n\n作者的理由是，线性 Transformer 不擅长局部的位移与比较，固定大小的状态也让检索变难，所以借用 Griffin、Samba 的做法，加上窗口为 2K 的局部注意力。\n\n## 讨论\n\n### 实验设置\n\n所有模型都在相同条件下训练：1.3B 参数、从 FineWeb-Edu 抽样的 100B tokens、Llama 2 分词器、训练长度 4K。比较对象有 RetNet、HGRN2、Mamba、Mamba2、DeltaNet、Samba 与 Transformer++。\n\n### 大海捞针：三种规则的分工\n\n最能看出设计用意的，是 RULER 里的单针大海捞针（S-NIAH）：\n\n| 模型 | S-NIAH-1 4K | S-NIAH-1 8K | S-NIAH-2 4K | S-NIAH-2 8K | S-NIAH-3 2K | S-NIAH-3 4K |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| DeltaNet | 99.0 | 98.8 | 18.6 | 14.4 | 47.0 | 22.4 |\n| Mamba2 | 65.4 | 30.4 | 56.2 | 17.0 | 47.6 | 4.6 |\n| Gated DeltaNet | 91.4 | 91.8 | 92.2 | 29.6 | 84.2 | 27.6 |\n\n*表：原论文 Table 2 节选，1.3B 模型。*\n\n这张表值得慢慢读，因为三个模型的名次会随设置反转。\n\n先看 S-NIAH-1：背景是不断重复的合成句子，模型只要记住一个密码。DeltaNet 在 4K、8K 都接近全对，Mamba2 到 8K 只剩 30.4，Gated DeltaNet 在 91 左右。对「把一件事记很久」来说，一路打折的衰减反而是负担。\n\n换到 S-NIAH-2，背景变成真实文章，名次倒了过来：4K 时 Gated DeltaNet 92.2，Mamba2 56.2，DeltaNet 只剩 18.6。这次要在一大堆看起来都可能有用的内容里留住答案，清不掉旧信息的 DeltaNet 最先撑不住。\n\nS-NIAH-3 把答案换成 UUID 这种又长又没规律的字符串。4K 时 Mamba2 只剩 4.6，DeltaNet 22.4，Gated DeltaNet 27.6。\n\n作者用「清记忆」和「记配对」两种能力之间的取舍来解释这些反转：衰减负责清掉不相关的内容，delta rule 负责把配对记准，两者结合的版本在多数设置下都不吃亏。\n\n也要注意，这些模型用 4K 长度训练，8K 那几列同时考查了长度外推。在 8K 的 S-NIAH-2 上，Gated DeltaNet 也掉到 29.6，检索仍有明显差距。\n\n### 语言建模与真实检索\n\n常识推理与困惑度（Table 3）：纯递归模型里，Gated DeltaNet 八项零样本准确率的平均值是 55.32，略高于 Mamba2 的 54.89，也高于 Transformer++ 的 52.25。两个混合版本更高。（原表 H1 的平均值标为 56.40，按同一行的八个数字重算约为 56.46，原表本身有小出入。）\n\n真实检索任务（Table 4，输入截断到 2K tokens）就是另一回事了：\n\n| 模型 | 六项平均 |\n| --- | ---: |\n| Mamba2 | 29.8 |\n| DeltaNet | 26.2 |\n| Gated DeltaNet | 30.6 |\n| Transformer++ | 37.0 |\n| Gated DeltaNet-H1 | 39.0 |\n| Gated DeltaNet-H2 | 40.1 |\n\n*表：原论文 Table 4 节选。项目包括 SWDE、SQuAD、FDA、TriviaQA、NQ、DROP。*\n\n纯递归模型之间，Gated DeltaNet 最好，但和 Transformer++ 还差 6 分以上。加上滑动窗口注意力的混合版本，才反过来超过纯注意力。\n\n作者在这里也很坦诚：在真实检索任务上，Gated DeltaNet 相对 Mamba2 的优势比 S-NIAH 上小很多。他们推测，这是因为这些小模型没有做指令微调，答题能力本身就有限。\n\n### 速度\n\n训练吞吐量方面（Figure 3：单张 H100、1.3B 模型、每批约 32K tokens），Gated DeltaNet 和 DeltaNet 差不多，比 Mamba2 慢一些，作者说大约每秒少 2 到 3 千个 token，原因是更新规则更复杂。混合了 SWA 的版本反而更快。\n\n## 结论\n\n这篇论文的贡献可以简短地概括为：\n\n1. **Mamba-2 的衰减与 DeltaNet 的 delta rule 是互补的**。前者擅长大范围地忘，后者擅长精准地改。\n2. **两者结合的 gated delta rule 可以并行训练**。在 1.3B、100B tokens 的比较里，语言评测与真实检索的平均分都比 Mamba2 和 DeltaNet 好；长度外推则因任务而异，S-NIAH-1 上仍是 DeltaNet 领先。\n3. **纯递归模型的检索差距仍然存在**。在真实检索任务上，要加回局部注意力，才能追过 Transformer。\n\n这篇论文发表后不到一年，就从学术比较走进了大模型。2025 年 9 月，阿里的 Qwen3-Next 采用了 48 层、每 4 层里 3 层 Gated DeltaNet 加 1 层 Gated Attention 的排法。[Qwen3-Next 模型卡](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) 同年 10 月，月之暗面的 Kimi Linear 又把 $\\alpha_t$ 从一个标量改成了逐通道的向量。\n\n从 Mamba 的选择性、Mamba-2 的标量衰减，到这里的先擦再写，名字换了，那条「让模型自己决定记什么、忘什么」的线一直没有断。\n\n\n> [!TIP]\n> \n> 这篇的前一步与后一步，可以参考：\n> \n> - **[[24.05] Mamba-2：注意力的远亲](/blog/papers/mamba/mamba-2)**\n> - **[[25.10] Kimi Linear：逐通道的遗忘](/blog/papers/mamba/kimi-linear)**\n","imageSizes":{"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png":{"width":1050,"height":544}}}