{"notes":[{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"先擦再寫","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"zh-hant","path":"/content/papers/zh-hant/mamba/gated-deltanet.1519c748e648.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"線性注意力是一本會撞號的電話簿","text":"線性注意力是一本會撞號的電話簿","depth":3},{"id":"mamba-2全部一起打折","text":"Mamba-2：全部一起打折","depth":3},{"id":"deltanet只改那一筆","text":"DeltaNet：只改那一筆","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"三種寫法算一次","text":"三種寫法，算一次","depth":3},{"id":"把兩者合起來","text":"把兩者合起來","depth":3},{"id":"讓它能平行訓練","text":"讓它能平行訓練","depth":3},{"id":"架構與混合版本","text":"架構與混合版本","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"大海撈針三種規則的分工","text":"大海撈針：三種規則的分工","depth":3},{"id":"語言建模與真實檢索","text":"語言建模與真實檢索","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"注意力的遠親","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"zh-hant","path":"/content/papers/zh-hant/mamba/mamba-2.0f4721961023.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"把-ssm-寫成一個矩陣","text":"把 SSM 寫成一個矩陣","depth":3},{"id":"把-a-簡化成一個數","text":"把 A 簡化成一個數","depth":3},{"id":"跟著四個-token-算一次","text":"跟著四個 token 算一次","depth":3},{"id":"兩條路各走一半","text":"兩條路各走一半","depth":3},{"id":"mamba-2-區塊","text":"Mamba-2 區塊","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"狀態大一點記得多一點","text":"狀態大一點，記得多一點","depth":3},{"id":"語言建模","text":"語言建模","depth":3},{"id":"原作者也建議混合","text":"原作者也建議混合","depth":3},{"id":"也不是全面升級","text":"也不是全面升級","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/kimi-linear","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Kimi Linear","subtitle":"逐通道的遺忘","author":"Z. Yuan","paperDate":"2025-10-30","publishedOn":"2026-08-06","world":"external/kimi-linear","paperTitle":"Kimi Linear: An Expressive, Efficient Attention Architecture","paperUrl":"https://arxiv.org/abs/2510.26692","bibliography":{"title":"Kimi Linear: An Expressive, Efficient Attention Architecture","authors":["Kimi Team","Yu Zhang","Zongyu Lin","Xingcheng Yao","Jiaxi Hu","Fanqing Meng","Chengyin Liu","Xin Men","Songlin Yang","Zhiyuan Li","Wentao Li","Enzhe Lu","Weizhou Liu","Yanru Chen","Weixin Xu","Longhui Yu","Yejie Wang","Yu Fan","Longguang Zhong","Enming Yuan","Dehao Zhang","Yizhi Zhang","T. Y. Liu","Haiming Wang","Shengjun Fang","Weiran He","Shaowei Liu","Yiwei Li","Jianlin Su","Jiezhong Qiu","Bo Pang","Junjie Yan","Zhejun Jiang","Weixiao Huang","Bohong Yin","Jiacheng You","Chu Wei","Zhengtao Wang","Chao Hong","Yutian Chen","Guanduo Chen","Yucheng Wang","Huabin Zheng","Feng Wang","Yibo Liu","Mengnan Dong","Zheng Zhang","Siyuan Pan","Wenhao Wu","Yuhao Wu","Longyu Guan","Jiawen Tao","Guohong Fu","Xinran Xu","Yuzhi Wang","Guokun Lai","Yuxin Wu","Xinyu Zhou","Zhilin Yang","Yulun Du"],"url":"https://arxiv.org/abs/2510.26692","pdfUrl":"https://arxiv.org/pdf/2510.26692v2","codeUrl":"https://github.com/MoonshotAI/Kimi-Linear","preprint":{"id":"2510.26692","submittedOn":"2025-10-30","url":"https://arxiv.org/abs/2510.26692"},"publication":null,"citation":"@misc{mambakimilinear2025,\n  title = {{Kimi Linear: An Expressive, Efficient Attention Architecture}},\n  author = {Kimi Team and Yu Zhang and Zongyu Lin and Xingcheng Yao and Jiaxi Hu and Fanqing Meng and Chengyin Liu and Xin Men and Songlin Yang and Zhiyuan Li and Wentao Li and Enzhe Lu and Weizhou Liu and Yanru Chen and Weixin Xu and Longhui Yu and Yejie Wang and Yu Fan and Longguang Zhong and Enming Yuan and Dehao Zhang and Yizhi Zhang and T. Y. Liu and Haiming Wang and Shengjun Fang and Weiran He and Shaowei Liu and Yiwei Li and Jianlin Su and Jiezhong Qiu and Bo Pang and Junjie Yan and Zhejun Jiang and Weixiao Huang and Bohong Yin and Jiacheng You and Chu Wei and Zhengtao Wang and Chao Hong and Yutian Chen and Guanduo Chen and Yucheng Wang and Huabin Zheng and Feng Wang and Yibo Liu and Mengnan Dong and Zheng Zhang and Siyuan Pan and Wenhao Wu and Yuhao Wu and Longyu Guan and Jiawen Tao and Guohong Fu and Xinran Xu and Yuzhi Wang and Guokun Lai and Yuxin Wu and Xinyu Zhou and Zhilin Yang and Yulun Du},\n  year = {2025},\n  eprint = {2510.26692},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2510.26692}\n}\n","citationPath":"/content/papers/citations/mamba/kimi-linear.38a5b561e63e.bib"},"thumbnail":{"src":"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png","width":1440,"height":960},"locale":"zh-hant","path":"/content/papers/zh-hant/mamba/kimi-linear.fc66b3c3e559.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"kda把衰減拆到每個通道","text":"KDA：把衰減拆到每個通道","depth":3},{"id":"其他零件","text":"其他零件","depth":3},{"id":"31-的混合以及沒有位置編碼的-mla","text":"3:1 的混合，以及沒有位置編碼的 MLA","depth":3},{"id":"手算一次快取","text":"手算一次快取","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"比例怎麼選","text":"比例怎麼選","depth":3},{"id":"同樣的配方14t-tokens","text":"同樣的配方，1.4T tokens","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"怎麼讀這些結果","text":"怎麼讀這些結果","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/gated-deltanet","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]}],"text":"副標題說得很清楚：這篇要改進 Mamba-2，用的是 delta rule。\n\ndelta rule 不是新東西。它最早可以追到 1960 年 Widrow 等人的研究，2021 年 Schlag 等人把它放進線性注意力，叫做 DeltaNet；2024 年又有人替它設計了能在 GPU 上平行訓練的演算法。\n\n作者是 MIT CSAIL 的 Songlin Yang，以及 NVIDIA 的 Jan Kautz 與 Ali Hatamizadeh。他們觀察到，Mamba-2 和 DeltaNet 各自解決了線性注意力的一半問題，而且兩者剛好互補。這篇發表在 ICLR 2025。\n\n## 定義問題\n\n### 線性注意力是一本會撞號的電話簿\n\n先把線性注意力寫成遞迴形式。拿掉 softmax，再省略正規化與 query/key 的特徵映射，它每讀一個 token，就把一組「鑰匙–值」寫進一個矩陣狀態：\n\n$$\nS_t=S_{t-1}+v_t k_t^\\top,\\qquad o_t=S_t q_t\n$$\n\n要查詢時，拿 $q_t$ 去乘狀態。假設狀態從零開始，所有鑰匙兩兩正交、長度都是 1，那麼查詢 $q_t=k_i$ 時，乘出來就是 $v_i$。\n\n問題在於，$d_k$ 維的空間最多只有 $d_k$ 個互相正交的方向。序列長度一超過 $d_k$，鑰匙之間一定會互相干擾，作者稱之為**記憶碰撞**（memory collision）：查一把鑰匙，拿回來的是好幾個值疊在一起的結果。\n\n### Mamba-2：全部一起打折\n\nMamba-2 的做法是加一個由資料決定的衰減：\n\n$$\nS_t=\\alpha_t S_{t-1}+v_t k_t^\\top,\\qquad \\alpha_t\\in(0,1)\n$$\n\n舊的東西會越來越淡，碰撞就沒那麼嚴重。\n\n但這個衰減是**一視同仁**的。想忘掉某一筆資料時，其他所有資料也跟著一起變淡。\n\n### DeltaNet：只改那一筆\n\nDeltaNet 換了一種寫法。寫入新的鑰匙–值之前，先用這把鑰匙把舊值讀出來，再把它換成新的：\n\n$$\nv^{\\text{old}}_t=S_{t-1}k_t,\\qquad\nv^{\\text{new}}_t=\\beta_t v_t+(1-\\beta_t)v^{\\text{old}}_t\n$$\n\n$$\nS_t=S_{t-1}-v^{\\text{old}}_t k_t^\\top+v^{\\text{new}}_t k_t^\\top\n=S_{t-1}\\big(I-\\beta_t k_t k_t^\\top\\big)+\\beta_t v_t k_t^\\top\n$$\n\n$\\beta_t$ 是寫入強度。鑰匙長度為 1、$\\beta_t=1$ 時，用這把鑰匙讀到的舊值會被整個換成新值。其他鑰匙只有在和它正交時才完全不受影響；不正交的話，讀值也會跟著被修正一點。\n\n這很精準，但一次只能改一筆。遇到情境切換、需要一口氣清掉大量舊資訊時，DeltaNet 就清不乾淨。作者引用先前的觀察：DeltaNet 在合成的檢索任務上很強，放到真實任務上卻只是普通。\n\n## 解決問題\n\n### 三種寫法，算一次\n\n我們用一個很小的例子比較三種更新方式。鑰匙是三維的單位向量，值是一個數。依序寫入三筆：\n\n1. 鑰匙 $k_1$，值 5\n2. 鑰匙 $k_2$，值 7\n3. 鑰匙 $k_1$，值改成 9\n\n最後查 $k_1$ 應該拿到 9，查 $k_2$ 應該拿到 7：\n\n```python\nimport numpy as np\n\nk1, k2 = np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 0.0])\nevents = [(k1, 5.0), (k2, 7.0), (k1, 9.0)]     # 鑰匙 k1 的值後來從 5 改成 9\n\ndef run(update):\n    S = np.zeros(3)                              # 值是 1 維，狀態就是 1×3\n    for k, v in events:\n        S = update(S, k, v)\n    return S @ k1, S @ k2\n\nlinear = lambda S, k, v: S + v * k                              # 線性注意力：直接累加\ndecay  = lambda S, k, v: 0.5 * S + v * k                        # Mamba-2：先整體打折再寫\ndelta  = lambda S, k, v: S - (S @ k) * k + 1.0 * v * k          # delta rule（β=1）：先擦掉舊值再寫\nfor name, f in [(\"線性注意力\", linear), (\"整體衰減 α=0.5\", decay), (\"delta rule\", delta)]:\n    a, b = run(f)\n    print(f\"{name:<14} 查 k1 → {a:5.2f}（應為 9）  查 k2 → {b:5.2f}（應為 7）\")\n```\n\n```text\n線性注意力          查 k1 → 14.00（應為 9）  查 k2 →  7.00（應為 7）\n整體衰減 α=0.5     查 k1 → 10.25（應為 9）  查 k2 →  3.50（應為 7）\ndelta rule     查 k1 →  9.00（應為 9）  查 k2 →  7.00（應為 7）\n```\n\n- **線性注意力**把兩次寫入 $k_1$ 的值加在一起，拿回 14。\n- **整體衰減**讓舊的 5 淡掉一些，拿回 10.25；可是 $k_2$ 也被打折成 3.5，它根本沒做錯什麼。\n- **delta rule** 先把 $k_1$ 的舊值擦掉，再寫入 9；$k_2$ 完全不受影響。\n\n這個例子的鑰匙剛好正交，是 delta rule 最舒服的情況。真實模型的鑰匙不會這麼乾淨，但方向是一樣的：delta rule 擅長精準地改一筆，衰減擅長大範圍地清掉。\n\n### 把兩者合起來\n\n作者提出的 **gated delta rule** 只是把衰減乘回 delta rule 前面：\n\n$$\nS_t=S_{t-1}\\Big(\\alpha_t\\big(I-\\beta_t k_t k_t^\\top\\big)\\Big)+\\beta_t v_t k_t^\\top\n$$\n\n兩個旋鈕各管一件事：\n\n- $\\alpha_t$ 管要保留多少舊狀態。趨近 0 時，舊資訊被清掉，只剩這一步寫入的 $\\beta_t v_t k_t^\\top$，適合情境切換；趨近 1 時，就回到純 delta rule。\n- $\\beta_t$ 管沿著目前這把鑰匙修正的力道。\n\n作者也從**線上學習**的角度解釋這幾種規則：每次更新狀態，都像是在解一個小的最佳化問題。線性注意力與 Mamba-2 對應的目標，是在提高 $S_t k_t$ 與 $v_t$ 內積的同時，懲罰新狀態離舊狀態（或打折後的舊狀態）太遠；delta rule 則對應把 $S_t k_t$ 和 $v_t$ 的平方誤差變小，並且只走一步梯度下降。後者更像是在做迴歸，存進去的配對比較不容易糊在一起。\n\n\n> [!NOTE]\n> \n> 論文註腳還提到，若讓 $\\beta_t$ 介於 0 到 2，$I-\\beta_t k_t k_t^\\top$ 就可能出現負的特徵值，這和線性 RNN 能不能做「狀態追蹤」（例如數奇偶）有關。這篇沒有往這個方向做，後來的 Mamba-3 從另一個角度處理了這個問題。\n\n\n### 讓它能平行訓練\n\n遞迴寫起來很簡單，但要在 GPU 上訓練，必須能平行。Mamba-2 的分塊演算法可以處理純量衰減；delta rule 的轉移矩陣 $I-\\beta_t k_t k_t^\\top$ 每一步都不同，連乘起來比較麻煩。\n\n前人已經用 **WY 表示法**（一種把一串 Householder 型矩陣的乘積壓成低秩形式的技巧）把 DeltaNet 改寫成分塊平行的形式。作者把衰減項也放進這套演算法裡，保留「區塊內用矩陣乘法、區塊之間傳遞狀態」的結構。\n\n### 架構與混合版本\n\n[![三張圖。左邊 Gated DeltaNet-H1：每個重複單元由 Gated DeltaNet 加 MLP、再由滑動視窗注意力 SWA 加 MLP 組成，重複 N 次。中間 Gated DeltaNet-H2：依序是 Mamba2 加 MLP、Gated DeltaNet 加 MLP、SWA 加 MLP。右邊是 Gated DeltaNet 區塊：q、k 經過線性投影、短卷積、SiLU 與 L2 正規化，v 經過線性投影、短卷積與 SiLU，α 與 β 只經過線性投影，一起送進 Gated Delta Rule；輸出經過正規化，再與另一支閘門相乘，最後線性投影](/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png)](/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png)\n\n*原論文 Figure 1，取自 [**v3 PDF 第 7 頁**](https://arxiv.org/pdf/2412.06464v3#page=7)，CC BY 4.0。*\n\n先看右邊的區塊設計。整體骨架沿用 Llama：token mixer 與 SwiGLU MLP 交替堆疊，只是把自注意力換成 gated delta rule。\n\n- $q$、$k$、$v$ 都經過線性投影、短卷積與 SiLU；$q$、$k$ 另外做 L2 正規化，論文給的理由是讓訓練穩定。（本文補充一點：鑰匙長度固定為 1 時，$\\beta_t=1$ 的 $I-k_tk_t^\\top$ 剛好把這個方向整個投影掉，「擦掉」才擦得乾淨。）\n- $\\alpha$、$\\beta$ 只用線性投影，$\\alpha$ 沿用 Mamba-2 的參數化。\n- 輸出先正規化，再乘上一支閘門，最後投影回去。\n\n左邊和中間是兩個混合版本：\n\n- **Gated DeltaNet-H1**：Gated DeltaNet 與滑動視窗注意力（SWA）交替。\n- **Gated DeltaNet-H2**：Mamba2、Gated DeltaNet、SWA 三種輪流。\n\n作者的理由是，線性 Transformer 不擅長局部的位移與比較，固定大小的狀態也讓檢索變難，所以借用 Griffin、Samba 的做法，加上視窗 2K 的局部注意力。\n\n## 討論\n\n### 實驗設定\n\n所有模型都用相同條件訓練：1.3B 參數、FineWeb-Edu 抽樣的 100B tokens、Llama 2 分詞器、訓練長度 4K。比較對象有 RetNet、HGRN2、Mamba、Mamba2、DeltaNet、Samba 與 Transformer++。\n\n### 大海撈針：三種規則的分工\n\n最能看出設計用意的，是 RULER 裡的單一大海撈針（S-NIAH）：\n\n| 模型 | S-NIAH-1 4K | S-NIAH-1 8K | S-NIAH-2 4K | S-NIAH-2 8K | S-NIAH-3 2K | S-NIAH-3 4K |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| DeltaNet | 99.0 | 98.8 | 18.6 | 14.4 | 47.0 | 22.4 |\n| Mamba2 | 65.4 | 30.4 | 56.2 | 17.0 | 47.6 | 4.6 |\n| Gated DeltaNet | 91.4 | 91.8 | 92.2 | 29.6 | 84.2 | 27.6 |\n\n*表：原論文 Table 2 節錄，1.3B 模型。*\n\n這張表值得慢慢讀，因為三個模型的名次會隨設定反轉。\n\n先看 S-NIAH-1：背景是不斷重複的合成句子，模型只要記住一個密碼。DeltaNet 在 4K、8K 都將近全對，Mamba2 到 8K 只剩 30.4，Gated DeltaNet 落在 91 左右。對「把一件事記很久」來說，一路打折的衰減反而是負擔。\n\n換到 S-NIAH-2，背景變成真實文章，名次倒了過來：4K 時 Gated DeltaNet 92.2，Mamba2 56.2，DeltaNet 只剩 18.6。這次要在一大堆看起來都可能有用的內容裡留住答案，清不掉舊資訊的 DeltaNet 最先撐不住。\n\nS-NIAH-3 把答案換成 UUID 這種又長又沒規律的字串。4K 時 Mamba2 只剩 4.6，DeltaNet 22.4，Gated DeltaNet 27.6。\n\n作者用「清記憶」和「記配對」兩種能力的取捨解釋這些反轉：衰減負責清掉不相關的內容，delta rule 負責把配對記準，兩者合起來的版本在多數設定都不吃虧。\n\n也要注意，這些模型用 4K 長度訓練，8K 的欄位同時考了長度外推。8K 的 S-NIAH-2 上，Gated DeltaNet 也掉到 29.6，檢索仍有明顯落差。\n\n### 語言建模與真實檢索\n\n常識推理與困惑度（Table 3）：純遞迴模型裡，Gated DeltaNet 的八項零樣本準確率平均是 55.32，略高於 Mamba2 的 54.89，也高於 Transformer++ 的 52.25。兩個混合版本更高。（原表 H1 的平均標為 56.40，按同列八個數字重算約是 56.46，原表本身有小出入。）\n\n真實的檢索任務（Table 4，輸入截到 2K tokens）就是另一回事：\n\n| 模型 | 六項平均 |\n| --- | ---: |\n| Mamba2 | 29.8 |\n| DeltaNet | 26.2 |\n| Gated DeltaNet | 30.6 |\n| Transformer++ | 37.0 |\n| Gated DeltaNet-H1 | 39.0 |\n| Gated DeltaNet-H2 | 40.1 |\n\n*表：原論文 Table 4 節錄。項目包括 SWDE、SQuAD、FDA、TriviaQA、NQ、DROP。*\n\n純遞迴模型之間，Gated DeltaNet 最好，但和 Transformer++ 還差了 6 分以上。加上滑動視窗注意力的混合版本，才反過來超過純注意力。\n\n作者在這裡也很老實：真實檢索任務上，Gated DeltaNet 對 Mamba2 的優勢比 S-NIAH 小很多。他們推測是因為這些小模型沒有做指令微調，答題能力本身就有限。\n\n### 速度\n\n訓練吞吐量方面（Figure 3：單張 H100、1.3B 模型、每批約 32K tokens），Gated DeltaNet 和 DeltaNet 差不多，比 Mamba2 慢一些，作者說大約差每秒 2 到 3 千個 token，因為更新規則比較複雜。混合 SWA 的版本反而更快。\n\n## 結論\n\n這篇論文的貢獻可以很簡短地說完：\n\n1. **Mamba-2 的衰減與 DeltaNet 的 delta rule 是互補的**。前者擅長大範圍地忘，後者擅長精準地改。\n2. **合在一起的 gated delta rule 可以平行訓練**。在 1.3B、100B tokens 的比較裡，語言評測與真實檢索的平均都比 Mamba2 和 DeltaNet 好；長度外推則依任務而異，S-NIAH-1 仍是 DeltaNet 領先。\n3. **純遞迴模型的檢索差距還在**。真實檢索任務上，要加回局部注意力，才追過 Transformer。\n\n這篇論文發表後不到一年，就從學術比較走進了大模型。2025 年 9 月，阿里的 Qwen3-Next 採用 48 層、每 4 層裡 3 層 Gated DeltaNet 加 1 層 Gated Attention 的排法。[Qwen3-Next 模型卡](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) 同年 10 月，月之暗面的 Kimi Linear 再把 $\\alpha_t$ 從一個純量改成逐通道的向量。\n\n從 Mamba 的選擇性、Mamba-2 的純量衰減，到這裡的先擦再寫，名字換了，那條「讓模型自己決定要記什麼、忘什麼」的線一直沒有斷。\n\n\n> [!TIP]\n> \n> 這篇的前一步與後一步，可以參考：\n> \n> - **[[24.05] Mamba-2：注意力的遠親](/blog/papers/mamba/mamba-2)**\n> - **[[25.10] Kimi Linear：逐通道的遺忘](/blog/papers/mamba/kimi-linear)**\n","imageSizes":{"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png":{"width":1050,"height":544}}}