{"notes":[{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"Erase, Then Write","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"en","path":"/content/papers/en/mamba/gated-deltanet.76d17f2230a0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"linear-attention-is-a-phone-book-where-entries-collide","text":"Linear Attention Is a Phone Book Where Entries Collide","depth":3},{"id":"mamba-2-everything-gets-discounted-together","text":"Mamba-2: Everything Gets Discounted Together","depth":3},{"id":"deltanet-change-only-that-entry","text":"DeltaNet: Change Only That Entry","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-update-rules-worked-through","text":"Three Update Rules, Worked Through","depth":3},{"id":"combining-the-two","text":"Combining the Two","depth":3},{"id":"making-it-trainable-in-parallel","text":"Making It Trainable in Parallel","depth":3},{"id":"architecture-and-hybrid-variants","text":"Architecture and Hybrid Variants","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"needle-in-a-haystack-how-the-three-rules-divide-the-work","text":"Needle in a Haystack: How the Three Rules Divide the Work","depth":3},{"id":"language-modeling-and-real-world-retrieval","text":"Language Modeling and Real-World Retrieval","depth":3},{"id":"speed","text":"Speed","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"Attention's Distant Cousin","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"en","path":"/content/papers/en/mamba/mamba-2.84790bbee84c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"writing-an-ssm-as-a-matrix","text":"Writing an SSM as a Matrix","depth":3},{"id":"reducing-a-to-a-single-number","text":"Reducing A to a Single Number","depth":3},{"id":"working-through-four-tokens","text":"Working Through Four Tokens","depth":3},{"id":"half-one-way-half-the-other","text":"Half One Way, Half the Other","depth":3},{"id":"the-mamba-2-block","text":"The Mamba-2 Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"a-bigger-state-remembers-more","text":"A Bigger State Remembers More","depth":3},{"id":"language-modeling","text":"Language Modeling","depth":3},{"id":"the-original-authors-recommend-hybrids-too","text":"The Original Authors Recommend Hybrids Too","depth":3},{"id":"not-an-across-the-board-upgrade","text":"Not an Across-the-Board Upgrade","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/kimi-linear","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Kimi Linear","subtitle":"Forgetting, Channel by Channel","author":"Z. Yuan","paperDate":"2025-10-30","publishedOn":"2026-08-06","world":"external/kimi-linear","paperTitle":"Kimi Linear: An Expressive, Efficient Attention Architecture","paperUrl":"https://arxiv.org/abs/2510.26692","bibliography":{"title":"Kimi Linear: An Expressive, Efficient Attention Architecture","authors":["Kimi Team","Yu Zhang","Zongyu Lin","Xingcheng Yao","Jiaxi Hu","Fanqing Meng","Chengyin Liu","Xin Men","Songlin Yang","Zhiyuan Li","Wentao Li","Enzhe Lu","Weizhou Liu","Yanru Chen","Weixin Xu","Longhui Yu","Yejie Wang","Yu Fan","Longguang Zhong","Enming Yuan","Dehao Zhang","Yizhi Zhang","T. Y. Liu","Haiming Wang","Shengjun Fang","Weiran He","Shaowei Liu","Yiwei Li","Jianlin Su","Jiezhong Qiu","Bo Pang","Junjie Yan","Zhejun Jiang","Weixiao Huang","Bohong Yin","Jiacheng You","Chu Wei","Zhengtao Wang","Chao Hong","Yutian Chen","Guanduo Chen","Yucheng Wang","Huabin Zheng","Feng Wang","Yibo Liu","Mengnan Dong","Zheng Zhang","Siyuan Pan","Wenhao Wu","Yuhao Wu","Longyu Guan","Jiawen Tao","Guohong Fu","Xinran Xu","Yuzhi Wang","Guokun Lai","Yuxin Wu","Xinyu Zhou","Zhilin Yang","Yulun Du"],"url":"https://arxiv.org/abs/2510.26692","pdfUrl":"https://arxiv.org/pdf/2510.26692v2","codeUrl":"https://github.com/MoonshotAI/Kimi-Linear","preprint":{"id":"2510.26692","submittedOn":"2025-10-30","url":"https://arxiv.org/abs/2510.26692"},"publication":null,"citation":"@misc{mambakimilinear2025,\n  title = {{Kimi Linear: An Expressive, Efficient Attention Architecture}},\n  author = {Kimi Team and Yu Zhang and Zongyu Lin and Xingcheng Yao and Jiaxi Hu and Fanqing Meng and Chengyin Liu and Xin Men and Songlin Yang and Zhiyuan Li and Wentao Li and Enzhe Lu and Weizhou Liu and Yanru Chen and Weixin Xu and Longhui Yu and Yejie Wang and Yu Fan and Longguang Zhong and Enming Yuan and Dehao Zhang and Yizhi Zhang and T. Y. Liu and Haiming Wang and Shengjun Fang and Weiran He and Shaowei Liu and Yiwei Li and Jianlin Su and Jiezhong Qiu and Bo Pang and Junjie Yan and Zhejun Jiang and Weixiao Huang and Bohong Yin and Jiacheng You and Chu Wei and Zhengtao Wang and Chao Hong and Yutian Chen and Guanduo Chen and Yucheng Wang and Huabin Zheng and Feng Wang and Yibo Liu and Mengnan Dong and Zheng Zhang and Siyuan Pan and Wenhao Wu and Yuhao Wu and Longyu Guan and Jiawen Tao and Guohong Fu and Xinran Xu and Yuzhi Wang and Guokun Lai and Yuxin Wu and Xinyu Zhou and Zhilin Yang and Yulun Du},\n  year = {2025},\n  eprint = {2510.26692},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2510.26692}\n}\n","citationPath":"/content/papers/citations/mamba/kimi-linear.38a5b561e63e.bib"},"thumbnail":{"src":"/content/papers/assets/2f970769c0aa4ffad72650699d1a2c3f9c71e6b262171a38ddf5b2e287a10124.png","width":1440,"height":960},"locale":"en","path":"/content/papers/en/mamba/kimi-linear.9f3e6ef90c77.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"kda-splitting-the-decay-across-channels","text":"KDA: Splitting the Decay Across Channels","depth":3},{"id":"other-components","text":"Other Components","depth":3},{"id":"a-31-mix-and-mla-without-positional-encoding","text":"A 3:1 Mix, and MLA Without Positional Encoding","depth":3},{"id":"working-out-the-cache-by-hand","text":"Working Out the Cache by Hand","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"choosing-the-ratio","text":"Choosing the Ratio","depth":3},{"id":"same-recipe-14t-tokens","text":"Same Recipe, 1.4T Tokens","depth":3},{"id":"speed","text":"Speed","depth":3},{"id":"how-to-read-these-results","text":"How to Read These Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/gated-deltanet","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]}],"text":"The subtitle spells it out: this paper sets out to improve Mamba-2, and its tool is the delta rule.\n\nThe delta rule is nothing new. It goes back to work by Widrow et al. in 1960; in 2021, Schlag et al. brought it into linear attention under the name DeltaNet; and in 2024, others designed an algorithm that lets it train in parallel on GPUs.\n\nThe authors are Songlin Yang of MIT CSAIL, together with Jan Kautz and Ali Hatamizadeh of NVIDIA. They observed that Mamba-2 and DeltaNet each solve half of linear attention's problem, and that the two happen to complement each other. The paper appeared at ICLR 2025.\n\n## Defining the Problem\n\n### Linear Attention Is a Phone Book Where Entries Collide\n\nFirst, let's write linear attention in recurrent form. Drop the softmax and leave out the normalization and the query/key feature maps; then, each time it reads a token, it writes a “key–value” pair into a matrix state:\n\n$$\nS_t=S_{t-1}+v_t k_t^\\top,\\qquad o_t=S_t q_t\n$$\n\nTo look something up, multiply the state by $q_t$. Suppose the state starts at zero and all keys are mutually orthogonal with length 1; then querying with $q_t=k_i$ returns exactly $v_i$.\n\nThe problem is that a $d_k$-dimensional space has at most $d_k$ mutually orthogonal directions. Once the sequence is longer than $d_k$, the keys are bound to interfere with one another. The authors call this **memory collision**: you look up one key, and what comes back is several values piled on top of each other.\n\n### Mamba-2: Everything Gets Discounted Together\n\nMamba-2's approach is to add a data-dependent decay:\n\n$$\nS_t=\\alpha_t S_{t-1}+v_t k_t^\\top,\\qquad \\alpha_t\\in(0,1)\n$$\n\nOld content gradually fades, so collisions become less severe.\n\nBut this decay treats **everything the same**. To forget one entry, every other entry has to fade along with it.\n\n### DeltaNet: Change Only That Entry\n\nDeltaNet writes in a different way. Before writing a new key–value pair, it uses the key to read out the old value, then replaces that value with the new one:\n\n$$\nv^{\\text{old}}_t=S_{t-1}k_t,\\qquad\nv^{\\text{new}}_t=\\beta_t v_t+(1-\\beta_t)v^{\\text{old}}_t\n$$\n\n$$\nS_t=S_{t-1}-v^{\\text{old}}_t k_t^\\top+v^{\\text{new}}_t k_t^\\top\n=S_{t-1}\\big(I-\\beta_t k_t k_t^\\top\\big)+\\beta_t v_t k_t^\\top\n$$\n\n$\\beta_t$ is the write strength. When the key has length 1 and $\\beta_t=1$, the old value read with this key is replaced entirely by the new one. Other keys are completely unaffected only if they are orthogonal to it; if they aren't, the values read with them get corrected a little as well.\n\nThis is precise, but it can only change one entry at a time. When the context switches and a large amount of old information needs to be cleared at once, DeltaNet can't wipe the slate clean. The authors cite an earlier observation: DeltaNet is strong on synthetic retrieval tasks but merely average on real-world tasks.\n\n## Solving the Problem\n\n### Three Update Rules, Worked Through\n\nLet's compare the three update rules on a tiny example. The keys are three-dimensional unit vectors, and each value is a single number. We write three entries in order:\n\n1. Key $k_1$, value 5\n2. Key $k_2$, value 7\n3. Key $k_1$, value changed to 9\n\nAt the end, looking up $k_1$ should return 9, and looking up $k_2$ should return 7:\n\n```python\nimport numpy as np\n\nk1, k2 = np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 0.0])\nevents = [(k1, 5.0), (k2, 7.0), (k1, 9.0)]     # key k1's value later changes from 5 to 9\n\ndef run(update):\n    S = np.zeros(3)                              # values are 1-D, so the state is 1×3\n    for k, v in events:\n        S = update(S, k, v)\n    return S @ k1, S @ k2\n\nlinear = lambda S, k, v: S + v * k                              # linear attention: just accumulate\ndecay  = lambda S, k, v: 0.5 * S + v * k                        # Mamba-2: discount everything, then write\ndelta  = lambda S, k, v: S - (S @ k) * k + 1.0 * v * k          # delta rule (β=1): erase the old value, then write\nfor name, f in [(\"Linear attn\", linear), (\"Decay α=0.5\", decay), (\"Delta rule\", delta)]:\n    a, b = run(f)\n    print(f\"{name:<14} look up k1 → {a:5.2f} (should be 9)  look up k2 → {b:5.2f} (should be 7)\")\n```\n\n```text\nLinear attn    look up k1 → 14.00 (should be 9)  look up k2 →  7.00 (should be 7)\nDecay α=0.5    look up k1 → 10.25 (should be 9)  look up k2 →  3.50 (should be 7)\nDelta rule     look up k1 →  9.00 (should be 9)  look up k2 →  7.00 (should be 7)\n```\n\n- **Linear attention** adds together the two values written to $k_1$ and returns 14.\n- **Uniform decay** lets the old 5 fade somewhat and returns 10.25; but $k_2$ also gets discounted to 3.5, even though it did nothing wrong.\n- **The delta rule** first erases $k_1$'s old value, then writes 9; $k_2$ is left completely untouched.\n\nIn this example the keys happen to be orthogonal, which is the most comfortable case for the delta rule. Real models' keys won't be this clean, but the tendency is the same: the delta rule is good at precisely changing one entry, and decay is good at clearing things out on a broad scale.\n\n### Combining the Two\n\nThe **gated delta rule** the authors propose simply multiplies the decay back in front of the delta rule:\n\n$$\nS_t=S_{t-1}\\Big(\\alpha_t\\big(I-\\beta_t k_t k_t^\\top\\big)\\Big)+\\beta_t v_t k_t^\\top\n$$\n\nEach of the two knobs handles one job:\n\n- $\\alpha_t$ controls how much of the old state to keep. As it approaches 0, old information is cleared away, leaving only this step's write $\\beta_t v_t k_t^\\top$, which suits a context switch; as it approaches 1, it reduces to the pure delta rule.\n- $\\beta_t$ controls how strongly to correct along the current key.\n\nThe authors also explain these rules from an **online learning** perspective: every state update is like solving a small optimization problem. The objectives behind linear attention and Mamba-2 raise the inner product of $S_t k_t$ and $v_t$ while penalizing the new state for straying too far from the old state (or the discounted old state); the delta rule corresponds to shrinking the squared error between $S_t k_t$ and $v_t$, taking just one step of gradient descent. The latter is closer to regression, so the stored pairs are less likely to blur together.\n\n\n> [!NOTE]\n> \n> A footnote in the paper also mentions that if $\\beta_t$ is allowed to range from 0 to 2, $I-\\beta_t k_t k_t^\\top$ can have negative eigenvalues, which bears on whether a linear RNN can do “state tracking” (such as computing parity). This paper doesn't pursue that direction; Mamba-3 later tackles the problem from a different angle.\n\n\n### Making It Trainable in Parallel\n\nThe recurrence is simple to write down, but training on GPUs requires parallelism. Mamba-2's chunked algorithm can handle scalar decay; the delta rule's transition matrix $I-\\beta_t k_t k_t^\\top$ is different at every step, which makes multiplying them together trickier.\n\nEarlier work had already used the **WY representation** (a technique that compresses a product of Householder-like matrices into a low-rank form) to rewrite DeltaNet in a chunkwise parallel form. The authors fold the decay term into this algorithm as well, keeping the structure of “matrix multiplication within chunks, state passing between chunks.”\n\n### Architecture and Hybrid Variants\n\n[![Three diagrams. Left, Gated DeltaNet-H1: each repeated unit consists of Gated DeltaNet plus MLP, then sliding-window attention (SWA) plus MLP, repeated N times. Middle, Gated DeltaNet-H2: Mamba2 plus MLP, Gated DeltaNet plus MLP, and SWA plus MLP, in that order. Right, the Gated DeltaNet block: q and k pass through a linear projection, short convolution, SiLU, and L2 normalization; v passes through a linear projection, short convolution, and SiLU; α and β pass only through linear projections; all of them go into the Gated Delta Rule. The output is normalized, multiplied by a separate gate branch, and finally passed through a linear projection](/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png)](/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png)\n\n*Figure 1 from the paper, excerpted from [**page 7 of the v3 PDF**](https://arxiv.org/pdf/2412.06464v3#page=7), CC BY 4.0.*\n\nStart with the block design on the right. The overall skeleton follows Llama: token mixers alternate with SwiGLU MLPs, except that self-attention is replaced by the gated delta rule.\n\n- $q$, $k$, and $v$ all pass through a linear projection, a short convolution, and SiLU; $q$ and $k$ also get L2 normalization, which the paper says is for training stability. (This note adds one point: when keys are fixed at length 1, setting $\\beta_t=1$ makes $I-k_tk_t^\\top$ project out that direction entirely, so the “erase” really does wipe it clean.)\n- $\\alpha$ and $\\beta$ use only linear projections, with $\\alpha$ following Mamba-2's parameterization.\n- The output is normalized first, then multiplied by a gate branch, and finally projected back.\n\nThe left and middle diagrams show two hybrid variants:\n\n- **Gated DeltaNet-H1**: Gated DeltaNet alternates with sliding-window attention (SWA).\n- **Gated DeltaNet-H2**: Mamba2, Gated DeltaNet, and SWA take turns.\n\nThe authors' rationale: linear Transformers are poor at local shifting and comparison, and a fixed-size state makes retrieval harder, so, following Griffin and Samba, they add local attention with a 2K window.\n\n## Discussion\n\n### Experimental Setup\n\nAll models are trained under the same conditions: 1.3B parameters, 100B tokens sampled from FineWeb-Edu, the Llama 2 tokenizer, and a training length of 4K. The baselines are RetNet, HGRN2, Mamba, Mamba2, DeltaNet, Samba, and Transformer++.\n\n### Needle in a Haystack: How the Three Rules Divide the Work\n\nWhat best reveals the intent of the design is RULER's single needle-in-a-haystack test (S-NIAH):\n\n| Model | S-NIAH-1 4K | S-NIAH-1 8K | S-NIAH-2 4K | S-NIAH-2 8K | S-NIAH-3 2K | S-NIAH-3 4K |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| DeltaNet | 99.0 | 98.8 | 18.6 | 14.4 | 47.0 | 22.4 |\n| Mamba2 | 65.4 | 30.4 | 56.2 | 17.0 | 47.6 | 4.6 |\n| Gated DeltaNet | 91.4 | 91.8 | 92.2 | 29.6 | 84.2 | 27.6 |\n\n*Table: Excerpted from Table 2 of the paper; 1.3B models.*\n\nThis table rewards a slow read, because the three models' rankings flip from one setting to the next.\n\nStart with S-NIAH-1: the background is a synthetic sentence repeated over and over, and the model only has to remember one passkey. DeltaNet is nearly perfect at both 4K and 8K, Mamba2 drops to 30.4 at 8K, and Gated DeltaNet sits around 91. When the job is “remember one thing for a long time,” a decay that keeps discounting everything is actually a burden.\n\nSwitch to S-NIAH-2, where the background becomes real essays, and the ranking reverses: at 4K, Gated DeltaNet scores 92.2, Mamba2 56.2, and DeltaNet only 18.6. This time the answer has to be held on to amid a pile of content that all looks potentially useful, and DeltaNet, which can't clear out old information, is the first to give way.\n\nS-NIAH-3 replaces the answer with a long, patternless string such as a UUID. At 4K, Mamba2 manages only 4.6, DeltaNet 22.4, and Gated DeltaNet 27.6.\n\nThe authors explain these reversals as a trade-off between two abilities, “clearing memory” and “remembering pairs”: decay clears out irrelevant content, the delta rule keeps pairs accurate, and the version that combines them doesn't lose out in most settings.\n\nNote, too, that these models were trained at 4K, so the 8K columns also test length extrapolation. On S-NIAH-2 at 8K, Gated DeltaNet also falls to 29.6, so a clear gap in retrieval remains.\n\n### Language Modeling and Real-World Retrieval\n\nCommonsense reasoning and perplexity (Table 3): among purely recurrent models, Gated DeltaNet's average zero-shot accuracy over eight tasks is 55.32, slightly above Mamba2's 54.89 and above Transformer++'s 52.25. The two hybrid variants score higher still. (The original table lists H1's average as 56.40; recomputing it from the eight numbers in the same row gives about 56.46, so the table itself has a small discrepancy.)\n\nReal-world retrieval tasks (Table 4, inputs truncated to 2K tokens) are another story:\n\n| Model | Six-task average |\n| --- | ---: |\n| Mamba2 | 29.8 |\n| DeltaNet | 26.2 |\n| Gated DeltaNet | 30.6 |\n| Transformer++ | 37.0 |\n| Gated DeltaNet-H1 | 39.0 |\n| Gated DeltaNet-H2 | 40.1 |\n\n*Table: Excerpted from Table 4 of the paper. The tasks are SWDE, SQuAD, FDA, TriviaQA, NQ, and DROP.*\n\nAmong purely recurrent models, Gated DeltaNet is best, but it is still more than 6 points behind Transformer++. Only the hybrid variants, with sliding-window attention added, turn the tables and surpass pure attention.\n\nThe authors are candid here too: on real-world retrieval tasks, Gated DeltaNet's advantage over Mamba2 is much smaller than on S-NIAH. They suspect this is because these small models weren't instruction-tuned, so their ability to answer questions is limited to begin with.\n\n### Speed\n\nOn training throughput (Figure 3: a single H100, 1.3B models, about 32K tokens per batch), Gated DeltaNet is about on par with DeltaNet and somewhat slower than Mamba2; the authors put the gap at roughly 2,000 to 3,000 tokens per second, due to the more complex update rule. The variants mixed with SWA are actually faster.\n\n## Conclusion\n\nThe contribution of this paper can be summed up briefly:\n\n1. **Mamba-2's decay and DeltaNet's delta rule are complementary**. The former is good at broad forgetting, the latter at precise edits.\n2. **The combined gated delta rule can be trained in parallel**. In the comparison at 1.3B parameters and 100B tokens, its averages on language benchmarks and real-world retrieval both beat Mamba2 and DeltaNet; length extrapolation varies by task, and DeltaNet still leads on S-NIAH-1.\n3. **The retrieval gap of purely recurrent models remains**. On real-world retrieval tasks, local attention has to be added back before the model overtakes the Transformer.\n\nLess than a year after publication, this paper had made its way from academic comparisons into large models. In September 2025, Alibaba's Qwen3-Next adopted a 48-layer layout in which every 4 layers consist of 3 Gated DeltaNet layers plus 1 Gated Attention layer. [Qwen3-Next model card](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) In October of the same year, Moonshot AI's Kimi Linear went further and turned $\\alpha_t$ from a scalar into a per-channel vector.\n\nFrom Mamba's selectivity and Mamba-2's scalar decay to the erase-then-write rule here, the names have changed, but the thread of “letting the model decide for itself what to remember and what to forget” has never been broken.\n\n\n> [!TIP]\n> \n> For the step before this paper and the step after it, see:\n> \n> - **[[24.05] Mamba-2: Attention's Distant Cousin](/blog/papers/mamba/mamba-2)**\n> - **[[25.10] Kimi Linear: Forgetting, Channel by Channel](/blog/papers/mamba/kimi-linear)**\n","imageSizes":{"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png":{"width":1050,"height":544}}}