{"notes":[{"id":"mamba/mamba-3","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-3","subtitle":"推理优先","author":"Z. Yuan","paperDate":"2026-03-16","publishedOn":"2026-04-02","world":"external/mamba3","paperTitle":"Mamba-3: Improved Sequence Modeling using State Space Principles","paperUrl":"https://arxiv.org/abs/2603.15569","bibliography":{"title":"Mamba-3: Improved Sequence Modeling using State Space Principles","authors":["Aakash Lahoti","Kevin Y. Li","Berlin Chen","Caitlin Wang","Aviv Bick","J. Zico Kolter","Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2603.15569","pdfUrl":"https://arxiv.org/pdf/2603.15569v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2603.15569","submittedOn":"2026-03-16","url":"https://arxiv.org/abs/2603.15569"},"publication":{"venue":"ICLR","year":2026,"kind":"conference","url":"https://mlanthology.org/iclr/2026/lahoti2026iclr-mamba3/"},"citation":"@misc{mambamamba32026,\n  title = {{Mamba-3: Improved Sequence Modeling using State Space Principles}},\n  author = {Aakash Lahoti and Kevin Y. Li and Berlin Chen and Caitlin Wang and Aviv Bick and J. Zico Kolter and Tri Dao and Albert Gu},\n  year = {2026},\n  eprint = {2603.15569},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2603.15569}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-3.7f42c2a217f2.bib"},"thumbnail":{"src":"/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png","width":1356,"height":644},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/mamba-3.239d16c21039.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"一更有表达力的离散化","text":"一、更有表达力的离散化","depth":3},{"id":"二复数状态转个圈","text":"二、复数状态：转个圈","depth":3},{"id":"三mimo让解码时的-gpu-忙一点","text":"三、MIMO：让解码时的 GPU 忙一点","depth":3},{"id":"架构","text":"架构","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"语言建模","text":"语言建模","depth":3},{"id":"检索纯模型还是不够","text":"检索：纯模型还是不够","depth":3},{"id":"速度注意比较条件","text":"速度：注意比较条件","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/mamba","category":"mamba","legacyPath":"mamba/2312-mamba","legacySlug":"2312-mamba","legacyPaths":[],"title":"Mamba","subtitle":"谁是继承人","author":"Z. Yuan","paperDate":"2023-12-01","publishedOn":"2024-12-25","paperTitle":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","paperUrl":"https://arxiv.org/abs/2312.00752","bibliography":{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","authors":["Albert Gu","Tri Dao"],"url":"https://arxiv.org/abs/2312.00752","pdfUrl":"https://arxiv.org/pdf/2312.00752","codeUrl":null,"preprint":{"id":"2312.00752","submittedOn":"2023-12-01","url":"https://arxiv.org/abs/2312.00752"},"publication":null,"citation":"@misc{mambamamba2023,\n  title = {{Mamba: Linear-Time Sequence Modeling with Selective State Spaces}},\n  author = {Albert Gu and Tri Dao},\n  year = {2023},\n  eprint = {2312.00752},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2312.00752}\n}\n","citationPath":"/content/papers/citations/mamba/mamba.5671f143cfd6.bib"},"thumbnail":{"src":"/content/papers/assets/8f6187d5a2a7a17c0687f2ca08ea302601584bb95f5cd5f52927c54d6e76089a.jpg","width":1224,"height":500},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/mamba.d492134c0eee.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"可选择机制","text":"可选择机制","depth":3},{"id":"s6-架构","text":"S6 架构","depth":3},{"id":"mamba-架构","text":"Mamba 架构","depth":3},{"id":"再谈选择机制","text":"再谈选择机制","depth":3},{"id":"训练方法与协定","text":"训练方法与协定","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"scaling-laws","text":"Scaling Laws","depth":3},{"id":"和其他语言模型的比较","text":"和其他语言模型的比较","depth":3},{"id":"推论效率比较","text":"推论效率比较","depth":3},{"id":"消融实验-选择机制","text":"消融实验-选择机制","depth":3},{"id":"消融实验-关键参数","text":"消融实验-关键参数","depth":3},{"id":"消融实验-其他设定","text":"消融实验-其他设定","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/s4","relation":"background","topic":"从状态空间记忆到视觉序列"},{"id":"mamba/vim","relation":"next","topic":"从状态空间记忆到视觉序列"}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"注意力的远亲","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/mamba-2.16b1612b4608.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"把-ssm-写成一个矩阵","text":"把 SSM 写成一个矩阵","depth":3},{"id":"把-a-简化成一个数","text":"把 A 简化成一个数","depth":3},{"id":"跟着四个-token-算一次","text":"跟着四个 token 算一次","depth":3},{"id":"两条路各走一半","text":"两条路各走一半","depth":3},{"id":"mamba-2-模块","text":"Mamba-2 模块","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"状态大一点记得多一点","text":"状态大一点，记得多一点","depth":3},{"id":"语言建模","text":"语言建模","depth":3},{"id":"原作者也建议混合","text":"原作者也建议混合","depth":3},{"id":"也不是全面升级","text":"也不是全面升级","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"先擦再写","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"zh-hans","path":"/content/papers/zh-hans/mamba/gated-deltanet.76941d1611bb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"线性注意力是一本会撞号的电话簿","text":"线性注意力是一本会撞号的电话簿","depth":3},{"id":"mamba-2全部一起打折","text":"Mamba-2：全部一起打折","depth":3},{"id":"deltanet只改那一条","text":"DeltaNet：只改那一条","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三种写法算一次","text":"三种写法，算一次","depth":3},{"id":"把两者合起来","text":"把两者合起来","depth":3},{"id":"让它能并行训练","text":"让它能并行训练","depth":3},{"id":"架构与混合版本","text":"架构与混合版本","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"大海捞针三种规则的分工","text":"大海捞针：三种规则的分工","depth":3},{"id":"语言建模与真实检索","text":"语言建模与真实检索","depth":3},{"id":"速度","text":"速度","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]}],"text":"距离 Mamba 第一版过去了两年多，Mamba-3 发表于 ICLR 2026。\n\n作者团队换了一批新面孔，指导者仍是 Albert Gu 与 Tri Dao。署名单位是卡内基梅隆、普林斯顿、Together AI 和 Cartesia AI：Tri Dao 来自普林斯顿与 Together AI，Albert Gu 来自卡内基梅隆与 Cartesia AI。\n\n读这篇的引言，会发现语气和 2023 年很不一样。Mamba 当年谈的是「能不能成为通用序列模型的骨干」；这一篇第一段谈的是推理成本：思维链、agent 工作流，让推理时的计算成为模型性能的关键。接着作者直接写出现状：Mamba-2 和 Gated DeltaNet 这类层，已经被放进 Kimi、NVIDIA、腾讯混元、Qwen 的大规模**混合**模型里。\n\n所以这篇要做的，是为这些混合模型提供一个更好的线性层。\n\n## 定义问题\n\n作者列了三个问题：\n\n1. **Mamba-2 为了训练速度，牺牲了表达能力**。在同样的推理成本下，Mamba-2 模型更差。这一点 Tri Dao 在 2024 年的博客里就提过。\n2. **做不了状态追踪**。给一串 0 和 1，问里面 1 的个数是奇数还是偶数，这种一层 LSTM 就能做的事，Mamba-2 做不到。前人的研究已经证明，原因在于状态转移的特征值被限制为非负实数。\n3. **解码时硬件很闲**。这些算法都是从训练的角度设计的。解码时每次只处理一个 token，计算量很小，却要把整个状态从显存读出来再写回去；GPU 大部分时间在等数据，算术强度（计算量除以访存量）很低。\n\n作者想从「推理优先」的角度出发，回到状态空间模型的原理，为每个问题各给出一个解法。\n\n## 解决问题\n\n### 一、更有表达力的离散化\n\n状态空间模型原本是连续时间系统，要拿来处理一个个 token，得先离散化。作者先指出一件事：Mamba-1 和 Mamba-2 论文里写的是一种公式，实际代码用的却是另一种近似。他们为这个近似补上了推导，称为 **exponential-Euler**：\n\n$$\nh_t=e^{\\Delta_t A_t}h_{t-1}+\\Delta_t B_t x_t\n$$\n\n它只用区间右端点的输入，是一阶近似。Mamba-3 改用梯形法的推广，同时看区间两端：\n\n$$\nh_t=\\alpha_t h_{t-1}+\\beta_t B_{t-1}x_{t-1}+\\gamma_t B_t x_t\n$$\n\n其中 $\\alpha_t=e^{\\Delta_t A_t}$、$\\beta_t=(1-\\lambda_t)\\Delta_t e^{\\Delta_t A_t}$、$\\gamma_t=\\lambda_t\\Delta_t$，$\\lambda_t\\in[0,1]$ 由数据决定。$\\lambda_t=1$ 时就退回 Mamba-2；$\\lambda_t=\\tfrac12$ 时，是对（乘上指数衰减后的）输入积分取两端点的平均，类似梯形法。要达到更高的误差阶，还得满足 $\\lambda_t$ 接近 $\\tfrac12$ 等条件；实际模型让 $\\lambda_t$ 自由学习，主要是为了增加递归的表达能力。\n\n这个式子多了一项前一个 token 的输入，相当于先在 SSM 的输入上做一个宽度为 2 的卷积。作者发现，再加上 $B$、$C$ 的可学习偏置项，几乎所有线性模型都会放的那个短卷积就可以拿掉了：\n\n| 变体（440M） | 困惑度 |\n| --- | ---: |\n| Mamba-3 去掉偏置与梯形法 | 16.68 |\n| Mamba-3 去掉偏置 | 16.49 |\n| Mamba-3 | 15.72 |\n| Mamba-3 再加回短卷积 | 15.85 |\n\n*表：原论文 Table 5a。*\n\n### 二、复数状态：转个圈\n\n先看奇偶这个问题。如果状态可以「旋转」，事情就很简单：读到 1 就转半圈，读到 0 不动，最后看停在哪一边。\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\nbits = rng.integers(0, 2, size=1000)\n\ndef rot(theta):\n    c, s = np.cos(theta), np.sin(theta)\n    return np.array([[c, -s], [s, c]])\n\nh = np.array([1.0, 0.0])\nfor x in bits:\n    h = rot(np.pi * x) @ h          # 读到 1 就转半圈，读到 0 不动\nprint(\"旋转状态：\", np.round(h, 6), \"→ 奇偶\", 0 if h[0] > 0 else 1)\nprint(\"直接计算：\", bits.sum() % 2)\n\n# 实数、非负的衰减：每一步乘上 0 到 1 之间的数，只能缩小，不能翻面\nh = 1.0\nfor x in bits[:20]:\n    h = (0.5 if x else 0.9) * h\nprint(\"非负衰减 20 步后：\", round(h, 6))\n```\n\n```text\n旋转状态： [-1.  0.] → 奇偶 1\n直接计算： 1\n非负衰减 20 步后： 3.2e-05\n```\n\n一千个比特走完，旋转的状态恰好停在答案上。反观 Mamba-2 这种非负的标量衰减，每一步只能乘上 0 到 1 之间的数：状态只会越来越小，永远不会翻到另一边。\n\n（最后一段只是示意。前人的定理讲的是更一般的情况：只要转移矩阵的特征值都是非负实数，这类线性 RNN 就无法在有限精度下解出任意长度的奇偶问题。）\n\n旋转对应到复数，就是乘上 $e^{i\\theta}$。作者把 Mamba-3 的 SSM 改成复数形式，再证明了两件事：\n\n1. 复数 SSM 可以改写成维度加倍的实数 SSM，转移矩阵是衰减系数乘上由一个个 2×2 旋转矩阵排成的块对角矩阵。\n2. 把到当前为止的累积旋转吸收到 $B$、$C$ 上，就等价于套用一个**由数据决定的旋转位置编码**（RoPE）。\n\n第二点很实用。RoPE 在 Transformer 里已经有非常成熟、廉价的实现，Mamba-3 可以直接沿用，额外成本很小。差别在于，普通 RoPE 每个位置转固定的角度，Mamba-3 转多少度由输入决定。\n\n形式语言任务的结果（缩放后的准确率，%）：\n\n| 模型 | 奇偶 | 模运算（无括号） | 模运算（有括号） |\n| --- | ---: | ---: | ---: |\n| Mamba-3 | 100.00 | 98.51 | 87.75 |\n| Mamba-3（改用普通 RoPE） | 1.56 | 20.70 | 2.62 |\n| Mamba-3（不用 RoPE） | 2.27 | 1.49 | 0.72 |\n| Mamba-2 | 0.90 | 47.81 | 0.88 |\n| GDN（特征值范围 [−1, 1]） | 100.00 | 99.25 | 93.50 |\n\n*表：原论文 Table 5b。*\n\n这组测试是用短序列训练、拿较长的序列测试。在 Mamba-3 的几个变体里，只有由数据决定角度的版本三项都做得好；固定角度的 RoPE 和不用 RoPE 的版本大多接近零，只有固定角度版本在无括号模运算上拿到 20.70。（表上是缩放后的准确率，0 大致对应瞎猜。）Mamba-2 在无括号模运算上有部分成功（47.81），奇偶与有括号模运算则接近零。\n\n表上最后一行也值得看：允许负特征值的 Gated DeltaNet 同样解得出来，有括号模运算甚至更好。状态追踪不只有复数这一条路，负特征值也是一种翻面的方式。\n\n### 三、MIMO：让解码时的 GPU 忙一点\n\n原本的 SSM 每一步做的是**单输入单输出**（SISO）更新：输入和 $B$ 做一次外积，把一个秩为 1 的矩阵加到状态上。解码时每一步都要读写整个状态，计算却只有这么一点点。\n\nMamba-3 把它换成**多输入多输出**（MIMO）：输入与 $B$ 各自扩展出 $R$ 个分支，每一步写入一个秩至多为 $R$ 的更新，读出时也用 $R$ 组 $C$，状态大小不变。当 $R$ 远小于状态维度与头维度时，SSM 主要运算的计算量大约随 $R$ 增加，访存量仍由那份固定大小的状态主导，算术强度因此随之提高。解码本来就卡在访存上，多出来的计算相当于利用了 GPU 闲置的时间。\n\n论文用 $R=4$。为了让参数量和 SISO 版本一致，1.5B 模型的 MLP 中间层从 4096 缩到 3824。训练时 MIMO 比较慢，作者说耗时最多是 SISO 的 $R$ 倍。\n\n### 架构\n\n[![两个模块并排。左边 Mamba-2 模块：输入投影出 X、B、C 与决定衰减的参数（图中标为 A），X 经过卷积与激活函数，进入 SSM，输出与门控相乘后归一化，再做输出投影。右边 Mamba-3 模块：没有卷积，B 与 C 先各自归一化，再经过 RoPE（角度 Θ 由输入投影产生），另外多了可选的 MIMO 投影](/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png)](/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png)\n\n*原论文 Figure 2，取自 [**v1 PDF 第 11 页**](https://arxiv.org/pdf/2603.15569v1#page=11)，CC BY 4.0。*\n\n对照左右两个模块：\n\n- **短卷积拿掉了**，由梯形法与偏置项取代。\n- **$B$、$C$ 先归一化**，作用类似 Transformer 的 QK 归一化。\n- **加上 RoPE**，角度 $\\Theta$ 由输入产生，这就是复数状态的实现。\n- **黄色的 MIMO 投影**是可选的。\n- 纯 Mamba-3 拿掉了 Mamba-2 输出前的那层归一化；放进混合模型时，作者发现加回来对长度泛化有帮助。\n\n整体骨架沿用 Llama，Mamba-3 模块与 SwiGLU 交替堆叠。\n\n## 讨论\n\n### 语言建模\n\n所有模型都在 FineWeb-Edu 的 100B tokens 上训练，上下文 2K，从 180M 到 1.5B 共四种规模。1.5B 的结果：\n\n| 模型 | FineWeb-Edu 困惑度 | 七项准确率平均 |\n| --- | ---: | ---: |\n| Transformer | 10.51 | 55.4 |\n| GDN | 10.45 | 55.8 |\n| Mamba-2 | 10.47 | 55.7 |\n| Mamba-3 SISO | 10.35 | 56.4 |\n| Mamba-3 MIMO（$R=4$） | 10.24 | 57.6 |\n\n*表：原论文 Table 3 节选。*\n\nSISO 比次优的 GDN 高 0.6 分，MIMO 再多 1.2 分。四种规模下，Mamba-3 SISO 都比 Mamba-2 好。\n\n[![折线图：横轴是相对的总状态大小（对数刻度），纵轴是预训练困惑度。三条线从上到下是 Mamba-2、Mamba-3、Mamba-3 MIMO，状态越大困惑度越低；Mamba-3 MIMO 的整条线都在最下方](/content/papers/assets/9c2757127eedcbc423ca747c871fa87a8c498a586a11782bf356497d5bf9510d.png)](/content/papers/assets/9c2757127eedcbc423ca747c871fa87a8c498a586a11782bf356497d5bf9510d.png)\n\n*原论文 Figure 3，取自 [**v1 PDF 第 15 页**](https://arxiv.org/pdf/2603.15569v1#page=15)，CC BY 4.0。440M 模型，训练到 Chinchilla 最优 token 数的两倍。*\n\n解码速度主要由状态大小决定，所以作者用状态大小作为推理速度的代理指标。图上在同样的状态大小下，Mamba-3 的困惑度都比 Mamba-2 低；MIMO 版本状态为 64 时，困惑度就和状态为 128 的 Mamba-2 相当。这就是摘要里「状态减半」的由来。\n\n### 检索：纯模型还是不够\n\n检索仍然是固定状态的弱点。1.5B 模型、截断到 2K tokens 的真实检索任务：\n\n| 模型（1.5B） | SWDE | SQuAD | FDA | TriviaQA |\n| --- | ---: | ---: | ---: | ---: |\n| Transformer | 48.9 | 46.6 | 58.4 | 67.5 |\n| Mamba-2 | 30.7 | 39.1 | 23.7 | 64.3 |\n| Mamba-3 SISO | 28.5 | 40.1 | 23.4 | 64.5 |\n| Mamba-3 MIMO | 36.3 | 41.7 | 29.3 | 64.5 |\n| 混合 Mamba-3 SISO | 58.5 | 47.0 | 65.9 | 64.8 |\n\n*表：原论文 Table 4 节选。混合模型是 5 层线性层配 1 层不用位置编码的注意力。*\n\n这些真实数据集都改写成截断到 2K tokens 的完形填空版本。问答类的 TriviaQA 差不多，但在 SWDE、FDA 这种要从半结构化网页或文档里抽取信息的任务上，纯 Mamba-3 SISO 分别落后 Transformer 20.4 与 35.0 分，MIMO 版本缩小到 12.6 与 29.1 分。加上注意力层的混合版本，在这些项目上就追平甚至超过了 Transformer。\n\n作者在这里写了一句很重要的话：由于固定大小的状态天生不擅长检索，**他们预计线性层将主要用于混合架构**，由平方复杂度的自注意力层来弥补这一弱点。\n\n这句话出自 Mamba 原作者自己的论文。和 2023 年相比，Mamba 的定位已经说得很清楚了。\n\n作者也承认，混合模型的行为不太符合直觉：归一化放在门控之前还是之后、用哪一种，会在长度泛化与真实检索之间相互拉扯，目前还没有定论。\n\n### 速度：注意比较条件\n\n解码 kernel 的延迟（Table 6）是在一张 H100 上、批大小 128、1.5B 模型下测的。bf16、状态 128 时，Mamba-3 SISO 每步 0.156 ms，Mamba-2 是 0.203 ms，GDN 是 0.257 ms；MIMO 是 0.179 ms，比 SISO 慢一点，仍比 Mamba-2 快。\n\n端到端的比较（Table 7）是预填充加上同样长度的解码。16K tokens 时，Mamba-3 SISO 约 141 秒，vLLM 上的 Llama-3.2-1B 约 977 秒，相差约 7 倍。新闻报道里「快 7 倍」的数字，应该就来自这两格。\n\n读这个数字要看清条件：\n\n- 比较对象是另一个模型（Llama-3.2-1B）跑在 vLLM 上，而不是同一套代码。\n- 批大小 128。附录说明，16K 时 vLLM 用批大小 128 会超出显存，所以实测用批大小 16，再乘以 8 推算。\n- 序列越短差距越小：512 tokens 时，两者只差 0.06 秒。\n\nHacker News 上，有读者不太买账「推理时多做计算是好事」这个结论：批大小为 1 时，解码确实卡在访存上，但服务商都会把请求凑成大批量一起算，算力未必闲着。另一位读者回应说，论文比较 Mamba-2 与 Mamba-3 的延迟时用的就是批大小 128，在这个批大小下 Mamba-2 看起来都还没吃满算力；质疑者又回复说，DeepSeek 的批大小大约是 8192，128 算不上大。也有人补充，本地推理的批大小通常是 1，闲着的算力正好可以拿来用。[Hacker News，2026-03-21](https://news.ycombinator.com/item?id=47419391) 这番来回大致点出了 MIMO 的适用范围：批大小越小、越受访存限制，它就越划算。\n\n## 结论\n\nMamba-3 从状态空间模型的老原理里，拿出了三个改动：\n\n1. **指数梯形离散化**：更有表达力的递归，配合 $B$、$C$ 的偏置项，让短卷积变得可有可无。\n2. **复数状态**：等价于由数据决定的 RoPE，能解出 Mamba-2 解不出的奇偶与模运算问题。\n3. **MIMO**：在受访存限制的解码阶段多做一些计算，质量变好，延迟几乎不变。\n\n在 1.5B、100B tokens 的比较中，它在语言建模上胜过 Transformer、Mamba-2 与 GDN；在另一组 440M、两倍 Chinchilla 最优 token 数的实验里，MIMO 版本只要一半的状态，就能达到 Mamba-2 的困惑度。\n\n但最值得记住的，或许是作者自己对检索的判断：固定状态在检索上的弱点，让他们预计线性层将主要用于混合架构。论文同时保留了纯模型的比较，也测试了它与注意力互补的效果。对照 2023 年 Mamba 论文期待它成为通用序列模型骨干的说法，重心明显移向了「怎样做好混合模型里的线性层」。\n\n主实验都在 1.5B 以下、100B tokens 的规模，状态大小与组件消融用的是更小的 440M 模型；混合模型也只试了 5:1 一种比例。它放进上百亿参数的混合模型之后表现如何，论文还没有回答。\n\n\n> [!TIP]\n> \n> Mamba 系列的前两代，以及同样处理状态更新的 Gated DeltaNet，可以参考：\n> \n> - **[[23.12] Mamba：谁是继承人](/blog/papers/mamba/mamba)**\n> - **[[24.05] Mamba-2：注意力的远亲](/blog/papers/mamba/mamba-2)**\n> - **[[24.12] Gated DeltaNet：先擦再写](/blog/papers/mamba/gated-deltanet)**\n","imageSizes":{"/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png":{"width":1356,"height":644},"/content/papers/assets/9c2757127eedcbc423ca747c871fa87a8c498a586a11782bf356497d5bf9510d.png":{"width":650,"height":345}}}