{"notes":[{"id":"mamba/mamba-3","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-3","subtitle":"Inference First","author":"Z. Yuan","paperDate":"2026-03-16","publishedOn":"2026-04-02","world":"external/mamba3","paperTitle":"Mamba-3: Improved Sequence Modeling using State Space Principles","paperUrl":"https://arxiv.org/abs/2603.15569","bibliography":{"title":"Mamba-3: Improved Sequence Modeling using State Space Principles","authors":["Aakash Lahoti","Kevin Y. Li","Berlin Chen","Caitlin Wang","Aviv Bick","J. Zico Kolter","Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2603.15569","pdfUrl":"https://arxiv.org/pdf/2603.15569v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2603.15569","submittedOn":"2026-03-16","url":"https://arxiv.org/abs/2603.15569"},"publication":{"venue":"ICLR","year":2026,"kind":"conference","url":"https://mlanthology.org/iclr/2026/lahoti2026iclr-mamba3/"},"citation":"@misc{mambamamba32026,\n  title = {{Mamba-3: Improved Sequence Modeling using State Space Principles}},\n  author = {Aakash Lahoti and Kevin Y. Li and Berlin Chen and Caitlin Wang and Aviv Bick and J. Zico Kolter and Tri Dao and Albert Gu},\n  year = {2026},\n  eprint = {2603.15569},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2603.15569}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-3.7f42c2a217f2.bib"},"thumbnail":{"src":"/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png","width":1356,"height":644},"locale":"en","path":"/content/papers/en/mamba/mamba-3.6ac69a9c8897.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"1-more-expressive-discretization","text":"1. More Expressive Discretization","depth":3},{"id":"2-complex-states-taking-a-spin","text":"2. Complex States: Taking a Spin","depth":3},{"id":"3-mimo-keeping-the-gpu-busier-during-decoding","text":"3. MIMO: Keeping the GPU Busier During Decoding","depth":3},{"id":"architecture","text":"Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"language-modeling","text":"Language Modeling","depth":3},{"id":"retrieval-pure-models-still-fall-short","text":"Retrieval: Pure Models Still Fall Short","depth":3},{"id":"speed-mind-the-conditions-of-the-comparison","text":"Speed: Mind the Conditions of the Comparison","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/mamba","category":"mamba","legacyPath":"mamba/2312-mamba","legacySlug":"2312-mamba","legacyPaths":[],"title":"Mamba","subtitle":"Who is the Successor","author":"Z. Yuan","paperDate":"2023-12-01","publishedOn":"2024-12-25","paperTitle":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","paperUrl":"https://arxiv.org/abs/2312.00752","bibliography":{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","authors":["Albert Gu","Tri Dao"],"url":"https://arxiv.org/abs/2312.00752","pdfUrl":"https://arxiv.org/pdf/2312.00752","codeUrl":null,"preprint":{"id":"2312.00752","submittedOn":"2023-12-01","url":"https://arxiv.org/abs/2312.00752"},"publication":null,"citation":"@misc{mambamamba2023,\n  title = {{Mamba: Linear-Time Sequence Modeling with Selective State Spaces}},\n  author = {Albert Gu and Tri Dao},\n  year = {2023},\n  eprint = {2312.00752},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2312.00752}\n}\n","citationPath":"/content/papers/citations/mamba/mamba.5671f143cfd6.bib"},"thumbnail":{"src":"/content/papers/assets/8f6187d5a2a7a17c0687f2ca08ea302601584bb95f5cd5f52927c54d6e76089a.jpg","width":1224,"height":500},"locale":"en","path":"/content/papers/en/mamba/mamba.53a616bdc015.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"selective-mechanism","text":"Selective Mechanism","depth":3},{"id":"s6-architecture","text":"S6 Architecture","depth":3},{"id":"mamba-architecture","text":"Mamba Architecture","depth":3},{"id":"revisiting-the-selective-mechanism","text":"Revisiting the Selective Mechanism","depth":3},{"id":"training-methods-and-protocols","text":"Training Methods and Protocols","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"scaling-laws","text":"Scaling Laws","depth":3},{"id":"comparison-with-other-language-models","text":"Comparison with Other Language Models","depth":3},{"id":"inference-efficiency-comparison","text":"Inference Efficiency Comparison","depth":3},{"id":"ablation-study---selective-mechanism","text":"Ablation Study - Selective Mechanism","depth":3},{"id":"ablation-study---key-parameters","text":"Ablation Study - Key Parameters","depth":3},{"id":"ablation-study---other-settings","text":"Ablation Study - Other Settings","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/s4","relation":"background","topic":"From state-space memory to visual sequences"},{"id":"mamba/vim","relation":"next","topic":"From state-space memory to visual sequences"}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"Attention's Distant Cousin","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"en","path":"/content/papers/en/mamba/mamba-2.84790bbee84c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"writing-an-ssm-as-a-matrix","text":"Writing an SSM as a Matrix","depth":3},{"id":"reducing-a-to-a-single-number","text":"Reducing A to a Single Number","depth":3},{"id":"working-through-four-tokens","text":"Working Through Four Tokens","depth":3},{"id":"half-one-way-half-the-other","text":"Half One Way, Half the Other","depth":3},{"id":"the-mamba-2-block","text":"The Mamba-2 Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"a-bigger-state-remembers-more","text":"A Bigger State Remembers More","depth":3},{"id":"language-modeling","text":"Language Modeling","depth":3},{"id":"the-original-authors-recommend-hybrids-too","text":"The Original Authors Recommend Hybrids Too","depth":3},{"id":"not-an-across-the-board-upgrade","text":"Not an Across-the-Board Upgrade","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"Erase, Then Write","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"en","path":"/content/papers/en/mamba/gated-deltanet.76d17f2230a0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"linear-attention-is-a-phone-book-where-entries-collide","text":"Linear Attention Is a Phone Book Where Entries Collide","depth":3},{"id":"mamba-2-everything-gets-discounted-together","text":"Mamba-2: Everything Gets Discounted Together","depth":3},{"id":"deltanet-change-only-that-entry","text":"DeltaNet: Change Only That Entry","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-update-rules-worked-through","text":"Three Update Rules, Worked Through","depth":3},{"id":"combining-the-two","text":"Combining the Two","depth":3},{"id":"making-it-trainable-in-parallel","text":"Making It Trainable in Parallel","depth":3},{"id":"architecture-and-hybrid-variants","text":"Architecture and Hybrid Variants","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"needle-in-a-haystack-how-the-three-rules-divide-the-work","text":"Needle in a Haystack: How the Three Rules Divide the Work","depth":3},{"id":"language-modeling-and-real-world-retrieval","text":"Language Modeling and Real-World Retrieval","depth":3},{"id":"speed","text":"Speed","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]}],"text":"More than two years after the first Mamba, Mamba-3 was published at ICLR 2026.\n\nThe author list brings in a new set of faces, still advised by Albert Gu and Tri Dao. The listed affiliations are Carnegie Mellon, Princeton, Together AI, and Cartesia AI: Tri Dao is affiliated with Princeton and Together AI, and Albert Gu with Carnegie Mellon and Cartesia AI.\n\nRead this paper's introduction and you'll notice a very different tone from 2023. Back then, Mamba was about whether it “could become the backbone of general sequence models”; this paper's first paragraph is about inference cost: chain-of-thought reasoning and agent workflows have made inference-time computation key to model performance. The authors then state the current situation outright: layers like Mamba-2 and Gated DeltaNet have already been built into large-scale **hybrid** models from Kimi, NVIDIA, Tencent Hunyuan, and Qwen.\n\nSo what this paper sets out to do is provide a better linear layer for those hybrid models.\n\n## Defining the Problem\n\nThe authors list three problems:\n\n1. **Mamba-2 sacrificed expressiveness for training speed**. At the same inference cost, Mamba-2 models are worse. Tri Dao had already mentioned this on his blog in 2024.\n2. **State tracking is out of reach**. Take a string of 0s and 1s and ask whether the number of 1s is odd or even: a single-layer LSTM can do this, but Mamba-2 can't. Earlier research has already proven why: the eigenvalues of the state transition are restricted to nonnegative real numbers.\n3. **The hardware sits idle during decoding**. These algorithms were all designed with training in mind. Decoding processes one token at a time, with very little computation, yet the entire state has to be read from memory and written back; the GPU spends most of its time waiting for data, and the arithmetic intensity (computation divided by memory traffic) is very low.\n\nTaking an “inference-first” view, the authors go back to the principles of state space models and offer a solution to each.\n\n## Solving the Problem\n\n### 1. More Expressive Discretization\n\nState space models are originally continuous-time systems; to process tokens one by one, they first have to be discretized. The authors begin by pointing out that the Mamba-1 and Mamba-2 papers write down one formula, while the actual code uses a different approximation. They supply a derivation for that approximation and call it **exponential-Euler**:\n\n$$\nh_t=e^{\\Delta_t A_t}h_{t-1}+\\Delta_t B_t x_t\n$$\n\nIt uses only the input at the right endpoint of the interval, which makes it a first-order approximation. Mamba-3 switches to a generalization of the trapezoidal rule that looks at both ends of the interval:\n\n$$\nh_t=\\alpha_t h_{t-1}+\\beta_t B_{t-1}x_{t-1}+\\gamma_t B_t x_t\n$$\n\nHere $\\alpha_t=e^{\\Delta_t A_t}$, $\\beta_t=(1-\\lambda_t)\\Delta_t e^{\\Delta_t A_t}$, and $\\gamma_t=\\lambda_t\\Delta_t$, with $\\lambda_t\\in[0,1]$ determined by the data. When $\\lambda_t=1$, it reduces to Mamba-2; when $\\lambda_t=\\tfrac12$, it approximates the integral of the (exponentially decayed) input by averaging its two endpoints, much like the trapezoidal rule. Reaching a higher order of error also requires conditions such as $\\lambda_t$ being close to $\\tfrac12$; the actual model lets $\\lambda_t$ be learned freely, mainly to make the recurrence more expressive.\n\nThis formula adds a term for the previous token's input, which amounts to applying a width-2 convolution to the SSM's input first. The authors found that, once learnable bias terms are also added to $B$ and $C$, the short convolution that almost every linear model includes can be removed:\n\n| Variant (440M) | Perplexity |\n| --- | ---: |\n| Mamba-3 without bias and trapezoid | 16.68 |\n| Mamba-3 without bias | 16.49 |\n| Mamba-3 | 15.72 |\n| Mamba-3 with short conv added back | 15.85 |\n\n*Table: Table 5a of the paper.*\n\n### 2. Complex States: Taking a Spin\n\nStart with the parity problem. If the state can “rotate,” things are simple: turn half a circle on reading a 1, stay put on reading a 0, and at the end see which side it stops on.\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\nbits = rng.integers(0, 2, size=1000)\n\ndef rot(theta):\n    c, s = np.cos(theta), np.sin(theta)\n    return np.array([[c, -s], [s, c]])\n\nh = np.array([1.0, 0.0])\nfor x in bits:\n    h = rot(np.pi * x) @ h          # half a turn on a 1, stay put on a 0\nprint(\"Rotating state:\", np.round(h, 6), \"→ parity\", 0 if h[0] > 0 else 1)\nprint(\"Direct count:  \", bits.sum() % 2)\n\n# Real, nonnegative decay: each step multiplies by a number between 0 and 1, so it can only shrink, never flip\nh = 1.0\nfor x in bits[:20]:\n    h = (0.5 if x else 0.9) * h\nprint(\"Nonnegative decay after 20 steps:\", round(h, 6))\n```\n\n```text\nRotating state: [-1.  0.] → parity 1\nDirect count:   1\nNonnegative decay after 20 steps: 3.2e-05\n```\n\nAfter a thousand bits, the rotating state lands exactly on the answer. By contrast, a nonnegative scalar decay like Mamba-2's can only multiply by a number between 0 and 1 at each step: the state just keeps getting smaller and never flips over to the other side.\n\n(The last part is only an illustration. The earlier theorem covers a more general case: as long as all the eigenvalues of the transition matrix are nonnegative real numbers, such linear RNNs cannot solve parity for arbitrary lengths with finite precision.)\n\nIn complex numbers, rotation means multiplying by $e^{i\\theta}$. The authors turn Mamba-3's SSM into a complex form and prove two things:\n\n1. A complex SSM can be rewritten as a real SSM of twice the dimension, whose transition matrix is the decay coefficient times a block diagonal made of 2×2 rotation matrices.\n2. Absorbing the rotation accumulated so far into $B$ and $C$ is equivalent to applying a **data-dependent rotary positional embedding** (RoPE).\n\nThe second point is very practical. RoPE already has highly mature, cheap implementations in Transformers, and Mamba-3 can reuse them directly at very little extra cost. The difference is that ordinary RoPE rotates each position by a fixed angle, while in Mamba-3 the angle is determined by the input.\n\nResults on formal-language tasks (scaled accuracy, %):\n\n| Model | Parity | Modular arithmetic (no brackets) | Modular arithmetic (with brackets) |\n| --- | ---: | ---: | ---: |\n| Mamba-3 | 100.00 | 98.51 | 87.75 |\n| Mamba-3 (with standard RoPE) | 1.56 | 20.70 | 2.62 |\n| Mamba-3 (no RoPE) | 2.27 | 1.49 | 0.72 |\n| Mamba-2 | 0.90 | 47.81 | 0.88 |\n| GDN (eigenvalue range [−1, 1]) | 100.00 | 99.25 | 93.50 |\n\n*Table: Table 5b of the paper.*\n\nThis test trains on short sequences and evaluates on longer ones. Among the Mamba-3 variants, only the version with data-dependent angles does well on all three; the fixed-angle RoPE and no-RoPE versions are mostly near zero, with only the fixed-angle version reaching 20.70 on modular arithmetic without brackets. (The table reports scaled accuracy, where 0 roughly corresponds to random guessing.) Mamba-2 partly succeeds on modular arithmetic without brackets (47.81), while its parity and bracketed modular arithmetic scores are near zero.\n\nThe last row of the table is also worth a look: Gated DeltaNet, which allows negative eigenvalues, can solve these tasks too, and does even better on modular arithmetic with brackets. Complex numbers aren't the only road to state tracking; negative eigenvalues are another way to flip over.\n\n### 3. MIMO: Keeping the GPU Busier During Decoding\n\nThe original SSM performs a **single-input, single-output** (SISO) update at each step: the input takes one outer product with $B$, adding a rank-1 matrix to the state. During decoding, each step has to read and write the entire state, yet the computation is tiny.\n\nMamba-3 replaces this with **multi-input, multi-output** (MIMO): the input and $B$ each get $R$ branches, so every step writes an update of rank at most $R$, and the readout also uses $R$ sets of $C$, while the state size stays the same. When $R$ is much smaller than the state dimension and the head dimension, the computation in the SSM's main operations grows roughly in proportion to $R$, while memory traffic is still dominated by that fixed-size state, so the arithmetic intensity rises along with it. Decoding is memory-bound to begin with, so the extra computation essentially puts the GPU's idle time to use.\n\nThe paper uses $R=4$. To keep the parameter count equal to the SISO version, the MLP's intermediate size in the 1.5B model shrinks from 4096 to 3824. MIMO is slower in training; the authors say it can take up to $R$ times as long as SISO.\n\n### Architecture\n\n[![Two blocks side by side. On the left, the Mamba-2 block: the input is projected into X, B, C, and the parameters that determine decay (labeled A in the figure); X passes through a convolution and an activation function into the SSM, whose output is multiplied by a gate, normalized, and then passed through the output projection. On the right, the Mamba-3 block: there is no convolution; B and C are each normalized first and then go through RoPE (with the angle Θ produced by the input projection), and there is also an optional MIMO projection](/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png)](/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png)\n\n*Figure 2 from the paper, excerpted from [**page 11 of the v1 PDF**](https://arxiv.org/pdf/2603.15569v1#page=11), CC BY 4.0.*\n\nComparing the two blocks:\n\n- **The short convolution is gone**, replaced by the trapezoidal rule and the bias terms.\n- **$B$ and $C$ are normalized first**, which works much like QK normalization in Transformers.\n- **RoPE is added**, with the angle $\\Theta$ produced from the input; this is how the complex state is implemented.\n- **The yellow MIMO projection** is optional.\n- Pure Mamba-3 drops the normalization layer that Mamba-2 had before the output; when Mamba-3 is placed in a hybrid model, the authors found that adding it back helps length generalization.\n\nThe overall skeleton follows Llama, with Mamba-3 blocks alternating with SwiGLU.\n\n## Discussion\n\n### Language Modeling\n\nAll models are trained on 100B tokens of FineWeb-Edu with a 2K context, at four scales from 180M to 1.5B. The 1.5B results:\n\n| Model | FineWeb-Edu perplexity | Seven-task average accuracy |\n| --- | ---: | ---: |\n| Transformer | 10.51 | 55.4 |\n| GDN | 10.45 | 55.8 |\n| Mamba-2 | 10.47 | 55.7 |\n| Mamba-3 SISO | 10.35 | 56.4 |\n| Mamba-3 MIMO ($R=4$) | 10.24 | 57.6 |\n\n*Table: Excerpted from Table 3 of the paper.*\n\nSISO is 0.6 points above the runner-up, GDN, and MIMO adds another 1.2 points. At all four scales, Mamba-3 SISO beats Mamba-2.\n\n[![Line chart: the x-axis is relative total state size (log scale), and the y-axis is pretraining perplexity. The three lines, from top to bottom, are Mamba-2, Mamba-3, and Mamba-3 MIMO; the larger the state, the lower the perplexity, and the Mamba-3 MIMO line is lowest along its entire length](/content/papers/assets/9c2757127eedcbc423ca747c871fa87a8c498a586a11782bf356497d5bf9510d.png)](/content/papers/assets/9c2757127eedcbc423ca747c871fa87a8c498a586a11782bf356497d5bf9510d.png)\n\n*Figure 3 from the paper, excerpted from [**page 15 of the v1 PDF**](https://arxiv.org/pdf/2603.15569v1#page=15), CC BY 4.0. 440M models, trained to twice the Chinchilla-optimal number of tokens.*\n\nDecoding speed is determined mainly by state size, so the authors use state size as a proxy for inference speed. In the figure, at every state size, Mamba-3 has lower perplexity than Mamba-2; the MIMO version with a state size of 64 already matches the perplexity of Mamba-2 with a state size of 128. That's where the abstract's “half of its predecessor's state size” comes from.\n\n### Retrieval: Pure Models Still Fall Short\n\nRetrieval remains the weak spot of a fixed state. Here are real-world retrieval tasks for the 1.5B models, truncated to 2K tokens:\n\n| Model (1.5B) | SWDE | SQuAD | FDA | TriviaQA |\n| --- | ---: | ---: | ---: | ---: |\n| Transformer | 48.9 | 46.6 | 58.4 | 67.5 |\n| Mamba-2 | 30.7 | 39.1 | 23.7 | 64.3 |\n| Mamba-3 SISO | 28.5 | 40.1 | 23.4 | 64.5 |\n| Mamba-3 MIMO | 36.3 | 41.7 | 29.3 | 64.5 |\n| Hybrid Mamba-3 SISO | 58.5 | 47.0 | 65.9 | 64.8 |\n\n*Table: Excerpted from Table 4 of the paper. The hybrid pairs 5 linear layers with 1 attention layer that uses no positional encoding.*\n\nThese real-world datasets are all rewritten as cloze versions truncated to 2K tokens. On question answering like TriviaQA, the models are about even, but on tasks like SWDE and FDA, which require extracting information from semi-structured web pages or documents, pure Mamba-3 SISO trails the Transformer by 20.4 and 35.0 points respectively, and the MIMO version narrows the gaps to 12.6 and 29.1. The hybrid version, with attention layers added, catches up with or surpasses the Transformer on these items.\n\nHere the authors write a very important sentence: “Because of the natural retrieval-based weaknesses of fixed state-size, **we predict that linear layers will be predominantly used in hybrid architectures** that mitigate this downside with quadratic self-attention layers.”\n\nThat sentence comes from a paper by Mamba's own original authors. Compared with 2023, Mamba's role has now been spelled out clearly.\n\nThe authors also admit that hybrid models behave somewhat unintuitively: whether the normalization goes before or after the gate, and which kind is used, pulls length generalization and real-world retrieval in opposite directions, and there's no settled answer yet.\n\n### Speed: Mind the Conditions of the Comparison\n\nDecoding-kernel latency (Table 6) is measured on one H100 at batch size 128 with 1.5B models. In bf16 with a state size of 128, Mamba-3 SISO takes 0.156 ms per step, Mamba-2 0.203 ms, and GDN 0.257 ms; MIMO takes 0.179 ms, slightly slower than SISO but still faster than Mamba-2.\n\nThe end-to-end comparison (Table 7) covers prefill plus decoding of the same length. At 16K tokens, Mamba-3 SISO takes about 141 seconds and Llama-3.2-1B on vLLM about 977 seconds, a difference of roughly 7 times. The “7 times faster” figure in news coverage most likely comes from these two cells.\n\nRead this number with its conditions in view:\n\n- The comparison is against a different model (Llama-3.2-1B) running on vLLM, not the same codebase.\n- The batch size is 128. The appendix explains that at 16K, vLLM runs out of memory at batch size 128, so it was measured at batch size 16 and multiplied by 8 as an estimate.\n- The shorter the sequence, the smaller the gap: at 512 tokens, the two differ by only 0.06 seconds.\n\nOn Hacker News, one reader wasn't sold on the conclusion that “doing more computation at inference time is a good thing”: decoding at batch size 1 is indeed memory-bound, but providers group requests into large batches and compute them together, so compute isn't necessarily sitting idle. Another reader responded that the paper's latency comparison between Mamba-2 and Mamba-3 used exactly that batch size of 128, and that at this batch size Mamba-2 didn't seem to be using all its compute yet; the skeptic replied that DeepSeek's batch sizes are around 8192, so 128 is hardly large. Someone else added that local inference usually runs at batch size 1, so the idle compute is there for the taking. [Hacker News, 2026-03-21](https://news.ycombinator.com/item?id=47419391) This back-and-forth roughly maps out where MIMO applies: the smaller the batch and the more memory-bound the workload, the more it pays off.\n\n## Conclusion\n\nMamba-3 draws three changes out of the old principles of state space models:\n\n1. **Exponential-trapezoidal discretization**: a more expressive recurrence that, together with bias terms on $B$ and $C$, makes the short convolution optional.\n2. **Complex states**: equivalent to data-dependent RoPE, solving the parity and modular arithmetic tasks that Mamba-2 can't.\n3. **MIMO**: doing more computation during memory-bound decoding, for better quality at almost the same latency.\n\nIn the comparison at 1.5B parameters and 100B tokens, it beats the Transformer, Mamba-2, and GDN at language modeling; in another experiment at 440M parameters with twice the Chinchilla-optimal number of tokens, the MIMO version reaches Mamba-2's perplexity with only half the state.\n\nBut perhaps the thing most worth remembering is the authors' own judgment on retrieval: the retrieval weakness of a fixed state leads them to predict that linear layers will be used mainly in hybrid architectures. The paper still keeps its comparisons of pure models, and it also tests how well the layer complements attention. Set against the 2023 Mamba paper's hope that it would become the backbone of general sequence models, the focus has clearly shifted toward “how to build a good linear layer for hybrid models.”\n\nThe main experiments are all at 1.5B parameters or below with 100B tokens, and the state-size and component ablations use an even smaller 440M model; the hybrid was also tried at only one ratio, 5:1. How it performs once placed in hybrid models with tens of billions of parameters is a question the paper has yet to answer.\n\n\n> [!TIP]\n> \n> For the first two generations of the Mamba series, and for Gated DeltaNet, which also works on how the state is updated, see:\n> \n> - **[[23.12] Mamba: Who is the Successor](/blog/papers/mamba/mamba)**\n> - **[[24.05] Mamba-2: Attention's Distant Cousin](/blog/papers/mamba/mamba-2)**\n> - **[[24.12] Gated DeltaNet: Erase, Then Write](/blog/papers/mamba/gated-deltanet)**\n","imageSizes":{"/content/papers/assets/bafe70ab05ff3f4100278e18abb0198af0081a8802873a746e8df5fef028e047.png":{"width":1356,"height":644},"/content/papers/assets/9c2757127eedcbc423ca747c871fa87a8c498a586a11782bf356497d5bf9510d.png":{"width":650,"height":345}}}