{"notes":[{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"Attention's Distant Cousin","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"en","path":"/content/papers/en/mamba/mamba-2.84790bbee84c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"writing-an-ssm-as-a-matrix","text":"Writing an SSM as a Matrix","depth":3},{"id":"reducing-a-to-a-single-number","text":"Reducing A to a Single Number","depth":3},{"id":"working-through-four-tokens","text":"Working Through Four Tokens","depth":3},{"id":"half-one-way-half-the-other","text":"Half One Way, Half the Other","depth":3},{"id":"the-mamba-2-block","text":"The Mamba-2 Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"a-bigger-state-remembers-more","text":"A Bigger State Remembers More","depth":3},{"id":"language-modeling","text":"Language Modeling","depth":3},{"id":"the-original-authors-recommend-hybrids-too","text":"The Original Authors Recommend Hybrids Too","depth":3},{"id":"not-an-across-the-board-upgrade","text":"Not an Across-the-Board Upgrade","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]},{"id":"mamba/mamba","category":"mamba","legacyPath":"mamba/2312-mamba","legacySlug":"2312-mamba","legacyPaths":[],"title":"Mamba","subtitle":"Who is the Successor","author":"Z. Yuan","paperDate":"2023-12-01","publishedOn":"2024-12-25","paperTitle":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","paperUrl":"https://arxiv.org/abs/2312.00752","bibliography":{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","authors":["Albert Gu","Tri Dao"],"url":"https://arxiv.org/abs/2312.00752","pdfUrl":"https://arxiv.org/pdf/2312.00752","codeUrl":null,"preprint":{"id":"2312.00752","submittedOn":"2023-12-01","url":"https://arxiv.org/abs/2312.00752"},"publication":null,"citation":"@misc{mambamamba2023,\n  title = {{Mamba: Linear-Time Sequence Modeling with Selective State Spaces}},\n  author = {Albert Gu and Tri Dao},\n  year = {2023},\n  eprint = {2312.00752},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2312.00752}\n}\n","citationPath":"/content/papers/citations/mamba/mamba.5671f143cfd6.bib"},"thumbnail":{"src":"/content/papers/assets/8f6187d5a2a7a17c0687f2ca08ea302601584bb95f5cd5f52927c54d6e76089a.jpg","width":1224,"height":500},"locale":"en","path":"/content/papers/en/mamba/mamba.53a616bdc015.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"selective-mechanism","text":"Selective Mechanism","depth":3},{"id":"s6-architecture","text":"S6 Architecture","depth":3},{"id":"mamba-architecture","text":"Mamba Architecture","depth":3},{"id":"revisiting-the-selective-mechanism","text":"Revisiting the Selective Mechanism","depth":3},{"id":"training-methods-and-protocols","text":"Training Methods and Protocols","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"scaling-laws","text":"Scaling Laws","depth":3},{"id":"comparison-with-other-language-models","text":"Comparison with Other Language Models","depth":3},{"id":"inference-efficiency-comparison","text":"Inference Efficiency Comparison","depth":3},{"id":"ablation-study---selective-mechanism","text":"Ablation Study - Selective Mechanism","depth":3},{"id":"ablation-study---key-parameters","text":"Ablation Study - Key Parameters","depth":3},{"id":"ablation-study---other-settings","text":"Ablation Study - Other Settings","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/s4","relation":"background","topic":"From state-space memory to visual sequences"},{"id":"mamba/vim","relation":"next","topic":"From state-space memory to visual sequences"}]},{"id":"mamba/gated-deltanet","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Gated DeltaNet","subtitle":"Erase, Then Write","author":"Z. Yuan","paperDate":"2024-12-09","publishedOn":"2026-02-12","world":"external/gated-deltanet","paperTitle":"Gated Delta Networks: Improving Mamba2 with Delta Rule","paperUrl":"https://arxiv.org/abs/2412.06464","bibliography":{"title":"Gated Delta Networks: Improving Mamba2 with Delta Rule","authors":["Songlin Yang","Jan Kautz","Ali Hatamizadeh"],"url":"https://arxiv.org/abs/2412.06464","pdfUrl":"https://arxiv.org/pdf/2412.06464v3","codeUrl":"https://github.com/NVlabs/GatedDeltaNet","preprint":{"id":"2412.06464","submittedOn":"2024-12-09","url":"https://arxiv.org/abs/2412.06464"},"publication":{"venue":"ICLR","year":2025,"kind":"conference","url":"https://mlanthology.org/iclr/2025/yang2025iclr-gated/"},"citation":"@misc{mambagateddeltanet2024,\n  title = {{Gated Delta Networks: Improving Mamba2 with Delta Rule}},\n  author = {Songlin Yang and Jan Kautz and Ali Hatamizadeh},\n  year = {2024},\n  eprint = {2412.06464},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.06464}\n}\n","citationPath":"/content/papers/citations/mamba/gated-deltanet.b65d8f168c74.bib"},"thumbnail":{"src":"/content/papers/assets/ffdf63101911e19f771ec701520a03d60259e0007937a0626ccebb8bc77c0f15.png","width":1050,"height":544},"locale":"en","path":"/content/papers/en/mamba/gated-deltanet.76d17f2230a0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"linear-attention-is-a-phone-book-where-entries-collide","text":"Linear Attention Is a Phone Book Where Entries Collide","depth":3},{"id":"mamba-2-everything-gets-discounted-together","text":"Mamba-2: Everything Gets Discounted Together","depth":3},{"id":"deltanet-change-only-that-entry","text":"DeltaNet: Change Only That Entry","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-update-rules-worked-through","text":"Three Update Rules, Worked Through","depth":3},{"id":"combining-the-two","text":"Combining the Two","depth":3},{"id":"making-it-trainable-in-parallel","text":"Making It Trainable in Parallel","depth":3},{"id":"architecture-and-hybrid-variants","text":"Architecture and Hybrid Variants","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"needle-in-a-haystack-how-the-three-rules-divide-the-work","text":"Needle in a Haystack: How the Three Rules Divide the Work","depth":3},{"id":"language-modeling-and-real-world-retrieval","text":"Language Modeling and Real-World Retrieval","depth":3},{"id":"speed","text":"Speed","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba-2","relation":"referenced","topic":""},{"id":"mamba/kimi-linear","relation":"referenced","topic":""}]}],"text":"The title of this paper is a little provocative: Transformers are SSMs.\n\nIt's actually a tribute. In 2020, Katharopoulos et al. published a paper called “Transformers are RNNs,” showing that linear attention, with the softmax removed, can be rewritten as an RNN. Tri Dao and Albert Gu also admit in a footnote that the title only holds for “certain flavors” of attention.\n\nSix months earlier, they had used Mamba to show that state space models could trade blows with Transformers. This paper (later published at ICML 2024) heads in a different direction: the two kinds of models are actually relatives, so what can that relationship be used for?\n\n## Defining the Problem\n\nThe authors' frustration comes from the ecosystem.\n\nTransformers have an entire community studying them, building up layer upon layer of work: theoretical analysis, hardware optimizations like FlashAttention, tensor parallelism, sequence parallelism. On the SSM side, people have mostly been figuring things out on their own, and those results are hard to apply directly.\n\nMore concretely, Mamba's selective scan has two practical problems:\n\n1. **It can't use the matrix-multiplication units**. The fastest units on a GPU are tensor cores, which do nothing but matrix multiplication. Mamba's recurrence can be computed with a parallel (associative) scan, but most of the work in a scan is elementwise and can't use tensor cores.\n2. **The state can't grow much**. A larger state dimension $N$ remembers more, but it also slows the scan. Mamba defaults to $N=16$.\n\nWhat the authors want is to find a correspondence between SSMs and attention, so that SSMs can be computed with matrix multiplication and can directly reuse the systems engineering built up around Transformers.\n\n## Solving the Problem\n\n### Writing an SSM as a Matrix\n\nFirst, recall the recurrence of a selective SSM. The parameters at each step are determined by the input:\n\n$$\nh_t=A_t h_{t-1}+B_t x_t,\\qquad y_t=C_t^\\top h_t\n$$\n\nSet the initial state to zero. Unrolling the recurrence, the output at step $j$ is a weighted sum of all inputs up to that step:\n\n$$\ny_j=\\sum_{i\\le j} C_j^\\top A_j A_{j-1}\\cdots A_{i+1} B_i\\, x_i\n$$\n\nSo the input–output map over the whole sequence is a single matrix multiplication $y=Mx$, where\n\n$$\nM_{ji}=C_j^\\top A_{j:i} B_i,\\qquad A_{j:i}=A_j\\cdots A_{i+1}\n$$\n\n[![On the left, an input sequence of length T passes through a lower-triangular T×T sequence transformation matrix M to produce the output; on the right, each entry of M is written as C transposed times a chain of A's times B, and a submatrix below the main diagonal is outlined with a dashed box](/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png)](/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png)\n\n*Figure 2 from the paper, excerpted from [**page 9 of the v1 PDF**](https://arxiv.org/pdf/2405.21060v1#page=9), CC BY 4.0.*\n\nThis $M$ is lower triangular and has a special structure: take any submatrix below the diagonal (the blue dashed box in the figure), and its rank is at most the state dimension $N$. In mathematics, matrices like this are called **semiseparable matrices**.\n\nThe authors' core observation is that **the various ways of computing an SSM are all different algorithms for multiplying by this matrix**. Step-by-step recurrence is one way to do the multiplication; computing the full $M$ and then multiplying is another.\n\n### Reducing $A$ to a Single Number\n\nMamba's $A_t$ is a diagonal matrix. Mamba-2 simplifies it one step further to $A_t=a_t I$, using a single scalar $a_t\\in[0,1]$ at each step to control the decay. This version is called **SSD** (state space dual).\n\nWith that, $A_{j:i}$ becomes a product of scalars, which can be pulled out of the matrix:\n\n$$\nM=L\\circ(CB^\\top),\\qquad\nL_{ji}=\\begin{cases}a_j\\times\\cdots\\times a_{i+1}, & j\\ge i\\\\ 0, & j<i\\end{cases}\n$$\n\nNow compare it with causal attention:\n\n$$\nY=\\text{softmax}\\big(\\text{mask}(QK^\\top)\\big)V\n$$\n\nTreat $C$ as $Q$, $B$ as $K$, and $x$ as $V$, and the two look almost identical. They differ in two places:\n\n1. **There's no softmax**.\n2. **The mask isn't just 0s and 1s**. An ordinary causal mask only says whether a position “can be seen”; $L$ also says “how much”: the farther away, the more decay gets multiplied in, and the decay at each step is determined by the input.\n\nThe authors interpret the second point as $L$ taking the place of the Transformer's positional encoding, becoming a data-dependent positional mask.\n\n### Working Through Four Tokens\n\nThe formulas alone don't give much of a feel for this, so let's take a tiny example with four tokens and state dimension $N=2$, and compute it both ways:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=3, suppress=True)\n\nx = np.array([1.0, 2.0, 3.0, 4.0])          # 4 tokens, 1 dimension each\na = np.array([0.9, 0.5, 0.8, 0.7])          # decay at each step, set by the input\nB = np.array([[1, 0], [0, 1], [1, 1], [1, -1]], dtype=float)   # write directions, N=2\nC = np.array([[1, 1], [1, 0], [0, 1], [1, 1]], dtype=float)    # read directions\n\n# Linear (recurrent) form: carry a 2-dimensional state forward\nh = np.zeros(2); y_rec = []\nfor t in range(4):\n    h = a[t] * h + B[t] * x[t]\n    y_rec.append(C[t] @ h)\nprint(\"Recurrence:\", np.array(y_rec))\n\n# Quadratic (attention) form: build the 4×4 matrix first, then multiply by x\nL = np.zeros((4, 4))\nfor i in range(4):\n    for j in range(i + 1):\n        L[i, j] = np.prod(a[j + 1:i + 1])   # a_i × … × a_{j+1}; equals 1 when i=j\nM = L * (C @ B.T)\nprint(\"Mask L:\\n\", L)\nprint(\"Matrix:    \", M @ x)\n```\n\n```text\nRecurrence: [1.  0.5 4.6 5.6]\nMask L:\n [[1.   0.   0.   0.  ]\n [0.5  1.   0.   0.  ]\n [0.4  0.8  1.   0.  ]\n [0.28 0.56 0.7  1.  ]]\nMatrix:     [1.  0.5 4.6 5.6]\n```\n\nBoth methods give the same answer.\n\nThe recurrent route carries only 2 numbers forward at a time, so memory stays fixed, which suits generation. The matrix route has to build the 4×4 table first, and for a sequence of length $T$ that table is $T\\times T$, but it's all matrix multiplication, which tensor cores compute very quickly, so it suits training.\n\nLook at the last row of the mask: when the 4th token looks at the 1st, it gets multiplied by $0.5\\times0.8\\times0.7=0.28$. If even one step in between has a small $a_t$, whatever came before is almost completely forgotten. This is what Mamba's “selectivity” looks like on the matrix.\n\n### Half One Way, Half the Other\n\nThe actual SSD algorithm splits the sequence into fixed-length chunks:\n\n- **Within each chunk**, it uses the matrix form, computing a small quadratic matrix on the tensor cores.\n- **Between chunks**, it uses the recurrent form, passing only the state at the end of each chunk on to the next.\n\nThis amounts to a block decomposition of $M$: the small blocks on the diagonal are computed directly, and the part below the diagonal, being low-rank, can be represented through the state. Listing 1 of the paper gives a complete implementation in just a short stretch of PyTorch.\n\nFigure 10 of the paper benchmarks the core operation on an A100 80GB PCIe: with the state dimension at 64 for both, SSD is 2 to 8 times faster than Mamba's fused scan; against FlashAttention-2, it breaks even at a sequence length of about 2K and is 6 times faster at 16K. And because it computes so fast, the state dimension can grow from Mamba's default of 16 to 64 or 256.\n\n### The Mamba-2 Block\n\n[![Two blocks side by side. On the left, the Sequential Mamba Block first produces X through a convolution and an activation function, then projects A, B, and C from X and feeds them into the SSM; on the right, the Parallel Mamba Block produces A, X, B, and C at the start of the block with a single linear projection, and after the SSM output is multiplied by the gate there is an extra normalization layer N before the output projection](/content/papers/assets/e422ea301e9616b502680c01dc5005caf009df35935c9aded0a0c868f52dc2bd.png)](/content/papers/assets/e422ea301e9616b502680c01dc5005caf009df35935c9aded0a0c868f52dc2bd.png)\n\n*Figure 6 from the paper, excerpted from [**page 23 of the v1 PDF**](https://arxiv.org/pdf/2405.21060v1#page=23), CC BY 4.0. Mamba's block is on the left, Mamba-2's on the right.*\n\nOnce you see SSMs as relatives of attention, block design can also borrow from attention's experience:\n\n1. **Parallel projections**. Mamba first computes $X$, then uses $X$ to project $B$, $C$, and $\\Delta$; Mamba-2 produces $A$, $X$, $B$, and $C$ with a single projection at the start of the block (in the figure, $A$ refers to the discretized decay, whose data dependence comes from the step size $\\Delta$ projected at the same time), just as attention produces $Q$, $K$, and $V$ all at once. This uses slightly fewer parameters and is friendlier to tensor parallelism, halving the number of synchronization points per block.\n2. **Multiple heads**. Mamba effectively has one head per channel (head dimension 1); Mamba-2 uses a head dimension of $P=64$ or 128, following Transformer convention. $B$ and $C$ are shared across heads, similar to several value heads sharing keys and queries.\n3. **An extra normalization layer**. In preliminary experiments, larger models were prone to unstable training, so the authors added a normalization layer after the gate and before the output projection.\n\n## Discussion\n\n### A Bigger State Remembers More\n\n[![Three line charts showing multi-query associative recall (MQAR) accuracy at sequence lengths 256, 512, and 1024, with model dimension from 32 to 256 on the x-axis. Mamba (N=16) is near 0 almost everywhere; Mamba-2 (N=16) rises as the model grows; Mamba-2 (N=64) and N=256 are close to 1 in most settings; attention is around 0.5 at length 1024 and dimension 128](/content/papers/assets/62bb9a2f1e1e9d3f11500404928dcac9606819430521e5da6b90659f81b5925f.png)](/content/papers/assets/62bb9a2f1e1e9d3f11500404928dcac9606819430521e5da6b90659f81b5925f.png)\n\n*Figure 8 from the paper, excerpted from [**page 28 of the v1 PDF**](https://arxiv.org/pdf/2405.21060v1#page=28), CC BY 4.0.*\n\nMQAR (multi-query associative recall) is a synthetic task: the model first sees a list of key–value pairs, then gets asked about many keys at once. It's a lot like looking up numbers in a phone book, and it has long been a weak spot for state models.\n\nA few things in the figure are clear:\n\n- Mamba ($N=16$) fails almost across the board.\n- Mamba-2 with the same $N=16$ does much better. The authors themselves aren't sure which part of the architecture makes the difference.\n- As $N$ goes from 16 to 64 and 256, accuracy keeps climbing; at length 1024, the $N=256$ version even beats standard attention.\n\nThe larger the state, the more pairs it can hold. SSD makes large states affordable, and that is exactly why it matters.\n\nKeep in mind that the models here have only two layers and a dimension of at most 256: this is a very small synthetic experiment.\n\n### Language Modeling\n\nTrained on 300B tokens of the Pile, Mamba-2 2.7B has a zero-shot average of 60.2, above Mamba 2.8B's 59.9 and Pythia 2.8B's 55.7, and even above Pythia 6.9B, more than twice its size, at 58.3 (Mamba-2 numbers from Table 1 of the paper; Pythia 6.9B from Table 10 in the appendix). A separate set of scaling-law experiments covers 125M to 1.3B parameters and 2.5B to 26B training tokens; in that setting, Mamba-2 beats the authors' Transformer++ baseline in perplexity, theoretical FLOPs, and actual training time.\n\n### The Original Authors Recommend Hybrids Too\n\nOne part of this paper is often overlooked: the authors themselves tried mixing SSD layers with attention layers.\n\nThey trained a 350M-parameter, 48-layer model on 7B tokens of the Pile, varying how many of its layers were attention:\n\n| Attention layers | 0 (pure Mamba-2) | 1 | 3 | 6 | 11 | 24 | Pure Transformer++ |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| Perplexity | 8.60 | 8.38 | 8.29 | 8.26 | 8.30 | 8.50 | 8.68 |\n\n*Table: Excerpted from Table 2 of the paper.*\n\nAdding just one attention layer drops perplexity from 8.60 to 8.38; 6 layers out of 48 (12.5%) is best, which the authors summarize as “around 10%.” Why does mixing work better? The authors offer a conjecture: the two kinds of layers divide up the work of memory. SSD keeps organizing the preceding text in a fixed-size state, while attention keeps open a path for looking back token by token, so the model doesn't have to cram every detail into the state.\n\nScaled up to 2.7B parameters, 64 layers, and 300B tokens, Mamba-2-Attention, with 58 SSD layers plus 6 attention layers, reaches a Pile perplexity of 5.95 and a zero-shot average of 61.0, better than both pure Mamba-2 and Transformer++ at 60.2.\n\n### Not an Across-the-Board Upgrade\n\nThe authors candidly flag limitations in several places:\n\n- **The 2 to 8 times speedup is relative to Mamba's scan**. At the level of the whole model, for short sequences like 2K, Mamba-2 doesn't necessarily train faster than a Transformer, because at the same parameter count half of a Transformer consists of highly hardware-efficient MLP layers.\n- **A little less expressive power**. Reducing $A$ from a diagonal matrix to a scalar trades expressiveness for training efficiency. Tri Dao puts it more bluntly on his blog: Mamba-2 isn't strictly better than Mamba-1, because inference speed depends only on state size, and at the same state size Mamba-1 may be more expressive. [Tri Dao's blog, Mamba-2 Part I](https://tridao.me/blog/2024/mamba2-part1-model/)\n\nIn the Hacker News discussion, one commenter boiled the paper down to a sentence: restrict $A$ to a scalar times the identity, and the SSM can be written as causal linear attention, with the benefit of much faster training on GPUs. Someone else simply replied, “tldr: mamba is not as good as transformer.” [Hacker News, 2024-06-03](https://news.ycombinator.com/item?id=40564067) The first summary accurately captures the mechanism of the duality and its training benefit; the second verdict never says on which tasks, or under what conditions, the comparison was made. The paper's trade-off is actually explicit: it limits the expressiveness of $A$ in exchange for hardware efficiency, and that efficiency in turn lets the state grow, with a benefit that shows up on MQAR.\n\n## Conclusion\n\nWhat Mamba-2 leaves behind can roughly be grouped into three levels:\n\n1. **Theory**: an SSM whose state transition is a scalar times the identity (SSD) and linear attention with a semiseparable mask are two algorithms for the same matrix. The authors also prove that, within the masked attention defined in the paper, any mask with a finite-order linear recurrence form must be a semiseparable matrix, which is to say some kind of SSM.\n2. **Engineering**: the block-decomposition algorithm lets SSMs use tensor cores, making the core operation 2 to 8 times faster than Mamba's scan; the state can grow to 64 or 256, and tensor and sequence parallelism apply directly.\n3. **Architecture**: the authors' own experiments show that SSD layers combined with around 10% attention layers beat both pure Mamba-2 and pure Transformers.\n\nWith Mamba, the question was “Who is the Successor.” This paper's answer is more like: the successor was family all along, and the best arrangement is to have it live under the same roof as the original head of the house.\n\nThe later Gated DeltaNet builds directly on this framework, joining Mamba-2's scalar decay to a different update rule.\n\n\n> [!TIP]\n> \n> For Mamba before SSD, and for a later extension of this framework, see:\n> \n> - **[[23.12] Mamba: Who is the Successor](/blog/papers/mamba/mamba)**\n> - **[[24.12] Gated DeltaNet: Erase, Then Write](/blog/papers/mamba/gated-deltanet)**\n","imageSizes":{"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png":{"width":1462,"height":556},"/content/papers/assets/e422ea301e9616b502680c01dc5005caf009df35935c9aded0a0c868f52dc2bd.png":{"width":1208,"height":623},"/content/papers/assets/62bb9a2f1e1e9d3f11500404928dcac9606819430521e5da6b90659f81b5925f.png":{"width":1384,"height":384}}}