{"notes":[{"id":"mamba/mamba-2-hybrid","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2-Hybrid","subtitle":"Fuzzy Memory","author":"Z. Yuan","paperDate":"2024-06-12","publishedOn":"2026-01-22","paperTitle":"An Empirical Study of Mamba-based Language Models","paperUrl":"https://arxiv.org/abs/2406.07887","bibliography":{"title":"An Empirical Study of Mamba-based Language Models","authors":["Roger Waleffe","Wonmin Byeon","Duncan Riach","Brandon Norick","Vijay Korthikanti","Tri Dao","Albert Gu","Ali Hatamizadeh","Sudhakar Singh","Deepak Narayanan","Garvit Kulshreshtha","Vartika Singh","Jared Casper","Jan Kautz","Mohammad Shoeybi","Bryan Catanzaro"],"url":"https://arxiv.org/abs/2406.07887","pdfUrl":"https://arxiv.org/pdf/2406.07887v1","codeUrl":"https://github.com/NVIDIA/Megatron-LM","preprint":{"id":"2406.07887","submittedOn":"2024-06-12","url":"https://arxiv.org/abs/2406.07887"},"publication":null,"citation":"@misc{mambamamba2hybrid2024,\n  title = {{An Empirical Study of Mamba-based Language Models}},\n  author = {Roger Waleffe and Wonmin Byeon and Duncan Riach and Brandon Norick and Vijay Korthikanti and Tri Dao and Albert Gu and Ali Hatamizadeh and Sudhakar Singh and Deepak Narayanan and Garvit Kulshreshtha and Vartika Singh and Jared Casper and Jan Kautz and Mohammad Shoeybi and Bryan Catanzaro},\n  year = {2024},\n  eprint = {2406.07887},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2406.07887}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2-hybrid.2add9f814f3a.bib"},"thumbnail":{"src":"/content/papers/assets/f2e0e285e0d5337a787c2e32537b37089640a036c9339c3574c147a3cbaad4eb.png","width":1245,"height":322},"locale":"en","path":"/content/papers/en/mamba/mamba-2-hybrid.cc3edd4a2b64.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"first-how-pure-ssms-score","text":"First, How Pure SSMs Score","depth":3},{"id":"the-phone-book-and-fuzzy-memory","text":"The Phone Book and Fuzzy Memory","depth":3},{"id":"how-much-attention-to-add-back","text":"How Much Attention to Add Back?","depth":3},{"id":"working-out-the-memory-by-hand","text":"Working Out the Memory by Hand","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"12-standard-tasks","text":"12 Standard Tasks","depth":3},{"id":"long-context-mostly-on-par-behind-on-multi-document-qa","text":"Long Context: Mostly on Par, Behind on Multi-Document QA","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/repeat-after-me","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]},{"id":"mamba/repeat-after-me","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Repeat After Me","subtitle":"The Phone Book Test","author":"Z. Yuan","paperDate":"2024-02-01","publishedOn":"2025-12-23","paperTitle":"Repeat After Me: Transformers are Better than State Space Models at Copying","paperUrl":"https://arxiv.org/abs/2402.01032","bibliography":{"title":"Repeat After Me: Transformers are Better than State Space Models at Copying","authors":["Samy Jelassi","David Brandfonbrener","Sham M. Kakade","Eran Malach"],"url":"https://arxiv.org/abs/2402.01032","pdfUrl":"https://arxiv.org/pdf/2402.01032v2","codeUrl":"https://github.com/sjelassi/transformers_ssm_copy","preprint":{"id":"2402.01032","submittedOn":"2024-02-01","url":"https://arxiv.org/abs/2402.01032"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/jelassi24a.html"},"citation":"@misc{mambarepeatafterme2024,\n  title = {{Repeat After Me: Transformers are Better than State Space Models at Copying}},\n  author = {Samy Jelassi and David Brandfonbrener and Sham M. Kakade and Eran Malach},\n  year = {2024},\n  eprint = {2402.01032},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2402.01032}\n}\n","citationPath":"/content/papers/citations/mamba/repeat-after-me.6f1018ade730.bib"},"thumbnail":{"src":"/content/papers/assets/6d45fd8ab61191e1263c400e01bf76bdd9ef3534ecc4485835332074ea553443.png","width":1378,"height":416},"locale":"en","path":"/content/papers/en/mamba/repeat-after-me.a596955e7451.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-copy-task","text":"The Copy Task","depth":3},{"id":"too-small-a-state-cant-copy-it-all","text":"Too Small a State Can't Copy It All","depth":3},{"id":"transformers-find-their-place-with-n-grams","text":"Transformers Find Their Place with n-grams","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"synthetic-data-learning-fast-generalizing-far","text":"Synthetic Data: Learning Fast, Generalizing Far","depth":3},{"id":"tell-it-what-to-look-for-first-and-its-fine","text":"Tell It What to Look For First, and It's Fine","depth":3},{"id":"pretrained-models-the-phone-book","text":"Pretrained Models: The Phone Book","depth":3},{"id":"what-this-paper-leaves-open","text":"What This Paper Leaves Open","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/jamba","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2","subtitle":"Attention's Distant Cousin","author":"Z. Yuan","paperDate":"2024-05-31","publishedOn":"2025-12-06","world":"external/mamba2","paperTitle":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","paperUrl":"https://arxiv.org/abs/2405.21060","bibliography":{"title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","authors":["Tri Dao","Albert Gu"],"url":"https://arxiv.org/abs/2405.21060","pdfUrl":"https://arxiv.org/pdf/2405.21060v1","codeUrl":"https://github.com/state-spaces/mamba","preprint":{"id":"2405.21060","submittedOn":"2024-05-31","url":"https://arxiv.org/abs/2405.21060"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/dao24a.html"},"citation":"@misc{mambamamba22024,\n  title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},\n  author = {Tri Dao and Albert Gu},\n  year = {2024},\n  eprint = {2405.21060},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2405.21060}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2.3152f9dfa9a1.bib"},"thumbnail":{"src":"/content/papers/assets/771f5c6bcb0b30ac06fac163812a3aa6aad0063df419f7ffd41bc60cef195e79.png","width":1462,"height":556},"locale":"en","path":"/content/papers/en/mamba/mamba-2.84790bbee84c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"writing-an-ssm-as-a-matrix","text":"Writing an SSM as a Matrix","depth":3},{"id":"reducing-a-to-a-single-number","text":"Reducing A to a Single Number","depth":3},{"id":"working-through-four-tokens","text":"Working Through Four Tokens","depth":3},{"id":"half-one-way-half-the-other","text":"Half One Way, Half the Other","depth":3},{"id":"the-mamba-2-block","text":"The Mamba-2 Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"a-bigger-state-remembers-more","text":"A Bigger State Remembers More","depth":3},{"id":"language-modeling","text":"Language Modeling","depth":3},{"id":"the-original-authors-recommend-hybrids-too","text":"The Original Authors Recommend Hybrids Too","depth":3},{"id":"not-an-across-the-board-upgrade","text":"Not an Across-the-Board Upgrade","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/mamba","relation":"referenced","topic":""},{"id":"mamba/gated-deltanet","relation":"referenced","topic":""}]}],"text":"By mid-2024, the news about Mamba was somewhat polarized.\n\nOn one side were the original authors' experiments at 2.8B and below: Mamba and Mamba-2 matched or even beat same-size Transformers on common benchmarks. On the other, more and more studies pointed out weaknesses in copying, retrieval, and in-context learning. AI21's Jamba simply kept a few attention layers between its Mamba layers.\n\nThe trouble was that these comparisons were run under quite different conditions. The original authors' experiments stayed under 3B parameters and 1T tokens; large models like Jamba, on the other hand, had no control Transformer trained on the same data.\n\nThis technical report from NVIDIA sets out to fill that gap: using **the same data, the same hyperparameters, and the same evaluation pipeline**, it trains 8B-parameter Mamba, Mamba-2, and Transformer models, plus a hybrid, on up to 3.5T tokens, then compares them across 35 tasks.\n\nBesides NVIDIA researchers, the author list includes Mamba's two authors, Tri Dao and Albert Gu.\n\n## Defining the Problem\n\nThe authors' questions are direct: with a bigger training budget, can Mamba keep up with Transformers? And where it can't, is adding back a few attention layers enough?\n\nTo keep the comparison fair, the four models share:\n\n- The same data: two versions, 1.1T and 3.5T tokens, both precursors of the Nemotron-4 training data, made up of 70% English, 15% other languages, and 15% code.\n- The same hyperparameters: the authors found at small scale that Mamba's best hyperparameters were close to the Transformer's, so all models share them.\n- The same evaluation: a version-pinned open-source evaluation suite with the same set of prompts.\n\nThe model configurations:\n\n| Model | Layers | Notes |\n| --- | ---: | --- |\n| Transformer | 32 | Each layer is attention + MLP; 32 attention heads, RoPE |\n| Mamba | 56 | Two Mamba layers have about as many parameters as one attention + MLP layer; state dimension 128 |\n| Mamba-2 | 56 | Same as above, with Mamba-2 blocks; head dimension 64, 8 groups |\n| Mamba-2-Hybrid | 56 | 24 Mamba-2 layers, 4 attention layers, 28 MLP layers |\n\n*Table: Compiled from §3.1 and Table 6 of the paper. All four have about 8B parameters and a hidden dimension of 4096.*\n\nThe 3.5T-token group trained only Mamba-2, the Transformer, and the hybrid. The authors' reason: pure Mamba's state is so large that, already in the 1.1T group, it was nearly three times slower than Mamba-2.\n\n## Solving the Problem\n\n### First, How Pure SSMs Score\n\nThe 1.1T-token results contain one jarring number:\n\n| Model | Avg. of five commonsense tasks | MMLU 0-shot | MMLU 5-shot |\n| --- | ---: | ---: | ---: |\n| Transformer | 67.87 | 38.32 | 46.28 |\n| Mamba | 68.07 | 28.63 | 28.00 |\n| Mamba-2 | 68.56 | 28.94 | 29.19 |\n\n*Table: Excerpted from Table 2 of the paper; 8B models, 1.1T tokens. The five tasks are WinoGrande, PIQA, HellaSwag, ARC-E, and ARC-C.*\n\nOn commonsense tasks, both Mambas edge out the Transformer. On MMLU 5-shot, though, they trail by more than 17 points, falling almost to the chance level of guessing among four options.\n\nTrained to 3.5T tokens, Mamba-2's MMLU 5-shot score recovers to 48.7, only 1.37 points below the Transformer's 50.07; its average score even pulls ahead. So is more training all it takes?\n\nThe authors didn't stop there. They posed MMLU in three different ways:\n\n1. **Standard format**: four options, A, B, C, and D, are listed after the question, and the comparison is which letter the model is most likely to output next.\n2. **Choice text as the target**: the options are listed the same way, but the comparison is over the full option text, letter included: for example, the probability of “B. sorted('ab')”.\n3. **Cloze**: no options are listed after the question; instead, the comparison is the probability of each answer text coming right after the question.\n\nWith the third format, the 1.1T-token models score almost identically across both Mambas and the Transformer (about 38 to 39 points at 5-shot).\n\nFrom this, the authors infer that pure SSMs may have learned knowledge similar to the Transformer's; what trips them up is the format. The model has to understand that “four options are listed above, and the answer is a single letter,” then carry each option's content over to the matching letter. Moving information between tokens like this is exactly what attention is good at and what pure SSMs find harder. This is the authors' interpretation; the scores across the three formats support it, but they don't yet amount to direct proof.\n\nOne more detail: for the 3.5T Mamba-2, going from 0-shot to 5-shot raises its MMLU score by only 1.45 points, while the Transformer gains 4.38. Mamba-2 gets less out of the extra examples.\n\n### The Phone Book and Fuzzy Memory\n\nAnother test is the phone book: the prompt contains a list of “name: phone number” entries and two example questions with answers, then asks for one person's number.\n\n[![Three line charts, each with phone-book length (thousands of tokens) on the x-axis and a vertical line marking the 4K pretraining length. (a) Standard phone book: the Transformer is near 100% within 4K and drops rapidly to zero beyond it; Mamba and Mamba-2 begin to decline at around 0.5K and are mostly below 20% after 2.5K. (b) Number of correct tokens in the answer: both Mambas fall gradually from 12 to between 4 and 6. (c) Reversed phone book, where the model is first told whom to remember: Mamba does somewhat better but still declines with length, Mamba-2 falls below 10% at around 3K, and the Transformer stays at 100% within 4K](/content/papers/assets/f2e0e285e0d5337a787c2e32537b37089640a036c9339c3574c147a3cbaad4eb.png)](/content/papers/assets/f2e0e285e0d5337a787c2e32537b37089640a036c9339c3574c147a3cbaad4eb.png)\n\n*Figure 3 from the paper, excerpted from [**page 8 of the v1 PDF**](https://arxiv.org/pdf/2406.07887v1#page=8). All three panels use the 8B models trained on 1.1T tokens; the vertical line marks the pretraining length of 4,096.*\n\nLeft panel: the Transformer gets almost everything right within its pretraining length of 4,096 and breaks down beyond it. Both Mambas start answering wrong at around 500 tokens. The authors stress that, unlike MMLU, this gap persists in the Mamba-2 trained on 3.5T tokens.\n\nThe middle panel is particularly interesting. Instead of only checking right or wrong, the authors compared each answer token by token to count how many were correct. A phone number is about 12 tokens, and even when the two Mambas answered wrong, they still often got 4 to 8 of them right.\n\nThe authors call this **fuzzy memory**: every entry has been squeezed into the state, just not precisely enough, so it comes back out distorted.\n\nThe right panel changes the setup: the model is first told whom it will be asked about, then given the phone book. In principle, Mamba could keep its eye on that one name the whole way through. Scores do improve somewhat, but they still fall with length. The authors' reading is that even when it knows the question in advance, the model still struggles to decide which information to keep precisely and which it can forget.\n\n\n> [!TIP]\n> \n> This report's phone-book task is borrowed from the study by Jelassi et al. at Harvard. That paper also compared versions with the question given first and given last; in small models trained from scratch, state models could perform very well when the question came first:\n> \n> - **[[24.02] Repeat After Me: The Phone Book Test](/blog/papers/mamba/repeat-after-me)**\n\n\n### How Much Attention to Add Back?\n\nNext, the authors add a few attention layers back. To keep it fair, they adjust the number of attention heads and the MLP expansion ratio so that every type of layer has about the same number of parameters; that way, the ratio of layers equals the ratio of parameters.\n\n[![Line chart: the x-axis is the percentage of all layers that are attention layers, from 0 to 100; the y-axis is validation loss. At 0% it is about 2.615, falling to a minimum of about 2.554 at around 8%, then rising gradually as the share increases, to about 2.63 at 100%](/content/papers/assets/5b19ee03675ebbd942f790c816960a5b57e28b46fad1020453f2a1f4e4a2c647.png)](/content/papers/assets/5b19ee03675ebbd942f790c816960a5b57e28b46fad1020453f2a1f4e4a2c647.png)\n\n*Figure 4 from the paper, excerpted from [**page 9 of the v1 PDF**](https://arxiv.org/pdf/2406.07887v1#page=9). Hybrid models with 130 million parameters and 24 layers.*\n\nIn a small 130-million-parameter, 24-layer model, validation loss is lowest when about 8% of the layers are attention; all attention and no attention are both worse than a mix. A model with 840 million parameters confirms the same trend.\n\nA few other design choices:\n\n- **Share of MLP layers**: with attention fixed at 8%, an MLP share anywhere from 30% to 50% doesn't hurt the loss. The higher the share, the faster training runs; 50% is about 20% faster than 5%.\n- **Positional encoding**: for the 840M hybrid, adding RoPE or leaving it out makes little difference in the base model; after extending to a 16K context, the version without RoPE actually averages higher, so RoPE is left out from then on.\n- **Layer ordering**: a Mamba layer goes first, so that it naturally learns positional information; the attention and MLP layers are spread evenly through the rest. The authors found no better ordering, and there's no need to repeat a fixed block.\n- **GQA**: switching the attention layers to GQA raises validation perplexity by only about 0.04% and saves a good deal of memory at inference time.\n\nThe final 8B hybrid has 56 layers: 24 Mamba-2 (42.9%), 4 attention (7.1%), and 28 MLP (50%). The ordering is:\n\n```text\nM+M+M++M+M*+M+M+M+M++M*+M+M+M+M+M*++M+M+M+M+M*+M++M+M+M+\n```\n\n`M` is Mamba-2, `*` is attention, and `+` is MLP. The four asterisks are spread evenly through the middle section.\n\n### Working Out the Memory by Hand\n\nUsing the paper's configurations, let's work out how much each model has to carry at inference time. The Transformer has 32 layers of 32-head multi-head attention; the hybrid has 4 layers of GQA with 8 KV groups; both use 128 dimensions per head, in bf16:\n\n```python\ndef kv(layers, kv_heads, head_dim=128, tokens=1, b=2):\n    return 2 * layers * kv_heads * head_dim * b * tokens\n\nMiB, GiB = 2**20, 2**30\nt = kv(32, 32); h = kv(4, 8)\nprint(f\"Transformer: {t // 1024} KiB per token; hybrid: {h // 1024} KiB per token ({t // h}×)\")\nfor n in (4096, 131072):\n    print(f\"{n:>6} tokens: Transformer {kv(32, 32, tokens=n) / GiB:.2f} GiB, hybrid {kv(4, 8, tokens=n) / GiB:.3f} GiB\")\n\nssm = 24 * (2 * 4096) * 128            # 24 Mamba-2 layers, inner width 8192, state dimension 128\nprint(f\"State of 24 Mamba-2 layers: {ssm:,} values, bf16 {ssm * 2 / MiB:.0f} MiB (independent of length)\")\n```\n\n```text\nTransformer: 512 KiB per token; hybrid: 16 KiB per token (32×)\n  4096 tokens: Transformer 2.00 GiB, hybrid 0.062 GiB\n131072 tokens: Transformer 64.00 GiB, hybrid 2.000 GiB\nState of 24 Mamba-2 layers: 25,165,824 values, bf16 48 MiB (independent of length)\n```\n\nOf that factor of 32, 8 comes from cutting the attention layers from 32 to 4, and the other 4 from GQA. Mamba-2's 48 MiB of state is about the same size as the KV cache at a 4K context; the longer the context, the less it matters.\n\nThis is only the memory bill for a single sequence. Figure 5 of the paper projects that, when generating one token at batch size 32, the two run at about the same speed for short contexts, and at contexts beyond a hundred thousand tokens the hybrid is nearly 8 times faster. These are projections supplied by the report; it doesn't include matching measurements on an inference system or a detailed latency model. Training efficiency, on the other hand, was measured: on 1,024 H100s, the hybrid's MFU is 29.9%, versus 30.7% for the Transformer.\n\n## Discussion\n\n### 12 Standard Tasks\n\nAt 3.5T tokens, the hybrid beats the Transformer on all 12 standard tasks. The paper's average column counts MMLU 0-shot and 5-shot as separate columns, 13 in all, and on that average the hybrid is 2.65 points higher:\n\n| Model | MMLU 5-shot | SQuAD v2 | Paper average (13 columns) |\n| --- | ---: | ---: | ---: |\n| Transformer | 50.07 | 53.4 | 53.17 |\n| Mamba-2 | 48.7 | 51.9 | 54.69 |\n| Mamba-2-Hybrid | 53.60 | 58.67 | 55.82 |\n\n*Table: Excerpted from Table 7 of the paper.*\n\nThe MMLU training curves are also worth a look (Figure 6 in the paper): the hybrid leads pure Mamba-2 throughout, but early in training it still trails the Transformer; only after catching up at 1.5T tokens does it stay in front for good.\n\n### Long Context: Mostly on Par, Behind on Multi-Document QA\n\nThe authors continued pretraining the 4K base models on 50B tokens, extending them to 16K and 32K. The four attention layers use full global attention.\n\nThe results fall into three groups:\n\n1. **Phone book**: the hybrid is not only perfect within its pretraining length but can also extrapolate a little beyond it. The 4K hybrid stays at 100% up to 5.5K, and the 16K and 32K versions also go past their respective training lengths. A hybrid extended separately to 128K behaves the same way.\n2. **RULER synthetic tasks**: on tasks like needle-in-a-haystack, multi-key lookup, and variable tracking, the 16K hybrid averages about 12 points higher than the 16K Transformer (74.19 vs. 62.33). On variable tracking, the Transformer often just copies the example answer from the prompt.\n3. **Real long-document tasks**: on the average of nine tasks, the 16K and 32K Transformers actually come out about 1 point higher, with the gap coming mainly from multi-document QA. On HotpotQA, for example, it's 48.61 vs. 42.17 at 16K.\n\nThe authors' explanation for the third point: LongBench's multi-document QA mixes the relevant passages with lots of random Wikipedia passages and puts the question at the end. The SSM layers read without knowing the question yet and may get led astray by irrelevant documents. They also suspect that during context extension, training on unrelated documents simply concatenated together may have led the SSM to treat them as related.\n\nAnother observation: the hybrid is more sensitive to prompt format. Take the test of the 4K base models on Musique: when the prompt was changed, the hybrid's score swung between 10.63 and 16.16, while the Transformer's only moved between 15.25 and 17.68. The authors also point out that LongBench's prompts were originally tuned for Transformers.\n\n## Conclusion\n\nUsing the same data and pipeline, this report sets the various 2024 claims about Mamba side by side:\n\n1. **Pure SSMs hold their own against Transformers on most language tasks**; at 3.5T tokens, pure Mamba-2's average over the five commonsense tasks is even slightly higher, and other tasks are a mix of wins and losses.\n2. **The weaknesses of pure SSMs are concentrated in format, copying, and in-context learning**. Most of the MMLU gap can be closed with more training, but the phone-book gap remains at both the 1.1T and 3.5T budgets, and the 3.5T pure Mamba-2 also trails on the average of the nine natural long-document tasks. When it answers wrong, it “remembers fuzzily” rather than not remembering at all.\n3. **About 8% attention layers is enough**. The 8B hybrid has only 4 attention layers out of 56, beats the Transformer on all 12 standard tasks, has a KV cache 32 times smaller, and is more reliable at long-context lookup.\n4. **The hybrid still has open questions**: multi-document QA, prompt sensitivity, and how to train for long contexts.\n\nMamba was first discussed as a replacement for the Transformer. After reading this report, a more reasonable place for it is this: Mamba-2 handles the sequence in most layers, while a few attention layers take care of precise lookback. NVIDIA's later Nemotron-H and Nemotron 3 series both follow this path.\n\n\n> [!TIP]\n> \n> For the Mamba-2 block used in this report and its relationship to linear attention, see:\n> \n> - **[[24.05] Mamba-2: Attention's Distant Cousin](/blog/papers/mamba/mamba-2)**\n","imageSizes":{"/content/papers/assets/f2e0e285e0d5337a787c2e32537b37089640a036c9339c3574c147a3cbaad4eb.png":{"width":1245,"height":322},"/content/papers/assets/5b19ee03675ebbd942f790c816960a5b57e28b46fad1020453f2a1f4e4a2c647.png":{"width":506,"height":303}}}