{"notes":[{"id":"mamba/jamba","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Jamba","subtitle":"An Eighth of Attention","author":"Z. Yuan","paperDate":"2024-03-28","publishedOn":"2026-01-10","world":"external/jamba","paperTitle":"Jamba: A Hybrid Transformer-Mamba Language Model","paperUrl":"https://arxiv.org/abs/2403.19887","bibliography":{"title":"Jamba: A Hybrid Transformer-Mamba Language Model","authors":["Opher Lieber","Barak Lenz","Hofit Bata","Gal Cohen","Jhonathan Osin","Itay Dalmedigos","Erez Safahi","Shaked Meirom","Yonatan Belinkov","Shai Shalev-Shwartz","Omri Abend","Raz Alon","Tomer Asida","Amir Bergman","Roman Glozman","Michael Gokhman","Avashalom Manevich","Nir Ratner","Noam Rozen","Erez Shwartz","Mor Zusman","Yoav Shoham"],"url":"https://arxiv.org/abs/2403.19887","pdfUrl":"https://arxiv.org/pdf/2403.19887v2","codeUrl":null,"preprint":{"id":"2403.19887","submittedOn":"2024-03-28","url":"https://arxiv.org/abs/2403.19887"},"publication":null,"citation":"@misc{mambajamba2024,\n  title = {{Jamba: A Hybrid Transformer-Mamba Language Model}},\n  author = {Opher Lieber and Barak Lenz and Hofit Bata and Gal Cohen and Jhonathan Osin and Itay Dalmedigos and Erez Safahi and Shaked Meirom and Yonatan Belinkov and Shai Shalev-Shwartz and Omri Abend and Raz Alon and Tomer Asida and Amir Bergman and Roman Glozman and Michael Gokhman and Avashalom Manevich and Nir Ratner and Noam Rozen and Erez Shwartz and Mor Zusman and Yoav Shoham},\n  year = {2024},\n  eprint = {2403.19887},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2403.19887}\n}\n","citationPath":"/content/papers/citations/mamba/jamba.11fef67c0c54.bib"},"thumbnail":{"src":"/content/papers/assets/fa186580641946194f81adf74136e29431c3f6dd3fa63e2ecf8a1c0c29d3e5a0.png","width":1056,"height":1000},"locale":"en","path":"/content/papers/en/mamba/jamba.129c94adb5f2.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-jamba-block","text":"The Jamba Block","depth":3},{"id":"working-out-the-kv-cache-by-hand","text":"Working Out the KV Cache by Hand","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-it-stacks-up-against-mixtral","text":"How It Stacks Up Against Mixtral","depth":3},{"id":"one-to-three-or-one-to-seven","text":"One to Three, or One to Seven?","depth":3},{"id":"where-does-pure-mamba-get-stuck","text":"Where Does Pure Mamba Get Stuck?","depth":3},{"id":"three-more-findings","text":"Three More Findings","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/repeat-after-me","relation":"referenced","topic":""},{"id":"mamba/mamba-2-hybrid","relation":"referenced","topic":""}]},{"id":"mamba/repeat-after-me","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Repeat After Me","subtitle":"The Phone Book Test","author":"Z. Yuan","paperDate":"2024-02-01","publishedOn":"2025-12-23","paperTitle":"Repeat After Me: Transformers are Better than State Space Models at Copying","paperUrl":"https://arxiv.org/abs/2402.01032","bibliography":{"title":"Repeat After Me: Transformers are Better than State Space Models at Copying","authors":["Samy Jelassi","David Brandfonbrener","Sham M. Kakade","Eran Malach"],"url":"https://arxiv.org/abs/2402.01032","pdfUrl":"https://arxiv.org/pdf/2402.01032v2","codeUrl":"https://github.com/sjelassi/transformers_ssm_copy","preprint":{"id":"2402.01032","submittedOn":"2024-02-01","url":"https://arxiv.org/abs/2402.01032"},"publication":{"venue":"ICML","year":2024,"kind":"conference","url":"https://proceedings.mlr.press/v235/jelassi24a.html"},"citation":"@misc{mambarepeatafterme2024,\n  title = {{Repeat After Me: Transformers are Better than State Space Models at Copying}},\n  author = {Samy Jelassi and David Brandfonbrener and Sham M. Kakade and Eran Malach},\n  year = {2024},\n  eprint = {2402.01032},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2402.01032}\n}\n","citationPath":"/content/papers/citations/mamba/repeat-after-me.6f1018ade730.bib"},"thumbnail":{"src":"/content/papers/assets/6d45fd8ab61191e1263c400e01bf76bdd9ef3534ecc4485835332074ea553443.png","width":1378,"height":416},"locale":"en","path":"/content/papers/en/mamba/repeat-after-me.a596955e7451.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-copy-task","text":"The Copy Task","depth":3},{"id":"too-small-a-state-cant-copy-it-all","text":"Too Small a State Can't Copy It All","depth":3},{"id":"transformers-find-their-place-with-n-grams","text":"Transformers Find Their Place with n-grams","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"synthetic-data-learning-fast-generalizing-far","text":"Synthetic Data: Learning Fast, Generalizing Far","depth":3},{"id":"tell-it-what-to-look-for-first-and-its-fine","text":"Tell It What to Look For First, and It's Fine","depth":3},{"id":"pretrained-models-the-phone-book","text":"Pretrained Models: The Phone Book","depth":3},{"id":"what-this-paper-leaves-open","text":"What This Paper Leaves Open","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/jamba","relation":"referenced","topic":""}]},{"id":"mamba/mamba-2-hybrid","category":"mamba","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Mamba-2-Hybrid","subtitle":"Fuzzy Memory","author":"Z. Yuan","paperDate":"2024-06-12","publishedOn":"2026-01-22","paperTitle":"An Empirical Study of Mamba-based Language Models","paperUrl":"https://arxiv.org/abs/2406.07887","bibliography":{"title":"An Empirical Study of Mamba-based Language Models","authors":["Roger Waleffe","Wonmin Byeon","Duncan Riach","Brandon Norick","Vijay Korthikanti","Tri Dao","Albert Gu","Ali Hatamizadeh","Sudhakar Singh","Deepak Narayanan","Garvit Kulshreshtha","Vartika Singh","Jared Casper","Jan Kautz","Mohammad Shoeybi","Bryan Catanzaro"],"url":"https://arxiv.org/abs/2406.07887","pdfUrl":"https://arxiv.org/pdf/2406.07887v1","codeUrl":"https://github.com/NVIDIA/Megatron-LM","preprint":{"id":"2406.07887","submittedOn":"2024-06-12","url":"https://arxiv.org/abs/2406.07887"},"publication":null,"citation":"@misc{mambamamba2hybrid2024,\n  title = {{An Empirical Study of Mamba-based Language Models}},\n  author = {Roger Waleffe and Wonmin Byeon and Duncan Riach and Brandon Norick and Vijay Korthikanti and Tri Dao and Albert Gu and Ali Hatamizadeh and Sudhakar Singh and Deepak Narayanan and Garvit Kulshreshtha and Vartika Singh and Jared Casper and Jan Kautz and Mohammad Shoeybi and Bryan Catanzaro},\n  year = {2024},\n  eprint = {2406.07887},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2406.07887}\n}\n","citationPath":"/content/papers/citations/mamba/mamba-2-hybrid.2add9f814f3a.bib"},"thumbnail":{"src":"/content/papers/assets/f2e0e285e0d5337a787c2e32537b37089640a036c9339c3574c147a3cbaad4eb.png","width":1245,"height":322},"locale":"en","path":"/content/papers/en/mamba/mamba-2-hybrid.cc3edd4a2b64.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"first-how-pure-ssms-score","text":"First, How Pure SSMs Score","depth":3},{"id":"the-phone-book-and-fuzzy-memory","text":"The Phone Book and Fuzzy Memory","depth":3},{"id":"how-much-attention-to-add-back","text":"How Much Attention to Add Back?","depth":3},{"id":"working-out-the-memory-by-hand","text":"Working Out the Memory by Hand","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"12-standard-tasks","text":"12 Standard Tasks","depth":3},{"id":"long-context-mostly-on-par-behind-on-multi-document-qa","text":"Long Context: Mostly on Par, Behind on Multi-Document QA","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/repeat-after-me","relation":"referenced","topic":""},{"id":"mamba/mamba-2","relation":"referenced","topic":""}]}],"text":"In December 2023, Mamba, a selective state space model that runs in linear time, matched same-size Transformers at language modeling. But the largest model in that paper was 2.8B, and everyone had the same question on their minds: would it hold up at production scale?\n\nThree months later, Israel's AI21 Labs delivered a 52B-parameter model under the Apache 2.0 license, named Jamba. It did use a lot of Mamba layers, but it still kept one attention layer in every eight.\n\nIn the Hacker News thread for the release, someone asked point-blank: now that we have Mamba, why include attention layers at all? The reply was short: “Mamba is bad with long context. It doesn't remember phone numbers.” [Hacker News, 2024-03-28](https://news.ycombinator.com/item?id=39853958)\n\nThat reply linked to a study testing tasks that require copying the original text exactly, such as long phone books; we've written a separate note on it. That study also lists situations where state models do well. The Jamba paper gives another, more concrete reason, drawn from its own ablations.\n\n## Defining the Problem\n\nFirst, picture the two bills a model serving long documents has to pay.\n\nOne is memory. An attention layer keeps a key and a value for every token it has read; the longer the document, the larger this cache grows, and in the end it determines how long a context fits on one GPU. The other is time: every new token generated has to be compared against the entire context again, which drags down throughput on long texts.\n\nModels like Mamba pay less on both bills, because they only update a fixed-size state. Yet in AI21's evaluations, at the same scale, pure-Mamba language models still trailed Transformers slightly.\n\nOthers had already tried putting both kinds of layers in one model, but only at small scale: experiments interleaving Mamba and attention stopped at around 1.3B, and StripedHyena, which reached 7B, still lost to the attention-only Mistral-7B.\n\nSo the questions this paper really sets out to answer are ones that can only be answered after scaling up: in what ratio should the two kinds of layers be mixed? Will adding MoE cause conflicts? And will training stay stable at tens of billions of parameters?\n\n## Solving the Problem\n\n### The Jamba Block\n\n[![Jamba architecture diagram: on the left is one Jamba block, made of eight layers from bottom to top: Mamba, Mamba MoE, Mamba, Mamba MoE, Transformer, Mamba MoE, Mamba, Mamba MoE; on the right are the internal structures of the four layer types, each consisting of RMSNorm plus Attention or Mamba, followed by RMSNorm plus MLP or MoE, with a residual connection around each part](/content/papers/assets/fa186580641946194f81adf74136e29431c3f6dd3fa63e2ecf8a1c0c29d3e5a0.png)](/content/papers/assets/fa186580641946194f81adf74136e29431c3f6dd3fa63e2ecf8a1c0c29d3e5a0.png)\n\n*Figure 1 from the paper, excerpted from [**page 3 of the v2 PDF**](https://arxiv.org/pdf/2403.19887v2#page=3), CC BY-SA 4.0. The left panel is one Jamba block; the right panel shows the internal structure of the four layer types.*\n\nStart with the right panel. Every layer has two parts: the first mixes information along the sequence, using Attention or Mamba; the second transforms features token by token, using an ordinary MLP or MoE. Each part has an RMSNorm in front and is wrapped in its own residual connection.\n\nPairing them up gives the four layer types on the right. One of them, Attention MoE, isn't actually used in Jamba.\n\nNow the left panel. A Jamba block has eight layers. Counting from the bottom, the fifth is the only Transformer layer, and the other seven are Mamba; MoE appears in every other layer.\n\nThe authors describe this design space with five parameters:\n\n- $l$: the number of layers in a block.\n- $a:m$: the ratio of attention layers to Mamba layers.\n- $e$: the interval, in layers, at which an MLP is replaced with MoE.\n- $n$: the number of experts in each MoE layer.\n- $K$: the number of experts chosen per token.\n\nThe released version uses $l=8$, $a:m=1:7$, $e=2$, $n=16$, $K=2$. Four blocks stacked together give 32 layers, of which only 4 are attention. The model has 52B total parameters, with 12B active per token. With int8 weights, it fits on a single 80GB GPU.\n\nMost other details follow the standard recipe of the time: GQA for attention, SwiGLU activations, a 64K vocabulary, and each digit split into its own token. Two choices stand out, and we'll come back to them in the discussion: RMSNorm is added inside the Mamba layers, and the model uses no positional encoding such as RoPE at all.\n\n### Working Out the KV Cache by Hand\n\nSo how much does keeping only 4 attention layers actually save? Table 1 of the paper gives the KV cache for a 256K context in 16-bit:\n\n| Model | Total params | Active params | KV cache (256K) |\n| --- | ---: | ---: | ---: |\n| Llama-2 | 6.7B | 6.7B | 128GB |\n| Mistral | 7.2B | 7.2B | 32GB |\n| Mixtral | 46.7B | 12.9B | 32GB |\n| Jamba | 52B | 12B | 4GB |\n\n*Table: Table 1 of the paper.*\n\nLet's compute it ourselves. Each attention layer stores one key and one value per token; Mixtral and Jamba both have 8 KV heads with 128 dimensions each (per their configuration files on Hugging Face), and bf16 takes 2 bytes per number:\n\n```python\ndef kv_bytes(attn_layers, kv_heads, head_dim, tokens, bytes_per=2):\n    return 2 * attn_layers * kv_heads * head_dim * bytes_per * tokens\n\nGiB = 2**30\nfor name, layers in [(\"Mixtral (32 attention layers)\", 32), (\"Jamba (4 attention layers)\", 4)]:\n    per_token = kv_bytes(layers, 8, 128, 1)\n    print(f\"{name}: {per_token // 1024} KiB per token; 256K tokens: {kv_bytes(layers, 8, 128, 262144) / GiB:.0f} GiB total\")\n\n# Jamba's 28 Mamba layers: inner width 2×4096, state dimension 16 per channel\nmamba_layers, d_inner, d_state = 28, 2 * 4096, 16\nssm = mamba_layers * d_inner * d_state\nprint(f\"SSM state of 28 Mamba layers: {ssm:,} values; in bf16, about {ssm * 2 / 2**20:.0f} MiB\")\n```\n\n```text\nMixtral (32 attention layers): 128 KiB per token; 256K tokens: 32 GiB total\nJamba (4 attention layers): 16 KiB per token; 256K tokens: 4 GiB total\nSSM state of 28 Mamba layers: 3,670,016 values; in bf16, about 7 MiB\n```\n\nBoth numbers match Table 1 (the original table says GB; here we use GiB, counted in powers of 2, throughout). The eightfold difference is exactly the difference between 32 attention layers and 4.\n\nThe last line is the state the Mamba side has to carry along. Whether the context is 1K or 256K, it stays this size. (Only the SSM state is counted here; the convolution buffer is extra, and also tiny.)\n\n\n> [!NOTE]\n> \n> The Llama-2 row is four times larger because its 7B version doesn't use GQA: each of its 32 attention heads has its own key and value. Mistral switches to 8 groups of KV heads, which cuts the KV cache to a quarter.\n\n\n## Discussion\n\n### How It Stacks Up Against Mixtral\n\nFirst, the standard benchmarks. The authors mainly compare against Mixtral 8x7B, whose total and active parameter counts are both close to Jamba's. A few selected results:\n\n| Benchmark | Mixtral | Jamba |\n| --- | ---: | ---: |\n| HellaSwag (10-shot) | 86.7 | 87.1 |\n| WinoGrande (5-shot) | 81.2 | 82.5 |\n| GSM8K (3-shot CoT) | 60.4 | 59.9 |\n| HumanEval (pass@1) | 34.8 | 29.3 |\n| MMLU (5-shot) | 70.6 | 67.4 |\n| BBH (3-shot) | 50.3 | 45.4 |\n\n*Table: Excerpted from Table 2 of the paper.*\n\nIn the authors' words, “in most tasks, Jamba performs comparably.” Item by item, it is slightly higher on HellaSwag and WinoGrande in this table; in the full table, it trails by a few points on ARC-E and ARC-C, as well as MMLU, BBH, and HumanEval.\n\nJamba's advantage lies mainly in the cost of long contexts. Figure 2 of the paper shows that the context length that fits on a single A100 80GB is, for Jamba, about twice Mixtral's and seven times Llama-2-70B's, around 140K tokens. Figure 3 measures end-to-end throughput for encoding plus decoding, generating 512 tokens in both settings: on a single A100 with int8 quantization and an 8K context, Jamba fits a batch of 16 while Mixtral doesn't, and the throughput gap is around threefold; on four A100s, unquantized, at batch size 1 with a 128K context, Jamba is again about three times faster.\n\nThe authors also caution that these throughput numbers are only meaningful as relative comparisons: the community has spent six years optimizing inference for pure Transformers, and Jamba had none of that yet.\n\nFor long-context ability, the authors ran two kinds of tests. On needle-in-a-haystack, the needle is found almost every time within 256K; they stress that this relies on just 4 attention layers. They also converted the five longest datasets in L-Eval to a 3-shot format: the average F1 is 0.44, versus 0.43 for Mixtral.\n\n### One to Three, or One to Seven?\n\nTo choose the ratio, the authors experimented with small models of 1.3B parameters trained on 250B tokens:\n\n| Model | OLLM | HellaSwag | WinoGrande | NQ |\n| --- | ---: | ---: | ---: | ---: |\n| Pure attention | 36.4 | 62.4 | 59.6 | 14.5 |\n| Pure Mamba | 36.1 | 62.6 | 59.4 | 14.5 |\n| Hybrid 1:3 | 37.2 | 65.1 | 61.7 | 16.5 |\n| Hybrid 1:7 | 37.2 | 65.1 | 61.7 | 16.0 |\n\n*Table: Excerpted from Table 4 of the paper; none of these use MoE. OLLM is the authors' reproduction of the Hugging Face Open LLM Leaderboard aggregate score.*\n\nThe two hybrid ratios are nearly identical, and both beat either single architecture. The authors chose 1:7 because it is more compute-efficient. In a second comparison scaled up to 7B and 50B tokens, pure Mamba's OLLM of 35.3 is slightly below pure attention's 36.1, while the hybrid scores highest at 36.6 and has the lowest training loss throughout.\n\nOne caveat: the two ablations trained on 250B and 50B tokens respectively, and the paper doesn't disclose the total training budget of the released model. For now, the conclusions about the ratio are limited to these ablation settings.\n\n### Where Does Pure Mamba Get Stuck?\n\nThis is the most interesting part of the paper.\n\nPure Mamba's perplexity and general benchmark scores aren't bad, but it drops sharply on three datasets:\n\n| Model | IMDB | QuAC | NarrativeQA |\n| --- | ---: | ---: | ---: |\n| Pure attention | 84.1 | 27.9 | 45.8 |\n| Pure Mamba | 48.8 | 20.2 | 27.7 |\n| Hybrid | 90.9 | 26.6 | 43.7 |\n\n*Table: Table 6 of the paper; 1.3B models, 250B tokens.*\n\nIMDB is sentiment classification of movie reviews, with only two possible answers: Positive and Negative. When the authors looked at pure Mamba's outputs, they found that it often ignored the format, writing “Very Good,” “Funny,” “Bad,” or even “3/10.”\n\nSome of these answers may have the right sentiment; they just weren't given in the format of the earlier examples.\n\nIn the few-shot prompt, every example is “review + Sentiment: label.” One way to stick to this format is, at the final colon, to look back at how the earlier examples wrote their labels. Earlier research has linked this kind of “look back and copy” ability to the **induction head** in Transformers: an attention head that finds an earlier occurrence of the same pattern and copies the token that came after it.\n\nThe authors' conjecture is that pure Mamba has no attention, so this mechanism doesn't easily emerge on its own during training. Switch to the 1:7 hybrid (this 1.3B model has three attention layers in total), and the problem disappears. (The “pure Mamba” here actually interleaves Mamba layers with MLPs; it just has no attention.)\n\n[![A fragment of an IMDB few-shot prompt: three examples each end with a Sentiment label, Negative, Negative, Positive, highlighted in different shades of blue; the prompt ends at the colon after Sentiment](/content/papers/assets/ab6a0dbf1cacc4a6045d01ccebacef40614338c0765fdb2052bd1fa19ffcd408.png)](/content/papers/assets/ab6a0dbf1cacc4a6045d01ccebacef40614338c0765fdb2052bd1fa19ffcd408.png)\n\n*Figure 8 from the paper, excerpted from [**page 12 of the v2 PDF**](https://arxiv.org/pdf/2403.19887v2#page=12), CC BY-SA 4.0. One head in the hybrid model's first attention layer, looking out from the final colon; darker colors mean stronger attention.*\n\nThe passage in the figure is an example that pure Mamba gets wrong and the hybrid gets right. Looking out from that final colon, the attention lands almost entirely on the earlier examples' labels: Negative, Negative, Positive. Across the three attention layers of the 1.3B hybrid, the authors found 12 heads like this.\n\nThe authors themselves call this anecdotal evidence. It shows that heads resembling induction heads appear in the hybrid model, but it falls short of proving that SSMs can never acquire in-context learning. From an engineering standpoint, though, the conclusion is already quite useful: **a small number of attention layers is enough to close this gap**.\n\n\n> [!TIP]\n> \n> The line “Mamba can't remember phone numbers” traces back to another study from February of the same year. It explains, through both theory and experiments, why a fixed-size state can't finish copying long strings:\n> \n> - **[[24.02] Repeat After Me: The Phone Book Test](/blog/papers/mamba/repeat-after-me)**\n\n\n### Three More Findings\n\n1. **MoE helps**. Adding MoE (top-2 of 16, every other layer) to the 7B hybrid trained on 50B tokens raises OLLM from 36.6 to 38.1 and NQ from 15.4 to 18.9.\n2. **Mamba needs extra normalization at scale**. Training is stable at 1.3B and below; when scaling up to the full model, large loss spikes appeared, caused by overly large activations inside the Mamba layers. Adding RMSNorm inside them stabilized training.\n3. **Positional encoding is optional**. Adding RoPE to the attention layers or leaving it out gives similar results (1.3B, 250B tokens). The authors' conjecture is that the Mamba layers placed before the attention layers already supply implicit positional information.\n\n## Conclusion\n\nAI21 positions Jamba as the first production-scale hybrid of attention and Mamba; what was released in March 2024 was a pretrained base model that had not yet been instruction-tuned or aligned. The paper's main takeaways are roughly these:\n\n1. **The ratio can be very low**. Hybrids at 1:3 and 1:7 work almost equally well, and both beat pure attention and pure Mamba; keeping only 4 attention layers out of 32 cuts the KV cache for a 256K context from 32GB to 4GB.\n2. **Pure Mamba's weakness lies in format and copying**. General benchmarks barely show a difference; it surfaces in tasks like IMDB that require answering in the few-shot format, and the 1:7 hybrid recovers it.\n3. **Performance close to Mixtral, at a much lower cost for long texts**. Jamba still trails Mixtral on some benchmarks, and its throughput advantage appears mainly at long contexts.\n\nBack to the question: with Mamba here, why keep attention at all? The paper's answer is pragmatic: the approach that works best at an acceptable cost is to use Mamba for most layers and reserve a few for attention. Three months later, NVIDIA ran a larger-scale comparison with 8B models and reached much the same conclusion.\n\n\n> [!TIP]\n> \n> For NVIDIA's study comparing Mamba, Mamba-2, and Transformers on the same data, see:\n> \n> - **[[24.06] Mamba-2-Hybrid: Fuzzy Memory](/blog/papers/mamba/mamba-2-hybrid)**\n","imageSizes":{"/content/papers/assets/fa186580641946194f81adf74136e29431c3f6dd3fa63e2ecf8a1c0c29d3e5a0.png":{"width":1056,"height":1000},"/content/papers/assets/ab6a0dbf1cacc4a6045d01ccebacef40614338c0765fdb2052bd1fa19ffcd408.png":{"width":1128,"height":778}}}