Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
- Publication
- ICML 2024Conference
- First submitted to arXiv
- arXiv:2401.09417
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{mambavim2024,
title = {{Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model}},
author = {Lianghui Zhu and Bencheng Liao and Qian Zhang and Xinlong Wang and Wenyu Liu and Xinggang Wang},
year = {2024},
eprint = {2401.09417},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2401.09417}
}
Remember when Transformer first appeared in the NLP field, and it took the CV field three years to produce ViT?
Now, Mamba has appeared in the NLP field, and with the prior experience, the CV field only took a month to develop Vim.
One can only say that the researchers in this field are truly hardworking and diligent.
Defining the Problem
Mamba shows promise for efficient long-sequence language modeling, particularly by avoiding an attention KV cache that grows during autoregressive generation. Savings are not guaranteed for every task, device or training setup.
For a moment, it seems like all researchers' attention is on Mamba, and everyone is wondering whether Mamba has the potential to become the foundation for the next-generation LLM, replacing Transformer at the top of the field.
Whether Mamba is the successor to Transformer may be debated in academia for a few more years, but the CV field isn't going to wait that long. The authors of this paper directly swapped the Transformer architecture inside ViT for Mamba to see what effect it would have.
Solving the Problem
Model Architecture
We believe you already possess the necessary knowledge from previous research, so let’s directly look at the architecture of Vim.
The model architecture shown in the image follows the design style of ViT. Since Mamba is also a sequence model, the image input is first converted into sequence data. The method of flattening the image into sequences here, like ViT, involves dividing the image into patches of size , and then unfolding them into vectors.
If the entire image is divided into patches, the result is:
Then, using a linear projection matrix , each patch is projected to a dimension . Continuing the approach from ViT, a CLS token is added at the beginning of the sequence to aggregate the information of the entire sequence for image classification.
Additionally, a learnable positional encoding is added to the sequence :
Finally, Vim blocks are stacked, each block mapping the output token sequence of the previous layer into , and the class token from the -th layer output is taken for the final classification:
At this point, it is essentially the same as ViT. The main difference is inside the Vim block, so let's continue.
Here denotes the transformation without the residual, which is included in the complete block output below. This schematic follows the paper’s head-class-token notation. The selected classification configuration instead places the class token in the middle, so its output index must change accordingly rather than always using .
Vision Mamba Block
Original Mamba is causal: the state at position depends on earlier inputs, not future tokens. This dependency does not require strictly serial execution on hardware; training can use a parallel scan. Vim changes the direction of information flow.
However, for images that "require spatial awareness," processing the sequence from start to finish is clearly not the best approach. Image sequences not only need "left-to-right" information, but also often require "right-to-left" or broader contextual understanding. Therefore, the authors introduce the concept of bidirectional sequence modeling.
After inputting the image sequence, the model must simultaneously perform "forward" and "backward" SSM operations, then merge them at the output, conceptually similar to "bidirectional RNN" in natural language processing.
Following the architecture diagram above, let's briefly describe the computation process:
- Normalization: The input is normalized, and the output is denoted as .
- Linear Projection: is projected into , with dimensionality .
- 1-D Convolution: Each direction applies a 1-D convolution and SiLU to , producing .
- SSM Parameters: depend on . Continuous-time is a learned parameter shared across tokens; its discretized counterpart , like , depends on the input.
- Forward/Backward SSM: Each direction gives .
- Gated Integration: Align the backward output with the original token order, gate both directions with , sum, project back to dimensions, and add the residual.
In mathematical form, assume the sequence length in the forward direction is , then for :
The backward direction recursively processes from to .
The full block output can be summarized as:
Parameter Settings
The hyperparameter design of the Vim architecture is as follows:
- : The number of stacked Vim blocks.
- : The hidden dimension of the input tokens.
- : Expanded channel width; for the Tiny/Small configurations here.
- : The state dimension of the SSM, default set to 16.
When compared with architectures like DeiT-Tiny, DeiT-Small, the corresponding hyperparameter settings are:
- Tiny-size: .
- Small-size: .
- Patch projection size is 16×16, non-overlapping, directly flattened and input to Vim blocks.
Vim uses no self-attention or separate 2D convolutional backbone, but its blocks do contain 1D convolution.
Discussion
Performance in Image Classification
ImageNet-1K training mainly follows the DeiT recipe with AdamW and a cosine schedule on eight A800 GPUs. After 300 epochs of supervised training, the authors add 30 epochs of long-sequence fine-tuning. This keeps 16×16 patches but changes extraction stride to 8, creating overlapping patches and longer sequences.
The table compares several backbone families. Vim-S has 26M parameters and 80.3% Top-1 accuracy, compared with ResNet-50's 25M and 76.2%. However, parameter counts, input resolutions and training recipes are not identical across every row; the results cannot all be attributed solely to SSM. The ViT-B/L rows use 384-pixel inputs, while Vim uses 224-pixel inputs, so these rows alone do not settle a like-for-like ViT comparison.
The DeiT comparison is more direct: Vim Tiny/Small/Base have about 7M/26M/98M parameters versus 6M/22M/86M for DeiT. They are comparable in scale, but Vim is larger in each pair. Top-1 differences are:
- Vim-Tiny is 3.9 percentage points higher than DeiT-Tiny.
- Vim-Small is 0.5 percentage points higher than DeiT-Small.
- Vim-Base is 0.1 percentage points higher than DeiT-Base.
This supports Vim’s competitiveness on this benchmark.
Finally, after long-sequence fine-tuning (denoted by †), the accuracy of Vim-Tiny, Vim-Small, and Vim-Base was further improved, demonstrating that fine-tuning with long sequences can yield better performance while maintaining smaller parameter counts.
Performance Comparison at High Resolution
The figure measures Tiny-backbone batch-inference throughput (FPS, on a logarithmic axis) and GPU memory for feature extraction, not end-to-end single-request latency or training memory:
- At 512×512 resolution: Vim and DeiT have similar FPS, but Vim uses less GPU memory (3.32 GB versus 4.56 GB).
- At 1248×1248 resolution:
- Vim's inference speed is 2.8 times faster than DeiT.
- GPU memory usage is only 13.2% of DeiT's (saving 86.8%).
The main reason for this is that self-attention mechanisms grow quadratically with sequence length , whereas Vim's SSM module grows linearly. When is large, the speed and memory advantages of Vim become more pronounced.
Performance in Image Segmentation
After testing image classification, the authors evaluated the performance of Vim in image segmentation. They conducted experiments on the ADE20K dataset, using UPerNet as the base segmentation framework, showing Vim's superiority over self-attention-based backbone networks.
The experimental results, shown in the table above, demonstrate that Vim outperforms DeiT in mIoU at the tested Tiny and Small scales. For example, Vim-Tiny is 1.8 mIoU points higher than DeiT-Tiny, and Vim-Small is 0.9 points higher than DeiT-Small. This supports Vim as a segmentation backbone in this setting.
UPerNet + Vim-Small and UPerNet + ResNet-101 both reach 44.9 mIoU, with roughly 46M versus 86M parameters for the segmentation systems in the table. The near-halving is not a backbone-only parameter comparison.
Performance in Object Detection
COCO 2017 detection and instance segmentation use Cascade Mask R-CNN. The ViT backbone follows ViTDet’s window/global-attention configuration; Vim does not need that modification.
Table 3 gives box AP 44.4→45.7 (+1.3) and mask AP 38.1→39.2 (+1.1). Small-object box AP is tied. The paper’s prose says +1.3 for both overall AP metrics; the values here follow the table.
Medium-object box/mask AP improve by 1.6/1.3 and large-object scores by 1.4/1.8. These results are consistent with the long-context explanation.
Through the COCO detection and instance segmentation experiments, the authors show that the Vim backbone achieves higher AP at the tested high-resolution input without the DeiT backbone's 2D window attention. For computational efficiency, we still need the earlier speed and memory measurements.
Ablation Experiment - Bidirectional Design
The authors describe five direction-handling strategies, and the table reports results for four of them, to understand their impact on ImageNet classification and ADE20K segmentation. For segmentation, a two-layer Segmenter head tests the transferability of backbone outputs. No class token is used in this ablation.
- None: Directly using the original Mamba's unidirectional block, only performing forward direction. For image sequences, this approach may perform well for classification, but it likely sacrifices the capture of "spatial positioning" or "contextual information," affecting downstream segmentation.
- Bidirectional Sequence: During training, the visible sequence is randomly flipped in direction. This essentially acts as data augmentation, not providing bidirectional processing in the model structure.
- Bidirectional Block: Every two blocks are paired: the first block handles forward, and the second handles backward. In other words, in the model stack, odd layers are forward, even layers are backward, but there is no simultaneous forward and backward processing within the same layer.
- Bidirectional SSM: In each block, besides the original forward SSM, an additional backward SSM is added, allowing each block to compute the sequence in both directions simultaneously.
- Bidirectional SSM + Conv1d: Building on the Bidirectional SSM, an additional backward Conv1d module is added, applying a 1D convolution before the backward SSM.
The experimental results are shown in the table below:
- None (Unidirectional Mamba): Classification is reasonable at 73.2% Top-1, but segmentation is relatively weak at 32.3 mIoU. The absence of backward context may matter more for the dense prediction task.
- Bidirectional Layer/Block: Classification falls from 73.2% to 70.9%, a drop of 2.3 percentage points. Segmentation rises by 1.3 to 33.6 mIoU, so even splitting the directions across blocks helps in this setting. The paper's prose says a seven-point classification drop, inconsistent with v3 Table 4; these values follow the table.
- Bidirectional SSM: The table gives 72.8% Top-1 and 33.2 mIoU.
- Bidirectional SSM + Conv1d (full bidirectional strategy): This is the best balance in the table, at 73.9% Top-1 and 35.9 mIoU. It improves classification over Bidirectional SSM's 72.8% and segmentation over the None baseline's 32.3 mIoU. Integrating the forward and backward paths within a block, with a backward 1D convolution, becomes the design of the Vim block.
The original illustration is retained; read these numbers against v3 Table 4. The next section’s 76.1 uses a different configuration with a class token.
Ablation Experiment - Classification Strategies
The authors wanted to test whether using different "classification strategies" under the SSM-based backbone would affect the final ImageNet results. They proposed 5 strategies for comparison:
- Mean pool: Perform average pooling on all the last-layer outputs (tokens) before classification. No class token is used.
- Max pool: Perform max pooling on all sequence tokens. Similarly, no class token is used, and each token has a "local" classification head, with max pooling deciding the final prediction.
- Head class token: Mimic DeiT by adding a class token at the beginning of the sequence, using this token for final classification.
- Double class token: Place a class token at both the "start" and "end" of the sequence and combine them for classification, intending to capture both "beginning" and "ending" information.
- Middle class token: Insert a class token in the "middle" of the sequence, using this token for final classification. The authors believe this design takes full advantage of the recursive nature of SSM and the prior knowledge that the main object is typically centered in ImageNet images.
The experimental results are shown in the table below:
The results show that the Middle class token provides the best Top-1 accuracy, achieving 76.1%, 0.9 percentage points above the head class token's 75.2%.
The middle token is best in this ablation. Information flow and ImageNet’s tendency toward centered objects are the authors’ explanation, not causal relationships established by this table alone.
Conclusion
In this study, the authors introduced Vision Mamba (Vim), aiming to extend the latest and most efficient Mamba into a general-purpose vision backbone network.
Vim does not rely on self-attention and achieves sufficient global context through bidirectional SSM. It maintains reasonable computational and memory consumption even when dealing with longer sequences or higher-resolution images, and performs excellently on multiple standard vision tasks.
The most impressive concept from this paper is undoubtedly the "bidirectional sequence modeling."
Consider this: since images are two-dimensional, how can we look only "left to right" or "right to left"? Could we look vertically? Could we look diagonally?
We can explore various ways to "look":
Additionally, concepts like MAE and CLIP could be applied here. MAE uses "masked image modeling," and CLIP-style multimodal pretraining might further enhance Vim's performance.
It seems there's a ton of papers waiting for us to read next!