Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

By Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, Xinggang Wang. International Conference on Machine Learning, 2024

1,963 citations recorded by the source corpus.

Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., the Mamba deep learning model, have shown great potential for long sequence modeling. Meanwhile building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance on self-attention for visual representation learning is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation&memory efficiency. For example, Vim is 2.8$\times$ faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248$\times$1248. The results demonstrate that Vim is capable of overcoming the computation&memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to be the next-generation backbone for vision foundation models. Code is available at https://github.com/hustvl/Vim.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/38c48a1cd296d16dc9c56717495d6e44cc354444

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Limin Wang
2. Yu Qiao
3. Hongpeng Liao
4. Chien-Yao Wang
5. I-Hau Yeh
6. Yali Wang
7. Kunchang Li
8. Yi Wang

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Dacheng Tao
3. Hongpeng Liao
4. Chien-Yao Wang
5. Xinchao Wang
6. Yali Wang
7. Luc Van Gool
8. Limin Wang

In Electrical and Electronic Engineering

1. Hongpeng Liao
2. Chien-Yao Wang
3. I-Hau Yeh
4. Yu Qiao
5. Qixiang Ye
6. Lefei Zhang
7. Lingxi Xie
8. Yali Wang

Citations in the corpus

• RS-Mamba for Large Remote Sensing Image Dense Prediction (2024, methodology): “Vim [14] proposed SS1D module to perform selective scanning in the horizontal direction, enabling each part of the image to perceive global information.”
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, neutral): “Recent efforts like Efficient-Former [18] and FasterViT [10] focused on optimizing efficiency-accuracy trade-offs, achieving competitive performance with high throughput through carefully designed hybrid architectures.”
• Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, positive): “Compared with transformers, Mamba scales linearly or near-linearly with sequence length while maintaining the capability of modeling long-range dependencies, offering higher accuracy, lower memory consumption, and less computational burden [33].”
• RSMamba: Remote Sensing Image Classification With State Space Model (2024, methodology): “Vim [16] and VMamba [17] have successfully introduced Mamba into the 2-”
• VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “Note that di ff erent from UMT, which employs multi-layer alignment between student and teacher models, we align only the final outputs due to VideoMamba’s unique architecture (SSM vs. Transformer), Regarding our masking strategy, we propose di ff erent row masking techniques, depicted in Fig.”
• YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
• ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Model (2024, background): “Very recently, Mamba architecture has been extended to image data and shown promising results on some visual tasks [30], [31].”
• LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, neutral): “For instance, as depicted in Figure 1 (a) and (b), the flattening approach utilized in Vim [60] disrupts these local dependencies, significantly increasing the distance between vertically adjacent tokens and hampering the model’s ability to capture local nuances.”
• MambaHSI: Spatial–Spectral Mamba for Hyperspectral Image Classification (2025, methodology): “Notably, to prove the potential of the SSM model in HSI classification, we only adopted the basic mamba layer as the basic unit, but did not adopt the newer and stronger mamba visual variant [19] designed for visual images.”
• U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, methodology): “Building on the success of Mamba, for visual application, Vision Mamba [53] and VMamba [95] use bidirectional Vim Block and the Cross-Scan Module respectively to gain data-dependent global visual context.”
• VMamba: Visual State Space Model (2024, background): “In terms of computational efficiency, while existing SSM-based vision models typically exhibit significantly better throughput only with large-scale inputs [68] such as 1024 × 1024 , VMamba-T achieves a throughput of 1,335 images/s even at an input resolution of 224 × 224 .”
• MambaIR: A Simple Baseline for Image Restoration with State-Space Model (2024, background): “Moreover, there are also pioneering works that adopt Mamba to vision tasks such as image classification [44, 92], video understanding [37, 66], biomedical image segmentation [48,71] and others [28,31,56,59,76].”
• Is Mamba Effective for Time Series Forecasting? (2024, positive): “In conjunction with our work, two main areas of related work are investigated: (1) time series forecasting, and (2) applications of Mamba.”
• PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, methodology): “Point Cloud Mamba (PCM) [63] combines an improved Mamba module, i.e., Vision Mamba [68], with PointMLP [37] (a strong point cloud analysis method), and incorporates consistent traverse serialization at each stage.”
• YOLOv12: Attention-Centric Real-Time Object Detectors (2025, positive): “Although Mamba-based vision models [38, 70] aim for linear complexity, they still fall short of real-time speeds [38].”
• VM-UNet: Vision Mamba UNet for Medical Image Segmentation (2024, positive): “Building on the foundation of classical SSM [17] research, the modern SSMs (e.g., Mamba [13]) not only establish long-distance dependencies but also exhibit linear complexity with respect to input size.”
• MambaOut: Do We Really Need Mamba for Vision?* (2024, positive): “For instance, Vision Mamba [112] integrates Mamba [28] to develop isotropic vision models akin to ViT [26], demonstrating its long-range modeling capability; VMamba [56] employs Mamba to construct hierarchical vision models similar to AlexNet [47] and ResNet [34]; LocalMamba [41] enhances visual…”
• SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation (2024, methodology): “Meanwhile, Vision Mamba [24] introduces the Vim block, which incorporates bidirectional SSM for data-dependent global visual context modeling and position embeddings for location-aware visual understanding.”