VMamba: Visual State Space Model

By Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Yunfan Liu. Neural Information Processing Systems, 2024

2,640 citations recorded by the source corpus.

Designing computationally efficient network architectures remains an ongoing necessity in computer vision. In this paper, we adapt Mamba, a state-space language model, into VMamba, a vision backbone with linear time complexity. At the core of VMamba is a stack of Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module. By traversing along four scanning routes, SS2D bridges the gap between the ordered nature of 1D selective scan and the non-sequential structure of 2D vision data, which facilitates the collection of contextual information from various sources and perspectives. Based on the VSS blocks, we develop a family of VMamba architectures and accelerate them through a succession of architectural and implementation enhancements. Extensive experiments demonstrate VMamba's promising performance across diverse visual perception tasks, highlighting its superior input scaling efficiency compared to existing benchmark models. Source code is available at https://github.com/MzeroMiko/VMamba.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b24e899ec0f77eef2fc87a9b8e50516367aa1f97

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Yu Qiao
3. Hongpeng Liao
4. Chien-Yao Wang
5. Xinchao Wang
6. Dacheng Tao
7. Luc Van Gool
8. Wenyu Liu

Citations in the corpus

• MambaOut: Do We Really Need Mamba for Vision?* (2024, positive): “Experimental results demonstrate that the simpler MambaOut model, in reality, already surpasses the performance of visual Mamba models [41, 56, 96, 112], which in turn verifies our Hypothesis 1 .”
• Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation (2024, methodology): “In each of encoder and decoder, 2 VSS blocks are utilized, and the pretrained VMamba-Tiny [16] is loaded in the encoder, following the same process that Swin-UNet load the pre-trained SwinViT-Tiny [3].”
• U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, methodology): “Building on the success of Mamba, for visual application, Vision Mamba [53] and VMamba [95] use bidirectional Vim Block and the Cross-Scan Module respectively to gain data-dependent global visual context.”
• YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
• Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, neutral): “05 following [16].”
• ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Model (2024, methodology): “Very recently, VMamba [30] and Vision Mamba [31] have extended Mamba architecture to 2D image data, showing superior performance on many computer vision tasks.”
• Gated Linear Attention Transformers with Hardware-Efficient Training (2023, neutral): “…(especially modalities with long-range dependencies), in line with recent work on applying state-of-the-art state-space models to other types of data (Yan et al., 2023; Zhu et al., 2024; Ma et al., 2024; Liu et al., 2024; Xing et al., 2024; Wang et al., 2024a;b; Yang et al., 2024, inter alia ).”
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, neutral): “VMamba [20] proposed a generic Mamba-based vision back-bone featuring a Cross-Scan Module (CSM).”
• MambaIR: A Simple Baseline for Image Restoration with State-Space Model (2024, methodology): “To better utilize the 2D spatial information, we follow [44] and introduce the 2D Selective Scan Module (2D-SSM).”
• Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, methodology): “Different from them, we build a pure-SSM-based model, which can be adopted as a generic vision backbone.”
• RSMamba: Remote Sensing Image Classification With State Space Model (2024, methodology): “Vim [16] and VMamba [17] have successfully introduced Mamba into the 2-”
• SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention (2024, neutral): “Additionally, VMamba [32] proposes a cross-scanning module that extends 1D sequence scanning to 2D image scanning, effectively capturing multi-semantic global context information from four directions.”
• YOLOv12: Attention-Centric Real-Time Object Detectors (2025, neutral): “…vision transformers, due to their complex designs ( e.g. , window partitioning/reversing in Swin trans-former [39]) and the introduction of additional modules ( e.g. , positional encoding), gradually accumulate speed overhead, resulting in an overall slower speed compared to CNN architectures [38].”
• LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, methodology): “( a) and (b): Previous methods Vim [60] and VMamba [32] traverse the entire row or column axis, resulting in significant distances for capturing dependencies between neighboring pixels within the same semantic region ( e.g. , the left eye in the image).”
• VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “This innovation has spurred its adoption in vision tasks, as evidenced by Vision Mamba [91] and VMamba [50], which leverage multi-directional SSMs for enhanced 2D image processing.”
• RS-Mamba for Large Remote Sensing Image Dense Prediction (2024, background): “VMamba [15] extends this with SS2D module by conducting selective scanning both horizontally and vertically, enhancing the model’s global effective receptive field in both dimensions.”
• PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, methodology): “In the vision domain, Vision Mamba [35] compresses the visual representation through bidirectional state space models.”
• SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation (2024, methodology): “Additionally, VMamba [14] designs a CSM module to bridge the gap between 1-”
• VM-UNet: Vision Mamba UNet for Medical Image Segmentation (2024, positive): “SSM-based models, such as HiPPO [14], LSSL [16], and others, which are derived from linear state-space equations in control theory, are primarily employed for sequence data modeling tasks.”