VideoMamba: State Space Model for Efficient Video Understanding

By Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, Yu Qiao. European Conference on Computer Vision, 2024

558 citations recorded by the source corpus.

Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3af7273d7ca20c0c63cbaa47e60b058840835052

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhikang Zou
2. Zhihao Ouyang
3. Jinmin Li
4. Xiang Bai
5. Xin Zhou
6. Xiao Tan
7. Hang Guo
8. Xingkui Zhu

Citations in the corpus

• MambaIR: A Simple Baseline for Image Restoration with State-Space Model (2024, background): “Moreover, there are also pioneering works that adopt Mamba to vision tasks such as image classification [44, 92], video understanding [37, 66], biomedical image segmentation [48,71] and others [28,31,56,59,76].”
• PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, background): “Besides, the great potential of Mamba motivates a series of work across diverse domains, including graph [50, 3], medical segmentation [36, 34], video understanding [30, 7] and generative models [29, 31].”