MambaOut: Do We Really Need Mamba for Vision?*

By Weihao Yu, Xinchao Wang. Computer Vision and Pattern Recognition, 2024

309 citations recorded by the source corpus.

Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism and subsequently applied to vision tasks. Nevertheless, the performance of Mamba for vision is often underwhelming when compared with convolutional and attention-based models. In this paper, we delve into the essence of Mamba, and conceptually conclude that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. For vision tasks, as image classification on ImageNet does not align with either characteristic, we hypothesize that Mamba is not necessary for this task; Detection and segmentation tasks on COCO or ADE20K are also not autoregressive, yet they adhere to the long-sequence characteristic, so we believe it is still worthwhile to explore Mamba’s potential for these tasks. To empirically verify our hypotheses, we construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM. Experimental results strongly support our hypotheses. Specifically, our MambaOut model surpasses all visual Mamba models on ImageNet image classification, indicating that Mamba is indeed unnecessary for this task. As for detection and segmentation, MambaOut cannot match the performance of state-of-the-art visual Mamba models, demonstrating the potential of Mamba for long-sequence visual tasks.

Subfields: Education

View Paper: https://www.semanticscholar.org/paper/31fdba3a68f286894f025e734a277e2ce94dd84c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

  1. Zhikang Zou
  2. Xiang Bai
  3. Xin Zhou
  4. Xiao Tan
  5. Xingkui Zhu
  6. Xiaoqing Ye
  7. Wei Xu
  8. Dingkang Liang

Citations in the corpus