Mamba: Linear-Time Sequence Modeling with Selective State Spaces
By Albert Gu, Tri Dao. arXiv.org, 2023
8,131 citations recorded by the source corpus.
Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5$\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.
Subfields: Artificial Intelligence, Model Architectures
View Paper: https://arxiv.org/abs/2312.00752
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yu Qiao
- Dacheng Tao
- Limin Wang
- Yonatan Belinkov
- Hongpeng Liao
- Chien-Yao Wang
- Xinchao Wang
- Mohit Bansal
In Model Architectures
- Yu Qiao
- Boris Ginsburg
- Shan You
- Zhikang Zou
- Samuel Kriman
- Shai Shalev-Shwartz
- Klaus Hermann Maier-Hein
- I-Hau Yeh
Citations in the corpus
- SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention (2024, neutral): “Currently, many studies develop efficient models [66, 35, 4, 53] based on multi-semantic ideas, reducing parameters and computation for enhanced inference efficiency.”
- Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, positive): “…et al., 2023), RWKV4/5 (Peng et al., 2023), and RetNet (Sun et al., 2023a), to incorporating data-dependent decay mechanisms in more recent architectures such as HGRN1/2 (Qin et al., 2024b; 2023b), Mamba1/2 (Gu & Dao, 2023; Dao & Gu, 2024a), RWKV6 (Peng et al., 2024), GSA (Zhang et al., 2024).”
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “Recent linear recurrent models typically incorporate a lightweight depthwise-separable convolution layer after the query/key/value projections [30, 9, 18].”
- A Survey on Mixture of Experts in Large Language Models (2024, positive): “Jamba [66] enhances efficiency by incorporating the architecture of Mamba [148], utilizing a 1:7 ratio of attention-to-Mamba layers.”
- MambaHSI: Spatial–Spectral Mamba for Hyperspectral Image Classification (2025, methodology): “The Mamba denotes the standard Mamba block proposed in [24].”
- nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation (2024, background): “Despite this, the attraction of innovative architectures from the broader computer vision domain, such as Transformers [33] and Mamba [13], persists.”
- Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation (2024, methodology): “The Visual Mamba further introduce Cross-Scan Module (CSM) then integrate convolutional operations into the block, which is detailed in [7,16].”
- YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “In this section, we define and compare various important metrics:”
- Rewrite the Stars (2024, positive): “Star operation exhibits promising performance and efficiency across various research fields, including Natural Language Processing ( i.e. , Monarch Mixer [16], Mamba [17], Hyena Hierarchy [44], 45, 60], with “ ?”
- VMamba: Visual State Space Model (2024, methodology): “Recently, State Space Models (SSMs) [14, 39, 53] have demonstrated great potential for long sequence modeling with linear complexity in natural language processing (NLP) tasks.”
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “This has generated significant interest in making attention faster in the long-context regime, including by approximation [13, 27, 55] and software optimization ([16, 29, 45]), or even alternative architectures [22, 42, 54].”
- Recent advances in 3D Gaussian splatting (2024, neutral): “It is promising to extract geometry, texture, and lighting with more advanced rendering techniques to facilitate independent editing and build the connection between 3DGS and mesh-based representation to enable efficient editing.”
- Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, positive): “Mamba [4] using space state sequential models (SSMs) [5] to reduce the complexity of attention from quadratic to linear for long-sequence modeling in natural language processing.”
- RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “We evaluate the effective context length for two models with non-Transformer architectures: RWKV-v5 (Peng et al., 2023) and Mamba-2.”
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, background): “…et al., 2015), quantization (Hubara et al., 2018; Shen et al., 2020; Kim et al., 2021; Zadeh et al., 2020; Zafrir et al., 2019), pruning (Gale et al., 2019; Sanh et al., 2020; Kur-tic et al., 2022; Voita et al., 2019), and innovative network architecture designs (Gu & Dao, 2023; Wu et al., 2020).”
- Long-context LLMs Struggle with Long In-context Learning (2024, positive): “Furthermore, architectural innovations, transitioning from traditional Transformer-based designs to recurrent models or state space models, have shown promise in facilitating long-range computations naturally Orvieto et al. (2023); Gu & Dao (2023); Peng et al. (2023a).”
- VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “For SSM in the B-Mamba layer, we adopt the default hyperparameters as in Mamba [25]. setting the state dimension and expansion ratio to 16 and 2, respectively.”
- Jamba: A Hybrid Transformer-Mamba Language Model (2024, methodology): “Jamba is based on a novel hybrid architecture, which combines Transformer layers [46] with Mamba layers [16], a recent state-space model [17, 18], as well as a mixture-of-experts (MoE) component [13, 41].”
- Is Mamba Effective for Time Series Forecasting? (2024, neutral): “Some researchers have also considered the application of Transformer-based time series forecasting models in specific domains, such as piezometric level prediction [42], forecasting crude oil returns [1], predicting the power generation by solar panels [49], etc.”