Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

By Tri Dao, Albert Gu. International Conference on Machine Learning, 2024

1,701 citations recorded by the source corpus.

While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ca9f5b3bf0f54ad97513e6175b30497873670fed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bailin Wang
2. Yoon Kim
3. Yikang Shen
4. Yu Zhang
5. Songlin Yang

Citations in the corpus

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “Recent linear recurrent models typically incorporate a lightweight depthwise-separable convolution layer after the query/key/value projections [30, 9, 18].”
• Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, neutral): “Since our first version was released, RNN layers with matrix (linear) hidden states have also been further advanced in Mamba 2 [17] and Gated DeltaNet [82].”
• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, neutral): “The equivalence between these parallel and recurrent forms is also referred to as the state space duality (SSD) described in Dao & Gu (2024a).”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “We repeat the benchmarks 10 times and take the average timing.”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “Recent advances in linear RNNs (Peng et al., 2023, 2024; Qin et al., 2024b), state space models (Gu and Dao, 2024; Dao and Gu, 2024), and linear attention (Katharopoulos et al., 2020; Yang et al., 2024) offer alternatives with theoretical linear complexity.”