Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

By Albert Gu, Isys Johnson, Karan Goel, Khaled Kamal Saab, Tri Dao, Atri Rudra, Christopher R'e. 2021

1,197 citations recorded by the source corpus.

Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings. The Linear State-Space Layer (LSSL) maps a sequence $u \mapsto y$ by simply simulating a linear continuous-time state-space representation $\dot{x} = Ax + Bu, y = Cx + Du$. Theoretically, we show that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a trainable subset of structured matrices $A$ that endow LSSLs with long-range memory. Empirically, stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use hand-crafted features on 100x shorter sequences.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “HiPPO [37] and its extensions, most notably S4 [33, 38, 39] projects the history on a polynomial basis, allowing accurate reconstruction of the history through state-space models.”
Simplified State Space Layers for Sequence Modeling (2022, background): “[18] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.”
MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, positive): “In Mamba, a 1D continuous input Discretization The continuous parameters A , B and C in the above formulation are further converted into discrete parameters for better computational efﬁciency [8].”
Jamba: A Hybrid Transformer-Mamba Language Model (2024, methodology): “Jamba is based on a novel hybrid architecture, which combines Transformer layers [46] with Mamba layers [16], a recent state-space model [17, 18], as well as a mixture-of-experts (MoE) component [13, 41].”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, neutral): “Some SSM-based methods, such as the linear state-space layers (LSSL) [22], structured state space sequence model (S4) [21], diagonal state space (DSS) [24], and S4D [23], are proposed to process sequence data across a wide range of tasks and modalities, particularly on modeling long-range…”
LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, background): “Structured State Space Models (SSMs) have recently gained prominence as a versatile architecture in sequence modeling, heralding a new era of balancing computational efficiency and model versatility [9,12,13,35].”
PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, background): “Linear state space equations [15, 18], combined with deep learning, offer a compelling approach for modeling sequential data, presenting an alternative to CNNs or Transformers.”
RS-Mamba for Large Remote Sensing Image Dense Prediction (2024, methodology): “SSMs have gained significant traction in the field of deep learning in recent years, marking a remarkable evolution in the way long-range dependencies and sequential data are handled [28], [29], [30].”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “Another intriguing view was provided by the SGConv model (Li et al., 2022a), which leverages the convolutional interpretation of SSMs (Gu et al., 2021b) to design a purely ﬁlter-based version of S4, with no latent continuous-time model or need for discretization.”
Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, positive): “For instance, Vim [33] proposed a generic vision backbone with bidirectional Mamba blocks, while VMamba [16] introduced a cross-scan module to solve the direction-sensitive problem due to the difference between 1D sequences and 2D images.”
Is Mamba Effective for Time Series Forecasting? (2024, positive): “The Mamba layer takes a sequence 𝑿 ∈ ℝ 𝐵 × 𝑉 × 𝐷 as input, where 𝐵 denotes the batch size, 𝑉 denotes the number of variates, and 𝐷 denotes hidden dimension.”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “The token mixer of Mamba is the structured state space models (SSM) [28–30], under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86,…”
MambaIR: A Simple Baseline for Image Restoration with State-Space Model (2024, background): “State Space Models (SSMs) [24, 25, 62], stemming from classics control theory [33], are recently introduced to deep learning as a competitive backbone for state space transforming.”
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation (2024, background): “Recently, state space sequence models (SSMs) [11,15], in particular structured state space sequence models (S4) [14], have emerged as an efficient and effective building block (or layer) for constructing deep networks, which obtained cutting-edge performance in continuous long-sequence data…”
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting (2022, methodology): “S 4 model Gu et al. (2021a) comes up with another recursive memory design for data representation and signiﬁcantly improves state-of-the-art results for Long-range forecasting benchmark (LRA) Tay et al. (2020).”
VMamba: Visual State Space Model (2024, methodology): “State Space Models (SSMs) are recently proposed models that are introduced into deep learning as state space transforming [18, 17, 43].”
MambaHSI: Spatial–Spectral Mamba for Hyperspectral Image Classification (2025, background): “Recently, state-space models (SSMs) [20], [21], in particular structured state-space sequence models (S4) [22], have emerged as an efficient and effective layer to construct deep networks, and have obtained superior performance in continuous long-sequence data analysis [22], [23].”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, neutral): “…(Wu et al., 2019), Long Convolu-tion (Fu et al., 2023c; Qin et al., 2023a; Poli et al., 2023; Massaroli et al., 2023; Li et al., 2023d; Romero et al., 2021), and State Space Models (Gu et al., 2021a; Gupta & Berant, 2022; Gu et al., 2021b; Hasani et al., 2022; Smith et al., 2023; Ma et al., 2023).”