Long Range Language Modeling via Gated State Spaces

By Harsh Mehta, Ankit Gupta, Ashok Cutkosky, Behnam Neyshabur. International Conference on Learning Representations, 2022

377 citations recorded by the source corpus.

State space models have shown to be effective at modeling long range dependencies, specially on sequence classification tasks. In this work we focus on autoregressive sequence modeling over English books, Github source code and ArXiv mathematics articles. Based on recent developments around the effectiveness of gated activation functions, we propose a new layer named Gated State Space (GSS) and show that it trains significantly faster than the diagonal version of S4 (i.e. DSS) on TPUs, is fairly competitive with several well-tuned Transformer-based baselines and exhibits zero-shot generalization to longer inputs while being straightforward to implement. Finally, we show that leveraging self-attention to model local dependencies improves the performance of GSS even further.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/eaef083b9d661f42cc0d89d9d8156218f33a91d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “Based on it, various models have been developed ( e.g. , S5 [67], H3 [21], and GSS [57]).”
PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, background): “This work underscores the potential of SSMs in 3D vision-related tasks and presents a simple yet effective Mamba-based baseline for future research.”
Hungry Hungry Hippos: Towards Language Modeling with State Space Models (2022, methodology): “The H3-attention hybrid also outperforms the GSS-attention hybrid (Mehta et al., 2022).”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, neutral): “Chunkwise Parallel”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “In addition to earlier applications in NLP (Mehta et al., 2022), more sophisticated architectures based on S4 recently showed great promise in language modeling (Dao et al.”
VMamba: Visual State Space Model (2024, background): “These models are then integrated into large representation models [39, 37, 13].”
Hyena Hierarchy: Towards Larger Convolutional Language Models (2023, methodology): “Comparing operators We repeat our associative recall experiment, this time benchmarking different 2 layer models rather than changing the convolution parametrization: an order 2 Hyena, GSS (Mehta et al., 2022), H3 (Dao et al.”
U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, background): “Recently, Mamba [18] has achieved a groundbreaking milestone with its linear-time inference and efficient training process by integrating selection mechanism and hardware-aware algorithms into previous works [19, 21, 57].”
MambaHSI: Spatial–Spectral Mamba for Hyperspectral Image Classification (2025, background): “Mehta et al. [52] introduced more gating units to improve the expressivity of S4.”
LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, background): “Structured State Space Models (SSMs) have recently gained prominence as a versatile architecture in sequence modeling, heralding a new era of balancing computational efficiency and model versatility [9,12,13,35].”
Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation (2024, background): “Recent developments in State Space Models (SSMs) [6,22,27], especially Structured SSMs (S4) [8], offer a promising solution with their efficient performance in processing long sequences.”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, neutral): “[46] builds the Gated State Space layer on S4 by introducing more gating units to improve the expressivity.”