Resurrecting Recurrent Neural Networks for Long Sequences

By Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, Soham De. International Conference on Machine Learning, 2023

516 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. In this paper, we show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, while also introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

Subfields: Artificial Intelligence, Control and Systems Engineering, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/f393aff1593c2d370ec0ae004910d18e40524967

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Qian
2. Fei Wang
3. Xiao Liu
4. Yuxiao Dong
5. Zhengxiao Du
6. Stella Biderman
7. Shan You
8. Bo Peng

In Control and Systems Engineering

1. Shan You
2. Bo Peng
3. Fei Wang

In Materials Chemistry

1. Fei Wang
2. Yuxiao Dong

Citations in the corpus

• Long-context LLMs Struggle with Long In-context Learning (2024, unknown): “Furthermore, architectural innovations, transitioning from traditional Transformer-based designs to recurrent models or state space models, have shown promise in facilitating long-range computations naturally Orvieto et al. (2023); Gu & Dao (2023); Peng et al. (2023a).”
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, background): “…2020; Beltagy et al., 2020; Zaheer et al., 2020; Wang et al., 2020; Fedus et al., 2022; Ding et al., 2023), recurrent and memory modules (Dai et al., 2019; Rae et al., 2020; Wu et al., 2022; Martins et al., 2022; Bulatov et al., 2022; Orvieto et al., 2023; Liang et al., 2023; Zhou et al., 2023).”
• RWKV: Reinventing RNNs for the Transformer Era (2023, methodology): “In parallel, RNN-style (Hochreiter and Schmidhuber, 1997; Chung et al., 2014) recursive components have also been modified to increase context length, such as the Recurrent Memory Transformer (Bulatov et al., 2022, 2023) and Linear Recurrent Units (Orvieto et al., 2023).”
• LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, background): “State Space Models (SSMs) [11, 13, 16, 27, 37], represent a paradigm in architecture designed for sequence-to-sequence transformation, adept at managing long dependency tokens.”
• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “…has rapidly evolved from using data-independent decay mechanisms, as exemplified by models like S4 (Gu et al., 2022), S5 (Smith et al., 2023), LRU (Orvieto et al., 2023), RWKV4/5 (Peng et al., 2023), and RetNet (Sun et al., 2023a), to incorporating data-dependent decay mechanisms in more recent…”