RWKV: Reinventing RNNs for the Transformer Era

By Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella and 21 more. Conference on Empirical Methods in Natural Language Processing, 2023

1,130 citations recorded by the source corpus.

Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/026b3396a63ed5772329708b7580d633bb86bec9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Limin Wang
2. Yu Qiao
3. Yali Wang
4. Kunchang Li
5. Boris Ginsburg
6. Yi Wang
7. Yinan He
8. Samuel Kriman

Citations in the corpus

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, neutral): “For example, while most recent works make use of depthwise-separable convolution layers (not shown in Table 4) [30, 76, 90, 9, 18], earlier works generally do not [47, 99, 79].”
• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, positive): “…data-independent decay mechanisms, as exemplified by models like S4 (Gu et al., 2022), S5 (Smith et al., 2023), LRU (Orvieto et al., 2023), RWKV4/5 (Peng et al., 2023), and RetNet (Sun et al., 2023a), to incorporating data-dependent decay mechanisms in more recent architectures such as HGRN1/2…”
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “In this section, we define and compare various important metrics:”
• Long-context LLMs Struggle with Long In-context Learning (2024, positive): “Furthermore, architectural innovations, transitioning from traditional Transformer-based designs to recurrent models or state space models, have shown promise in facilitating long-range computations naturally Orvieto et al. (2023); Gu & Dao (2023); Peng et al. (2023a).”
• Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, neutral): “Transformer-based Pre-trainedLLM,AM,VLM General : Time-LLM [46], OFA [124], LLM4TS [11], PromptCast [105], TEMPO [10], LLMTime [35], Voice2Series [108], AutoTimes [64], UniTime [61] Finance : Yu et al. [112], Chen et al. [19], Xie et al. [103], Wimmer et al. [99] Healthcare : Liu et al. [63] Self-su”
• RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “We evaluate the effective context length for two models with non-Transformer architectures: RWKV-v5 (Peng et al., 2023) and Mamba-2.”
• U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, background): “In contrast, structured state-space sequence models (SSMs) [16, 18, 67] have recently shown high efficiency and effectiveness in long-sequence modeling, and are promising solutions for long-term dependency modeling in visual tasks.”
• MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “More recently, a new wave of RNN-like models has emerged [28, 29, 45, 65, 106], drawing significant interest from the community for their capability of paral-lelizable training and performing efficient inference on long sequences.”
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, positive): “Probabilistic sequence modeling plays a crucial role in diverse machine learning applications including natural language processing [6, 47], video prediction [31, 69] and decision making [3, 22].”
• ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, neutral): “Retrieve.”
• VMamba: Visual State Space Model (2024, methodology): “Inspired by continuous state space models in control systems, combined with HiPPO [15] initialization, LSSL [18] showcases the potential in handling long range dependency problems.”
• VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “The emergence of low-cost operators such as S4 [26], RWKV [74], and Ret-Net [71] in the NLP domain, has carved a novel pathway for the vision model.”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “This has generated significant interest in making attention faster in the long-context regime, including by approximation [13, 27, 55] and software optimization ([16, 29, 45]), or even alternative architectures [22, 42, 54].”
• Gated Linear Attention Transformers with Hardware-Efficient Training (2023, methodology): “Such models have been the focus of much recent work as a competitive sub-quadratic alternative to the Transformer architecture (Peng et al., 2023; Gu & Dao, 2023; Qin et al., 2023c;b; Sun et al., 2023a; Wang et al., 2022).”