Retentive Network: A Successor to Transformer for Large Language Models

By Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei. arXiv.org, 2023

719 citations recorded by the source corpus.

In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/240103933ffe3dac2179cc160a2bd91299357a53

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “This has generated signiﬁcant interest in making attention faster in the long-context regime, including by approximation [13, 27, 55] and software optimization ([16, 29, 45]), or even alternative architectures [22, 42, 54].”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “The chunkwise linear attention algorithm has also been independently discovered in several works [105, 44, 18].”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, background): “We believe that our work not only offers practical insights for those working on LLM training, but also paves the way for future research in this rapidly evolving field.”
RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “Finally, novel architectures (Gu et al., 2022; Fu et al., 2023a; Poli et al., 2023; Fu et al., 2023b; Sun et al., 2023a; Beck et al., 2024; Sun et al., 2024) such as Mamba (Gu & Dao, 2023) and RWKV (Peng et al., 2023) have also been proposed to efficiently handle long-context input.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, background): “More recently, retentive networks have been proposed as a successor to transformers [65].”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Efforts, such as retentive networks [193], can be integrated within VLMs for high throughput processing.”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, methodology): “We first discuss aspects of hardware that should be taken into account for a practically efficient implementation.”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, neutral): “Following the notation of Sun et al. (2023b); Yang et al. (2024a;b), we take the query block, q , as an example.”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, background): “Add the subsequence x i,j = { x i , x i +1 , ..., x j } to the response y rec .”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, background): “There are plenty of works [7, 8, 11, 32, 48, 55, 65] to address this long-standing and prominent challenge, but few of them focus on visual applications.”
VMamba: Visual State Space Model (2024, methodology): “Inspired by continuous state space models in control systems, combined with HiPPO [15] initialization, LSSL [18] showcases the potential in handling long range dependency problems.”
LLM-Based Doppelgänger Models: Leveraging Synthetic Data for Human-Like Responses in Survey Simulations (2024, unknown): “To address the high memory usage issue of Transformer-based LLM models when processing long input tokens, RETNET [21] has been proposed.”