LongNet: Scaling Transformers to 1, 000, 000, 000 Tokens

By Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei. 2023

265 citations recorded by the source corpus.

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. In this work, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between tokens; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.

Subfields: Artificial Intelligence, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Scaling & Learning Theory

Boris Ginsburg

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, background): “Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
A foundation model for clinical-grade computational pathology and rare cancers detection (2024, unknown): “…of patterns, including cellular morphology, tissue architecture, staining characteristics, nuclear morphology, mitotic figures, necrosis, inflammatory response, neovascularization and biomarker expression and therefore would be well-suited to predicting a wide variety of WSI characteristics.”
A whole-slide foundation model for digital pathology from real-world data (2024, unknown): “Prov-GigaPath slide encoder used the LongNet model architecture with standard settings 5 .”
Evaluating Large Language Models in Class-Level Code Generation (2024, background): “Although there is a trend that LLMs can take longer and longer inputs [25], the effectiveness of utilization remains questionable.”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, background): “…include extending the context window (Press et al., 2022; Chen et al., 2023), utilizing recurrent memory (Dai et al., 2019; Bulatov et al., 2023), using sparsed attention (Ding et al., 2023; Mohtashami and Jaggi, 2023), and augmenting with an external memory (Liang et al., 2023; Zhou et al., 2023).”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “2: while l < y .len () do 3: if Substring y l ∈ (cid:101) x then 4: Find the longer substring (cid:101) y key ,l = { y l , y l +1 , ..., y r } ∈ (cid:101) x .”
RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “Various sparse attention mechanisms (Child et al., 2019; Jaszczur et al., 2021) such as shifted sparse attention (Chen et al., 2024), dilated attention (Ding et al., 2023), and attention sinks (Han et al., 2023; Xiao et al., 2024b) were employed to enable efficient context scaling.”
A Comprehensive Overview of Large Language Models (2023, unknown): “Reward ranked fine-tuning RAFT [150] fine-tunes the model on ranked responses by the reward model.”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, unknown): “There are plenty of works [7, 8, 11, 32, 48, 55, 65] to address this long-standing and prominent challenge, but few of them focus on visual applications.”