Longformer: The Long-Document Transformer
By Iz Beltagy, Matthew E. Peters, Arman Cohan. 2020
5,720 citations recorded by the source corpus.
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
Subfields: Artificial Intelligence, Large Language Models, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Yoshua Bengio
- Yejin Choi
- Graham Neubig
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Mohit Bansal
- Furu Wei
- Jianfeng Gao
- Yejin Choi
- Hannaneh Hajishirzi
- Timo Schick
In Model Architectures
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
- Mohit Bansal
- Kyunghyun Cho
- Alan L. Yuille
- Tri Dao
- Richard Socher
Citations in the corpus
- Mistral 7B (2023, background): “Mistral 7B leverages grouped-query attention (GQA) [1], and sliding window attention (SWA) [6, 3].”
- Rethinking Attention with Performers (2020, methodology): “…(Roy et al., 2020) which applies k -means clustering to learn dynamic sparse attention regions, or (Kitaev et al., 2020), where locality sensitive hashing is used to group together tokens of similar embeddings), sliding windows (Beltagy et al., 2020), or truncated targeting (Chelba et al., 2020).”
- Big Bird: Transformers for Longer Sequences (2020, background): “We compare performance in predicting the masked out tokens in terms of bits per character, following [8].”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “√ For large sequence lengths 𝑁 , 𝑠 is often set to 𝑁 − 1 / 2 [12] or 𝑁 − 1 log 𝑁 [3, 18, 96], resulting in Θ ( 𝑁 𝑁 ) or Θ ( 𝑁 log 𝑁 ) IO complexity.”
- SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “Longformer avoids computing quadratic-complexity attention via sliding-window attention, where each word only attends to a constant number of nearby tokens, in addition to a few tokens that compute global attention over the entire input.”
- Code Llama: Open Foundation Models for Code (2023, background): “Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “Longformer [3] further introduces dilated sliding window patterns to increase attention’s receptive field and manually picks the window sizes for each layer.”
- HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2020) Longformer (Beltagy et al., 2020)”
- R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “In preliminary experiments of this work we used a Longformer encoder (Beltagy et al., 2020) with concatenated passages at it’s input to benefit from the early fusion between passages.”
- Adaptive Semiparametric Language Models (2021, background): “, 2019), (b) longformer (Beltagy et al., 2020), and (c) compressive transformer (Rae et al.”
- Making Pre-trained Language Models Better Few-shot Learners (2021, background): “Issues (2) and (3) might be ameliorated with longer-context language models (e.g., Beltagy et al., 2020).”
- It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “While there are some variants of the Transformer that can deal with much longer contexts (e.g., Kitaev et al., 2020; Beltagy et al., 2020), it has yet to be investigated to what extent such models are able to actually make good use of priming examples over such a long context span.”
- Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “While repetition often helps in augmenting human understanding, short and concise instructions are often more effective for computers due to their limited attention span (Beltagy et al., 2020).”
- When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, methodology): “While the method may be limited by the fact that some models can face difficulties in processing long pieces of text (Beltagy et al., 2020), this issue is partly mitigated by marginalizing over k sets of explanations.”