Do Long-Range Language Models Actually Use Long-Range Context?

By Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, Mohit Iyyer. 2021

107 citations recorded by the source corpus.

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the Routing Transformer, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “A growing amount of literature shows that predicting the next token is mostly a local task that does not require modeling long-range dependencies (Khandelwal et al., 2018; Sun et al., 2021), and that masking or down-weighting distant tokens can actually improve perplexity (Press et al., 2021a,b).”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, background): “This effect has been noted in previous work (Press et al., 2021; Sun et al., 2021) and further provides evidence that current language model performance on small benchmarks is not bottlenecked by their limited range context.”
The NLP Task Effectiveness of Long-Range Transformers (2022, background): “These metrics may not be ideal (Sun et al., 2021) and may not reﬂect performance on complex NLP tasks (Arutiunian et al., 2020; Thorne et al., 2021).”
The Power of Noise: Redefining Retrieval for RAG Systems (2024, unknown): “However, they often struggle with understanding semantic relationships between words [22].”
Memorizing Transformers (2022, background): “Experiments on the PG19 dataset (Sun et al., 2021) have shown that relative position does not appear to matter at long range, and the T5 relative bias puts all long-range tokens in the same bucket anyway.”
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024, unknown): “However, as Liu et al. (2023) and Sun et al. (2021) have noted, models tend to underutilize long-range context and see diminishing performance as context length increases, especially when pertinent information is embedded within a lengthy context.”
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (2023, background): “Long-context models often struggle to beneﬁt from the full context (Sun et al., 2021).”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, background): “However, as suggested in (Sun et al., 2021), the perplexity metric may not necessarily reflect the model’s performance on sequence-level tasks in real applications Meanwhile, some works assess long text modeling through artificial tasks such as retrieval (Tay et al., 2021; Chen et al., 2023; Li et…”
Lost in the Middle: How Language Models Use Long Contexts (2023, background): “Tay et al. (2021) propose the Long Range Arena, which evaluates long-context models on a variety of natural language, visual reasoning, and synthetic tasks.”
RankGen: Improving Text Generation with Large Ranking Models (2022, background): “This issue, combined with the observation that LMs overly rely on local context (Khandelwal et al., 2018; Sun et al., 2021), contributes to the generation of sequences that break coherence or consistency within a larger discourse-level context (Wang et al., 2022).”