Large Language Models Struggle to Learn Long-Tail Knowledge

By Nikhil Kandpal, H. Deng, Adam Roberts, Eric Wallace, Colin Raffel. 2022

694 citations recorded by the source corpus.

The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training. We identify these relevant documents by entity linking pre-training datasets and counting documents that contain the same entities as a given question-answer pair. Our results demonstrate strong correlational and causal relationships between accuracy and relevant document count for numerous question answering datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models are better at learning long-tail knowledge, we estimate that today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data. Finally, we show that retrieval-augmentation can reduce the dependence on relevant pre-training information, presenting a promising approach for capturing the long-tail.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “We further investigate this phenomenon across model checkpoints and model sizes by adapting arithmetic tasks of multiplication and addition (Razeghi et al., 2022) and a QA task (Kandpal et al., 2022) using natural language prompts evaluated over a set of k - shot settings.”
Explainability for Large Language Models: A Survey (2023, background): “The impact of different demonstration types appears to vary depending on the model scale and task type.”
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, background): “Second, as LLMs grow in size, their ability to re-member facts and statements from pre-training improves (Kandpal et al., 2022; Tirumala et al., 2022; Carlini et al., 2023, 2020).”
Extracting Training Data from Diffusion Models (2023, background): “One area of recent interest is memorization in language models for text, where past work shows that adversaries can extract training samples using two-step attack techniques that resemble our approach [11, 47, 41, 40].”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “, 2021), or focuses on question answering with short answers (Kadavath et al., 2022; Kandpal et al., 2022; Mallen et al., 2022; Nori et al., 2023).”
A Survey of Deep Learning for Mathematical Reasoning (2022, neutral): “, 2022) or factual question answering (Kandpal et al., 2022).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “• Long-tail Knowledge Within the vast knowledge landscape that LLMs draw upon, a notable challenge emerges in the form of long-tail knowledge utilization (Kandpal et al., 2023; Mallen et al., 2023).”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, methodology): “, 2022) further improve performance on diverse knowledge-intensive tasks, leveraging their strong parametric memories (Kandpal et al., 2022; Yu et al., 2022).”
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024, neutral): “…encoded within their parameters (Petroni et al., 2019; Jiang et al., 2020; Talmor et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Chowdhery et al., 2022; Bubeck et al., 2023; Kandpal et al., 2023) and models can be further improved with fine-tuning on downstream tasks (Roberts et al., 2020).”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, background): “It has been shown that LMs have difficulty solving problems involving long-tail entities (Kandpal et al., 2022; Mallen et al., 2022) and complex mathematical reasoning challenges (Mishra et al.”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, methodology): “Meanwhile, LMs are shown to have limited memorization for less frequent entities (Kandpal et al., 2022), are prone to hallucinations (Shuster et al., 2021), and suffer from temporal degradation (Luu et al., 2022; Jang et al., 2022).”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “Despite these advancements, LLMs exhibit notable limitations, particularly in handling domain-specific or highly specialized queries [ Kandpal et al. , 2023 ] .”
The rise and potential of large language model based agents: a survey (2025, neutral): “Although LLMs have been aligned with human values to mitigate biased outputs, the models still struggle to portray minority groups well due to the long-tail eﬀect of the training data [466–468].”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, neutral): “8), the gold program is quite simple thus it is not surprising that et al., 2022; Peng et al., 2023; Kandpal et al., 2022a; Magar and Schwartz, 2022).”
Retrieval-based Language Models and Applications (2023, background): “However, relying solely on their parameters to encode a wealth of world knowledge requires a prohibitively large number of parameters and hence massive computing, and they often struggle to learn long-rail knowledge (Roberts et al., 2020; Kandpal et al., 2022; Mallen et al., 2022).”
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, background): “Second, even the largest models struggle to memorise knowledge that is only rarely mentioned in the training corpus (Kandpal et al., 2022; Mallen et al., 2023).”