Large Language Models Can Be Easily Distracted by Irrelevant Context

By Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Scharli, Denny Zhou. International Conference on Machine Learning, 2023

1,096 citations recorded by the source corpus.

Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3d68522abfadfc8ee6b7ec9edaaf91f1b2f38e5e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, background): “Hallucinations refer to instances where the language model starts generating inaccurate or irrelevant information based on its insufficient understanding (Wei et al., 2022; Kojima et al., 2022; Shi et al., 2023).”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “The question of “when to retrieve” has led to strategies ranging from single [ Wang et al. , 2023e, Shi et al. , 2023 ] to adaptive [Jiang et al. , 2023b, Huang et al. , 2023] and multiple retrieval [Izacard et al. , 2022] methods.”
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023, methodology): “We can find that our method consistently improves reasoning performance across multiple models and datasets, indicating its capa-Standard Figure 3: Evaluation results on GSMIC (Shi et al., 2023).”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, background): “In the context of arithmetric reasoning, Shi et al. (2023) showed that adding irrelevant context to exemplars or task specific instructions can help, suggesting the model may be equipped with such skills from pre-training.”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, result): “This observation aligns with the study by Shi et al. (2023a) on how LLMs might be distracted by irrelevant context in mathematics problems.”
VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, positive): “First, using too many frames introduces extensive information and noise, which leads to performance degradation because LLMs suffer from long contexts and can be easily distracted [24,40].”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, background): “(2) Longer prompts introduce irrelevant and redundant information, which can weaken LLMs’ performance (Shi et al., 2023), as illustrated in Figure 1a. (3) LLMs exhibit position bias (Kamradt, 2023), also known as the "lost in the middle" issue (Liu et al., 2024), suggesting that the placement of…”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “Such knowledge is also referred to as context knowledge (Shi et al., 2023b).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Irrelevant evidence can be propagated into the generation phase, possibly tainting the output (Li et al., 2023a; Shi et al., 2023a; Cho et al., 2023; Xu et al., 2023).”
Can Language Models Solve Graph Problems in Natural Language? (2023, methodology): “Particularly, simple math problem datasets such as AQUA Ling et al. [2017], GSM8K [Cobbe et al., 2021], and SVAMP [Patel et al., 2021] are used for evaluating arithmetic reasoning [He-Yueya et al., 2023, Touvron et al., 2023, Shi et al., 2023].”
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023, methodology): “Previous research has found similar inconsistencies in LLMs (Fluri et al., 2023; Elazar et al., 2021; Press et al., 2023; Hosseini et al., 2021; Lin et al., 2022; Shi et al., 2023) Forward vs backward recall in humans Does the Reversal Curse apply to humans?”
Benchmarking Cognitive Biases in Large Language Models as Evaluators (2023, methodology): “In addition to the original instruction, we follow a similar setup from (Shi et al., 2023) where we include irrelevant information about one of the comparand models to test the ability of evaluators.”
Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024, neutral): “All four of the hospitalists are internal medicine physicians with emergency department experience.”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, positive): “Small changes in input tokens can drastically alter model outputs, indicating a strong token bias and suggesting that these models are highly sensitive and fragile (Jiang et al., 2024; Shi et al., 2023).”
Agentless: Demystifying LLM-based Software Engineering Agents (2024, background): “That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [84, 107].”