How Language Model Hallucinations Can Snowball

By Muru Zhang, Ofir Press, William Merrill, Alisa Liu, Noah A. Smith. International Conference on Machine Learning, 2023

448 citations recorded by the source corpus.

A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements. Hallucinations are often attributed to knowledge gaps in LMs, but we hypothesize that in some cases, when justifying previously generated hallucinations, LMs output false claims that they can separately recognize as incorrect. We construct three question-answering datasets where ChatGPT and GPT-4 often state an incorrect answer and offer an explanation with at least one incorrect claim. Crucially, we find that ChatGPT and GPT-4 can identify 67% and 87% of their own mistakes, respectively. We refer to this phenomenon as hallucination snowballing: an LM over-commits to early mistakes, leading to more mistakes that it otherwise would not make.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/6825ba09383bc758f9a2feaebabe35a6cd4adc4c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “The fixer can be either another LLM (Peng et al., 2023a; Zhang et al., 2023d; Chern et al., 2023; Gou et al., 2023) or a specific small model (Chen et al., 2023a).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Such inconsistency can result in hallucinations (Wang and Sennrich, 2020), especially when an erroneous token generated by the model cascades errors throughout the subsequent sequence, akin to a snowball effect (Zhang et al., 2023c).”
DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping (2023, background): “…in terms of scalability due to the labor-intensive nature of the annotation process or in terms of data quality due to the hallucination problem (Zhang et al., 2023; Zheng et al., 2023) associated with LLMs. Recent research (Köksal et al., 2023; Li et al., 2023a) has provided a more potential…”
How is ChatGPT's behavior changing over time? (2023, methodology): “…code has also been open-sourced. u Those papers focus on ML services for simple classification tasks such as sentiment analysis, while this work studies generative LLM services. s Chen et al., 2022), and studied how to efficiently estimate ML service performance shifts (L. (MLaaS) more broadly.”
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (2023, result): “…language models specified in parentheses: MiniGPT-4 (Vicuna 13B) (Zhu et al., 2023), LLaVa (LLaMA 13B) (Liu et al., 2023d), MMGPT (LLaMA 7B) (Gong et al., 2023), LLaMA-Adapter (LLaMA 7B) (Zhang et al., 2023b), mPLUG-Owl (LLaMA 7B) (Ye et al., 2023), and InstructBLIP (Vicuna 7B) (Dai et al., 2023).”
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, unknown): “The hallucination [21, 43] refers to the generation of text that is either irrelevant, factually incorrect, or nonsensical in the given context, which is quite severe in current large foundation models.”
A Survey of Hallucination in Large Foundation Models (2023, methodology): “However, (Zhang et al., 2023a) proposed a hypothesis that in certain instances when language models attempt to rationalize previously generated hallucinations, they may produce false statements that they can independently identify as inaccurate.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “It is also possible that these are due to memorization: the model could have memorized the specific input-output mapping of a task, without understanding how to derive the output from the input, and when explicitly instructed to spell out that process, it makes more errors (Zhang et al., 2023b).”