On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?

By Nouha Dziri, Sivan Milton, Mo Yu, Osmar R Zaiane, Siva Reddy. 2022

237 citations recorded by the source corpus.

Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of > 60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.

Subfields: Artificial Intelligence, Social Psychology, Sociology and Political Science, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

In Sociology and Political Science

In Natural Language Processing

Citations in the corpus

Improving alignment of dialogue agents via targeted human judgements (2022, background): “Measuring and optimising towards the supportedness of evidence is important for assessing and increasing the rate at which responses are faithfully-grounded in external knowledge, and reducing the problem of hallucinations (Dziri et al., 2022).”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, background): “This is consistent with recent analyses on WOW, CMU-DOG, and TOPICALCHAT which revealed that more than 60% of the ground-truth responses are unattributable to the knowledge (Dziri et al., 2022b; Rashkin et al., 2021a).”
Generative artificial intelligence (2023, background): “The underlying causes of hallucinations are still being researched, with early findings suggesting that the training data that might contain contradictory or fictional content besides factual information (Dziri et al., 2022).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Besides, Dziri et al. (2022) argue that hallucination is also present in human-generated corpora (can be reflected as out-dated (Liska et al., 2022; Luu et al., 2022), biased (Chang et al., 2019; Garrido-Muñoz et al., 2021), or fabricated (Penedo et al., 2023) expression).”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “For example, prior work has shown that RAMs generate text that either includes additional information not present in the retrieved document (Dziri et al., 2022), ignores the document altogether (Krishna et al.”
Explainability for Large Language Models: A Survey (2023, background): “This additional textual information helps to provide more accurate and explainable diagnoses from CLIP [Radford et al. 2021].”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “For example, prior work has shown that RAMs generate text that either includes additional information not present in the retrieved document (Dziri et al., 2022), ignores the document altogether (Krishna et al.”