Detecting Hallucinated Content in Conditional Neural Sequence Generation

By Chunting Zhou, Jiatao Gu, Mona T. Diab, P. Guzmán, Luke S. Zettlemoyer, Marjan Ghazvininejad. 2020

234 citations recorded by the source corpus.

Neural sequence models can generate highly fluent sentences but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input, which can cause a lack of trust in the model. To better assess the faithfulness of the machine outputs, we propose a new task to predict whether each token in the output sequence is hallucinated conditioned on the source input, and collect new manually annotated evaluation sets for this task. We also introduce a novel method for learning to model hallucination detection, based on pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations. Experiments on machine translation and abstract text summarization demonstrate the effectiveness of our proposed approach -- we obtain an average F1 of around 0.6 across all the benchmark datasets and achieve significant improvements in sentence-level hallucination scoring compared to baseline methods. We also release our annotated data and code for future research at this https URL.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “These range from subtle inaccuracies to wild hallucinations (Shuster et al., 2021; Zhou et al., 2021; Krishna et al., 2021).”
You Only Need One Model for Open-domain Question Answering (2021, background): “More importantly however, these language models often suffer from Hallucinations (Shuster et al., 2021; Maynez et al., 2020; Zhou et al., 2021; Zellers et al., 2019; Roller et al., 2021; Mielke et al., 2020), generating factually incorrect but plausiblesounding outputs.”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, background): “ai/projects/hallucination/ machine translation (Zhou et al., 2020), and news generation (Zellers et al.”
Survey of Hallucination in Natural Language Generation (2022, methodology): “[168] ine-tune pre-trained language models on synthetic data with automatically inserted hallucinations to detect the hallucinatory content in summaries.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Other possible ﬁxes include learning task-speciﬁc metrics to measure semantic overlap (Sellam et al., 2020) or metrics to check factual correctness (Zhang et al., 2020) and faithfulness to input (Wang et al., 2020; Durmus et al., 2020; Zhou et al., 2020).”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “…texts that match given facts is a known challenge (Fan et al., 2020; Kryscinski et al., 2020; Lewis et al., 2020b; Parikh et al., 2020; Shah et al., 2020; Tian et al., 2020) as language models tend to degenerate and hallucinate (Holtzman et al., 2020; Schuster et al., 2020; Zhou et al., 2020).”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, background): “Prior works have highlighted the potential for misuse [24, 4], bias [46], and hallucination [48] in neural language models.”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, methodology): “…token-level embedding matching (Zhang et al., 2020a) and leverage extrinsic capabilities such as question answering (Eyal et al., 2019; Wang et al., 2020) and entailment classiﬁ-cation (Falke et al., 2019; Kry´sci´nski et al., 2019; Zhou et al., 2020) to align variable spans and entire sentences.”
Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “Domain Adaptation/Generalization Domain Adaptation (DA) (Wang & Deng, 2018) and Domain Generalization (DG) (Zhou et al., 2021b) also follow “open-world” assumption.”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, methodology): “…detect hallucination in the generated text using statistical n-gram matching (Dhingra et al., 2019; Liu et al., 2019), edit distance heuristics (Zhou et al., 2021), natural language inference (Kryscin-ski et al., 2020; Falke et al., 2019), information extraction (Zhang et al., 2020; Goodrich…”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “Such capabilities attract increasing attention (Zhou et al., 2021) as they enable both better evaluation and better generation models via ﬁltering training data (Gehrmann et al., 2021) or training data augmentation for controlled generation (Rashkin et al., 2021b).”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, background): “, 2020), there has been significant effort to develop automated metrics that can detect such errors (Zhou et al., 2021; Gabriel et al., 2021; Pagnoni et al., 2021a).”
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports (2022, background): “Hallucinations are an intrinsic problem of generative models like LLMs [15] and difficult to remedy [16, 17].”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “Prior to the advent of LLMs, the NLP community has primarily defined “hallucination” as the generation of non-sensical content or content that deviates from its sources [28, 32, 39, 57, 74, 76].”
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey (2021, unknown): “In ﬁxed schema extraction, a relation is predicted Metric Category Summarization Dataset Code Goodrich et al. [2019] Triple-based Wikipedia Falke et al. [2019] Textual-Entailment-based CNN/DM Mishra et al. [2020] Textual-Entailment-based CNN/DM QAGS [Wang et al. , 2020] QA-based CNN/DM, XSum FEQA [D”
Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation (2021, background): “…been studied in multiple generation tasks such as summarization (Maynez et al., 2020; Lux et al., 2020; Kryscinski et al., 2020; Nan et al., 2021; Zhou et al., 2020), document-level generation (Massarelli et al., 2020) and natural language generation (NLG) (Dušek and Kasner, 2020; Thomson and…”