Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization

By Mengyao Cao, Yue Dong, Jackie Chi Kit Cheung. 2021

193 citations recorded by the source corpus.

State-of-the-art abstractive summarization systems often generate hallucinations; i.e., content that is not directly inferable from the source text. Despite being assumed to be incorrect, we find that much hallucinated content is actually consistent with world knowledge, which we call factual hallucinations. Including these factual hallucinations in a summary can be beneficial because they provide useful background information. In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities. Our method is based on an entity’s prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively. Empirical results suggest that our method vastly outperforms two baselines in both accuracy and F1 scores and has a strong correlation with human judgments on factuality classification tasks.Furthermore, we use our method as a reward signal to train a summarization system using an off-line reinforcement learning (RL) algorithm that can significantly improve the factuality of generated summaries while maintaining the level of abstractiveness.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Weizhu Chen
3. Yiming Yang
4. Nan Duan
5. Mohit Bansal
6. Peter Clark
7. Aman Madaan
8. Yelong Shen

In Natural Language Processing

1. Weizhu Chen
2. Jianfeng Gao
3. Nan Duan
4. Yiming Yang
5. Peter Clark
6. Aman Madaan
7. Yelong Shen
8. Niket Tandon

In Safety, Robustness & Interpretability

1. Yelong Shen
2. Weizhu Chen
3. Peter Clark

Citations in the corpus

• Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Hallucinating remains a critical limitation of generative models (Cao et al., 2022), therefore additional heuristics and model calibration might be necessary to make our approach foolproof.”
• Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering (2022, background): “This could be aligned with research areas like measuring factual consistency (Kryściński et al., 2019; Cao et al., 2022) and hallucinations (Cao et al.”
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2023), including abstractive summarization (Maynez et al., 2020; Cao et al., 2022), dialogue generation (Shuster et al.”