The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
By Yi-Chong Huang, Xiachong Feng, Xiaocheng Feng, Bing Qin. 2021
128 citations recorded by the source corpus.
Recently, various neural encoder-decoder models pioneered by Seq2Seq framework have been proposed to achieve the goal of generating more abstractive summaries by learning to map input text to output text. At a high level, such neural models can freely generate summaries without any constraint on the words or phrases used. Moreover, their format is closer to human-edited summaries and output is more readable and fluent. However, the neural model's abstraction ability is a double-edged sword. A commonly observed problem with the generated summaries is the distortion or fabrication of factual information in the article. This inconsistency between the original text and the summary has caused various concerns over its applicability, and the previous evaluation methods of text summarization are not suitable for this issue. In response to the above problems, the current research direction is predominantly divided into two categories, one is to design fact-aware evaluation metrics to select outputs without factual inconsistency errors, and the other is to develop new summarization systems towards factual consistency. In this survey, we focus on presenting a comprehensive review of these fact-specific evaluation methods and text summarization models.
Subfields: Artificial Intelligence, Natural Language Processing, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Caiming Xiong
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Richard Socher
- Yulia Tsvetkov
In Natural Language Processing
- Mohit Bansal
- Yulia Tsvetkov
- Weizhu Chen
- Jianfeng Gao
- Hannaneh Hajishirzi
- Vidhisha Balachandran
- Caiming Xiong
- Xiaojun Wan
In Summarization
- Vidhisha Balachandran
- Xiaojun Wan
- Yulia Tsvetkov
- Tiezheng Yu
- Hannaneh Hajishirzi
- Pascale Fung
- William W. Cohen
- Dragomir R. Radev
Citations in the corpus
- Survey of Hallucination in Natural Language Generation (2022, methodology): “These metrics are based on the idea that only the source knowledge reference should entail the entirety of the information in faithful and hallucination-free generation [37, 41, 44, 72, 75, 86, 89, 123, 190].”
- Evaluating Object Hallucination in Large Vision-Language Models (2023, background): “Especially, LLMs tend to hallucinate unintended text (Huang et al., 2021; Bang et al., 2023), and VLPMs might generate nonexistent objects in a target image (Rohrbach et al.”
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[43] Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin.”
- Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022, background): “[SEP] T he first vaccine for Ebola …….. ready this year, although clinical trials have already started.”
- Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “Thus, evaluating the factual consistency of the generated summaries with respect to the source is an important task (Falke et al., 2019; Cao et al., 2020; Gabriel et al., 2021; Durmus et al., 2020; Huang et al., 2021; Pagnoni et al., 2021).”
- DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, background): “Although there are automatic evaluation metrics for measuring the quality of all aspects of summaries on conventional summarization tasks, especially for factual consistency (Huang et al., 2021), it is difficult to guarantee that they will still perform well on dialogue summarizarion.”
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Generally, hallucinations in natural language generation tasks can be categorized into two primary types: intrinsic hallu-cination and extrinsic hallucination (Huang et al., 2021; Li et al., 2022b; Ji et al., 2023a).”
- Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024, background): “A conventional classification of hallucination is the intrinsic-extrinsic dichotomy [14, 25, 29, 75].”
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, background): “Hallucination has been studied in text generation tasks, including summarization (Huang et al., 2021) and dialogue generation (Shuster et al.”