Entity-level Factual Consistency of Abstractive Text Summarization

By Feng Nan, Ramesh Nallapati, Zhiguo Wang, C. D. Santos, Henghui Zhu, Dejiao Zhang, K. McKeown, Bing Xiang. 2021

197 citations recorded by the source corpus.

A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Precision-source (Nan et al., 2021), defined as [|NER(x) ∩ NER(c)|]/|NER(x)| , the percentage of named entities in the summary that can be found in the source.”
Survey of Hallucination in Natural Language Generation (2022, methodology): “[134] propose an entity-based metric relying on the Named-Entity Recognition model, which is relatively more robust.”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, methodology): “In this work, we speciﬁcally adopt the NE-based metric [64] that is designed with a belief that a model is hallucinating (making factual errors) if it generates a NE that does not appear in the ground-truth knowledge source.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Nan et al. (2021) introduced a metric to quantify the extent of entity hallucination, which calculates the precision of named-entities in the summary against the source entities.”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, background): “Three types of remedies have been proposed: running a separately learned error correction component (Dong et al., 2020), removing noisy training samples (Nan et al., 2021; Goyal and Durrett, 2021), and modifying the Transformer architecture (Huang et al., 2020; Zhu et al., 2021).”
Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search (2022, methodology): “…correcting already-generated summaries (Dong et al., 2020; Zhu et al., 2021), using a knowledge graph (Zhu et al., 2021), filtering training data (Nan et al., 2021), constraining generation with keywords (Mao et al., 2020), using NLI models (Barrantes et al., 2020; Mishra et al., 2020), among…”
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization (2021, methodology): “Nan et al. (2021) show that data filtering159 and use a summary-worthy entity classification task160 as an auxiliary training objective can help improve161 model’s entity-level factuality.162 Filippova (2020) proposed a method for control-163 ling hallucination in data-to-text generation task.164…”
Improving Factual Consistency of Abstractive Summarization via Question Answering (2021, background): “In another related work, factual consistency metrics at the entity level have been proposed (Nan et al., 2021).”
Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation (2020, unknown): “In automatic evaluation, CAS improves BERTSum by 0.7/3.9 on entity-level F1 (Nan et al., 2021) and 4.9/0.7 on support score (Matsumaru et al., 2020) for CNNDM/XSum, respectively.”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, background): “However, studies have found that 30–50% of summaries created this way contain facts that are unsupported by the rest of the article (Tejaswin et al., 2021; Nan et al., 2021).”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, methodology): “In an attempt to improve factuality in an end-to-end fashion, Nan et al. (2021) and Goyal and Durrett (2021) explore a useful method of removing nonfactual examples during training, but this only allows the model to be trained on a small portion of the training data.”
Controlling Conditional Language Models without Catastrophic Forgetting (2021, methodology): “Constraints Following (Nan et al., 2021), we define an entity-level factual consistency constraint as a product of two constraints: there must be at least four named entities in the summary x and all the named entities x must have occurred in the source c.”