Improving Factual Consistency of Abstractive Summarization via Question Answering

By Feng Nan, C. D. Santos, Henghui Zhu, Patrick Ng, K. McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, Bing Xiang. 2021

103 citations recorded by the source corpus.

A commonly observed problem with the state-of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The fact that automatic summarization may produce plausible-sounding yet inaccurate summaries is a major concern that limits its wide application. In this paper we present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency; next, we propose a novel learning algorithm that maximizes the proposed metric during model training. Through extensive experiments, we confirm that our method is effective in improving factual consistency and even overall quality of the summaries, as judged by both automatic metrics and human evaluation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Studies on these tasks have explored (1) modifying attention mechanisms to specifically capture relationships between entities (Nan et al., 2021; Zhu et al., 2021), (2) improving coreference to mitigate gender bias in translation (Levy et al., 2021), and (3) using text entailment to develop loss…”
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022, unknown): “QA-based methods (Wang et al., 2020; Durmus et al., 2020; Honovich et al., 2021; Nan et al., 2021) generate questions from the document and summary, and compare the corresponding answers in order to assess factuality.”
Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation (2021, background): “…correctness has been studied in multiple generation tasks such as summarization (Maynez et al., 2020; Lux et al., 2020; Kryscinski et al., 2020; Nan et al., 2021; Zhou et al., 2020), document-level generation (Massarelli et al., 2020) and natural language generation (NLG) (Dušek and Kasner,…”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Furthermore, in text summarization, some studies proposed evaluation metrics or correction methods to measure and revise the generated text for preserving factuality [46, 172].”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, methodology): “…triples extracted from source documents (Cao et al., 2018; Zhu et al., 2021), optimizing summarization models towards a factual consistency metrics (Nan et al., 2021; Cao and Wang, 2021), learning a post-editing error corrector (Dong et al., 2020) and removing noisy training samples (Kang and…”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “…GPT-3 In previous studies, different metrics and methods have been proposed to measure and improve the factual accuracy of language generation within different tasks [11], including text summarization [e.g., 12–15], question answering [e.g., 16–18], and table-to-text generation [e.g., 19, 20].”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2021, background): “Arumae and Liu (2019) optimize a QA-based consistency reward, and Nan et al. (2021) streamline a QAG reward by combining the QG and QA model, making it more efficient for RL training.”