Faithful to the Original: Fact Aware Neural Abstractive Summarization

By Ziqiang Cao, Furu Wei, Wenjie Li, Sujian Li. 2017

393 citations recorded by the source corpus.

Unlike extractive summarization, abstractive summarization has to fuse different parts of the source text, which inclines to create fake facts. Our preliminary study reveals nearly 30% of the outputs from a state-of-the-art neural summarization system suffer from this problem. While previous abstractive summarization approaches usually focus on the improvement of informativeness, we argue that faithfulness is also a vital prerequisite for a practical abstractive summarization system. To avoid generating fake facts in a summary, we leverage open information extraction and dependency parse technologies to extract actual fact descriptions from the source text. The dual-attention sequence-to-sequence framework is then proposed to force the generation conditioned on both the source text and the extracted fact descriptions. Experiments on the Gigaword benchmark dataset demonstrate that our model can greatly reduce fake summaries by 80%. Notably, the fact descriptions also bring significant improvement on informativeness since they often condense the meaning of the source text.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “[19] extract fact descriptions from the source text and apply a dualattention seq-to-seq framework to force the summaries to be conditioned on both source documents and the extracted fact descriptions.”
What Makes Good In-Context Examples for GPT-3? (2021, background): “…dialogue generation (Yan et al., 2016; Cai et al., 2018; Song et al., 2016; Pandey et al., 2018; We-ston et al., 2018; Wu et al., 2019), text summarization (Cao et al., 2017; Peng et al., 2019), data-to-text generation (Peng et al., 2019), and text-to-code generation (Hashimoto et al., 2018).”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, background): “Abstractive summarization is particularly interesting because factual consistency with the original text is crucial to usability, and a lack of such consistency has plagued abstractive neural summarization models (Cao et al., 2018; Falke et al., 2019; Kryscinski et al., 2019b, i.a.).”
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (2019, background): “However, these summaries often contain fabricated and repeated content (Cao et al., 2018). Fan et al. (2018) show that, for content selection, existing models rely on positional information and can be easily fooled by adversarial content present”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Recent studies have shown that around 30% of generated summaries contain unfaithful information (Cao et al., 2018; Falke et al., 2019a; Kry´sci´nski et al., 2019), especially when the sentence combines content from multiple source sentences (Lebanoff et al., 2019).”
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (2019, methodology): “Similar to previous work by Cao et al. (2018) and Li et al. (2018), we argue that the correctness of a generated summary can only be reliably evaluated by manual inspection.”
Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization (2018, methodology): “FTSum Cao et al. (2017b) encoded the facts extracted from the source sentence to improve both the faithfulness and informativeness of generated summaries.”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, background): “…attracted increasing attention in recent years (Maynez et al., 2020) both in improving factual consistency of ab-stractive summarization systems (Cao et al., 2018) and in evaluating the factual consistency of generated summaries (Goodrich et al., 2019; Kry´sci´nski et al., 2019; Xu et al.,…”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, background): “, 2020), ∼30% of summaries have factual inconsistencies (Cao et al., 2018).”
Assessing The Factual Accuracy of Generated Text (2019, methodology): “ive summarization models such as those described in Liu et al. [15] may generate sentences with factual inaccuracies (e.g. incorrect month in date of birth, wrong city in the state, etc.). Cao et al. [4] found that 30% of summaries generated by a state-of-the-art summarization model contained factua”
Guiding Extractive Summarization with Question-Answering Rewards (2019, background): “Moreover, the summaries may contain inaccurate factual details and introduce new meanings not present in the original text (Cao et al., 2018; Song et al., 2018).”
Factual Error Correction for Abstractive Summarization Models (2020, methodology): “Previous work on factual consistency in abstractive summarization can be divided into two categories: abstractive summarization models tailored towards factual consistency (Cao et al., 2018; Zhang et al., 2019b; Li et al., 2018), and evaluation models for factual consistency in abstractive summariza”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Recent efforts by NLP researchers have drawn attention to the issue of factual errors and hallucinations in the output of neural summarization models (Cao et al., 2018; Massarelli et al., 2019; Zhao et al., 2020; Falke et al., 2019b; Goodrich et al., 2019; Celikyilmaz et al., 2020).”
Reinforced Extractive Summarization with Question-Focused Rewards (2018, background): “Failing to reproduce factual details has been revealed as one of the main obstacles for neural abstractive summarization (Cao et al., 2018; Song and Liu, 2018).”
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022, methodology): “We use BART-base (Lewis et al., 2020) as our sequence-to-sequence model for the inﬁlling based data generator and the fact correction model.”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, background): “Diverse Generation Models There has been a surge of interest in making language models generate more diverse and human-like outputs.”
Factual Error Correction for Abstractive Summaries Using Entity Retrieval (2022, neutral): “However, factual inconsistency between the original text and the summary is frequently observed in the abstractive summarization system (Cao et al., 2018; Zhao et al., 2020; Maynez et al., 2020).”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, background): “tual consistency, including applying attentions to fact triples extracted from source documents (Cao et al., 2018; Zhu et al., 2021), optimizing summarization models towards a factual consistency metrics (Nan et al.”