Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference

By Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, Iryna Gurevych. 2019

344 citations recorded by the source corpus.

While recent progress on abstractive summarization has led to remarkably fluent summaries, factual errors in generated summaries still severely limit their use in practice. In this paper, we evaluate summaries produced by state-of-the-art models via crowdsourcing and show that such errors occur frequently, in particular with more abstractive models. We study whether textual entailment predictions can be used to detect such errors and if they can be reduced by reranking alternative predicted summaries. That leads to an interesting downstream application for entailment models. In our experiments, we find that out-of-the-box entailment models trained on NLI datasets do not yet offer the desired performance for the downstream task and we therefore release our annotations as additional test data for future extrinsic evaluations of NLI.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “analyze mispredictions on a set of previously annotated summarization corpora (Kryscinski et al., 2020; Wang et al., 2020; Falke et al., 2019; Maynez et al., 2020).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Some other notable directions in this space involve fact-aware pretraining (Falke et al., 2019; Wan and Bansal, 2022) and structured learning frameworks (Wiseman et al., 2018).”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “In line with our ﬁndings, Falke et al. (2019) observed that the BERT-based NLI models substantially improved summaries reranking in terms of their correctness.”
Survey of Hallucination in Natural Language Generation (2022, background): “[45] discover that models trained on NLI datasets can not transfer well to abstractive summarization tasks, degrading the reliability of NLI-based hallucination metrics.”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, methodology): “This aligns with the conclusions of (Falke et al., 2019), where the authors showed that for the task of factual consistency the inter-annotator agreement coefﬁcient κ reached 0.75 only when 12 annotations were collected for each example.”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, methodology): “6 Re-ranking with QAGS Several works explore the use of natural language inference (NLI) models to detect factual consistency in generated text (Welleck et al., 2019; Falke et al., 2019).”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “In addition to QA, recent work has used relation extraction and textual entailment models for faithfulness evaluation (Falke et al., 2019a; Goodrich et al., 2019).”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “Recently, several efforts have been made on automatic evaluation of faithfulness and truthfulness in abstract text summarization (Wang et al., 2020a; Maynez et al., 2020; Durmus et al., 2020; Falke et al., 2019) by leveraging existing semantic inference systems: (1) question answering (QA) system-ba”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, background): “65 between three crowd annotators and expert annotations of factuality (Falke et al., 2019).”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “5, we observe that (1) BARTS CORE + CNN can almost match human baseline on Rank19 and outperform all other metrics, including the most recent top-performing factuality metrics FactCC and QAGS by a large margin.”
Factual Error Correction for Abstractive Summarization Models (2020, methodology): “…fact triples (Cao et al., 2018; Zhang et al., 2019b; Goodrich et al., 2019), applying natural language inference or question answering models for consistency checking (Falke et al., 2019; Li et al., 2018; Wang et al., 2020) and training the model on artiﬁcial datasets (Kry´sci´nski et al., 2019).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “with up to 30% factual inconsistencies (Kryscinski et al., 2019; Falke et al., 2019a; Zhu et al., 2020).”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, neutral): “(2020) and report on textual entailment (Pasunuru and Bansal, 2018; Falke et al., 2019; Kryscinski et al., 2020).”
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2019, methodology): “Falke et al. (2019) explored using natu-ral language inference systems to evaluate the correctness of generated summaries, and found models trained on existing datasets to be inadequate.”
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2020, methodology): “…recent work in summarization, dialogue, or data-to-text generation have attempted at ﬁxing this problem by using natural language inference (NLI) (Falke et al. 2019; Welleck et al. 2019), question answering (QA) (Wang, Cho, and Lewis 2020), or information extraction (Goodrich et al. 2019)…”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, neutral): “SummaC benchmark consists of 6 summarization datasets: CogenSum (Falke et al., 2019), XSumFaith (Maynez et al., 2020), Polytope (Huang et al., 2020), FactCC (Kryscinski et al., 2020), SummEval (Fabbri et al., 2021) and FRANK (Pagnoni et al., 2021).”
Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search (2022, background): “These methods tend to reduce the problem of generating consistent text to another difﬁcult problem (e.g. information extraction (IE) or NLI).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “In line with this, numerous studies (Falke et al., 2019; Maynez et al., 2020) have trained classifiers on NLI datasets to identify factual inaccuracies, especially in the context of abstract summarization.”