Neural Text Summarization: A Critical Evaluation

By Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher. 2019

403 citations recorded by the source corpus.

Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original document. Despite increased interest in the community and notable research effort, progress on benchmark datasets has stagnated. We critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary shortcomings: 1) automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited diversity in their outputs.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, background): “However, at the time of writing, ROUGE remains a dominant metric in summarization research (Kryściński et al., 2019).”
Fine-Tuning Language Models from Human Preferences (2019, background): “Kry´sci´nski et al. (2019) found that 30% of samples from the supervised summarization models they tested contained inconsistencies, and Khandelwal et al. (2019) found that their pretrained encoder-decoder model “hallucinates facts...which are topical but never appear in the source”.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “In a more recent study, Kryscinski et al. (2019) 5- way annotated system outputs on 100 samples from the test set of the CNN-Dailymail summarization corpus (CNNDM, Hermann et al., 2015; Nallapati et al., 2016) along two measures of content quality (relevance of the content, and faithfulness) and…”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “Although ROUGE clearly has its draw-backs (Kryscinski et al., 2019), over-penalizing abstractive approaches com- pared to extractive ones and having no sense of linguistic quality, we found that choosing perplexity-optimized models using aggregated ROUGE (rather than directly optimizing ROUGE as in…”
Learning to summarize from human feedback (2020, background): “Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “However, summarization is much more subjective, which can make the task underconstrained and difﬁ-cult to evaluate (Kry´sci´nski et al., 2019).”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “inate training signal (Kryściński et al., 2019).”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, methodology): “This could be because summarization has a large search space and is often an underconstrained task for the model as showed by Kryscinski et al. (2019).”
Beyond Goldfish Memory: Long-Term Open-Domain Conversation (2021, background): “Summarization is a rich field where the vast majority of work focuses on summarizing documents (Kaikhah, 2004; Kryściński et al., 2019; Cheng and Lapata, 2016), for example summarizing in order to predict other relevant information (West et al.”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, methodology): “This inadequacy leaves human evaluation as the primary method for evaluating the factual consistencies, which has been noted to be challenging even for humans (Daume III and Marcu, 2005; Kryscinski et al., 2019b), in addition to being slow and costly.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Recent studies have shown that around 30% of generated summaries contain unfaithful information (Cao et al., 2018; Falke et al., 2019a; Kry´sci´nski et al., 2019), especially when the sentence combines content from multiple source sentences (Lebanoff et al., 2019).”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “…while recent state-of-the-art generative models (Lewis et al., 2019; Zhang et al., 2019a) produce ﬂuent summaries, they frequently contain false or unsupported information (Kry´sci´nski et al., 2019), a phenomenon also known as neural hallucination (Rohrbach et al., 2018; Zhao et al., 2020).”
Evaluation of Text Generation: A Survey (2020, methodology): “As a result, factual-consistency-aware text generation research has drawn a lot of attention in the community in recent years (Kryscinski et al., 2019; Gunel et al., 2019; Kryściński et al., 2019; Zhang et al., 2019b; Wang et al., 2020).”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, background): “Previous studies of factuality in summarization only distinguish factual and hallucinated content (Kryscinski et al., 2019; Maynez et al., 2020) and provide limited insights on the fine-grained types of factual errors.”
Fill in the BLANC: Human-free quality estimation of document summaries (2020, neutral): “human evaluation (Kryściński et al., 2019a).”