SummEval: Re-evaluating Summarization Evaluation

By Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Richard Socher, Dragomir R. Radev. 2020

1,023 citations recorded by the source corpus.

Abstract The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Many works like that of Fabbri et al. (2021) would not be possible without access to model outputs, and such corpora can be used for metric development and validation, and to conduct meta evaluations.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “However, recent work by Fabbri et al. (2020) demonstrates that this may not hold in summarization, where the automated metric BERTScore (Zhang et al., 2020b) does not improve upon the correlation of ROUGE.”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “To evaluate QuestEval , we measure its correlation with human judgments on different datasets: 2020), it is one of the largest human-annotated datasets for summarization.”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, methodology): “Kryscinski et al. (2019) and Fabbri et al. (2020) find that standard n-gram based metrics have low correlation with human judgements of factuality.”
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2020, methodology): “We collect human judgment annotations for two of the four quality dimensions studied in Kry´sci´nski et al. (2019); Fabbri et al. (2020), namely consistency and relevance.”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “We can observe that: (1) Simply vanilla BARTS CORE can outperform BERTScore and MoverScore by a large margin on 8 settings except the I NFO perspective on SummEval.”
CTRLsum: Towards Generic Controllable Text Summarization (2020, background): “…reliable human judgement for summarization from non-expert crowdsourcing workers remains an active research direction (Shapira et al., 2019), and it has been shown before that non-expert human judgement of summarization can exhibit poor correlation with expert judgement (Fabbri et al., 2021).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Prior work concerning evaluation of automatic metrics and human evaluation for NLG systems has mainly focused on general analysis of output quality or coherence and ﬂuency (Callison-Burch et al., 2007; Graham, 2015; Fabbri et al., 2021), rather than factuality.”
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022, neutral): “Outside of the medical domain, our work is comparable to Fabbri et al. (2021), who run an automatic metrics correlation study for news article summaries for the CNN/DailyMail dataset (Nallap-ati et al., 2016).”
InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation (2021, methodology): “To compare the different metrics previous work (Bhandari et al. 2020; Fabbri et al. 2020) either relies on the TAC datasets (Dang and Owczarzak 2008; McNamee and Dang 2009) or on new summarization datasets extracted from CNN/DailyMail (Nallapati et al.”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, background): “…most of them cannot reliably substitute human evaluation due to the unstable performance across datasets (Bhandari et al., 2020), weak to moderate correlations with human judgment (Fabbri et al., 2021), or more indication of topic similarity than information overlap (Deutsch and Roth, 2021).”
Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “Translation [24], Question Answering [94, 128], Information Retrieval [52, 130], Text Summarization [47, 49], Word Sense Disambiguation [6, 7, 14, 34, 86], Entity Linking [8, 25, 112], Semantic Role Labeling [18, 32, 33, 35, 105], Semantic Parsing [13, 85], and Natural Language Inference [90, 129], ”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Unlike previous studies that predominantly focused on specific tasks such as summarization (Fabbri et al., 2021; Tang et al., 2022)or particular domains such as world knowledge (Miao et al., 2023), this benchmark assesses factuality across five domains: world knowledge, science and technology,…”
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021, background): “Other summaries might contain signiﬁcant hallucinations, including words that are not found in the source document (Kry´sci´nski et al., 2020; Maynez et al., 2020).”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “…reference-based such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020) are unreliable when small improvements are reported (Peyrard, 2019; Fabbri et al., 2021); however large differences, on the order of say 5 R OUGE points or greater, are considered to be correlated with human…”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “For SMS (Clark et al., 2019), WMDo (Chow et al., 2019), and Mover-Score (Zhao et al., 2019), we use the implementation provided by Fabbri et al. (2020).”
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022, background): “Factuality Evaluation Standard n-gram based metrics do not correlate well with human judgements of factuality and are unsuitable for evaluating factuality (Kryscinski et al., 2019; Fabbri et al., 2020).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, result): “Summarization - FRANK (Pagnoni et al., 2021) 671 + 33.2% - SummEval (Fabbri et al., 2021) 1,600 - 81.6% - MNBM (Maynez et al., 2020) 2,500 - 10.2% - QAGS-CNNDM (Wang et al., 2020) 235 - 48.1% - QAGS-XSum (Wang et al., 2020) 239 - 48.5% Dialogue - BEGIN (Dziri et al., 2021) 836 + 33.7% -”