Re-evaluating Evaluation in Text Summarization

By Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, Graham Neubig. 2020

216 citations recorded by the source corpus.

Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -- for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both system-level and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Bhandari et al. (2020a) revisits this conclusion, demonstrating that metrics disagree whenever the quality range is narrow, regardless of whether the summaries are good or bad.”
Are We Done with MMLU? (2024, neutral): “In NLP, similar issues have been found in summarisation [20, 21] and natural language inference [22, 23, 24, 25] benchmarks.”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “Therefore, to see whether such biases exist, we inspected the rank differences given by human annotators and BARTScore (ﬁne-tuned on CNNDM dataset) on the REALSumm dataset where 24 systems are considered, including both abstractive models and extractive models as well as models based on pre-trained ”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, methodology): “System outputs on text summarization are expanded based on the previous work’s collection (Bhandari et al., 2020) as well as recently state-of-the-art systems (Liu and Liu, 2021) while outputs from machine translation are collected from the WMT20.”
InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation (2021, methodology): “To compare the different metrics previous work (Bhandari et al. 2020) either relies on the TAC datasets (Dang and Owczarzak 2008; McNamee and Dang 2009) or on new summarization datasets extracted from CNN/Dai-lyMail (Nallapati et al. 2016).”
SMART: Sentences as Basic Units for Text Evaluation (2022, result): “Note that, unlike most of previous work on summarization evaluation (Lin, 2004; Clark et al., 2019; Bhandari et al., 2020; Fab-bri et al., 2021), we also take into account source documents S when calculating metric f q ( · ) .”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “∗ Equal contribution for manual evaluation (Bhandari et al., 2020; Fabbri et al., 2022a; Gao and Wan, 2022).”
Measuring Similarity of Opinion-bearing Sentences (2021, background): “Although ROUGE correlates well with human judgments at the system level but it performs poorly at the summary text level (Bhandari et al., 2020).”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, unknown): “…tasks including machine translation (Freitag et al., 2021; Mathur et al., 2020), data-to-text generation (Dhingra et al., 2019), summarization (Bhandari et al., 2020; Pagnoni et al., 2021; Durmus et al., 2020; Gabriel et al., 2021; Fabbri et al., 2021; Durmus et al., 2022), and dialogue…”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, methodology): “…have been a number of manual evaluation datasets and analytical studies for conventional summarization tasks ((Dang and Owczarzak, 2008); Fabbri et al., 2021b; Bhandari et al., 2020), but very little work has been done on systematic analysis of dialogue summarization models and evaluation metrics.”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, methodology): “Different from this method, we keep the exhaustive set (instead of a ﬁxed-size sample) of SCUs for each example (also used by Bhandari et al. (2020)).”
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021, background): “Meta-evaluation for Automated Metrics Metaevaluation aims to evaluate the reliability of automated metrics based on their correlation with human judgments (Graham, 2015; Peyrard, 2019; Bhandari et al., 2020).”
How well do you know your summarization datasets? (2021, unknown): “…2017), (7) P OINTER G ENERATOR C OPYING (See et al., 2017) For CNN/DM, we use the outputs of 11 top-performing summarization systems collected by Bhandari et al. (2020) 4 : (1) H ETER G RAPH (Wang et al., 2020), (2) M ATCH S UMM (Lewis et al., 2020), (3) R EFRESH (Narayan et al., 2018b) , (4) T…”
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022, background): “Furthermore, Bhandari et al. (2020) has studied the limitations of reference-based evaluation metrics of text summarization, comparing these metrics across different datasets and application scenarios.”
Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics (2020, background): “Generally, there are two meta-evaluation strategies: (i) assessing how well each metric correlates with human judgments (Lin, 2004; Ng and Abrecht, 2015; Louis and Nenkova, 2013; Peyrard et al., 2017; Bhandari et al., 2020), which requires procuring manual annotations that are expensive and time-con”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, background): “…al. (2019) note biases and failure modes of abstractive summarization models, while other work analyzes and collects annotations over the output of recent summarization models across multiple dimensions, including factual consistency (Fabbri et al., 2021; Bhandari et al., 2020; Huang et al., 2020).”
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022, methodology): “Although the above deﬁnition of the system-level correlation has been used by recent meta-evaluation studies of metrics (Bhandari et al., 2020; Fabbri et al., 2021; Deutsch et al., 2021b), there is a disconnect between how the automatic metrics are evaluated and how they are used in practice.”