Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE

By Yvette Graham. 2015

149 citations recorded by the source corpus.

We provide an analysis of current evaluation methodologies applied to summarization metrics and identify the following areas of concern: (1) movement away from evaluation by correlation with human assessment; (2) omission of important components of human assessment from evaluations, in addition to large numbers of metric variants; (3) absence of methods of significance testing improvements over a baseline. We outline an evaluation methodology that overcomes all such challenges, providing the first method of significance testing suitable for evaluation of summarization metrics. Our evaluation reveals for the first time which metric variants significantly outperform others, optimal metric variants distinct from current recommended best variants, as well as machine translation metric BLEU to have performance on-par with ROUGE for the purpose of evaluation of summarization systems. We subsequently replicate a recent large-scale evaluation that relied on, what we now know to be, suboptimal ROUGE variants revealing distinct conclusions about the relative performance of state-of-the-art summarization systems.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Analyzing DUC 2004 data (Over and Yen, 2004), where human raters annotated the language quality and the coverage of a summary, i.e., how well it covered the meaning of the source, Graham (2015) found that there was almost no correlation between the two measures.”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “The effectiveness of ROUGE was previously evaluated (Lin, 2004; Graham, 2015) through statistical correlations with human judgment on the DUC datasets (Over and Yen, 2001, 2002, 2003).”
A Structured Review of the Validity of BLEU (2018, background): “Five of these correlations are Low; the sixth (reported in Graham (2015)) is High.”
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020, neutral): “Further, Graham (2015) ﬁnds that ROUGE-L is poorly correlated with human judgements when compared to ROUGE-1,2,3.”
Evaluation of Text Generation: A Survey (2020, methodology): “Outside of machine translation, BLEU has been used for other text generation tasks, such as document summarization (Graham, 2015), image captioning (Vinyals et al., 2014), human-machine conversation (Gao et al., 2019), and language generation (Semeniuta et al., 2019).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Prior work concerning evaluation of automatic metrics and human evaluation for NLG systems has mainly focused on general analysis of output quality or coherence and fluency (Callison-Burch et al., 2007; Graham, 2015; Fabbri et al., 2021), rather than factuality.”
The limits of automatic summarisation according to ROUGE (2017, background): “Previous research on evaluation metrics for automatic summarisation has tried to empirically show a correlation between human judgments and system output quality (Lin, 2004; Lin and Hovy, 2003; Liu and Liu, 2008; Graham, 2015).”
Recent advances in document summarization (2017, methodology): “Also, a recent study [65] reveals some inconsistency for using different ROUGE variants, using pairwise Williams significant test to show that previ-”
BLEU is Not Suitable for the Evaluation of Text Simplification (2018, methodology): “BLEU has also been applied to monolingual translation tasks, such as grammatical error correction (Park and Levy, 2011), summarization (Graham, 2015) and text simplification (Narayan and Gardent, 2014; Štajner et al.”
Language Agnostic Automatic Summarization Evaluation (2020, methodology): “Similar to methods proposed in the AESOP task from 2009 to 2011 (Rankel et al., 2013) none of these methods gained wide-spread use, despite various studies showing problems with ROUGE (see for example (Graham, 2015)).”
Recent automatic text summarization techniques: a survey (2016, methodology): “Conclusion: Results obtained by quantitatively evaluating the summarizers are similar to the results obtained by qualitatively analyzing them through ROUGE.”
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021, background): “Meta-evaluation for Automated Metrics Metaevaluation aims to evaluate the reliability of automated metrics based on their correlation with human judgments (Graham, 2015; Peyrard, 2019; Bhandari et al., 2020).”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, background): “Schluter (2017) and Graham (2015) illustrate the limitations of ROUGE in evaluating summarization tasks.”
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, unknown): “Reference text quality issues and length disparities further reduce reliability (Graham, 2015; Sai et al., 2019; Celiky-ilmaz et al., 2020).”
SUM-QE: a BERT-based Summary Quality Estimation Model (2019, methodology): “Many ROUGE versions are available, but it remains hard to decide which one to use (Graham, 2015).”
Re-evaluating Evaluation in Text Summarization (2020, methodology): “Thus, in meta-evaluation studies, calculating correlation of automatic metrics with human judgments at the system level is a commonly-used setting (Novikova et al., 2017; Bojar et al., 2016; Graham, 2015).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “Within metric meta-analysis, several studies have focused on the analysis of ROUGE (Lin, 2004b), and its variations (Rankel et al., 2013; Graham, 2015), across domains such as news”
Re-evaluating Automatic Metrics for Image Captioning (2016, background): “The test has shown to be valuable for evaluation of document and segment-level machine translation (Graham and Baldwin, 2014; Graham et al., 2015; Graham and Liu, 2016) and summarization metrics (Graham, 2015).”