The limits of automatic summarisation according to ROUGE

By Natalie Schluter. 2017

125 citations recorded by the source corpus.

This paper discusses some central caveats of summarisation, incurred in the use of the ROUGE metric for evaluation, with respect to optimal solutions. The task is NP-hard, of which we give the first proof. Still, as we show empirically for three central benchmark datasets for the task, greedy algorithms empirically seem to perform optimally according to the metric. Additionally, overall quality assurance is problematic: there is no natural upper bound on the quality of summarisation systems, and even humans are excluded from performing optimal summarisation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “Human Evaluation In addition to automatic evaluation using ROUGE which can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017), we also evaluated system output by eliciting human judgments in two ways.”
Learning to summarize from human feedback (2020, background): “Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
On Faithfulness and Factuality in Abstractive Summarization (2020, methodology): “ness of summaries (Schluter, 2017).”
Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, methodology): “In addition to ROUGE which can be misleading when used as the only means to assess the informativeness of summaries (Schluter, 2017), we also evaluated system output by eliciting human judgments in two ways.”
Learning by Semantic Similarity Makes Abstractive Summarization Better (2020, background): “rics are known to be have high correlation with human judgement, they have limitations and may mislead the actual quality of a generated text (Schluter, 2017; Maynez et al., 2020; Gehrmann et al., 2021).”
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, result): “Despite our generally superior results on the ROUGE metric, past work (Schluter, 2017; Hoang et al., 2019; Sun et al., 2019) has explored the limitations of ROUGE for evaluating summarization tasks, including the issue that ROUGE measures n-gram overlap without distinguishing between salient n-grams”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, methodology): “Automatic metrics for evaluating text summarization have been well-documented as correlating poorly with various human judgments of quality (Schluter, 2017; Kryscinski et al., 2019; Durmus et al., 2020).”
Towards Summarization for Social Media - Results of the TL;DR Challenge (2019, background): “Furthermore, a key issue of ROUGE is that it does not provide any upper bounds for the quality of a summarization system (Schluter, 2017), thus warranting an extensive manual evaluation of the systems.”
Cut the CARP: Fishing for zero-shot story evaluation (2021, methodology): “Unfortunately, methods for automatically evaluating stories, such as ROUGE (Lin, 2004), are extremely limited in their ability to accurately assess models and are easily Goodharted (Belz & Reiter, 2006; Cohan & Goharian, 2016; Schluter, 2017; Schluter & Alonso, 2016; Eyal et al., 2019).”
HighRES: Highlight-based Reference-less Evaluation of Summarization (2019, methodology): “measure performance in summarization (Schluter, 2017), also known for other tasks in which the goal is to generate natural language (Novikova et al.”