Looking for a Few Good Metrics: Automatic Summarization Evaluation - How Many Samples Are Enough?

By Chin-Yew Lin. 2004

124 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Get To The Point: Summarization with Pointer-Generator Networks (2017, background): “This inﬂexibility of ROUGE is exacerbated by only having one reference summary, which has been shown to lower ROUGE’s reliability compared to multiple reference summaries (Lin, 2004a).”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, methodology): “Only systems that are capable of abstractive reasoning can outperform the ROUGE scores of Fragments.”
Automatically Assessing Machine Summary Content Without a Gold Standard (2013, methodology): “These overlap scores have been shown to correlate well with human assessment (Lin 2004b) and so ROUGE removes the need for manual judgments in this part of the evaluation.”
Efficient Adaptation of Pretrained Transformers for Abstractive Summarization (2019, result): “First, echoing the results of Sun et al. [24], the most common summarization evaluation metric, ROUGE [10], is highly sensitive to summary length, providing an advantage to methods that produce longer summaries, either through learning or with minimum summary length constraints.”
Enhancing Single-Document Summarization by Combining RankNet and Third-Party Sources (2007, methodology): “Recall-Oriented Understudy for Gisting Evaluation (Lin, 2004b), known as ROUGE, measures the quality of a model-generated summary or sentence by comparing it to a “gold-standard”, typically human-generated, summary or sentence.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “Fig.4, Fig.5, and Fig.6 show the power analysis results on CNNDM, XSum and SamSum respectively, where the system pairs are grouped by their performance difference in either ACU or ROUGE1 recall scores.”
A Reinforcement Learning Approach for Adaptive Single- and Multi-Document Summarization (2015, methodology): “This is based on our choice of R-1 as the optimality score, which was based on the correlation between human scores and R-1 (Lin, 2004a Table 5 shows the results on the other two MDS tasks (DUC 2001 and 2002), compared to the best result in the literature and the best baseline system.”
SummEval: Re-evaluating Summarization Evaluation (2020, methodology): “Other work has examined the problems with applying ROUGE in settings such as meeting summarization (Liu and Liu, 2008) and summarization of scientific articles (Cohan and Goharian, 2016).”