A Decade of Automatic Content Evaluation of News Summaries: Reassessing the State of the Art

By Peter A. Rankel, John M. Conroy, Hoa Trang Dang, Ani Nenkova. 2013

58 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Rankel et al. (2013) find that rarely used configurations of ROUGE are outperforming commonly used one, and in an investigation of all 192 ROUGE configurations, Graham (2015) find that none of them outperformed BLEU and that best performance was achieved with the precision vari- 6Note though that…”
MLSUM: The Multilingual Summarization Corpus (2020, background): “A distinctive feature of the DUC datasets is the availability of multiple reference summaries: this is a valuable characteristic since, as found by Rankel et al. (2013), the correlation between qualitative and automatic metrics, such as ROUGE (Lin, 2004), decreases signiﬁcantly when only a single…”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, neutral): “ROUGE-1 and ROUGE-L fail to accurately measure factual inconsistency across domains in our controlled analysis.”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, methodology): “For ROUGE, we ﬁnd that increasing the size of n in ROUGE-n is not better in almost all dimensions, which is different from the ﬁndings of Rankel et al. (2013) and Fabbri et al. (2021b).”
Learning to Score System Summaries for Better Content Selection Evaluation. (2017, methodology): “However, for the other language dependent features, we used the German word embeddings developed by Reimers et al. (2014). For the English datasets, we use dependency-based word embeddings (Levy and Goldberg, 2014).”
Re-evaluating Evaluation in Text Summarization (2020, background): “Workers are paid similar to Shapira et al. (2019), scaling the rates for fewer SCUs and shorter summary texts.”
System Combination for Multi-document Summarization (2015, unknown): “1 Among automatic evaluation metrics, ROUGE-1 (R-1) can predict that one system performs signiﬁcantly better than the other with the highest recall (Rankel et al., 2013).”
Language Agnostic Automatic Summarization Evaluation (2020, methodology): “Similar to methods proposed in the AESOP task from 2009 to 2011 (Rankel et al., 2013) none of these methods gained wide-spread use, despite various studies showing problems with ROUGE (see for example (Graham, 2015)).”
Studying Summarization Evaluation Metrics in the Appropriate Scoring Range (2019, methodology): “Yet, this remains an open problem (Rankel et al., 2013).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “Within metric meta-analysis, several studies have focused on the analysis of ROUGE (Lin, 2004b), and its variations (Rankel et al., 2013; Graham, 2015), across domains such as news (Lin, 2004a), meeting summarization (Liu and Liu, 2008), and scientific articles (Cohan and Goharian, 2016).”