Looking for a Few Good Metrics: ROUGE and its Evaluation

By Chin-Yew Lin. 2004

188 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Iryna Gurevych
2. Mohit Bansal
3. Maosong Sun
4. Zhiyuan Liu
5. Percy Liang
6. Yang Gao
7. Graham Neubig
8. Jianfeng Gao

In Evaluation & Benchmarks

1. Iryna Gurevych
2. Yang Gao
3. Zhiyuan Liu
4. Tatsunori B. Hashimoto
5. Maosong Sun
6. Percy Liang
7. Steffen Eger
8. Wei Zhao

Citations in the corpus

• Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “As a baseline, we also consider the feature-rich reward learning method proposed by Peyrard and Gurevych (2018) (see §2).”
• Unifying Human and Statistical Evaluation for Natural Language Generation (2019, methodology): “Automatic metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin and Rey, 2004) capture quality better than perplexity but still correlate poorly with human evaluation and fail to capture diversity (Novikova et al., 2017; Chaganty et al., 2018).”
• The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “As the gold standard of performing human evaluation is often too expensive, there has been a large effort develop-∗ ing automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin and Rey, 2004), ME-TEOR (Lavie and Denkowski, 2009; Denkowski and Lavie, 2014) and CiDER (Vedantam et al., 2015).”
• Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions (2010, methodology): “In our experiments, we use ROUGE-1, ROUGE-2 and ROUGE-SU4 measures.”
• SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, methodology): “We show it outperforms the state-of-the-art unsupervised summarization methods (in multiple ROUGE metrics).”
• ROUGE: A Package for Automatic Evaluation of Summaries (2004, background): “The stability and reliability of ROUGE at different sample sizes was reported by the author in ( Lin , 2004 ).”
• Neural Headline Generation with Sentence-wise Optimization (2016, methodology): “Then we demonstrate that our model performs the best compared with state-of-the-art baseline systems.”