ROUGE: A Package for Automatic Evaluation of Summaries

By Chin-Yew Lin. 2004

21,161 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, background): “The maximum-likelihood training objective is the minimization of the following loss: However, minimizing L ml does not always produce the best results on discrete evaluation metrics such as ROUGE (Lin, 2004).”
Convolutional Sequence to Sequence Learning (2017, methodology): “We also evaluate on a Gigaword test set of 2000 pairs which is identical to the one used by Rush et al. (2015) and we report F1 ROUGE similar to prior work.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “For CNN/Daily Mail, we find the performance of models on the ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F metrics [Lin, 2004] to be highly correlated so we”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “report ROUGE-1 and -L recall scores (Lin, 2004).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “We average ROUGE-1, ROUGE-2, and ROUGE-L scores (Lin, 2004) to compute an overall ROUGE score.”
Generating Wikipedia by Summarizing Long Sequences (2018, methodology): “RNN-based encoder-decoder models with attention (seq2seq) perform very well on this task in both ROUGE (Lin, 2004), an automatic metric often used in summarization, and human evaluation (Chopra et al., 2016).”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “Our model also outperforms the best extractive model (Liu, 2019) by 1 point in ROUGE-L.”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “We evaluate our models with the standard ROUGE metric (Lin, 2004b), reporting the F 1 scores for ROUGE-1, ROUGE-2 and ROUGE-L (which respectively measure the word-overlap, bigram-overlap, and longest common sequence between the reference summary and the summary to be evaluated).”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Here we report the drops on ROUGE-1, ROUGE-2, and ROUGE-L on the nonanonymized validation set.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “To evaluate, we use the standard ROUGE metric (Lin, 2004) and report ROUGE-1, ROUGE-2, and ROUGE-L.”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “These validation accuracies are shown in Table 1 below, which includes additional comparisons of GPT-judge to alternate metrics that make use of ROUGE1 (Lin, 2004) or BLEURT (Sellam et al.”
Learning Compact Metrics for MT (2021, neutral): “Recent improvements in Machine Translation (MT) and multilingual Natural Language Generation (NLG) have led researchers to question the use of n-gram overlap metrics such as BLEU and ROUGE (Papineni et al., 2002; Lin, 2004).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “However, recent work by Fabbri et al. (2020) demonstrates that this may not hold in summarization, where the automated metric BERTScore (Zhang et al., 2020b) does not improve upon the correlation of ROUGE.”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, background): “trained on large, monolingual corpora in 25 languages. The model uses a shared sub-word vocabulary, encoder, and decoder across all 25 languages, and is trained as a denoising auto-encoder during the pre-training step. Liu et al. (2020) showed that this pre-training method provides a good initial-”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “This explains the relatively lower reported ROUGE, compared to the model with the full preprocessing.”
Towards a Human-like Open-Domain Chatbot (2020, background): “BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), or other related metrics in translation and summarization, while popular and easy to compute, have been shown to be unsuitable for dialog (Liu et al., 2016) or more broadly language generation systems (Novikova et al., 2017).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “Algorithm 1 Sequential Sentence Selection 1: S := ∅ 2: for j ← 1 to m do 3: si := rouge ( S ∪ {xi}, D \ (S ∪ {xi}) ) ∀i s.t. xi /∈ S 4: k := argmaxi{si}n 5: S := S ∪ {xk} 6: end for When calculating ROUGE1-F1, we also consider n-grams as a set (Uniq) instead of double-counting identical n-grams as i”
Semantic Noise Matters for Neural Natural Language Generation (2019, methodology): “We use freely available word-overlap-based evaluation metrics (WOM) scripts that come with the E2E data (Duˇsek et al., 2019), 9 supporting BLEU (Papineni et al., 2002), NIST (Doddington, 2002), ROUGE-L (Lin, 2004), METEOR (Lavie and Agarwal, 2007) and CIDEr (Vedantam et al., 2015).”