Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics
By Chin-Yew Lin, Eduard H. Hovy. 2003
1,926 citations recorded by the source corpus.
Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Summarization, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Luke S. Zettlemoyer
- Graham Neubig
- Noah A. Smith
- Caiming Xiong
In Information Systems
- Mirella Lapata
- Hamed Zamani
- Aliaksei Severyn
- Ryan T. McDonald
- Rada Mihalcea
- Graham Neubig
- Shay B. Cohen
- Ani Nenkova
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Mirella Lapata
- Graham Neubig
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
In Summarization
- Mirella Lapata
- Ryan T. McDonald
- Rada Mihalcea
- Ani Nenkova
- Shay B. Cohen
- Anya Belz
- Pengfei Liu
- L. Elisa Celis
In Signal Processing
- Rada Mihalcea
- Mirella Lapata
- Yoshua Bengio
- Ani Nenkova
- Aaron C. Courville
- Ryan T. McDonald
- Anya Belz
- Pengfei Liu
Citations in the corpus
- RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, background): “, 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the”
- Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “We evaluated summarization quality using F1 ROUGE (Lin and Hovy, 2003).”
- Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “We also show that standard automatic metrics such as Bleu (Papineni, Roukos, Ward, & Zhu, 2002) or Rouge (Lin, 2004) that have also been used to evaluate caption generation systems show poor correlation with human judgments, leading us to believe that the evaluation of caption generation system shou”
- Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “ROUGE (Lin and Hovy, 2003), which is also used for chatbot evaluation, is a popular metric for text summarization.”
- Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “These works directly optimize for specific metrics, such as BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003), using REINFORCE (Williams, 1992).”
- An Actor-Critic Algorithm for Sequence Prediction (2016, methodology): “…answer as the target for the network at step k , whereas Ranzato et al. (2015) rely on the REINFORCE algorithm (Williams, 1992) to decide whether or not the tokens from a sampled prediction lead to a high task-specific score, such as BLEU (Papineni et al., 2002) or ROUGE (Lin & Hovy, 2003).”
- On Faithfulness and Factuality in Abstractive Summarization (2020, background): “Interestingly, all summaries are topical and fluent, and perform well in terms of ROUGE scores (Lin and Hovy, 2003).”
- Neural Summarization by Extracting Sentences and Words (2016, methodology): “ROUGE scores for the word extraction model are less promising.”
- A Hierarchical Neural Autoencoder for Paragraphs and Documents (2015, methodology): “We first adopt two standard evaluation metrics, ROUGE (Lin, 2004; Lin and Hovy, 2003) and BLEU (Papineni et al., 2002).”
- Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2019, methodology): “We report on the ROUGEF1 scores (Lin and Hovy, 2003); in particular, we report on ROUGE-1 and ROUGE-2 for informativeness and ROUGE-L for fluency in Table 5.”
- LexRank: Graph-based Lexical Centrality as Salience in Text Summarization (2004, background): “Among these different scores, unigram-based ROUGE score (ROUGE-1) has been shown to agree with human judgements most (Lin & Hovy, 2003).”
- Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints (2016, methodology): “We evaluate our system along two axes: first, on content selection, using ROUGE9 (Lin and Hovy, 2003), and second, on clarity of language and referential structure, using annotators from Amazon Mechanical Turk.”
- Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, methodology): “directly optimizing the final evaluation metric, namely ROUGE (Lin and Hovy, 2003).”
- LCSTS: A Large Scale Chinese Short Text Summarization Dataset (2015, methodology): “5https://pypi.python.org/pypi/jieba/ For evaluation, we adopt the ROUGE metrics (Lin, 2004) proposed by (Lin and Hovy, 2003), which have been proved strongly correlated with human evaluations.”
- Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, neutral): “They observe a marginal improvement when the tasks are performed in parallel, but achieve the best performance in a revision-based architecture, where syntactic mapping is followed by referring expression insertion, followed by a revision of the syntax. Classification cascades for nlg maintain a cle”
- Improved Image Captioning via Policy Gradient optimization of SPIDEr (2016, methodology): “Common choices include BLEU, CIDEr, METEOR and ROUGE.”