Manual and automatic evaluation of summaries
By Chin-Yew Lin, Eduard H. Hovy. 2002
309 citations recorded by the source corpus.
In this paper we discuss manual and automatic evaluations of summaries using data from the Document Understanding Conference 2001 (DUC-2001). We first show the instability of the manual evaluation. Specifically, the low inter-human agreement indicates that more reference summaries are needed. To investigate the feasibility of automated summary evaluation based on the recent BLEU method from machine translation, we use accumulative n-gram overlap scores between system and human summaries. The initial results provide encouraging correlations with human judgments, based on the Spearman rank-order correlation coefficient. However, relative ranking of systems needs to take into account the instability.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ani Nenkova
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Caiming Xiong
- Mirella Lapata
- Graham Neubig
In Evaluation & Benchmarks
- Ani Nenkova
- Chin-Yew Lin
- Dragomir R. Radev
- K. McKeown
- Ryan T. McDonald
- Mohit Bansal
- Donna K. Harman
- Paul Over
In Summarization
- Ani Nenkova
- Ryan T. McDonald
- Chin-Yew Lin
- K. McKeown
- Rebecca J. Passonneau
- Dragomir R. Radev
- Ji Ma
- Eduard H. Hovy
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We report metrics for BLEU [54], METEOR [4] ROUGE [42], and CIDEr [69].”
- R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “The performance is evaluated by ROUGE F1 score [37].”
- Evaluating DUC 2005 using Basic Elements (2005, background): “We have not experimented with different weights based on words’ information content, etc., although one can obviously do so.”
- The Effects of Human Variation in DUC Summarization Evaluation (2004, background): “This large difference in scores is coming solely from the model difference since judgment is being made by the same person (although some self-inconsistency is involved (Lin and Hovy, 2002)).”
- Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation (2020, methodology): “Below we measured generation quality (via Rouge-based metrics (Lin and Hovy, 2002)) versus retrieval quality for three systems.”
- Introduction to the Special Issue on Summarization (2002, methodology): “Simply using a variant of the Bilingual Evaluation Understudy (BLEU) scoring method (based on a linear combination of matching n -grams between the system output and the ideal summary) developed for machine translation (Pap-ineni et al. 2001) is promising but not sufficient (Lin and Hovy 2002b).”
- Evaluating Content Selection in Summarization: The Pyramid Method (2004, methodology): “However, in contrast to translation, where the evaluation criterion can be defined fairly precisely it is difficult to elicit stable human judgments for summarization (Rath et al., 1961) (Lin and Hovy, 2002).”
- Looking for a Few Good Metrics: Automatic Summarization Evaluation - How Many Samples Are Enough? (2004, background): “For question (a), Lin and Hovy [4, 5] showed that stable evaluation results could be obtained but the variability of human judgments and evaluation metrics must be factored in.”
- Looking for a Few Good Metrics: ROUGE and its Evaluation (2004, background): “For question (a), Lin and Hovy [4, 5] showed that stable evaluation results could be obtained but the variability of human judgments and evaluation metrics must be factored in.”