Manual and automatic evaluation of summaries

By Chin-Yew Lin, Eduard H. Hovy. 2002

309 citations recorded by the source corpus.

In this paper we discuss manual and automatic evaluations of summaries using data from the Document Understanding Conference 2001 (DUC-2001). We first show the instability of the manual evaluation. Specifically, the low inter-human agreement indicates that more reference summaries are needed. To investigate the feasibility of automated summary evaluation based on the recent BLEU method from machine translation, we use accumulative n-gram overlap scores between system and human summaries. The initial results provide encouraging correlations with human judgments, based on the Spearman rank-order correlation coefficient. However, relative ranking of systems needs to take into account the instability.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Ani Nenkova
  2. Mohit Bansal
  3. Percy Liang
  4. Luke S. Zettlemoyer
  5. Jianfeng Gao
  6. Caiming Xiong
  7. Mirella Lapata
  8. Graham Neubig

In Evaluation & Benchmarks

  1. Ani Nenkova
  2. Chin-Yew Lin
  3. Dragomir R. Radev
  4. K. McKeown
  5. Ryan T. McDonald
  6. Mohit Bansal
  7. Donna K. Harman
  8. Paul Over

In Summarization

  1. Ani Nenkova
  2. Ryan T. McDonald
  3. Chin-Yew Lin
  4. K. McKeown
  5. Rebecca J. Passonneau
  6. Dragomir R. Radev
  7. Ji Ma
  8. Eduard H. Hovy

Citations in the corpus