Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

By Chin-Yew Lin, Eduard H. Hovy. 2003

1,926 citations recorded by the source corpus.

Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Summarization, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yejin Choi
  4. Percy Liang
  5. Luke S. Zettlemoyer
  6. Graham Neubig
  7. Noah A. Smith
  8. Caiming Xiong

In Information Systems

  1. Mirella Lapata
  2. Hamed Zamani
  3. Aliaksei Severyn
  4. Ryan T. McDonald
  5. Rada Mihalcea
  6. Graham Neubig
  7. Shay B. Cohen
  8. Ani Nenkova

In Natural Language Processing

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yejin Choi
  4. Mirella Lapata
  5. Graham Neubig
  6. Luke S. Zettlemoyer
  7. Percy Liang
  8. Noah A. Smith

In Summarization

  1. Mirella Lapata
  2. Ryan T. McDonald
  3. Rada Mihalcea
  4. Ani Nenkova
  5. Shay B. Cohen
  6. Anya Belz
  7. Pengfei Liu
  8. L. Elisa Celis

In Signal Processing

  1. Rada Mihalcea
  2. Mirella Lapata
  3. Yoshua Bengio
  4. Ani Nenkova
  5. Aaron C. Courville
  6. Ryan T. McDonald
  7. Anya Belz
  8. Pengfei Liu

Citations in the corpus