ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation

By Chin-Yew Lin, Franz Joseph Och. 2004

426 citations recorded by the source corpus.

Comparisons of automatic evaluation metrics for machine translation are usually conducted on corpus level using correlation statistics such as Pearson's product moment correlation coefficient or Spearman's rank order correlation coefficient between human scores and automatic scores. However, such comparisons rely on human judgments of translation qualities such as adequacy and fluency. Unfortunately, these judgments are often inconsistent and very expensive to acquire. In this paper, we introduce a new evaluation method, Orange, for evaluating automatic machine translation evaluation metrics automatically without extra human involvement other than using a set of reference translations. We also show the results of comparing several existing automatic metrics and three new automatic metrics using Orange.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Graham Neubig
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Noah A. Smith
  7. Kyunghyun Cho
  8. Yejin Choi

In Evaluation & Benchmarks

  1. Graham Neubig
  2. Mohit Bansal
  3. Rico Sennrich
  4. Mitesh M. Khapra
  5. Noah A. Smith
  6. Jianfeng Gao
  7. Pengfei Liu
  8. Kevin Gimpel

In Machine Translation

  1. Spence Green
  2. Kevin Gimpel
  3. Christopher D. Manning
  4. Jeffrey Heer
  5. Jason Chuang
  6. Graham Neubig
  7. Rico Sennrich
  8. George F. Foster

Citations in the corpus