ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation
By Chin-Yew Lin, Franz Joseph Och. 2004
426 citations recorded by the source corpus.
Comparisons of automatic evaluation metrics for machine translation are usually conducted on corpus level using correlation statistics such as Pearson's product moment correlation coefficient or Spearman's rank order correlation coefficient between human scores and automatic scores. However, such comparisons rely on human judgments of translation qualities such as adequacy and fluency. Unfortunately, these judgments are often inconsistent and very expensive to acquire. In this paper, we introduce a new evaluation method, Orange, for evaluating automatic machine translation evaluation metrics automatically without extra human involvement other than using a set of reference translations. We also show the results of comparing several existing automatic metrics and three new automatic metrics using Orange.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
- Kyunghyun Cho
- Yejin Choi
In Evaluation & Benchmarks
- Graham Neubig
- Mohit Bansal
- Rico Sennrich
- Mitesh M. Khapra
- Noah A. Smith
- Jianfeng Gao
- Pengfei Liu
- Kevin Gimpel
In Machine Translation
- Spence Green
- Kevin Gimpel
- Christopher D. Manning
- Jeffrey Heer
- Jason Chuang
- Graham Neubig
- Rico Sennrich
- George F. Foster
Citations in the corpus
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “We show code summarization results of smoothed BLEU-4 on six PL data in Table 2.”
- Unified Pre-training for Program Understanding and Generation (2021, methodology): “We use corpus level BLEU (Papineni et al., 2002) score for all the generation tasks, except code summarization where we use smoothed BLEU-4 score (Lin and Och, 2004) following Feng et al. (2020).”
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “[18], who use smoothed BLEU score [49] as evaluation metric, because this is suitable for evaluating short documents.”
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020, methodology): “Since the generated documentations are short and higher order n-grams may not overlap, we remedy this problem by using smoothed BLEU score (Lin and Och, 2004).”
- Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics (2004, positive): “In Lin and Och (2004), we proposed a framework that automatically evaluated automatic MT evaluation metrics using only manual translations without further human involvement.”
- Summarizing Source Code using a Neural Attention Model (2016, methodology): “We remedy this problem by using +1 smoothing (Lin and Och, 2004).”
- Classical Structured Prediction Losses for Sequence to Sequence Learning (2017, background): “…either BLEU or ROUGE with respect to t , the gold reference: As is common practice when computing BLEU at the sentence-level, we smooth all initial counts to one (except for unigram counts) so that the geometric mean is not dominated by zero-valued n - gram match counts (Lin and Och, 2004).”
- Analyzing Uncertainty in Neural Machine Translation (2018, methodology): “Sentence-level BLEU is computed similarly to corpus BLEU, but with smoothed n -gram counts (+1) for n > 1 (Lin & Och, 2004).”
- RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation (2018, methodology): “However, most MTE metrics are obtained by computing the similarity between an MT hypothesis and a reference based on the character or word N-grams, such as SentBLEU (Lin and Och, 2004), which is a smoothed version of BLEU (Papineni et al.”
- A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation (2007, methodology): “Metrics in the Rouge family allow for skipn-grams (Lin and Och, 2004a); Kauchak and Barzilay (2006) take paraphrasing into account; metrics such as METEOR (Banerjee and Lavie, 2005) and GTM (Melamed et al., 2003) calculate both recall and precision; METEOR is also similar to SIA (Liu and Gildea,…”
- Re-evaluating Evaluation in Text Summarization (2020, methodology): “To characterize the relative goodness of evaluation metrics, it is necessary to perform metaevaluation (Graham, 2015; Lin and Och, 2004), where a dataset annotated with human judgments (e.”
- CoTexT: Multi-task Learning with Code-Text Transformer (2021, methodology): “, 2021b), we employ smoothed BLEU-4 score (Lin and Och, 2004) for Code Summarization and corpuslevel BLEU score for all remain tasks.”
- Phrase Dependency Machine Translation with Quasi-Synchronous Tree-to-Tree Features (2014, methodology): “For example, for a monolingual task, Wu et al. (2009) used a shallow parser to convert lexical dependencies from a dependency parser into phrase dependencies.”
- The Meteor metric for automatic evaluation of machine translation (2009, methodology): “We also used the smoothed Bleu described in meteor-mtj-2009.tex; 16/05/2009; 0:52; p.12 (Lin and Och, 2004) to ensure meaningful segment-level Bleu scores.”
- Further Meta-Evaluation of Broad-Coverage Surface Realization (2010, methodology): “, 2001), with smoothing as described in (Lin and Och, 2004)”
- Comparing Rating Scales and Preference Judgements in Language Evaluation (2010, background): “It is not surprising then that stable averages of quality judgements, let alone high levels of agreement, are hard to achieve, as has been observed for MT (Turian et al., 2003; Lin and Och, 2004), text summarisation (Trang Dang, 2006), and language generation (Belz and Reiter, 2006).”
- Predictive translation memory: a mixed-initiative system for human language translation (2014, methodology): “For Fr-En, the LMEM predicts a mean (intercept) BLEU+1 score of 33.7 for post-edit and 34.6 for PTM.”