Automatic evaluation of machine translation quality using n-gram co-occurrence statistics

By G. Doddington. 2002

1,847 citations recorded by the source corpus.

Evaluation is recognized as an extremely helpful forcing function in Human Language Technology R&D. Unfortunately, evaluation has not been a very powerful tool in machine translation (MT) research because it requires human judgments and is thus expensive and time-consuming and not easily factored into the MT research agenda. However, at the July 2001 TIDES PI meeting in Philadelphia, IBM described an automatic MT evaluation technique that can provide immediate feedback and guidance in MT research. Their idea, which they call an "evaluation understudy", compares MT output with expert reference translations in terms of the statistics of short sequences of words (word N-grams). The more of these N-grams that a translation shares with the reference translations, the better the translation is judged to be. The idea is elegant in its simplicity. But far more important, IBM showed a strong correlation between these automatically generated scores and human judgments of translation quality. As a result, DARPA commissioned NIST to develop an MT evaluation facility based on the IBM work. This utility is now available from NIST and serves as the primary evaluation measure for TIDES MT research.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “N-gram based metrics (Papineni et al., 2002; Doddington, 2002; Wang et al., 2016; Popović, 2015) that measure lexical overlap require tokenization which is highly affected by language specific factors such as alphabet size and morphological complexity.”
Semantic Noise Matters for Neural Natural Language Generation (2019, positive): “, 2002), NIST (Doddington, 2002), ROUGE-L (Lin, 2004), METEOR (Lavie and Agarwal, 2007) and CIDEr (Vedantam et al.”
Neural Generation for Czech: Data and Baselines (2019, methodology): “See Section 4.2 for metrics; scores are averaged over 5 different random ini-tializations, all scores except for NIST and CIDEr are percentages.”
The E2E Dataset: New Challenges For End-to-End Generation (2017, background): “BLEU (Papineni et al., 2002) 0.6925 NIST (Doddington, 2002) 8.4781 METEOR (Lavie and Agarwal, 2007) 0.4703 ROUGE-L (Lin, 2004) 0.7257 CIDEr (Vedantam et al., 2015) 2.3987 Table 6: TGen results on the development set. cent E2E data-driven systems.”
Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “We also compute various n -gram similarity metrics (BLEU, ME-TEOR, NIST) comparing the noisy questions to the original questions and show that these metrics do not correlate well with human judgments.”
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation (2019, positive): “NIST is a variant of BLEU that weights n-gram matches by their information gain, i.e., it indirectly penalizes uninformative n-grams.”
Why We Need New Evaluation Metrics for NLG (2017, methodology): “However, WOMs rate this utterance low, i.e. scores of BLEU1-4, NIST, LEPOR, CIDEr, ROUGE and METEOR normalised into the 1-6 range all stay below 1.5.”
Minimum Error Rate Training in Statistical Machine Translation (2003, methodology): “As a result, we expect that actual ’true’ translation quality is improved, as previous work has shown that for some evaluation criteria there is a correlation with human subjective evaluation of ﬂuency and adequacy (Papineni et al., 2001; Doddington, 2002).”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “A recurring theme is that metrics like BLEU and NIST (Doddington, 2002) are not suitable for judging content quality in NLG. Recently, Novikova et al. (2017a) did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable…”
Minimum Risk Training for Neural Machine Translation (2015, methodology): “Gao et al. (2014) leverage MRT for learning continuous phrase representations for statistical machine translation.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “NIST (Doddington, 2002) is a version of BLEU with higher weighting for less frequent (i.”
Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings (2016, methodology): “(4)We treat the two paraphrases for the same DA as separate instances in the training set but use them together as two references to measure BLEU and NIST scores (Papineni et al., 2002; Doddington, 2002) on the validation and test sets.”
(Meta-) Evaluation of Machine Translation (2007, methodology): “Bleu was used exclusively since it is the most widely used metric in the field and has been shown to correlate with human judgments of translation quality in many instances (Doddington, 2002; Coughlin, 2003; Przybocki, 2004).”
A Structured Review of the Validity of BLEU (2018, background): “I did not look at the NIST metric (Doddington 2002), that is I did not consider NIST to be a standard version of BLEU.”
Convolutional Neural Network for Paraphrase Identification (2015, methodology): “…as a kind of mutual translation, hence combined eight kinds of machine translation metrics including BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover et al., 2006), TERp (Snover et al., 2009), METEOR (Denkowski and Lavie, 2010), SEPIA (Habash and Elkholy, 2008),…”