An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research

By Sonja Nießen, Franz Joseph Och, Gregor Leusch, Hermann Ney. International Conference on Language Resources and Evaluation, 2000

355 citations recorded by the source corpus.

In this paper we present a tool for the evaluation of translation quality. First, the typical requirements of such a tool in the framework of machine translation (MT) research are discussed. We define evaluation criteria which are more adequate than pure edit distance and we describe how the measurement along these quality criteria is performed semi-automatically in a fast, convenient and above all consistent way using our tool and the corresponding graphical user interface.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/15315ed05451c88f83c50d56a66a0b85517c5f4f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Minimum Error Rate Training in Statistical Machine Translation (2003, methodology): “Examples of such methods are word error rate, position-independent word error rate (Tillmann et al., 1997), generation string accuracy (Bangalore et al., 2000), multi-reference word error rate (Nießen et al., 2000), BLEU score (Pap-ineni et al., 2001), NIST score (Doddington, 2002).”
Natural language processing: state of the art, current trends and challenges (2017, background): “…word error rate (Tillmann et al., 1997) [138], generation string accuracy (Bangalore et al., 2000) [8], multi-reference word error rate (Nießen et al., 2000) [95], BLEU score (Papineni et al., 2002) [101], NIST score (Doddington, 2002) [35] All these criteria try to approximate human…”
Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation (2010, methodology): “WER (Word Error Rate) (Nießen et al., 2000) We use −WER to make this into a precision measure.”
The Alignment Template Approach to Statistical Machine Translation (2004, methodology): “In the case of the multireference word error rate for each test sentence, not just a single reference translation is used, as for the WER, but a whole set of most similar sentence is calculated (Nießen et al. 2000).”
Discriminative Training and Maximum Entropy Models for Statistical Machine Translation (2002, methodology): “Each translated sentence was judged by a human examiner according to an error scale from 0.0 to 1.0 (Nießen et al., 2000).”
Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics (2004, background): “Nießen et al. (2000) calculated the length-normalized edit distance, called word error rate (WER), between a candidate and multiple reference translations.”
Semi-supervised model adaptation for statistical machine translation (2007, methodology): “We evaluated the generated translations using three different evaluation metrics: BLEU score (Papineni et al., 2002), mWER (multi-reference word error rate), and mPER (multi-reference position-independent word error rate) (Nießen et al., 2000).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “The original formula is based on the fraction of word edits as given below: Since WER relies heavily on the reference sentence, Nießen et al. [89] propose enhanced WER that takes into account multiple references.”
Transductive learning for statistical machine translation (2007, methodology): “We evaluated the generated translations using three different evaluation metrics: BLEU score (Pa-pineni et al., 2002), mWER (multi-reference word error rate), and mPER (multi-reference position-independent word error rate) (Nießen et al., 2000).”
Evaluation of Text Generation: A Survey (2020, background): “, 2015) calculates the edit distance between several references for each sentence and chooses the smallest one (Nießen et al., 2000).”
ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation (2004, background): “Nießen et al. (2000) calculated the length-normalized edit distance, called word error rate (WER), between a candidate and multiple reference translations.”
Using monolingual source-language data to improve MT performance (2006, methodology): “In the following, we present results measuring translation quality using the BLEU-4 score [3], multi-reference word error rate (mWER) and position-independent word error rate (mPER) [12].”