Error Analysis of Statistical Machine Translation Output

By David Vilar, Jia Xu, Luis Fernando D'Haro, Hermann Ney. International Conference on Language Resources and Evaluation, 2006

333 citations recorded by the source corpus.

Evaluation of automatic translation output is a difficult task. Several performance measures like Word Error Rate, Position Independent Word Error Rate and the BLEU and NIST scores are widely use and provide a useful tool for comparing different systems and to evaluate improvements within a system. However the interpretation of all of these measures is not at all clear, and the identification of the most prominent source of errors in a given system using these measures alone is not possible. Therefore some analysis of the generated translations is needed in order to identify the main problems and to focus the research efforts. This area is however mostly unexplored and few works have dealt with it until now. In this paper we will present a framework for classification of the errors of a machine translation system and we will carry out an error analysis of the system used by the RWTH in the first TC-STAR evaluation.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f282db3b17282483821066cc9db0848624febd1f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “Following [40], we use 9 categories: Missing Words, Word Repetition, Named Entity, Word Order, Incorrect Words, Unknown Words, Collocation, Factoid, and Ungrammatical.”
Edinburgh’s Syntax-Based Systems at WMT 2015 (2014, methodology): “Are our syntax-based systems tackling these problems better? And what are the main remaining problems? For both German–English and English– German, we analyzed 100 sentences, we carried out an error analysis using linguistic error categories that roughly match other efforts in this area (Vilar et al”
A Grain of Salt for the WMT Manual Evaluation (2011, methodology): “Bojar (2011) applies the error-flagging manual evaluation by Vilar et al. (2006) to four systems of WMT09 English-to-Czech task.”
Can Automatic Post-Editing Make MT More Meaningful (2012, background): “Similarly, Vilar et al. (2006) found that 22% of Chinese-English MT errors were content deletion.”
Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output (2012, methodology): “We use a reﬁned version of the classiﬁcation described in (Vilar et al., 2006).”
LIMSI @ WMT’14 Medical Translation Task (2014, methodology): “The high level of scores for automatic metrics encouraged us to perform a detailed, small-scale analysis of our system output, using the error types proposed by Vilar et al. (2006).”
A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation (2018, background): “…deﬁned in Bentivogli et al. (2018), where multiple post-edits are exploited in order to quantify morphological, lexical, and word order errors, a simpliﬁed error classiﬁcation with respect to that proposed in Vilar et al. (2006), which settles two additional classes, namely missing and extra words.”
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (2019, methodology): “…rate analysis is inspired by previous work on automatic error analysis , which takes a typology of errors (Flanagan, 1994; Murata et al., 2005; Vilar et al., 2006), and attempts to automatically predict which sentences contain these errors (Popovi´c and Ney, 2011; Zeman et al., 2011; Fishel…”
Neural versus Phrase-Based Machine Translation Quality: a Case Study (2016, background): “Regarding error classiﬁ-cation, Hjerson (Popovi´c, 2011) detects ﬁve main types of word-level errors as deﬁned in (Vilar et al., 2006): morphological, reordering, missing words, extra words, and lexical choice errors.”
Evaluating the morphological competence of Machine Translation Systems (2017, methodology): “Error analysis protocols, as proposed eg. by Vilar et al. (2006); Popovi´c and Ney (2011) for PBMT, are obvious candidates for such studies and have been used eg. in (Bentivogli et al., 2016).”
Improving the Robustness of Speech Translation (2018, background): “And It is known that over 50% of the machine translation errors are associated with substitution errors which have a greater impact on translation quality than deletion or insertion errors (Vilar et al. 2006; Ruiz and Federico 2014).”