A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions

By Antonio Toral, Víctor M. Sánchez-Cartagena. 2017

161 citations recorded by the source corpus.

We aim to shed light on the strengths and weaknesses of the newly introduced neural machine translation paradigm. To that end, we conduct a multifaceted evaluation in which we compare outputs produced by state-of-the-art neural machine translation and phrase-based machine translation systems for 9 language directions across a number of dimensions. Specifically, we measure the similarity of the outputs, their fluency and amount of reordering, the effect of sentence length and performance across different error categories. We find out that translations produced by neural machine translation systems are considerably different, more fluent and more accurate in terms of word order compared to those produced by phrase-based systems. Neural machine translation systems are also more accurate at producing inflected forms, but they perform poorly when translating very long sentences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

What do Neural Machine Translation Models Learn about Morphology? (2017, background): “The improved translation quality is often attributed to better handling of non-local dependencies and morphology generation (Luong 1Our code is available at https://github.com/ boknilev/nmt-repr-analysis. and Manning, 2015; Bentivogli et al., 2016; Toral and Sánchez-Cartagena, 2017).”
Six Challenges for Neural Machine Translation (2017, background): “Bentivogli et al. (2016) considered different linguistic categories for English–German and Toral and S´anchez-Cartagena (2017) compared different broad aspects such as ﬂuency and reordering for nine language directions.”
Dynamic Data Selection for Neural Machine Translation (2017, background): “Finally, recent work comparing various aspects for PBMT and NMT includes (Bentivogli et al., 2016; Farajian et al., 2017; Toral and S´anchez-Cartagena, 2017; Koehn and Knowles, 2017).”
LIUM-CVC Submissions for WMT17 Multimodal Translation Task (2017, background): “, 2014) have received a lot of attention because of their competitive performance (Toral and Sánchez-Cartagena, 2017).”
Neural Machine Translation (2020, unknown): “(2016) considered different linguistic categories for English– German and Toral and Sánchez-Cartagena (2017) compared different broad aspects such as fluency and reordering for nine language directions.”
Neural Machine Translation: A Review (2019, background): “+ Translation quality degrades less on very long sentences than NMT [608, 609].”
Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder (2017, background): “These improvements are attributed to more ﬂuent output (Toral and S´anchez-Cartagena, 2017) and better handling of morphology and long-range dependencies (Bentivogli et al., 2016).”
Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation (2019, result): “Part of this success comes from the impressive improvements in ﬂuency of NMT output compared to previous MT paradigms (Bentivogli et al., 2016; Toral and Sánchez-Cartagena, 2017; Koehn and Knowles, 2017).”
Pinyin as Subword Unit for Chinese-Sourced Neural Machine Translation (2017, background): “NMT systems have outperformed the state-of-the-art SMT model on various language pairs in terms of translation quality [13, 2, 7, 22, 21, 5].”