Neural versus Phrase-Based Machine Translation Quality: a Case Study
By Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, Marcello Federico. 2016
344 citations recorded by the source corpus.
Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particular, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-of-the-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural versus phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models -- such as the reordering of verbs -- while pointing out other aspects that remain to be improved.
Subfields: Artificial Intelligence, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Graham Neubig
- Felix Stahlberg
- Philipp Koehn
- Mohit Bansal
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Jianfeng Gao
In Machine Translation
- Yonatan Belinkov
- Philipp Koehn
- Felix Stahlberg
- Nadir Durrani
- Hassan Sajjad
- James R. Glass
- Fahim Dalvi
- Matteo Negri
Citations in the corpus
- What do Neural Machine Translation Models Learn about Morphology? (2017, background): “The improved translation quality is often attributed to better handling of non-local dependencies and morphology generation (Luong and Manning, 2015; Bentivogli et al., 2016; Toral and S´anchez-Cartagena, 2017).”
- Six Challenges for Neural Machine Translation (2017, background): “Bentivogli et al. (2016) considered different linguistic categories for English–German and Toral and Sánchez-Cartagena (2017) compared different broad aspects such as fluency and reordering for nine language directions.”
- A Challenge Set Approach to Evaluating Machine Translation (2017, methodology): “t NMT performs better on average than previous technologies, but they do not help us understand what aspects of the translation have improved. Some groups have conducted more detailed error analyses. Bentivogli et al. (2016) carried out a number of experiments on IWSLT 2015 EnglishGerman evaluation ”
- An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “Neural MT (NMT) has become the de-facto standard with a performance that clearly outperforms the alternative approach of Statistical Machine Translation (Luong et al., 2015b; Bojar et al., 2016; Bentivogli et al., 2016).”
- Analyzing Uncertainty in Neural Machine Translation (2018, methodology): “Recent work on analyzing machine translation has focused on topics such as comparing neural translation to phrase-based models (Bentivogli et al., 2016; Toral & Sanchez-Cartagena, 2017).”
- Dynamic Data Selection for Neural Machine Translation (2017, background): “Finally, recent work comparing various aspects for PBMT and NMT includes (Bentivogli et al., 2016; Farajian et al., 2017; Toral and S´anchez-Cartagena, 2017; Koehn and Knowles, 2017).”
- Evaluating the morphological competence of Machine Translation Systems (2017, methodology): “Error analysis protocols, as proposed eg. by Vilar et al. (2006); Popovi´c and Ney (2011) for PBMT, are obvious candidates for such studies and have been used eg. in (Bentivogli et al., 2016).”
- The Lazy Encoder: A Fine-Grained Analysis of the Role of Morphology in Neural Machine Translation (2018, background): “The advent of Neural Machine Translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2014) has led to remarkable improvements in machine translation quality (Bentivogli et al., 2016) but has also produced models that are much less interpretable.”
- Neural Machine Translation with Reconstruction (2016, background): “Particularly, NMT has significantly enhanced the performance of translation between a language pair involving rich morphology prediction and/or significant word reordering (Luong and Manning 2015; Bentivogli et al. 2016).”
- The IWSLT 2016 Evaluation Campaign (2016, result): “9For a detailed analysis of the outputs of the systems participating in the IWSLT 2015 MT EnDe task, see [35] ing mTER and TER results given in the tables, which largely confirm previous years’ findings.”
- Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, background): “…by enabling to tackle sequence-to-sequence problems with a single end-to-end model, achieved state-of-the-art results on machine translation (MT) (Bentivogli et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018) and obtained increasingly good performance in automatic…”
- Exploiting Deep Representations for Neural Machine Translation (2018, positive): “B ASE goes up with the increase of input sentence lengths, which is superior to the performance of RNN-based NMT models on long sentences reported by (Bentivogli et al., 2016).”
- How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse? (2017, background): “Different research groups have shown that NMT can generate natural and fluent translations (Bentivogli et al., 2016), achieving human-like performance in certain settings (Wu et al.”
- The University of Cambridge’s Machine Translation Systems for WMT18 (2018, background): “There is a large body of research comparing NMT and SMT (Schnober et al., 2016; Toral and S´anchez-Cartagena, 2017; Koehn and Knowles, 2017; Menacer et al., 2017; Dowling et al., 2018; Bentivogli et al., 2016, 2018).”
- Assessing the Tolerance of Neural Machine Translation Systems Against Speech Recognition Errors (2017, methodology): “[4] compare neural machine translation against three topperforming statistical machine translation systems in the TED talk machine translation track from IWSLT 2015.”
- Investigating Backtranslation in Neural Machine Translation (2018, background): “…machine translation (MT) paradigm that has quickly become dominant in both academic and industry MT communities, achieving state-of-the-art results [Bentivogli et al., 2016; Bojar et al., 2016; Junczys-Dowmunt et al., 2016; Wu et al., 2016; Castilho et al., 2017; Shte-rionov et al., 2017] on a…”
- A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions (2017, background): “The latter was not the case in the work by Bentivogli et al. (2016).”