A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation

By Surafel Melaku Lakew, Mauro Cettolo, Marcello Federico. 2018

111 citations recorded by the source corpus.

Recently, neural machine translation (NMT) has been extended to multilinguality, that is to handle more than one translation direction with a single system. Multilingual NMT showed competitive performance against pure bilingual systems. Notably, in low-resource settings, it proved to work effectively and efficiently, thanks to shared representation space that is forced across languages and induces a sort of transfer-learning. Furthermore, multilingual NMT enables so-called zero-shot inference across language pairs never seen at training time. Despite the increasing interest in this framework, an in-depth analysis of what a multilingual NMT model is capable of and what it is not is still missing. Motivated by this, our work (i) provides a quantitative and comparative analysis of the translations produced by bilingual, multilingual and zero-shot systems; (ii) investigates the translation quality of two of the currently dominant neural architectures in MT, which are the Recurrent and the Transformer ones; and (iii) quantitatively explores how the closeness between languages influences the zero-shot translation. Our analysis leverages multiple professional post-edits of automatic translations by several different systems and focuses both on automatic standard metrics (BLEU and TER) and on widely used error categories, which are lexical, morphology, and word order errors.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, methodology): “We restrict our experiments to Transformer models (Vaswani et al., 2017) as they were shown to be very effective in recent benchmarks (Ott et al., 2018), also in the context of multilingual models (Lakew et al., 2018; Sachan and Neubig, 2018).”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “As Transformer was originally proposed as an NMT system [1], it has been widely studied on NMT tasks including hyperparameter search [9], parallelism implementation [5] and in comparison with RNN [10].”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, positive): “Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018; Tan et al., 2019), improve low-resource translation (Ha et al., 2016; Arivazhagan et al., 2019b), 1We release our code at https://github. com/bzhangGo/zero.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “All of these architectures are based on either the recurrent model with attention [91] or the Transformer-based model [182], and the latter has shown to be better [109, 111, 160] and is the base for most of the recent multi-NMT architectures.”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, result): “A comparison between Transformer and LSTM has been done in [43–46].”
Three Strategies to Improve One-to-Many Multilingual Translation (2018, background): “Owing to excellent translation performance and ease of use, many researchers (Blackwood et al., 2018; Lakew et al., 2018) have conduct translation of multiple languages based on the framework of Johnson et al. (2017) and Ha et al. (2016).”
Multilingual training for Software Engineering (2021, background): “Several papers show that multilingual-trained models show better performance [36, 62] and are more practical to deploy [9].”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “Lakew et al. (2018b) dynamically updated the vocabulary of the parent model with the low-resource language pair before transferring parameters.”
Neural Machine Translation: A Review (2019, background): “current capabilities of multilingual and zero-shot translation systems see (Aharoni et al., 2019; Cettolo et al., 2017; Lakew et al., 2018).”
One-to-Many Multilingual End-to-End Speech Translation (2019, methodology): “Although this approach has been proposed for RNN-based NMT, it works even better [30] when using the Transformer [6] architecture.”
Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, methodology): “Previous works have also shown similar findings for the Transformer model when it comes to zero-shot translation [17, 6].”