Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism

By Orhan Firat, Kyunghyun Cho, Yoshua Bengio. 2016

644 citations recorded by the source corpus.

We propose multi-way, multilingual neural machine translation. The proposed approach enables a single neural translation model to translate between multiple languages, with a number of parameters that grows only linearly with the number of languages. This is made possible by having a single attention mechanism that is shared across all language pairs. We train the proposed multi-way, multilingual model on ten language pairs from WMT'15 simultaneously and observe clear performance improvements over models trained on only one language pair. In particular, we observe that the proposed model significantly improves the translation quality of low-resource language pairs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “When pivots are not easy to obtain, Firat et al. (2016a); Lee et al. (2016); Johnson et al. (2016) have shown that the structure of NMT is suitable for multilingual machine translation.”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “Most previous literature in multilingual NMT focuses on improving the performance of low resource languages (Zoph et al., 2016; Firat et al., 2016; Neubig and Hu, 2018), often ignoring the source language performance of the adapted model.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Early multilingual NMT models utilized separate vo-cabularies for each language (Dong et al., 2015; Luong et al., 2015; Firat et al., 2016a); later ones used shared multilingual vocabularies (Sennrich et al., 2016; Ha et al., 2016c; Johnson et al., 2017) which are shared across languages.”
Beyond English-Centric Multilingual Machine Translation (2020, background): “, 2017) and more recently, neural MMT models have shown promising results (Firat et al., 2016; Zhang et al., 2020).”
Complete Multilingual Neural Machine Translation (2020, neutral): “We hypothesise that this imbalance in the learning signal with respect to the target language is one of the roots of poor translation quality of multilingual models when translating out of English (Firat et al., 2016a; Johnson et al., 2017; Arivazhagan et al., 2019b).”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “We chose multilingual neural machine translation (MT) [39, 40, 41] to validate our design for efficient training with GShard.”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, methodology): “In this section, we explore two ways of leveraging available parallel data to improve zero-shot translation quality, similar in spirit to what was reported in Firat et al., (2016c). For our multilingual architecture”
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling (2016, background): “Neural network models have recently made tremendous progress in a variety of NLP applications such as speech recognition (Irie et al., 2016), sentiment analysis (Socher et al., 2013), text summarization (Rush et al., 2015; Nallapati et al., 2016), and machine translation (Firat et al., 2016).”
Unsupervised Neural Machine Translation (2017, background): “It has however been shown that the use of more advanced models like a teacher-student framework can bring considerable improvements over this basic baseline (Firat et al., 2016b; Chen et al., 2017).”
Zero-Resource Translation with Multi-Lingual Neural Machine Translation (2016, methodology): “First, we showed that the multi-way, multilingual neural translation model by Firat et al. (2016) is able to exploit common, underlying structures across many languages in order to better translate when a source sentence is given in multiple languages.”
Transfer Learning for Low-Resource Neural Machine Translation (2016, background): “The parent model trained on a large amount of bilingual data can be considered an anchor point, the peak of our prior distribution in model space.”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, background): “…to consider pre-training in multilingual translation systems that share an encoder or decoder between multiple languages (Johnson et al., 2016; Firat et al., 2016), which is another promising way to use additional data (this time from another language) as a way to improve NMT. Speciﬁcally, we…”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, methodology): “of our multilingual models. Data scheduling For the multilingual models, an appropriate scheduling of data from different languages is crucial to avoid overﬁtting to one language too soon. Following (Firat et al., 2016a; Firat et al., 2016b), each minibatch is balanced, in that the proportion of eac”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Multilingual NMT (Dong et al., 2015; Firat et al., 2016; Ha et al., 2016; Johnson et al., 2017; Tan et al., 2019) can be framed as a multi-task learning problem.”