Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

By Thanh-Le Ha, Jan Niehues, A. Waibel. 2016

301 citations recorded by the source corpus.

In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework under a unified approach in which the information shared among languages can be helpful in the translation of individual language pairs. We are then able to employ attention-based Neural Machine Translation for many-to-many multilingual translation tasks. Our approach does not require any special treatment on the network architecture and it allows us to learn minimal number of free parameters in a standard way of training. Our approach has shown its effectiveness in an under-resourced translation scenario with considerable improvements up to 2.6 BLEU points. In addition, we point out a novel way to make use of monolingual data with Neural Machine Translation using the same approach with a 3.15-BLEU-score gain in IWSLT’16 English→German translation task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, methodology): “We follow the method of Ha et al. (2016); Johnson et al. (2017) and add a target-language preﬁx token to each source sentence to enable many-to-many translation.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “There are two categories of meta-learning: 1. learning a meta-policy for updating model parameters (see, e.g., Andrychowicz et al., 2016; Ha et al., 2016a; Mishra et al., 2017) 2. learning a good parameter initialization for fast adaptation (see, e.g., Finn et al., 2017; Vinyals et al., 2016; Snell…”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “…threads of work include research on meta-learning to learn model hyper-parameters (Nichol et al., 2018; Baydin et al., 2017), model parameters (Ha et al., 2016a; Platanios et al., 2018) and models that learn new tasks with high sample ef-ﬁciency (Finn et al., 2017) without forgetting existing…”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “Similarly to Ha et al. (2016) and Johnson et al. (2017), we add a special token in the encoder indicating the source language and a special token in the decoder indicating the target language.”
Complete Multilingual Neural Machine Translation (2020, neutral): “…non-English pairs either resort to bridging (pivoting) through a pivot language (Habash and Hu, 2009), or make use of synthetic parallel data (via back-translation) (Firat et al., 2016b; Chen et al., 2017) or study the problem under zero-shot settings (Johnson et al., 2017; Ha et al., 2016).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “Language Indicator Inspired by (Johnson et al., 2017; Ha et al., 2016), to distinguish from different translation pairs, we simply add two artificial language tokens to indicate languages at the source and target side.”
WARP: Word-level Adversarial ReProgramming (2021, methodology): “In particular, multilingual neural machine translation models use special tokens in the input to control the target language (Ha et al., 2016; Johnson et al., 2017) or politeness of the translation (Sennrich et al., 2016).”
Unsupervised Neural Machine Translation (2017, methodology): “Our system makes use of one and only one encoder that is shared by both languages involved, similarly to Ha et al. (2016), Lee et al. (2017) and Johnson et al. (2017).”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “More specifically, the authors explored the adaptation through continued training, where an NMT model trained using large corpora in one domain can later initialize a new NMT model for another domain.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Therefore, multilingual NMT (Johnson et al., 2017; Firat et al., 2016; Ha et al., 2016; Lu et al., 2018) is developed which handles multiple language pairs in one model, greatly reducing the ofﬂine training and online serving cost.”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, background): “While adapting MT systems to new languages is a long-standing challenge (Schultz and Black, 2006; Jabaian et al., 2013), multilingual NMT is highly promising in its ability to abstract across language boundaries (Firat et al., 2016; Ha et al., 2016; Johnson et al., 2016).”
A Teacher-Student Framework for Zero-Resource Neural Machine Translation (2017, background): “Several authors propose a universal encoder-decoder network in multilingual scenarios to perform zero-shot learning (Johnson et al., 2016; Ha et al., 2016).”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, positive): “Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018; Tan et al., 2019), improve low-resource translation (Ha et al., 2016; Arivazhagan et al., 2019b), 1We release our code at https://github. com/bzhangGo/zero.”
A neural interlingua for multilingual machine translation (2018, background): “The universal encoder-decoder approach (Ha et al., 2016; Johnson et al., 2017) constructs a shared vocabulary”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, methodology): “The universal model proposed by Ha et al. (2016) requires a new coding scheme for the input sentences, which results in large vocabulary sizes that are difﬁ-cult to scale.”
Learning Language Representations for Typology Prediction (2017, methodology): “NMT Language Vector In our second set of feature vectors, MTV EC , we similarly use a language embedding vector, but instead learn a multi-lingual neural MT model trained to translate from many languages to English, in a similar fashion to Johnson et al. (2016); Ha et al. (2016).”
An Empirical Analysis of NMT-Derived Interlingual Embeddings and Their Use in Parallel Sentence Identification (2017, methodology): “A simpler approximation (Ha et al., 2016; Johnson et al., 2016) considers exactly the same architecture as the one-to-one NMT for many-to-many NMT using multilingual data with some additional labelling.”
Multilingual Neural Machine Translation with Language Clustering (2019, background): “Multilingual NMT NMT has recently been extended from the translation of a language pair to multilingual translation (Dong et al., 2015; Lu-ong et al., 2015a; Firat et al., 2016; Lu et al., 2018; Johnson et al., 2017; He et al., 2019; Ha et al., 2016; Platanios et al., 2018; Tan et al., 2019).”
Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation (2020, background): “, 2019) as well as zero-shot translation (Ha et al., 2016; Johnson et al., 2017; Arivazhagan et al., 2019b).”