Multi-Task Learning for Multiple Language Translation

By Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, Haifeng Wang. 2015

653 citations recorded by the source corpus.

In this paper, we investigate the problem of learning a machine translation model that can simultaneously translate sentences from one source language to multiple target languages. Our solution is inspired by the recently proposed neural machine translation model which generalizes machine translation as a sequence learning problem. We extend the neural machine translation to a multi-task learning framework which shares source language representation and separates the modeling of different target language translation. Our framework can be applied to situations where either large amounts of parallel data or limited parallel data is available. Experiments show that our multi-task learning model is able to achieve significantly higher translation quality over individually learned model in both situations on the data sets publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Though this can be addressed in principle by training a “copy model” to mimic a traditional alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent…”
Multi-task Sequence to Sequence Learning (2015, methodology): “To the best of our knowledge, there is only one recent publication by Dong et al. (2015) which applies a seq2seq models for machine translation, where the goal is to translate from one language to multiple languages.”
Massively Multilingual Neural Machine Translation (2019, background): “Recent works (Dong et al., 2015; Firat et al., 2016a; Ha et al., 2016; Johnson et al., 2017) extended the approach to support multilingual translation, i.”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “, 2018), performance on out-of-domain data or low resource languages, still remains pretty poor (Duh et al., 2013; Koehn and Knowles, 2017; Farajian et al., 2017; Dong et al., 2015; Zoph et al., 2016).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Early multilingual NMT models utilized separate vo-cabularies for each language (Dong et al., 2015; Luong et al., 2015; Firat et al., 2016a); later ones used shared multilingual vocabularies (Sennrich et al., 2016; Ha et al., 2016c; Johnson et al., 2017) which are shared across languages.”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “Dong et al. (2015) use periodic task alternations with equal training ratios for every task.”
Complete Multilingual Neural Machine Translation (2020, neutral): “Multilingual machine translation (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019), which can serve multiple language pairs with a single model, has attracted much attention.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, background): “multilingual data to improve NMT for all languages involved, which has been extensively studied in a number of papers such as Dong et al. (2015); Johnson et al.”
A Knowledge-Grounded Neural Conversation Model (2017, positive): “Multi-task learning can be helpful in tasks ranging from query classification to machine translation (Caruana 1997; Dong et al. 2015; Liu et al. 2015; Luong et al. 2016).”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, neutral): “ormultilingualNMT,inthiscasewithoutanattentionmechanism. Tomakeproper useofmultilingualdata,theyextendtheirmodelwithmultipleencodersanddecoders,oneforeachsupported sourceandtargetlanguage. Similarlyin[6],theauthorsmodifyanattention-basedencoder-decoderapproach toperformmultilingualNMTbyaddingasepara”
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism (2016, methodology): “The proposed model is a step forward from the recent works on multilingual neural translation, in the sense that we support attention mechanism, compared to (Luong et al., 2015a) and multi-way translation, compared to (Dong et al., 2015).”
Zero-Resource Translation with Multi-Lingual Neural Machine Translation (2016, background): “Furthermore, neural machine translation has been found to translate between multiple languages, achieving better translation quality by exploiting positive language transfer (Dong et al., 2015; Firat et al., 2016; Zoph and Knight, 2016).”
Transfer Learning for Low-Resource Neural Machine Translation (2016, background): “The English embeddings should be kept but the Uzbek embeddings should be modiﬁed during training of the child model.”
Recurrent Neural Network for Text Classification with Multi-Task Learning (2016, background): “More recently, several multi-task encoder-decoder networks were also proposed for neural machine translation [ Dong et al. , 2015; Firat et al. , 2016 ] , which can make use of cross-lingual information.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Dong et al. (2015) use a shared encoder but different decoders to translate the same source language to multiple target languages.”
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder (2016, background): “Similar to the multi-target translation(Dong et al., 2015), we exploit further the correlation in semantics of those target sentences across different languages.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Multilingual MT (Dong et al., 2015; Firat et al., 2016a,b; Wang et al., 2020) addresses the transfer of MT from high-resourced to low-resourced languages by training multilingual models for all languages at once.”