Transfer Learning for Low-Resource Neural Machine Translation

By Barret Zoph, Deniz Yuret, Jonathan May, Kevin Knight. 2016

925 citations recorded by the source corpus.

The encoder-decoder framework for neural machine translation (NMT) has been shown effective in large data scenarios, but is much less effective for low-resource languages. We present a transfer learning method that significantly improves Bleu scores across a range of low-resource languages. Our key idea is to first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. Using our transfer learning method we improve baseline NMT models by an average of 5.6 Bleu on four low-resource language pairs. Ensembling and unknown word replacement add another 2 Bleu which brings the NMT performance on low-resource machine translation close to a strong syntax based machine translation (SBMT) system, exceeding its performance on one language pair. Additionally, using the transfer learning model for re-scoring, we can improve the SBMT system by an average of 1.3 Bleu, improving the state-of-the-art on low-resource machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “Zoph et al. (2016); Firat et al. (2016) pre-train the model on similar rich-resource language pairs and ﬁne-tuned on the target language pair, which relies on supervised data on other language pairs.”
Massively Multilingual Neural Machine Translation (2019, background): “The answer is not trivial: on the one hand, training multiple language pairs together may result in transfer learning (Zoph et al., 2016; Nguyen and Chiang, 2017).”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “Its variant, transfer learning, was also proposed by Zoph et al. (2016), in which an NMT system is pretrained on a high-resource language pair before being ﬁne-tuned on a target low-resource language pair.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “, 2017), and high-resource machine translation can support low-resource machine translation (Zoph et al., 2016).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “Most previous literature in multilingual NMT focuses on improving the performance of low resource languages (Zoph et al., 2016; Firat et al., 2016; Neubig and Hu, 2018), often ignoring the source language performance of the adapted model.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…improve performance on low and zero-resource language pairs due to joint training and consequent positive transfer from higher-resource languages (Zoph et al., 2016; Firat et al., 2016b; Nguyen and Chiang, 2017; Johnson et al., 2017; Neubig and Hu, 2018; Aharoni et al., 2019; Escolano et al.,…”
Transfer Learning in Natural Language Processing (2019, methodology): “We will then describe supervised approaches which make use of large annotated datasets (Zoph et al., 2016; Yang et al., 2017; Wi-eting et al., 2016; Conneau et al., 2017; McCann et al., 2017) before turning to distant supervision approaches which use heuristics to automatically label datasets…”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “Various forms of transfer or multitask learning with seq2seq framework also have the flavors of our algorithm (Zoph et al., 2016; Luong et al., 2015; Firat et al., 2016).”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “To learn our sentence representation we train a unified bilingual NMT system similar to [53] that can translate between Chinese and English in both directions.”
Efficient Neural Architecture Search via Parameter Sharing (2018, background): “The idea has apparent complications, as different child models might utilize their weights differently, but was encouraged by previous work on transfer learning and multitask learning, which established that parameters learned for a particular model on a particular task can be used for other models ”
Pre-trained language model representations for language generation (2019, background): “In neural machine translation, there has been work on transferring representations from high-resource language pairs to low-resource settings (Zoph et al., 2016).”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “In transfer learning and multi-task learning, the weights obtained by training a model designed for a specific task on a dataset are also applicable to other models designed for other tasks [86, 87, 88].”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, background): “Bilinear Upsample Parameter sharing [40, 73] shows to effectively boost the efﬁciency of NAS [46].”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “Other works (Zoph et al., 2016; Lakew et al., 2018, 2019; Escolano et al., 2019) have studied repurposing translation models as initializations for bilingual models for a target low-resource lan-”
Universal Neural Machine Translation for Extremely Low Resource Languages (2018, background): “Multi-lingual NMT has been extensively studied in a number of papers such as Lee et al. (2017), Johnson et al. (2017), Zoph et al. (2016) and Firat et al. (2016).”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, methodology): “Lastly, in order to check whether the learned information can be transferred across models, we use the encoder weights of a high-resource language pair to initialize a low-resource language pair, inspired by the work of Zoph et al. (2016).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Through the lens of a machine learning researcher, “low-resourced” identiﬁes languages for which few digital or computational data resources exist, often classiﬁed in comparison to another language (Gu et al., 2018; Zoph et al., 2016).”
Exploiting Source-side Monolingual Data in Neural Machine Translation (2016, background): “Zoph et al. (2016), Zoph and Knight (2016) and Firat et al. (2016) further deal with more complicated cases (e.g. multi-source languages).”