Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

By Toan Q. Nguyen, David Chiang. 2017

228 citations recorded by the source corpus.

We present a simple method to improve neural translation of a low-resource language pair using parallel data from a related, also low-resource, language pair. The method is based on the transfer method of Zoph et al., but whereas their method ignores any source vocabulary overlap, ours exploits it. First, we split words using Byte Pair Encoding (BPE) to increase vocabulary overlap. Then, we train a model on the first language pair and transfer its parameters, including its source word embeddings, to another model and continue training on the second language pair. Our experiments show that transfer learning helps word-based translation only slightly, but when used on top of a much stronger BPE baseline, it yields larger improvements of up to 4.3 BLEU.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “The answer is not trivial: on the one hand, training multiple language pairs together may result in transfer learning (Zoph et al., 2016; Nguyen and Chiang, 2017).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…language pairs due to joint training and consequent positive transfer from higher-resource languages (Zoph et al., 2016; Firat et al., 2016b; Nguyen and Chiang, 2017; Johnson et al., 2017; Neubig and Hu, 2018; Aharoni et al., 2019; Escolano et al., 2019; Arivazhagan et al., 2019; Hokamp et…”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, methodology): “Previous work on multilingual training has preferred the former (Nguyen and Chiang, 2017), but in this paper we use the latter for two reasons: (1) because data in the LRL will not affect the subword units”
Investigating Multilingual NMT Representations at Scale (2019, background): “…models have demonstrated great improvements for cross-lingual transfer, on tasks including lowresource language translation (Zoph et al., 2016; Nguyen and Chiang, 2017; Neubig and Hu, 2018) and zero or few-shot transfer learning for downstream tasks (Eriguchi et al., 2018; Lample and ∗Work…”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “, 2019) while also improving low-resource translation via transfer learning from higher-resource languages (Zoph et al., 2016; Nguyen and Chiang, 2017; Neubig and Hu, 2018).”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “…(Nakov and Ng (2009) for multilingual SMT), (b) using a common subword-vocabulary across all languages e.g. character (Lee et al., 2017) and BPE (Nguyen and Chiang, 2017), (c) representing words by both character encoding and a latent embedding space shared by all languages (Wang et al., 2019).”
Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (2020, background): “Transferring a pretrained model (source) to a new model (target) requires the use of a shared vocabulary (Nguyen and Chiang, 2017).”
Revisiting Low-Resource Neural Machine Translation: A Case Study (2019, unknown): “Similarly, parallel data from other language pairs can be used to pre-train the network or jointly learn representations (Zoph et al., 2016; Chen et al., 2017; Nguyen and Chiang, 2017; Neu-big and Hu, 2018; Gu et al., 2018a,b; Kocmi and Bojar, 2018).”
Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary (2018, background): “Later, in [18], the parent-child approach has been extended to analyze the effect of using related languages on the source side.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “For example, instead of words, using sub-word representations such as Byte Pair Encoding [126, 140] or phrase-based alignment [115] has shown more promise.”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, unknown): “Continued training has also been applied to cross-lingual transfer learning for NMT, with Zoph et al. (2016) and Nguyen and Chiang (2017) using it for transfer between high-and low-resource language pairs.”
Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, background): “Thus, transfer learning for LRL can be defined in two main forms; i) “vertically”, aggregating data from several language pairs to train a single model [3], ii) “horizontally”, pre-training a model with the available pairs and finetuning it using the test (LRL) language data [33, 34, 12], or iii) wi”
CUNI Submissions in WMT18 (2018, methodology): “As such, this method is very similar to the transfer learning proposed by Zoph et al. (2016) and improved by the using the shared vocabulary as in Nguyen and Chiang (2017).”
Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation (2019, background): “…based transfer learning has been studied for transferring proper parameters (Zoph et al., 2016; Gu et al., 2018b), lexical (Zoph et al., 2016; Nguyen and Chiang, 2017; Gu et al., 2018a; Lakew et al., 2018), and syntactic (Gu et al., 2018a; Murthy et al., 2018) knowledge from a resource-rich…”
The University of Maryland’s Kazakh-English Neural Machine Translation System at WMT19 (2019, background): “Speciﬁcally, Nguyen and Chiang (2017) showed consistent improvements in two Turkic languages via transfering from another related, low-resource language.”
Neural Machine Translation: A Review (2019, methodology): “Search errors in NMT have been studied by Niehues et al. (2017); Stahlberg & Byrne (2019); Stahlberg et al.”
Choosing Transfer Languages for Cross-Lingual Learning (2019, unknown): “translation (Zoph et al., 2016; Johnson et al., 2017; Nguyen and Chiang, 2017; Neubig and Hu, 2018), parsing (Täckström et al.”