Rapid Adaptation of Neural Machine Translation to New Languages

By Graham Neubig, Junjie Hu. 2018

223 citations recorded by the source corpus.

This paper examines the problem of adapting neural machine translation systems to new, low-resourced languages (LRLs) as effectively and rapidly as possible. We propose methods based on starting with massively multilingual “seed models”, which can be trained ahead-of-time, and then continuing training on data related to the LRL. We contrast a number of strategies, leading to a novel, simple, yet effective method of “similar-language regularization”, where we jointly train on both a LRL of interest and a similar high-resourced language to prevent over-fitting to small LRL data. Experiments demonstrate that massively multilingual models, even without any explicit adaptation, are surprisingly effective, achieving BLEU scores of up to 15.5 with no data from the LRL, and that the proposed similar-language regularization method improves over other adaptation methods by 1.7 BLEU points average over 4 LRL settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, methodology): “We also compare our massively multilingual models to bilingual baselines and to two recently published results on this dataset (Neubig and Hu (2018); Wang et al. (2019)).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “While several approaches have been explored in literature (Chu and Wang, 2018), full ﬁne-tuning of model parameters remains the dominant approach for adapting to new domains and languages (Luong and Manning, 2015; Neubig and Hu, 2018).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Temperature based sampling or co-training with related languages to improve adaptation (Zoph et al., 2016; Neubig and Hu, 2018) fall within the class of static strategies.”
Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, methodology): “Following Neubig and Hu (2018), we have chosen TR as a helper source language, because it is related to KK (both belong to the same language family) and TR is higher-resourced than KK.”
Masked Language Model Scoring (2019, methodology): “TED Talks gl → en sk → en ar → en Neubig and Hu (2018) 16 We also consider a non-English, higher-resource target by rescoring a pre-existing WMT 2014 English-German system (trained on 4.5M sentence pairs) with German BERT (base) models 1 trained on 16GB of text, similar to English BERT.”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts (Gu et al., 2018; Neubig and Hu, 2018; Lakew et al., 2019; Chronopoulou et al., 2020).”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “In Gu et al. (2018a;b) and Neubig & Hu (2018), multilingual translation is leveraged to boost the accuracy of low-resource language pairs with better model structure or training mechanism.”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, neutral): “We wanted to compare with the latest low-resource works of (Neubig and Hu, 2018; Aharoni et al., 2019) on the TED Talks corpus (Qi et al., 2018).”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, background): “Poor initialization can be attributed to to the monolingual corpora used to train word embeddings which do not have sufficient number of overlapping strings, and are not comparable (Neubig and Hu, 2018; Søgaard et al., 2018).”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, background): “For instance, low resource languages are often distant and often times corresponding corpora are not comparable, conditions which are far from the simulation with high resource European languages, as has been recently pointed out (Neubig and Hu, 2018).”
Investigating Multilingual NMT Representations at Scale (2019, background): “Although there exist empirical results that suggest that transfer is stronger across similar languages (Zoph et al., 2016; Neubig and Hu, 2018; Wu and Dredze, 2019), the mechanism of generalization in multilingual representations is poorly understood.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “, 2019) while also improving low-resource translation via transfer learning from higher-resource languages (Zoph et al., 2016; Nguyen and Chiang, 2017; Neubig and Hu, 2018).”
Balancing Training for Multilingual Neural Machine Translation (2020, methodology): “As stated previously, recent results have demonstrated the importance of balancing HRLs and LRLs during multilingual training (Arivazhagan et al., 2019; Conneau et al., 2019), which is largely done with heuristic sam-0 pling using a temperature term; MultiDDS provides a more effective and less heuri”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “the child is not overwhelmed by the parent [102, 139].”
A Survey of Multilingual Neural Machine Translation (2019, background): “Neubig and Hu (2018) used language relatedness to prevent overﬁtting when rapidly adapting pre-trained MNMT model for low-resource scenarios.”
Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation (2020, background): “…(Devlin et al., 2019; Raffel et al., 2019; Conneau et al., 2019), but their success for unsupervised NMT (Conneau and Lample, 2019; Song et al., 2019) currently lacks robustness when languages are distant or monolingual data domains are mismatched (Neubig and Hu, 2018; Vuli´c et al., 2019).”
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (2019, methodology): “Speciﬁcally, we use the example of comparing phrase-based (Koehn et al., 2003) and neural (Bahdanau et al., 2015) Slovak-English machine translation systems from Neubig and Hu (2018).”
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, methodology): “For MT various approaches have been explored, with a common method training on high-resource language(s) and fine-tuning on a low-resource language (Neubig and Hu, 2018).”