Universal Neural Machine Translation for Extremely Low Resource Languages

By Jiatao Gu, Hany Hassan, Jacob Devlin, Victor O. K. Li. 2018

304 citations recorded by the source corpus.

In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multi-lingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multi-lingual training and back-translation. Furthermore, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “, 2019) where the ultimate goal is to jointly train one translation model that translates multiple language directions at the same time, and shares representations to improve the translation performance on low-resource languages (Gu et al., 2018).”
Massively Multilingual Neural Machine Translation (2019, background): “This corroborates the results of Gu et al. (2018) that showed the sensitivity of such models to similar low resource conditions and the improvements gained from using many-to-one models (however with much fewer language pairs).”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “The proposed MetaNMT differs from the existing framework of multilingual translation (Lee et al., 2016; Johnson et al., 2016; Gu et al., 2018b) or transfer learning (Zoph et al., 2016).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “This is also reﬂected in other works on low-resource machine translation where Any → En typically bene-ﬁts the most (Zoph et al., 2016; Nguyen and Chi-ang, 2017; Gu et al., 2018a,b).”
Beyond English-Centric Multilingual Machine Translation (2020, background): “Strong performance for low-resource languages remains a critical area for future improvement (Gu et al., 2018; Sennrich and Zhang, 2019).”
Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “In particular, on ro → en , our unsupervised PB-SMT model obtains a BLEU score of 23.9, outper-forming Gu et al. (2018)’s method by 1 point, despite its use of 6,000 parallel sentences, a seed dictionary, and a multi-NMT system combining parallel resources from 5 different languages.”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “The longstanding goal of multilingual machine translation (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Gu et al., 2018) has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “In Gu et al. (2018a;b) and Neubig & Hu (2018), multilingual translation is leveraged to boost the accuracy of low-resource language pairs with better model structure or training mechanism.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Through the lens of a machine learning researcher, “low-resourced” identiﬁes languages for which few digital or computational data resources exist, often classiﬁed in comparison to another language (Gu et al., 2018; Zoph et al., 2016).”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, background): “Results on multi-lingual training for low-resource translation (Gu et al., 2018; Qi et al., 2018) further demonstrates this potential, although these works do not consider adaptation to new languages, the main focus of our work.”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, background): “For extremely low resource languages, Gu et al. (2018) proposed sharing lexical and sentence-level representations across multiple source languages with a single target language.”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “…attention head (Vaswani et al., 2017); we use the Pytorch’s default method 4 to initialize all the weights for pytorch-init ; we also try to change the conventional architecture with a layer-wise attention (Gu et al., 2018a) between the encoder and decoder, and it is denoted as layerwise-attn .”
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, background): “, 2019) aims to jointly train one translation model that translates multiple language directions, and shares representations to improve the translation performance on low-resource languages (Gu et al., 2018).”
XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering (2019, unknown): “More recently, similar techniques have been adapted for extremely low resource languages (Gu et al., 2018).”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Strategies involve allocating language specific hidden states, attention modules, decoders or additional specialized layers ((Hokamp et al., 2019; Wang et al., 2018; Gu et al., 2018; Bapna et al., 2019)).”
Revisiting Low-Resource Neural Machine Translation: A Case Study (2019, result): “97 BLEU reported by Gu et al. (2018b).”
VaLaR NMT : Vastly Lacking Resources Neural Machine Translation (2019, background): “Sharing lexical and sentence-level representations across multiple source languages into one target language [4] or sharing parameters across different language pairs in Google NMT [3] are some successful approaches.”