Meta-Learning for Low-Resource Neural Machine Translation

By Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, Victor O. K. Li. 2018

364 citations recorded by the source corpus.

In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML, Finn, et al., 2017) for low-resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem where we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation (Gu et al., 2018b) to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro,Lv, Fi, Tr and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach (Zoph et al., 2016) and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT’16 by seeing only 16,000 translated words (~600 parallel sentences)

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “This is also reﬂected in other works on low-resource machine translation where Any → En typically bene-ﬁts the most (Zoph et al., 2016; Nguyen and Chi-ang, 2017; Gu et al., 2018a,b).”
Transfer Learning in Natural Language Processing (2019, background): “As part of this overview, we will also highlight connections to other related and promising directions of research such as meta-learning (Gu et al., 2018), multilingual transfer learning, and continual learning (Lopez-Paz and Ranzato, 2017).”
Few-Shot Text Generation with Natural Language Instructions (2020, background): “…– for example, GEN P ET requires manually designed patterns and some amount of unlabeled examples, whereas meta learning approaches (e.g., Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019) require large annotated datasets for related tasks –, or they cannot be transferred to a…”
Zero-shot Text Classification With Generative Language Models (2019, background): “Some works have looked at meta-learning for machine translation in low resource languages (Gu et al., 2018), iteratively guiding policies with language (Co-Reyes et al., 2018) for instruction following (Branavan et al., 2009; Chen and Mooney, 2011), and generating WikiSQL-style structured queries…”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “MAML is widely applied into vast learning scenarios, e.g., robotics con-trol [49], neural translation [50] and uncertainty estimation [51], and many improved algorithms [52], [53], [54], [55], [56] have been made to further boost its performance.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Many more tasks have since been tackled, including text classification [135], neural program induction [263] and synthesis [264], English to SQL program synthesis [265], text-based relationship graph extractor [266], machine translation [267], and quickly adapting to new personas in dialogue”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “Domain transfer has also been studied on many tasks [3, 4], including dialogue intent detection & slot tagging [31], sentiment classification [77], NLI [21], and machine translation [27, 58].”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, background): “…language pairs often involve distant languages that do not share the same alphabet, or have very different morphology and syntax; accordingly, recent work has begun to 1 https://paracrawl.eu/ explore language-independent lexical representations to improve transfer learning (Gu et al., 2018).”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, background): “FSL has achieved promising results in various domains such as visual recognition [13, 38, 57], domain adaptation [14, 24], neural machine translation [19], data compression [64], and density estimation [48].”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “…attention head (Vaswani et al., 2017); we use the Pytorch’s default method 4 to initialize all the weights for pytorch-init ; we also try to change the conventional architecture with a layer-wise attention (Gu et al., 2018a) between the encoder and decoder, and it is denoted as layerwise-attn .”
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems (2019, methodology): “Moreover, few-shot learning in natural language applications has been applied in semantic parsing (Huang et al., 2018), machine translation (Gu et al., 2018), and text clas-siﬁcation (Yu et al., 2018) with meta-learning approaches (Schmidhuber, 1987; Finn et al., 2017).”
Domain Adaptive Dialog Generation via Meta Learning (2019, methodology): “, 2017), machine translation (Gu et al., 2018), robot manipulation (Duan et al.”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, background): “Common settings include few-shot adaptation to new relations (Han et al., 2018), words (Holla et al., 2020), domains (Bao et al., 2020; Yu et al., 2018; Geng et al., 2019), and language pairs (Gu et al., 2018).”
Hierarchically Structured Meta-learning (2019, background): “Despite their early success in few-shot image classification (Ravi & Larochelle, 2016) and machine translation (Gu et al., 2018), most of the existing meta-learning algorithms assume the transferable knowledge to be globally shared across all tasks.”