Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

By Zi-Yi Dou, Keyi Yu, Antonios Anastasopoulos. 2019

130 citations recorded by the source corpus.

Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based meta-learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “Other approaches for few-shot learning in NLP include exploiting examples from related tasks (Yu et al., 2018; Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019; Yin et al., 2019) and using data augmentation (Xie et al.”
Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “Other approaches to few-shot learning in NLP commonly require large sets of examples from related tasks (Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019; Ye et al., 2020), parallel data for”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, neutral): “…NLP. Recent works have ex- plored meta-learning methods for relation classi- fication (Han et al., 2018; Gao et al., 2019), gen- eral text classification (Dou et al., 2019; Bansal et al., 2020a,b), low-resource machine transla- tion (Gu et al., 2018), cross-lingual NLI/QA (Nooralahzadeh et al.,…”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “• Domain Transfer: MR [49], CR [32], SNLI [9], and SciTail [35] are only in the meta-testing phase, but the corresponding sentiment and NLI datasets exist in the meta-training phase (MNLI [74], QNLI [52], and SST-2 [62]).”
Meta-learning via Language Model In-context Tuning (2021, methodology): “Meta-learning for FSL Meta-learning is a widely used technique in NLP to improve cross-domain transfer (Yu et al., 2018; Geng et al., 2019; Holla et al., 2020; Deng et al., 2020) and cross-task transfer (Gu et al., 2018; Bansal et al., 2020; Dou et al., 2019).”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, background): “Transfer learning [29] has enjoyed the prosperity recently in a number of areas [1, 6, 7, 18].”
Few-Shot Text Generation with Pattern-Exploiting Training (2020, background): “Other approaches for few-shot learning in NLP commonly require large sets of examples from re-lated tasks (Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019), parallel data for consistency training (Xie et al., 2019; Chen et al., 2020), or highly specialized methods tailored towards a speciﬁc…”
Controllable Semantic Parsing via Retrieval Augmentation (2021, methodology): “Fast update with meta learning has been applied to NLP models for generalizing to unseen tasks or domains (Gu et al., 2018a; Dou et al., 2019; Bansal et al., 2020; Chen et al., 2020; Athiwaratkun et al., 2020; Wang et al., 2021).”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, background): “On the other hand, many works deal with few-shot adaptation in settings with no clear task distribution (Dou et al., 2019; Bansal et al., 2020a) but do not address meta-overfitting, and thus are complementary to our work.”
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning (2020, background): “…learning problems in NLP – speciﬁcally machine translation (Gu et al., 2018), relation classiﬁcation (Obamuyide and Vlachos, 2019b), sentence-level semantic tasks (Dou et al., 2019; Bansal et al., 2019), text classiﬁcation (Jiang et al., 2018), and word sense disambiguation (Holla et al., 2020).”
Meta-learning for Few-shot Natural Language Processing: A Survey (2020, methodology): “Dou et al. (2019) compared three typical optimization-based meta-learning, including MAML , First-order MAML and Reptile on GLUE benchmarks: treating the low-resource tasks CoLA, MRPC, STS-B and RTE as the testing tasks, and the four high-resource tasks SST-2, QQP, MNLI and QNLI as the training…”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, methodology): “…NLP, this approach has been evaluated on tasks such as sentiment analysis (Yu et al., 2018), user intent classification (Yu et al., 2018; Chen et al., 2020b), natural language understanding (Dou et al., 2019), text classification (Bansal et al., 2020) and dialogue generation (Huang et al., 2020).”