Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

By Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, Andrew McCallum. 2020

100 citations recorded by the source corpus.

Self-supervised pre-training of transformer models has revolutionized NLP applications. Such pre-training with language modeling objectives provides a useful initial point for parameters that generalize well to new tasks with fine-tuning. However, fine-tuning is still data inefficient -- when there are few labeled examples, accuracy can be low. Data efficiency can be improved by optimizing pre-training directly for future fine-tuning with few examples; this can be treated as a meta-learning problem. However, standard meta-learning techniques require many training tasks in order to generalize; unfortunately, finding a diverse set of such supervised tasks is usually difficult. This paper proposes a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text. This is achieved using a cloze-style objective, but creating separate multi-class classification tasks by gathering tokens-to-be blanked from among only a handful of vocabulary terms. This yields as many unique meta-training tasks as the number of subsets of vocabulary terms. We meta-train a transformer model on this distribution of tasks using a recent meta-learning framework. On 17 NLP tasks, we show that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning. Furthermore, we show how the self-supervised tasks can be combined with supervised tasks for meta-learning, providing substantial accuracy gains over previous supervised meta-learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…Recent works have ex- plored meta-learning methods for relation classi- fication (Han et al., 2018; Gao et al., 2019), gen- eral text classification (Dou et al., 2019; Bansal et al., 2020a,b), low-resource machine transla- tion (Gu et al., 2018), cross-lingual NLI/QA (Nooralahzadeh et al., 2020).”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, background): “…work in few-shot learning has used different approaches, including semi-supervised learning with data augmentation or consistency training (Miyato et al., 2017; Clark et al., 2018; Xie et al., 2020; Chen et al., 2020) and meta learning (Finn et al., 2017; Huang et al., 2018; Bansal et al., 2020).”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, methodology): “…NLP, this approach has been evaluated on tasks such as sentiment analysis (Yu et al., 2018), user intent classification (Yu et al., 2018; Chen et al., 2020b), natural language understanding (Dou et al., 2019), text classification (Bansal et al., 2020) and dialogue generation (Huang et al., 2020).”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “, 2020a), metalearning (Bansal et al., 2020), self-training (Du et al.”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, methodology): “Relatedly, Bansal et al. (2020) used self-supervised meta-learning for fewshot text classification rather than in-context fewshot learning.”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, methodology): “…(Vinyals et al., 2016; Sung et al., 2018; Zhao et al., 2020), and has recently been adapted to different NLP tasks such as text classiﬁcation (Wang et al., 2018; Geng et al., 2019; Bansal et al., 2020), machine translation (Gu et al., 2018) and relation classiﬁcation (Han et al., 2018).”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, methodology): “We experiment with 4-shot adaptation on GLUE-SciTail to match the evaluation setup from Bansal et al. (2020b).”