Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

By Trapit Bansal, Rishikesh Jha, Andrew McCallum. 2019

127 citations recorded by the source corpus.

Pre-trained transformer models have shown enormous success in improving performance on several downstream tasks. However, fine-tuning on a new task still requires large amounts of task-specific labeled data to achieve good performance. We consider this problem of learning to generalize to new tasks, with a few examples, as a meta-learning problem. While meta-learning has shown tremendous progress in recent years, its application is still limited to simulated problems or problems with limited diversity across tasks. We develop a novel method, LEOPARD, which enables optimization-based meta-learning across tasks with a different number of classes, and evaluate different methods on generalization to diverse NLP classification tasks. LEOPARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label. Across 17 NLP tasks, including diverse domains of entity typing, natural language inference, sentiment analysis, and several other text classification tasks, we show that LEOPARD learns better initial parameters for few-shot learning than self-supervised pre-training or multi-task training, outperforming many strong baselines, for example, yielding 14.6% average relative gain in accuracy on unseen tasks with only 4 examples per label.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning (2021, methodology): “, 2019a) Nu =9 Few-shot (Tv) DivFSL (Bansal et al., 2020) Nv =17”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, methodology): “Our benchmark consists of episodes with variable number of shots in the range [1-5] and with class imbalance.”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “Figure 3 illustrates the 2D vector projections of task embeddings using PCA (Wold et al., 1987 Prior work also proposed meta-learning or Bayesian approaches to generate softmax layer parameters for new settings (Bansal et al., 2020; Ponti et al., 2020).”
Meta-learning via Language Model In-context Tuning (2021, methodology): “Meta-learning for FSL Meta-learning is a widely used technique in NLP to improve cross-domain transfer (Yu et al., 2018; Geng et al., 2019; Holla et al., 2020; Deng et al., 2020) and cross-task transfer (Gu et al., 2018; Bansal et al., 2020; Dou et al., 2019).”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “Few shot (Bansal et al., 2019) and few sample (Zhang et al., 2021) learning and evaluation could potentially require more word order signal, and thereby encouraging the model to leverage its own learned syntax better.”
Meta-learning for Few-shot Natural Language Processing: A Survey (2020, methodology): “, 2018), set task-agnostic parameters (Bansal et al., 2019) and task-specific parameters • Use BERT as text encoder”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…Recent works have ex- plored meta-learning methods for relation classi- fication (Han et al., 2018; Gao et al., 2019), gen- eral text classification (Dou et al., 2019; Bansal et al., 2020a,b), low-resource machine transla- tion (Gu et al., 2018), cross-lingual NLI/QA (Nooralahzadeh et al., 2020).”
Controllable Semantic Parsing via Retrieval Augmentation (2021, methodology): “Fast update with meta learning has been applied to NLP models for generalizing to unseen tasks or domains (Gu et al., 2018a; Dou et al., 2019; Bansal et al., 2020; Chen et al., 2020; Athiwaratkun et al., 2020; Wang et al., 2021).”
Zero-shot Text Classification With Generative Language Models (2019, background): “Bansal et al. (2019) combine prototypical networks and MAML to adapt to NLP tasks with different numbers of labels.”
Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks (2020, methodology): “Hyper-parameters: We follow the approach of Bansal et al. (2019) and use validation tasks for estimating hyper-parameters during ﬁne-tuning for all baseline models.”
Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (2021, methodology): “The proposed approach corresponds to the other few-shot NLP methods, including: (1) Meta-learning (Yu et al. (2018) 2020)), which leverages unlabeled samples.”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, methodology): “We experiment with 4-shot adaptation on GLUE-SciTail to match the evaluation setup from Bansal et al. (2020b).”
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning (2020, background): “…learning problems in NLP – speciﬁcally machine translation (Gu et al., 2018), relation classiﬁcation (Obamuyide and Vlachos, 2019b), sentence-level semantic tasks (Dou et al., 2019; Bansal et al., 2019), text classiﬁcation (Jiang et al., 2018), and word sense disambiguation (Holla et al., 2020).”