Meta-learning for Few-shot Natural Language Processing: A Survey

By Wenpeng Yin. 2020

89 citations recorded by the source corpus.

Few-shot natural language processing (NLP) refers to NLP tasks that are accompanied with merely a handful of labeled examples. This is a real-world challenge that an AI system must learn to handle. Usually we rely on collecting more auxiliary information or developing a more efficient learning algorithm. However, the general gradient-based optimization in high capacity models, if training from scratch, requires many parameter-updating steps over a large number of labeled examples to perform well (Snell et al., 2017). If the target task itself cannot provide more information, how about collecting more tasks equipped with rich annotations to help the model learning? The goal of meta-learning is to train a model on a variety of tasks with rich annotations, such that it can solve a new task using only a few labeled samples. The key idea is to train the model's initial parameters such that the model has maximal performance on a new task after the parameters have been updated through zero or a couple of gradient steps. There are already some surveys for meta-learning, such as (Vilalta and Drissi, 2002; Vanschoren, 2018; Hospedales et al., 2020). Nevertheless, this paper focuses on NLP domain, especially few-shot applications. We try to provide clearer definitions, progress summary and some common datasets of applying meta-learning to few-shot NLP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Meta & Continual Learning

Citations in the corpus

FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “line of work (not noted by Yin [73]) is pretraining transfer, the problem of whether pretrained language models can perform well at meta-test time without any meta-training.”
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020, methodology): “Prototypical network is essentially a nearest neighbor algorithm (Yin, 2020) • Comparing UFO-E NTAIL with the SOTA technique STILTs in k -shot settings.”
Want To Reduce Labeling Cost? GPT-3 Can Help (2021, background): “One concurrent with our work, Yoo et al. (2021) consider distilling knowledge from GPT-3 with synthetic data.”
A Survey on Deep Semi-Supervised Learning (2021, background): “Meta-learning [41], [42], [43], [44], [45], also known as “learning to learn”, aims to learn new skills or adapt to new tasks rapidly with previous knowledge and a few training examples.”
AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization (2021, methodology): “Domain adaption for natural language processing and computer vision tasks is widely studied (Blitzer et al., 2007; Mansour et al., 2008; Daumé III, 2009; Sandu et al., 2010; Foster et al., 2010; Wang and Cardie, 2013; Sun et al., 2016; Liu et al., 2019b, 2020b; Gururangan et al., 2020; Winata et al.”