Generalizing from a Few Examples

By Yaqing Wang, Quanming Yao, James T. Kwok, Lionel Ming-shuan Ni. 2019

2,151 citations recorded by the source corpus.

Machine learning has been highly successful in data-intensive applications but is often hampered when the data set is small. Recently, Few-shot Learning (FSL) is proposed to tackle this problem. Using prior knowledge, FSL can rapidly generalize to new tasks containing only a few samples with supervised information. In this article, we conduct a thorough survey to fully understand FSL. Starting from a formal definition of FSL, we distinguish FSL from several relevant machine learning problems. We then point out that the core issue in FSL is that the empirical risk minimizer is unreliable. Based on how prior knowledge can be used to handle this core issue, we categorize FSL methods from three perspectives: (i) data, which uses prior knowledge to augment the supervised experience; (ii) model, which uses prior knowledge to reduce the size of the hypothesis space; and (iii) algorithm, which uses prior knowledge to alter the search for the best hypothesis in the given hypothesis space. With this taxonomy, we review and discuss the pros and cons of each category. Promising directions, in the aspects of the FSL problem setups, techniques, applications, and theories, are also proposed to provide insights for future research.1

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Carbon Emissions and Large Neural Network Training (2021, background): “One potential energy benefit of a large language model like GPT-3 is that they exhibit few-shot generalization , which means that they don’t need to be retrained for every new task like smaller models [Wan20].”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “Few-Shot Learning aims to extract the inductive bias from base classes and generalize it to unseen classes [44], [45], [46], [47], [48].”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “As we assume that only a few supporting hand-drawn sketch exemplars will be provided corresponding to every novel class, we design the weight generator G ψ under a few-shot paradigm [61].”
Continual Learning via Local Module Composition (2021, methodology): “[94] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni.”
Meta-Learning in Neural Networks: A Survey (2020, background): “and/or focus on algorithm selection for data mining [29], [33], [35], [36], AutoML [34], or particular applications of meta-learning such as few-shot learning [37] or NAS [38].”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “Few-shot learning, the challenge of learning from a small number of examples, is critical for developing efficient, robust NLP techniques [71, 76].”
Understanding Emails and Drafting Responses - An Approach Using GPT-3 (2021, background): “It learns based on only a few commands on what to do and ideally, some examples of the desired output (few-shot learning) (Wang et al., 2020).”
How to Train Your MAML to Excel in Few-Shot Classification (2021, background): “There have been an abundance of “novel” algorithms proposed for few-shot classification (Hospedales et al., 2020; Wang et al., 2020).”
Deep Learning for Anomaly Detection (2020, background): “This scenarios is fundamentally different from the general few-shot learning [159], in which the limited examples are class-specific and assumed to share the same manifold/class structure.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “To handle this issue, one of the possible solutions is to take advantage of few-shot learning techniques [380].”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, background): “GPT-2, developed in 2019 [22], mainly introduces the idea of multi-task learning [23] with more network parameters and data than GPT for training, so that the pretrained generative language model can be generalized to most of the supervised subtasks without further fine-tuning.”
Large Language Models in Finance: A Survey (2023, methodology): “Few-shot learning has shown advantages in various previous works [5, 48].”
A Survey on Few-Shot Class-Incremental Learning (2023, neutral): “Data augmentation can be categorized by its source: transforming samples from the training set, transforming samples from a weakly labeled or unlabeled data set, or transforming samples from similar data sets (Wang et al., 2020b).”
Symbolic Behaviour in Artificial Intelligence (2021, background): “[31] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni.”