Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks

By Micah Goldblum, Steven Reich, Liam H. Fowl, Renkun Ni, Valeriia Cherepanova, Tom Goldstein. 2020

83 citations recorded by the source corpus.

Meta-learning algorithms produce feature extractors which achieve state-of-the-art performance on few-shot classification. While the literature is rich with meta-learning methods, little is known about why the resulting feature extractors perform so well. We develop a better understanding of the underlying mechanics of meta-learning and the difference between models trained using meta-learning and models which are trained classically. In doing so, we introduce and verify several hypotheses for why meta-learned models perform better. Furthermore, we develop a regularizer which boosts the performance of standard training routines for few-shot classification. In many cases, our routine outperforms meta-learning while simultaneously running an order of magnitude faster.

Subfields: Artificial Intelligence, Cancer Research, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Virginia Smith
2. Ameet Talwalkar
3. Quoc V. Le
4. Tian Li
5. Sergey Levine
6. Ruslan Salakhutdinov
7. Honglak Lee
8. Dacheng Tao

In Cancer Research

1. Liam H. Fowl
2. Micah Goldblum
3. Tom Goldstein

In Representation & Self-Supervised Learning

1. Ting-Li Chen
2. Mohammad Norouzi
3. Simon Kornblith
4. Honglak Lee
5. Tom Goldstein
6. Geoffrey E. Hinton
7. Micah Goldblum
8. Quoc V. Le

In Meta & Continual Learning

1. Tian Li
2. Virginia Smith
3. Liam H. Fowl
4. Micah Goldblum
5. Ameet Talwalkar
6. Honglak Lee
7. Tom Goldstein

Citations in the corpus

• Federated Optimization in Heterogeneous Networks (2018, background): “…interesting connection between meta-learning methods and federated optimization methods (Khodak et al., 2019), and similar proximal terms have recently been investigated in the context of meta-learning for improved performance on few-shot learning tasks (Goldblum et al., 2020; Zhou et al., 2019).”
• Orthogonal Projection Loss (2021, unknown): “We explore the effects of OPL in few-shot learning settings, and visualize the novel class embeddings learned with OPL in Appendix B.1 using LDA [36] to preserve the inter-class to intra-class variance ratio as suggested in [14].”
• Adversarially Robust Few-Shot Learning: A Meta-Learning Approach (2019, background): “One simple approach is transfer learning , in which pre-trained feature extractors are trained on large data sets and fine-tuned on new tasks [2, 11].”
• Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “[25] find that regularization that increases class separation improves few-shot classification performance on mini-ImageNet and CIFAR-FS, whereas Liu et al.”