Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

By Yonglong Tian, Yue Wang, Dilip Krishnan, Josh Tenenbaum, Phillip Isola. 2020

1,074 citations recorded by the source corpus.

The focus of recent meta-learning research has been on the development of learning algorithms that can quickly adapt to test time tasks with limited data and low computational cost. Few-shot learning is widely used as one of the standard benchmarks in meta-learning. In this work, we show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, followed by training a linear classifier on top of this representation, outperforms state-of-the-art few-shot learning methods. An additional boost can be achieved through the use of self-distillation. This demonstrates that using a good learned embedding model can be more effective than sophisticated meta-learning algorithms. We believe that our findings motivate a rethinking of few-shot image classification benchmarks and the associated role of meta-learning algorithms. Code is available at: this http URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Learning to Prompt for Vision-Language Models (2021, background): “CoOp also outperforms the linear probe model, which is known as a strong few-shot learning baseline (Tian et al., 2020).”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “In optimization-based approaches, the model takes many [51, 46, 5] or few [9, 52, 2] gradient steps on the context examples, and the updated model then classifies the target examples.”
Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Prominent works either resort to meta-learning [ Vinyals et al. , 2016; Snell et al. , 2017 ] or build upon a deep pre-trained feature extractor [ Tian et al. , 2020 ] .”
Constrained Few-shot Class-incremental Learning (2022, methodology): “Motivated by [49] to derive good visual representations in the embedding, we pretrain the feature extractor in the standard supervised classification on the base session by replacing the EM with an additional auxiliary fully connected layer of dimension d× 60.”
Subspace Regularizers for Few-Shot Class Incremental Learning (2021, methodology): “We use a ResNet-18 model that is identical to the one described in Tian et al. (2020).”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, background): “…when we apply smaller (30) number of epochs during the second phase and compare it to a longer second phase (150 epochs). shorten the second Due to the negligible differences we use the initially chosen value that equals 150 epochs throughout the main paper. phase 0.2 points higher accuracy”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, background): “This observation is also demonstrated in (Tian et al., 2020; Ericsson et al., 2021).”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, methodology): “Constructing a cache model from few-shot training data has been explored by many previous methods, including Matching Network [63], Prototypical Networks [55], MAML [16], Relation Network [58] and others [6,7,12,60].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “Instead, pre-training with a diverse set of tasks [8,66] has been observed to improve the performance on downstream tasks [26, 63] and potentially learn a better representation.”
The Platonic Representation Hypothesis (2024, neutral): “• 49 ResNet-18 models trained with Alignment and Uniformity contrastive loss (Wang & Isola, 2020) on ImageNet-100, Places-365, and 47 realistic and synthetic image datasets from Baradad et al. (2021).”
Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, methodology): “Born-Again Neural Networks [15] uses KD proposes to consecutively distill knowledge from an identical teacher network to a student network, which is further applied to few-shot learning in [51] and multi-task learning in [10].”
Intriguing Properties of Vision Transformers (2021, methodology): “We use the extracted features to learn a linear classiﬁer over the support set for each query (similar to [46]), and evaluate using the standard FSL protocol deﬁned in [45].”
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning (2020, methodology): “As a method, similar ideas to Classifier-Baseline are concurrently reported in recent works [29, 26].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “19% RFS-distill [95] ECCV’20 Embedding ResNet-12 ✓ ✓ ✓ ✓ 63.”
Curious Representation Learning for Embodied Intelligence (2021, neutral): “Such a technique has also been noted to be useful in few-shot learning [44].”