TADAM: Task dependent adaptive metric for improved few-shot learning

By Boris N. Oreshkin, Pau Rodr'iguez L'opez, Alexandre Lacoste. 2018

1,482 citations recorded by the source corpus.

Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100. Our code is publicly available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “Following previous works [29, 34, 26, 41, 9], we use a ResNet12 as our backbone: the network consists of 4 residual blocks, where each has 3 convolutional layers with 3×3 kernel; a 2×2 max-pooling layer is applied after each of the first 3 blocks; and a global average-pooling layer is on top of the ”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, background): “The experimental protocol of barely supervised learning (BSL) shares similarities to those of few-shot learning (FSL) [37] as they both assume a limited availability (e.”
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes (2019, methodology): “Moreover, TADAM was applied to simple few-shot learning benchmarks (e.g. CIFAR100 and mini-ImageNet) and sees little gain from feature extractor adaptation.”
Meta-Learning Probabilistic Inference for Prediction (2018, background): “Approaches that employ pre-training and/or residual networks (Bauer et al., 2017; Qiao et al., 2017; Rusu et al., 2018; Gidaris and Komodakis, 2018; Oreshkin et al., 2018; Garcia and Bruna, 2017; Lacoste et al., 2018) have been excluded so that the quality of the learning algorithm can be assessed s”
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, methodology): “Others relate to Prototypical Networks by learning a representation on which differentiable training can be performed on some form of classifier (Bertinetto et al., 2019; Gidaris & Komodakis, 2018; Oreshkin et al., 2018).”
Meta-Learning with Latent Embedding Optimization (2018, methodology): “69% TADAM (Oreshkin et al., 2018) 58 .”
Meta-Learning With Differentiable Convex Optimization (2019, methodology): “Prior work in few-shot learning [20, 3, 10] suggest that adjusting the prediction score by a learnable scale parameter γ provides better performance under nearest class mean and ridge regression base learners.”
Diversity With Cooperation: Ensemble Methods for Few-Shot Classification (2019, background): “Speciﬁcally, better results were obtained by training the network on the classical classiﬁ-cation task using the meta-training data in a ﬁrst step, and then only ﬁne-tuning with meta-learning in a second step [22, 28, 34].”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, neutral): “1, 2 [7] Boris Oreshkin, Pau Rodrı́guez López, and Alexandre Lacoste.”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, methodology): “Implementation Details: We use a ResNet-12 [12] following [23] as our feature extractor φw.”
Meta-Transfer Learning for Few-Shot Learning (2018, methodology): “study and a comparison to state-of-the-art methods. 5.1. Datasets and implementation details We conduct few-shot learning experiments on two benchmarks, miniImageNet [53] and Fewshot-CIFAR100 (FC100) [34]. miniImageNet is widely used in related works [9,39,13,11,32]. FC100 is newly proposed in [34] ”
Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning (2020, background): “Similar to the few-shot learning literature [78, 58, 52, 62], the transitions of the context variables in OSAKA are not structured, i.”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, background): “Outside of RL, meta-learning methods for few-shot supervised learning problems have explored a wide variety of approaches and architectures (Santoro et al., 2016; Vinyals et al., 2016; Ravi & Larochelle, 2017; Oreshkin et al., 2018).”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “Vanilla methods in this family have been further enhanced by making the embedding task-conditional [99], [115], learning a more elaborate comparison metric [89], [90], or combiningwith gradient-basedmeta-learning to train other hyper-parameters such as stochastic regularizers [116].”
A Baseline for Few-Shot Image Classification (2019, methodology): “This indicates that CIFAR-FS, FC-100 and Mini-ImageNet may be good benchmarks for applications with few classes.”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “g objective that transfer the task-level inductive bias from SEEN classes to the new task composed by UNSEEN classes. For example, one line of works use meta-learned discriminative feature embeddings [16, 20, 25, 27, 29, 34, 39] together with non-parametric nearest neighbor classiﬁers, to 2 recogniz”
Learning to Self-Train for Semi-Supervised Few-Shot Classification (2019, background): “They can be roughly divided into four categories: (1) data augmentation based methods [15, 29, 37, 38] generate data or features in a conditional way for few-shot classes; (2) metric learning methods [36, 31, 33] learn a similarity space of image features in which the classification should be effici”
Meta-Curvature (2019, methodology): “The state-of-the-art results are achieved by additionally training 64-way classification task for pretraining (Qiao et al., 2018; Rusu et al., 2019; Oreshkin et al., 2018) with larger ResNet models (Qiao et al.”