Meta-learning with differentiable closed-form solvers

By Luca Bertinetto, João F. Henriques, Philip H. S. Torr, Andrea Vedaldi. 2018

1,076 citations recorded by the source corpus.

Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “We train 100 epochs for miniImageNet, 60 epochs for tieredImageNet, and 90 epochs for both CIFAR-FS and FC100.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Few-shot learning aims to reduce these demands by training models to recognize completely novel objects from only a few examples [9, 49, 40, 2, 38, 11, 46].”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “The meta-learner either learns to produce new parameters directly from the new data [9, 33, 56, 62, 64, 72, 73], or learns to produce an update rule to iteratively optimize the base learner to fit the new data [2, 6, 8, 38, 63, 97].”
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, background): “Others relate to Prototypical Networks by learning a data representation as well as a compact representation for a classiﬁer of data under that representation (Bertinetto et al., 2019; Gidaris & Komodakis, 2018; Oreshkin et al., 2018; Gidaris & Komodakis, 2018).”
A Closer Look at Few-shot Classification (2019, methodology): “(2018), ridge regression Bertinetto et al. (2019), and graph neural network Garcia & Bruna (2018). In this paper, we compare the performance of three distance metric learning methods. Our results show that a simple baseline method with a distancebased classifier (without training over a collection o”
Meta-Learning With Differentiable Convex Optimization (2019, methodology): “Prior work in few-shot learning [20, 3, 10] suggest that adjusting the prediction score by a learnable scale parameter γ provides better performance under nearest class mean and ridge regression base learners.”
Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (2020, methodology): “, 2019) and R2-D2 (Bertinetto et al., 2018), we find that meta-learning tends to cluster object classes more tightly in feature space.”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “To investigate the sensitivity of baselines and our method to dataset order, we also performed comparisons on two other dataset sequences, including (i) Butterfly, CUB, CIFARFS, Plantae, MiniIm-agenet, Aircraft and (ii) CUB, CIFARFS, Plantae, MiniImagenet, Butterfly, Aircraft.”
Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, methodology): “CIFAR-FS [6] is created by dividing CIFAR-100 into 64 training, 16 validation and 20 testing classes; images are of size 32 × 32.”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, methodology): “Compared to [4], our proposed method using recurrent back-propagation [18, 1, 25] is more general as it does not require a closed-form update, and the inner loop solver can employ any existing continuous optimizers.”
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “Other optimization-based algorithms have also since been developed, for example [18, 10, 4, 17, 38], which learn functions to embed the support set and test examples of a few-shot learning task, and then learn the weights of a task-specific classifier (using the support set) to perform few-shot clas”
Few-Shot Learning via Learning the Representation, Provably (2020, methodology): “In fact, the scheme we analyze in this paper is closely related to Lee et al. (2019), Bertinetto et al. (2018).”
A Baseline for Few-Shot Image Classification (2019, background): “This indicates that CIFAR-FS, FC-100 and Mini-ImageNet may be good benchmarks for applications with few classes.”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “Following the setting of most recent methods [45, 52, 70], we use ResNet variants [5, 22] to implement the embedding backbone φ.”
Boosting Few-Shot Visual Learning With Self-Supervision (2019, methodology): “We perform this study using the MiniImageNet and CIFAR-FS datasets and report results in Tables 1 and 2 respectively.”
Generalizing from a Few Examples (2019, neutral): “Improved upon Learnet, the classification layer of the learner is replaced by ridge regression in [13], such that parameters can be efficiently obtained in closed-form.”
Meta-Transfer Learning Through Hard Tasks (2019, background): “Examples of distance metrics include cosine similarity [26], [33], Euclidean distance to the prototypical representation of a class [31], CNN-based relation module [32], ridge regression based [38], and graph model based [39], [40].”
Transformers learn in-context by gradient descent (2022, methodology): “The way MLPs transform data in Transformers diverges from the standard meta-learning approach, where a task-shared input embedding network is optimized by backpropagation-through-training to improve the learning performance of a task-specific readout (e.g., Raghu et al., 2020; Lee et al., 2019; Bert”