Meta-Learning With Differentiable Convex Optimization

By Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, Stefano Soatto. Computer Vision and Pattern Recognition, 2019

1,446 citations recorded by the source corpus.

Many meta-learning approaches for few-shot learning rely on simple base learners such as nearest-neighbor classifiers. However, even in the few-shot regime, discriminatively trained linear predictors can offer better generalization. We propose to use these predictors as base learners to learn representations for few-shot learning and show they offer better tradeoffs between feature size and performance across a range of few-shot recognition benchmarks. Our objective is to learn feature embeddings that generalize well under a linear classification rule for novel categories. To efficiently solve the objective, we exploit two properties of linear classifiers: implicit differentiation of the optimality conditions of the convex problem and the dual formulation of the optimization problem. This allows us to use high-dimensional embeddings with improved generalization at a modest increase in computational overhead. Our approach, named MetaOptNet, achieves state-of-the-art performance on miniImageNet, tieredImageNet, CIFAR-FS, and FC100 few-shot learning benchmarks.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Optimization & Training

View Paper: https://arxiv.org/abs/1904.03758

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “Following previous works [29, 34, 26, 41, 9], we use a ResNet12 as our backbone: the network consists of 4 residual blocks, where each has 3 convolutional layers with 3×3 kernel; a 2×2 max-pooling layer is applied after each of the first 3 blocks; and a global average-pooling layer is on top of the ”
Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (2020, background): “Such models include R2-D2 and MetaOptNet (Bertinetto et al., 2018; Lee et al., 2019).”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Various approaches have been proposed [56,54,21,22,24,50,5,40,4,39,34,66,69,65].”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, methodology): “The former methods [6, 10, 20, 36] generally define few-shot learning as "learning to finetune", which aims to learn proper model initialization or predict network parameters.”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “In [86,87] both loops use gradient descent , while inner-loops can also be nearest-neighbor matching [88,89], recurrent neural representations [8,90,91], probabilistic inference [82,92], or mixed approaches [93 – 96].”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, methodology): “Meta-learning [36, 22, 31] for few-shot learning [24, 6, 30, 38, 35, 19, 25, 10] uses a relatively “meta training” dataset from which several few-shot tasks are sampled to mimic phenomena at the tail.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Finally, another family of approaches accelerate meta-training via closed-form solvers in the inner loop [164], [165].”
Few-Shot Learning via Learning the Representation, Provably (2020, methodology): “In fact, the scheme we analyze in this paper is closely related to Lee et al. (2019), Bertinetto et al. (2018).”
A Baseline for Few-Shot Image Classification (2019, methodology): “For MAML [2] and MetaOpt SVM [3], we use the number of episodes in the author’s Github implementation.”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “For example, one line of works uses meta-learned discriminative feature embeddings [31, 44, 52, 55, 62, 70] together with the nonparametric nearest neighbor classifiers, to recognize novel classes given a few exemplars.”
Learning to Self-Train for Semi-Supervised Few-Shot Classification (2019, background): “They are based on either data augmentation [15, 29] or gradient descent [3, 23, 7, 5, 42, 19, 27, 32, 10].”
Meta-Curvature (2019, critical): “Our methods also outperformed the very latest state of the art results [20] that used extensive data-augmentation, regularization, and 15-shot meta-training schemes with different backbone networks.”
Meta-Transfer Learning Through Hard Tasks (2019, methodology): “It also surpasses both TADAM and MetaOptNet by about 5% .”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, background): “1 MetaOptNet [37] ResNet-12 62.”
Meta-Learning with Warped Gradient Descent (2019, background): “Empirical evidence from gradient-based few-shot learning suggests enforcing some form of strict parameter sharing improve performance by reducing overfitting [22, 31, 33], more robust convergence [48, 38], or superior final performance [35, 42, 21].”
Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions (2018, methodology): “A bit different from the ResNet-12 in [11], we apply a global average pooling after the final layer, which leads to 640 dimensional embeddings.”