Meta-Learning with Latent Embedding Optimization
By Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell. International Conference on Learning Representations, 2018
1,522 citations recorded by the source corpus.
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
View Paper: https://arxiv.org/abs/1807.05960
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Trevor Darrell
- Mohit Bansal
- Dacheng Tao
- Bernt Schiele
- Quoc V. Le
- Chelsea Finn
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Bernt Schiele
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
In Meta & Continual Learning
- Han-Jia Ye
- De-chuan Zhan
- Chelsea Finn
- Bernt Schiele
- Sergey Levine
- Wei-Lun Chao
- Qianru Sun
- Yaoyao Liu
Citations in the corpus
- Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “LEO [44] proposed that it is beneficial to decouple the optimization-based meta-learning algorithms from highdimensional model parameters.”
- Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes (2019, methodology): “LEO [23], Proto-MAML [6] CNAPS, TADAM [27]”
- Meta-Learning Probabilistic Inference for Prediction (2018, neutral): “Approaches that employ pre-training and/or residual networks (Bauer et al., 2017; Qiao et al., 2017; Rusu et al., 2018; Gidaris and Komodakis, 2018; Oreshkin et al., 2018; Garcia and Bruna, 2017; Lacoste et al., 2018) have been excluded so that the quality of the learning algorithm can be assessed…”
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, background): “Some bear similarity with MAML and correspond to a meta-learner trained to quickly adapt its parameters to various novel tasks (Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Rusu et al., 2019; Yoon et al., 2018).”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “The core idea has inspired a number of extensions (Antoniou et al., 2019; Nichol et al., 2018; Rusu et al., 2019; Grant et al., 2018; Rajeswaran et al., 2019; Raghu et al., 2020), which broadly use a two-level optimization structure to perform meta-learning.”
- Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “74:65% accuracy on the 5-shot 5-way task. This implies that our approach not only drastically improved performance of other meta-learners but also can compete with state-of-the-art techniques such as [23] for few shot classification tasks. The best cell architectures found in these tasks were shown i”
- Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “There are three major swim lanes of the FSL problem: (a) recurrent-based [40, 48] (b) optimisation-based [43, 59], and (c) metric-based frameworks [17, 24].”
- Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, background): “LEO [43] has its classifier weights generated by a low-dimensional latent embedding updated in the inner-loop.”
- Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning (2021, methodology): “Meta-based methods [7, 22, 37, 10] aims to learn a learning strategy to adjust well to new samples.”
- Incremental Object Detection via Meta-Learning (2020, neutral): “Meta-learning algorithms can be broadly classified into optimization based methods [10], [11], [26], black-box adaptation methods [27], [28] and non-parametric methods [12], [29].”
- Incremental Meta-Learning via Indirect Discriminant Alignment (2020, neutral): “Meta-learning [36, 22, 31] for few-shot learning [24, 6, 30, 38, 35, 19, 25, 10] uses a relatively “meta training” dataset from which several few-shot tasks are sampled to mimic phenomena at the tail.”
- RLBench: The Robot Learning Benchmark & Learning Environment (2019, methodology): “gradient based methods [48], [49] have not been tested on such”
- Task Agnostic Continual Learning via Meta Learning (2019, positive): “[29] proposed Latent Embedding Optimization (LEO) which combines the encoder/decoder structure with the idea of inner loop fine-tuning from MAML.”
- Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “This algorithm, with deep neural networks as the underlying model, has been highly influential, with significant follow on work [19, 6, 23, 11, 5, 9, 28, 31, 12].”