Meta-SGD: Learning to Learn Quickly for Few Shot Learning
By Zhenguo Li, Fengwei Zhou, Fei Chen, Hang Li. 2017
1,259 citations recorded by the source corpus.
Few-shot learning is challenging for learning algorithms that learn each task in isolation and from scratch. In contrast, meta-learning learns from many related tasks a meta-learner that can learn a new task more accurately and faster with fewer examples, where the choice of meta-learners is crucial. In this paper, we develop Meta-SGD, an SGD-like, easily trainable meta-learner that can initialize and adapt any differentiable learner in just one step, on both supervised learning and reinforcement learning. Compared to the popular meta-learner LSTM, Meta-SGD is conceptually simpler, easier to implement, and can be learned more efficiently. Compared to the latest meta-learner MAML, Meta-SGD has a much higher capacity by learning to learn not just the learner initialization, but also the learner update direction and learning rate, all in a single meta-learning process. Meta-SGD shows highly competitive performance for few-shot learning on regression, classification, and reinforcement learning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Chelsea Finn
- Mohit Bansal
- Trevor Darrell
- Quoc V. Le
- Dacheng Tao
- Bernt Schiele
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Bernt Schiele
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Jitendra Malik
In Meta & Continual Learning
Citations in the corpus
- Fast Model Editing at Scale (2021, methodology): “Various methods for meta-learning also use gradient transforms to achieve better model updates for few-shot learning (Ravi and Larochelle, 2017; Li et al., 2017; Lee and Choi, 2018; Park and Oliva, 2019; Flennerhag et al., 2020).”
- How to train your MAML (2018, methodology): “Meta-SGD showcases significantly improved generalization performance (when compared to MAML) across all few-shot learning tasks, whilst only requiring a single inner loop update step.”
- Meta-Learning with Latent Embedding Optimization (2018, methodology): “The learning rates for these spaces were meta-learned in a similar fashion to Meta-SGD (Li et al., 2017), after being initialized to 1 and v Function g φ r from Eq.”
- Personalized Federated Learning: A Meta-Learning Approach (2020, background): “…formulation was first proposed by Finn et al. (2017) and followed by a number of papers studying its empirical characteristics (Antoniou et al., 2019; Li et al., 2017; Grant et al., 2018; Nichol et al., 2018; Zintgraf et al., 2019; Behl et al., 2019) as well as its convergence properties (Fallah et…”
- Task Agnostic Meta-Learning for Few-Shot Learning (2018, background): “Meta-SGD [13] extends the MAML, which not only learns weight initialization but also the learner’s update step size.”
- Meta-learning autoencoders for few-shot prediction (2018, background): “This brings the key innovation of MeLA: for a class of tasks, MeLA does not attempt to learn a single good initialization for multiple tasks as in [7], or learn an update function [19, 2, 1], or learn an update function together with a single good initialization [13, 17].”
- Deep Meta-Learning: Learning to Learn in the Concept Space (2018, methodology): “…few-shot learning problems, ranging from classification (Santoro et al., 2016; Vinyals et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017; Li et al., 2017; Snell et al., 2017), reinforcement learning (Wang et al., 2016; Duan et al., 2016; Nikhil Mishra, 2018; Kevin Frans, 2018),…”
- Meta-Learning Representations for Continual Learning (2019, methodology): “Some meta-learning algorithms have similarly considered optimizations through multiple steps of updating for the few-shot learning setting (Finn et al., 2017; Li et al., 2017; Al-Shedivat et al., 2018; Nagabandi et al., 2019) for learning model initializations.”
- Meta-Transfer Learning for Few-Shot Learning (2018, background): ““Meta” means that the parameters of these operations can be viewed as hyper-parameters trained on few-shot learning tasks [30, 25].”
- Federated Meta-Learning with Fast Convergence and Efficient Communication (2018, methodology): “We first discuss the meta-learning approach and present Model-Agnostic Meta-Learning (MAML) [5] and Meta-SGD [12] algorithms as running examples.”
- Defining Benchmarks for Continual Few-Shot Learning (2020, methodology): “, 2017), Meta-SGD (Li et al., 2017) and MAML++ (Antoniou et al.”
- Meta-Learning in Neural Networks: A Survey (2020, background): “The trainable component v can span simple hyper-parameters such as a fixed step size [77], [78] to more sophisticated pre-conditioning matrices [104], [105].”
- La-MAML: Look-ahead Meta Learning for Continual Learning (2020, background): “Learning LRs in meta-learning: Meta-SGD [16] proposed learning the LRs in MAML for few-shot learning.”
- Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, background): “…few-shot tasks, e.g., the model initialization to a pre-specified model configuration could be adapted rapidly usingfixed steps of gradient descents over the few-shot training data from unseen categories (Finn et al. 2017; Li et al. 2017; Nichol et al. 2018; Lee et al. 2018; Antoniou et al. 2019).”
- Meta-Curvature (2019, background): “Meta-SGD (Li et al., 2017) suggests to learn coordinatewise learning rates.”