How to train your MAML
By Antreas Antoniou, Harrison Edwards, Amos J. Storkey. International Conference on Learning Representations, 2018
887 citations recorded by the source corpus.
The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
View Paper: https://arxiv.org/abs/1810.09502
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Bernt Schiele
- Han-Jia Ye
- Jianfeng Gao
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
In Meta & Continual Learning
Citations in the corpus
- CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “to hyper-parameters and even random seeds (Antoniou et al., 2019), which we do not tune exhaustively for practical reasons.”
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, neutral): “Similarly to our work, Antoniou et al. (2019) have also proposed improvements to MAML.”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “The core idea has inspired a number of extensions (Antoniou et al., 2019; Nichol et al., 2018; Rusu et al., 2019; Grant et al., 2018; Rajeswaran et al., 2019; Raghu et al., 2020), which broadly use a two-level optimization structure to perform meta-learning.”
- Personalized Federated Learning: A Meta-Learning Approach (2020, background): “…version of this formulation was first proposed by Finn et al. (2017) and followed by a number of papers studying its empirical characteristics (Antoniou et al., 2019; Li et al., 2017; Grant et al., 2018; Nichol et al., 2018; Zintgraf et al., 2019; Behl et al., 2019) as well as its…”
- Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, methodology): “algorithms (Finn et al., 2017; Antoniou et al., 2018), or other non-MAML/FL methods (Zhang et al.”
- Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Various approaches have been proposed [56,54,21,22,24,50,5,40,4,39,34,66,69,65].”
- Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “MAML is widely applied into vast learning scenarios, e.g., robotics con-trol [49], neural translation [50] and uncertainty estimation [51], and many improved algorithms [52], [53], [54], [55], [56] have been made to further boost its performance.”
- Forward Compatible Few-Shot Class-Incremental Learning (2022, methodology): “Optimizationbased methods try to enable fast model adaptation with few-shot data [3, 4, 15, 30].”
- Continual Learning via Local Module Composition (2021, background): “[7] Antreas Antoniou, Harrison Edwards, and Amos Storkey.”
- Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, background): “Related Work: Although the MAML algorithm has seen significant recent developments [1, 6, 9, 23], there is little work investigating why the double-optimization loop structure of the algorithm is empirically highly effective.”
- Online Meta-Learning (2019, positive): “, in Ut), which is consistent with prior works (Finn et al., 2017; Grant et al., 2018; Antoniou et al., 2018; Shaban et al., 2018).”
- Defining Benchmarks for Continual Few-Shot Learning (2020, methodology): “SCA and High-End MAML++ (which utilize both embeddings and gradient-based optimization) produce the best performance across all settings.”
- Meta-Learning in Neural Networks: A Survey (2020, methodology): “The trainable component v can span simple hyper-parameters such as a fixed step size [77], [78] to more sophisticated pre-conditioning matrices [104], [105].”
- Learning to Self-Train for Semi-Supervised Few-Shot Classification (2019, background): “They can be roughly divided into four categories: (1) data augmentation based methods [15, 29, 37, 38] generate data or features in a conditional way for few-shot classes; (2) metric learning methods [36, 31, 33] learn a similarity space of image features in which the classification should be effici”
- Meta-Curvature (2019, background): “comparison to (Antoniou et al., 2018), we also reported the results of model ensemble.”
- Meta-Transfer Learning Through Hard Tasks (2019, methodology): “We report the results of TADAM [36] and MetaOptNet [46] using the numbers given in original papers, and obtain the results of classical methods – MAML [7], MAML++ [24], RelationNets [32] and MatchineNets [26] by implementing their open-sourced code with our deep networks.”
- Meta-learning via Language Model In-context Tuning (2021, background): “Even worse, few-shot task adaptation is known to be highly sensitive to optimization hyperparameters (Antoniou et al., 2019), while a large labeled validation set for hyperparameter tuning may not be available under a FSL setting (Perez et al.”
- An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “4 MAML++ [1] 4CONV 52.”