Meta-Learning in Neural Networks: A Survey
By Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, Amos J. Storkey. 2020
2,723 citations recorded by the source corpus.
The field of meta-learning, or learning-to-learn, has seen a dramatic rise in interest in recent years. Contrary to conventional approaches to AI where tasks are solved from scratch using a fixed learning algorithm, meta-learning aims to improve the learning algorithm itself, given the experience of multiple learning episodes. This paradigm provides an opportunity to tackle many conventional challenges of deep learning, including data and computation bottlenecks, as well as generalization. This survey describes the contemporary meta-learning landscape. We first discuss definitions of meta-learning and position it with respect to related fields, such as transfer learning and hyperparameter optimization. We then propose a new taxonomy that provides a more comprehensive breakdown of the space of meta-learning methods today. We survey promising applications and successes of meta-learning such as few-shot learning and reinforcement learning. Finally, we discuss outstanding challenges and promising areas for future research.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Dacheng Tao
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrea Vedaldi
- Yi Yang
In Meta & Continual Learning
- Zheda Mai
- Massimo Caccia
- Scott Sanner
- Hyunwoo J. Kim
- Xialei Liu
- Hakan Bilen
- Timothée Lesort
- Irina Rish
Citations in the corpus
- First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, background): “Nevertheless, when training data availability is scarce relative to the size of the model then full-body adaptation ceases to be suitable, and few-shot learning techniques [33], such as meta-learning [10] and transfer learning [40] become favorable.”
- Exploring Example Influence in Continual Learning (2022, positive): “Fortunately, the second-order derivation in IF is not necessary under the popular meta learning paradigm such as [18], instead we can easily get the derivation like IF through a one-step pseudo update.”
- Learngene: From Open-World to Your Learning Task (2021, background): “There are currently some learning paradigms that can solve some of the above problems, such as continual learning (Hadsell et al. 2020), transfer learning (Perkins, Sa-lomon et al. 1992) and meta-learning (Hospedales et al. 2020).”
- Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Meta-learning is an emerging learning paradigm where a neural network evolves from multiple related learning episodes and generalizes the learned knowledge to unseen tasks [122].”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “See (Vilalta and Drissi, 2002; Hospedales et al., 2020) for an in-depth survey of meta-learning.”
- Continual Lifelong Learning in Natural Language Processing: A Survey (2020, neutral): “, 2018) (Obamuyide and Vlachos, 2019) (Hospedales et al., 2020) (Beaulieu et al.”
- Reordering Examples Helps during Priming-based Few-Shot Learning (2021, background): “Note that the scope of this work is distinct from metalearning approaches (Hospedales et al., 2020), where the goal is improve the learning algorithm using several learning episodes.”
- Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (2020, methodology): “In a meta-learning framework (Hospedales et al., 2020) this corresponds to a Bayesian treatment for the inner loop cycle.”
- A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, background): “See Vilalta & Drissi (2002) and Hospedales et al. (2020) for in-depth surveys of meta-learning.”
- Membership Inference Attacks on Machine Learning: A Survey (2021, background): “Meta-learning, also known as learning to learn, refers to the process of improving a learning algorithm over multiple learning episodes [71].”
- Fine-Grained Image Analysis With Deep Learning: A Survey (2021, background): “auxiliary categories with well-labeled training data to the test categories, which usually employs zero-shot learning [63] or meta learning [163].”
- Reward is enough (2021, background): “Generalisation in meta-learning [61,20] has recently focused upon the problem of transferring an agent from one environment to another environment.”
- Subgraph Federated Learning with Missing Neighbor Generation (2021, background): “The ecently proposed meta-learning framework [9, 23, 14] that exploits information from different data sources to obtain a general model attracts FL researchers [8].”
- Memory Efficient Meta-Learning with Large Images (2021, background): “In comparison, meta-learning approaches [23] generally require orders of magnitude fewer MACs and steps to learn a new task at test time.”
- A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, background): “The challenge in meta-learning is to learn from experience in a systematic, data-driven manner (Hospedales et al., 2021).”