Meta Networks
By Tsendsuren Munkhdalai, Hong Yu. 2017
1,161 citations recorded by the source corpus.
Neural networks have been successfully applied in applications with a large amount of labeled data. However, the task of rapid generalization on new concepts with small training data while preserving performances on previously learned ones still presents a significant challenge to neural network models. In this work, we introduce a novel meta learning method, Meta Networks (MetaNet), that learns a meta-level knowledge across tasks and shifts its inductive biases via fast parameterization for rapid generalization. When evaluated on Omniglot and Mini-ImageNet benchmarks, our MetaNet models achieve a near human-level performance and outperform the baseline approaches by up to 6% accuracy. We demonstrate several appealing properties of MetaNet relating to generalization and continual learning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Trevor Darrell
- Dacheng Tao
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Bernt Schiele
- Jitendra Malik
- Yi Yang
In Meta & Continual Learning
- Chelsea Finn
- Sergey Levine
- Han-Jia Ye
- Bernt Schiele
- Qianru Sun
- Yaoyao Liu
- De-chuan Zhan
- Wei-Lun Chao
Citations in the corpus
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, neutral): “Such networks have been applied to few-shot image recognition (Santoro et al., 2016; Munkhdalai & Yu, 2017) and learning “fast” reinforcement learning agents (Duan et al., 2016b; Wang et al., 2016).”
- CrossTransformers: spatially-aware few-shot transfer (2020, background): “In the limit, attention over long-term experiences accumulated in memories [43, 56, 69, 76] can augment more traditional learning.”
- Meta-learning with differentiable closed-form solvers (2018, methodology): “…forgetting typical of deep neural networks (McCloskey & Cohen, 1989), several recent methods (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017; Sprechmann et al., 2018) make use of memory-augmented models, which can first retain and then access important and previously…”
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, background): “Some bear similarity with MAML and correspond to a meta-learner trained to quickly adapt its parameters to various novel tasks (Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Rusu et al., 2019; Yoon et al., 2018).”
- How to train your MAML (2018, methodology): “The application of meta-learning in the few-shot learning setting has enabled the overwhelming majority of the current state of the art few-shot learning methods (Vinyals et al., 2016; Ravi & Larochelle, 2016; Edwards & Storkey, 2016; Finn et al., 2017; Li et al., 2017; Munkhdalai & Yu, 2017).”
- Task Agnostic Meta-Learning for Few-Shot Learning (2018, neutral): “[21] 98.4% 99.6% Prototypical Nets [19] 98.8% 99.7% Meta Nets [16] 98.9% - Snail [14] 99.07 ± 0.16% 99.78 ± 0.09% MAML [2] 98.”
- Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, methodology): “In our comparisons to recurrent meta-learners, we will use two state-of-the-art meta-learning models: TCML (Mishra et al., 2017) and meta-networks (Munkhdalai & Yu, 2017).”
- Dynamic Few-Shot Visual Learning Without Forgetting (2018, background): “Meta-learning approaches typical involve a meta-learner model that given a few training examples of a new task it tries to quickly learn a learner model that “solves” this new task [20, 24, 1, 13, 19].”
- Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “Baselines We compare against various state of the art baselines for few-shot recognition, including neural statistician [8], Matching Nets with and without fine-tuning [39], MANN [32], Siamese Nets with Memory [18], Convolutional Siamese Nets [20], MAML [10], Meta Nets [27], Prototypical Nets [36] a”
- Few-Shot Learning with Graph Neural Networks (2017, neutral): “1-shot 5-shot Matching Networks Vinyals et al. (2016) 43.6% 55.3% Prototypical Networks Snell et al. (2017) 46.61% ± 0.78% 65.77% ± 0.70% Model Agnostic Meta-learner Finn et al. (2017) 48.70% ± 1.84% 63.1% ± 0.92% Meta Networks Munkhdalai & Yu (2017) 49.21% ± 0.96 - Ravi & Larochelle Ravi & Larochel”
- Learning Rapid-Temporal Adaptations (2017, methodology): “Previous work on metalearning has formulated the problem as two-level learning; specifically, “slow” learning of a meta-level model that occurs across several tasks, and “fast” learning of a base-level model that acts within each task [24, 41, 29, 7, 26, 23].”
- Meta-Learning with Temporal Convolutions (2017, background): “70% Munkhdalai and Yu [18] 49.21% ± 0 .”
- Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Various approaches have been proposed [56,54,21,22,24,50,5,40,4,39,34,66,69,65].”
- Continual Few-Shot Learning for Text Classification (2021, background): “Optimization approaches aim to learn to optimize model parameters based on the gradients computed from limited labeled examples (Ravi and Larochelle, 2017; Munkhdalai and Yu, 2017; Finn et al., 2017).”
- Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “In the use of dynamic representations, our work is also related to single task and multi-task models that learn to generate weights for an optimal neural network (Ha et al., 2016), (Ravi & Larochelle, 2017), (Munkhdalai & Yu, 2017).”
- Memory-based Parameter Adaptation (2018, background): “Learning to remember rare events (Kaiser et al., 2017) augments an LSTM with a key-value memory structure, and meta networks (Munkhdalai & Yu, 2017) combines fast weights with regular weights.”