Deep Meta-Learning: Learning to Learn in the Concept Space

By Fengwei Zhou, Bin Wu, Zhenguo Li. 2018

134 citations recorded by the source corpus.

Few-shot learning remains challenging for meta-learning that learns a learning algorithm (meta-learner) from many related tasks. In this work, we argue that this is due to the lack of a good representation for meta-learning, and propose deep meta-learning to integrate the representation power of deep learning into meta-learning. The framework is composed of three modules, a concept generator, a meta-learner, and a concept discriminator, which are learned jointly. The concept generator, e.g. a deep residual net, extracts a representation for each instance that captures its high-level concept, on which the meta-learner performs few-shot learning, and the concept discriminator recognizes the concepts. By learning to learn in the concept space rather than in the complicated instance space, deep meta-learning can substantially improve vanilla meta-learning, which is demonstrated on various few-shot image recognition problems. For example, on 5-way-1-shot image recognition on CIFAR-100 and CUB-200, it improves Matching Nets from 50.53% and 56.53% to 58.18% and 63.47%, improves MAML from 49.28% and 50.45% to 56.65% and 64.63%, and improves Meta-SGD from 53.83% and 53.34% to 61.62% and 66.95%, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Meta-Learning with Latent Embedding Optimization (2018, methodology): “Recent state-of-the-art approaches use the penultimate layer representation (Zhou et al., 2018; Qiao et al., 2017; Bauer et al., 2017; Gidaris & Komodakis, 2018); however, we choose the intermediate feature representation in layer 21 , given that higher layers tend to specialize to the training…”
Fast Parametric Learning with Activation Memorization (2018, methodology): “Meta-learning of neural networks has seen promising results for applications such as parameter optimization (Andrychowicz et al., 2016; Finn et al., 2017) and classification (Santoro et al., 2016; Vinyals et al., 2016; Zhou et al., 2018).”
Meta-Learning with Warped Gradient Descent (2019, background): “Alternatively, works in gradient-based few-shot learning extend MAML by meta-learning latent space for concepts or task inference, which implicitly modulates task gradients (Zhou et al., 2018; Rusu et al., 2019).”
Delta-encoder: an effective sample synthesis method for few-shot object recognition (2018, methodology): “In approaches such as MAML [10], Meta-SGD [26], DEML+Meta-SGD [52], Meta-Learn LSTM [34] and Meta-Networks [31], the meta-learned classiﬁers are optimized to be easily ﬁne-tuned on new few-shot tasks using small training data.”
Baby steps towards few-shot learning with multiple semantics (2019, unknown): “their mean as the category embedding. 4 Experimental results miniImageNet CUB Method 1-shot 5-shot 1-shot 5-shot Human performance 4.5 years old 70.0 – – – Adult 99.0 – – – No semantics DEML+Meta-SGD [42] 58:5 71:3 66:9 77:1 CAML [16] 59:2 72:4 – – Dist. ensemble [8] 59:3 76:9 68:7 83:5 [-encoder 29”
Fast Context Adaptation via Meta-Learning (2018, unknown): “Other meta-learning methods are also motivated by the practical difficulties of learning in high-dimensional parameter spaces, and the relative ease of fast adaptation in lower dimensional space (e.g., Sæmundsson et al., 2018; Zhou et al., 2018).”
RepMet: Representative-Based Metric Learning for Classification and Few-Shot Object Detection (2018, unknown): “Recently, even more compelling results were obtained on the standard few-shot classification benchmarks using meta-learning methods [9, 19, 24, 44] and synthesis methods [6, 10, 29, 40, 44].”
CAML: Fast Context Adaptation via Meta-Learning (2018, background): “Other meta-learning methods are also motivated by the fact that learning in a high-dimensional parameter space can pose practical difﬁculties, and fast adaptation in lower dimensional space can be better (e.g., Sæmundsson et al. (2018); Zhou et al. (2018); Rusu et al. (2018)).”
Federated Meta-Learning for Recommendation (2018, background): “…for regression (Finn et al., 2017; Li et al., 2017), image classification (Vinyals et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017; Li et al., 2017; Snell et al., 2017; Zhou et al., 2018) and reinforcement learning (Duan et al., 2016; Wang et al., 2016; Finn et al., 2017; Li et al., 2017).”
RepMet: Representative-based metric learning for classification and one-shot object detection (2018, background): “This has been demonstrated for domain-specific tasks such as face recognition [27], as well as for the classification of general categories [5, 10, 30, 35, 37, 40].”
MetaReg: Towards Domain Generalization using Meta-Regularization (2018, methodology): “A similar approach for training meta-learning algorithms in feature space has been explored in a recent work [34].”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, unknown): “, [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], target at effectively adapting model parameters to new tasks in the low-shot regime.”
Transductive Episodic-Wise Adaptive Metric for Few-Shot Learning (2019, methodology): “We further split the whole dataset into 64 classes as seen categories for training, 16 and 20 classes for validation and testing respectively [40].”
Meta-Learning with Implicit Gradients (2019, unknown): “…as input and produce weight updates [25, 5, 33, 48] or predictions for new inputs [50, 12, 58, 40, 38], and optimization-based approaches that use bi-level optimization to embed learning procedures, such as gradient descent, into the meta-optimization problem [15, 13, 8, 60, 34, 17, 59, 23].”