Meta-Learning with Adaptive Layerwise Metric and Subspace

By Yoonho Lee, Seungjin Choi. 2018

27 citations recorded by the source corpus.

Recent advances in meta-learning demonstrate that deep representations combined with the gradient descent method have sufficient capacity to approximate any learning algorithm. A promising approach is the model-agnostic meta-learning (MAML) which embeds gradient descent into the meta-learner. It optimizes for the initial parameters of the learner to warm-start the gradient descent updates, such that new tasks can be solved using a small number of examples. In this paper we elaborate the gradient-based meta-learning, developing two new schemes. First, we present a feedforward neural network, referred to as T-net, where the linear transformation between two adjacent layers is decomposed as T W such that W is learned by task-specific learners and the transformation T, which is shared across tasks, is meta-learned to speed up the convergence of gradient updates for task-specific learners. Second, we present MT-net where gradient updates in the T-net are guided by a binary mask M that is meta-learned, restricting the updates to be performed in a subspace. Empirical results demonstrate that our method is less sensitive to the choice of initial learning rates than existing meta-learning methods, and achieves the state-of-the-art or comparable performance on few-shot classification and regression tasks.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

Citations in the corpus

Incremental Object Detection via Meta-Learning (2020, positive): “A scalable way to achieve such a gradient preconditioning is to embed it directly into the task learner [33], [36], [37].”
Meta-Learning with Warped Gradient Descent (2019, methodology): “T-Nets [23] precondition feed-forward networks, h(x, θ) = Wx, by inserting non-learnable projections T .”
Transferring Knowledge across Learning Processes (2018, unknown): “The approach most closely related to us learns an initialization such that the model can adapt to a new task through one or a few gradient updates (Finn et al., 2017; Nichol et al., 2018; Al-Shedivat et al., 2017; Lee & Choi, 2018).”
Learning to Propagate Labels: Transductive Propagation Network for Few-Shot Learning (2018, background): “This significant gap between human and deep learning has reawakened the research interest on few-shot learning (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Ravi and Larochelle, 2017; Lee and Choi, 2018; Xu et al., 2017; Wang et al., 2018).”