Meta-Curvature

By Eunbyung Park, Junier B. Oliva. 2019

142 citations recorded by the source corpus.

We propose meta-curvature (MC), a framework to learn curvature information for better generalization and fast model adaptation. MC expands on the model-agnostic meta-learner (MAML) by learning to transform the gradients in the inner optimization such that the transformed gradients achieve better generalization performance to a new task. For training large scale neural networks, we decompose the curvature matrix into smaller matrices in a novel scheme where we capture the dependencies of the model's parameters with a series of tensor products. We demonstrate the effects of our proposed method on several few-shot learning tasks and datasets. Without any task specific techniques and architectures, the proposed method achieves substantial improvement upon previous MAML variants and outperforms the recent state-of-the-art methods. Furthermore, we observe faster convergence rates of the meta-training process. Finally, we present an analysis that explains better generalization performance with the meta-trained curvature.

Subfields: Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

Incremental Object Detection via Meta-Learning (2020, positive): “This allows for stronger representational capability than previous works that consider preconditioning via block diagonal matrices [38], [39].”
Meta-Learning in Neural Networks: A Survey (2020, background): “The trainable component v can span simple hyper-parameters such as a fixed step size [77], [78] to more sophisticated pre-conditioning matrices [104], [105].”
Meta-Learning with Warped Gradient Descent (2019, methodology): “Meta-Curvature (Park & Oliva, 2019) allows for a block-diagonal P at the cost of added complexity.”
Meta-Neighborhoods (2019, methodology): “To further improve flexibility, Meta-SGD (Li et al. 2017) learns coordinate-wise inner learning rates, and curvature information is considered in (Park and Oliva 2019) to transform the gradients in the inner optimization.”
Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning (2022, background): “Recent studies revealed that the choice of the lower-level optimizer affects the generalization performance of MAML (Grant et al., 2018a; Antoniou et al., 2019; Park & Oliva, 2019).”
Transformers learn in-context by gradient descent (2022, methodology): “This kind of meta-learning to improve upon plain gradient descent has been leveraged in numerous previous approaches for deep neural networks (Li et al., 2017; Lee & Choi, 2018; Park & Oliva, 2019; Zhao et al., 2020; Flennerhag et al., 2020).”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, methodology): “Those methods, including previous work (Li et al., 2017; Park and Oliva, 2019; Lee and Choi, 2018) and the proposed meta-kfo algorithm (§5.3), improve meta-learnability of shallower networks that otherwise do not meta-learn well.”
On Modulating the Gradient for Meta-learning (2020, methodology): “During training, the meta-parameters of both methods [22, 27] change in each meta-update, however, they remain unchanged when the algorithm is applied to unseen tasks (see Fig.”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, unknown): “, [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], target at effectively adapting model parameters to new tasks in the low-shot regime.”
Bootstrapped Meta-Learning (2021, background): “The meta-objective is the validation loss in expectation over the task distribution: Several works have extended this setup by altering the update rule ( ϕ ) (Lee & Choi, 2018; Zintgraf et al., 2019; Park & Oliva, 2019; Flennerhag et al., 2020).”
Learning where to learn: Gradient sparsity in meta and continual learning (2021, background): “It is updated similarly to θ. Meta-SGD (Li et al., 2017) considers a diagonal M , i.e. learnable learning rates, and MetaCurvature (Park & Oliva, 2019) considers a block matrix.”