Meta-Learning with Neural Tangent Kernels

By Yufan Zhou, Zhenyi Wang, Jiayi Xian, Changyou Chen, Jinhui Xu. 2021

21 citations recorded by the source corpus.

Model Agnostic Meta-Learning (MAML) has emerged as a standard framework for meta-learning, where a meta-model is learned with the ability of fast adapting to new tasks. However, as a double-looped optimization problem, MAML needs to differentiate through the whole inner-loop optimization path for every outer-loop training step, which may lead to both computational inefficiency and sub-optimal solutions. In this paper, we generalize MAML to allow meta-learning to be defined in function spaces, and propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK). Within this paradigm, we introduce two meta-learning algorithms in the RKHS, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory. Extensive experimental studies demonstrate advantages of our paradigm in both efficiency and quality of solutions compared to related meta-learning algorithms. Another interesting feature of our proposed methods is that they are demonstrated to be more robust to adversarial attacks and out-of-distribution adaptation than popular baselines, as demonstrated in our experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Li Shen
2. Mingchen Gao
3. Zhenyi Wang
4. Le Fang
5. Tiehang Duan
6. Donglin Zhan
7. Qiuling Suo

In Computer Vision and Pattern Recognition

1. Li Shen
2. Mingchen Gao
3. Zhenyi Wang
4. Le Fang
5. Tiehang Duan
6. Donglin Zhan
7. Qiuling Suo

Citations in the corpus

• Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Various approaches have been proposed [56,54,21,22,24,50,5,40,4,39,34,66,69,65].”
• Learning to Learn and Remember Super Long Multi-Domain Task Sequence (2022, background): “Most existing works of meta-learning [6, 19, 21, 29, 38, 65, 70, 75, 81] focus on stationary task distributions.”
• Meta-learning without data via Wasserstein distributionally-robust model fusion (2022, unknown): “…Finn et al., 2017, Snell et al., 2017, Ravi and Beatson, 2019, Rajeswaran et al., 2019, Finn et al., 2019b, Raghu et al., 2020, Wang et al., 2020b, Zhou et al., 2021, Bohdal et al., 2021, Rothfuss et al., 2021, CHEN et al., 2021, Bronskill et al., 2021, Sun et al., 2021] focus on the offline…”
• Meta Learning on a Sequence of Imbalanced Domains with Difficulty Awareness (2021, unknown): “Representative works include [57, 52, 20, 21, 23, 49, 7, 42, 6, 41, 37, 61, 66, 46, 53, 65], etc.”