Provable Guarantees for Gradient-Based Meta-Learning

By Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar. 2019

163 citations recorded by the source corpus.

We study the problem of meta-learning through the lens of online convex optimization, developing a meta-algorithm bridging the gap between popular gradient-based meta-learning and classical regularization-based multi-task transfer methods. Our method is the first to simultaneously satisfy good sample efficiency guarantees in the convex setting, with generalization bounds that improve with task-similarity, while also being computationally scalable to modern deep learning architectures and the many-task setting. Despite its simplicity, the algorithm matches, up to a constant factor, a lower bound on the performance of any such parameter-transfer method under natural task similarity assumptions. We use experiments in both convex and deep learning settings to verify and demonstrate the applicability of our theory.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Differentially Private Meta-Learning (2019, methodology): “Our learning guarantees scale with task-similarity, as measured by the closeness of the task-speciﬁc optimal parameters (Denevi et al., 2019; Khodak et al., 2019b).”
Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “We see great potential for applying ARUBA to derive many other new LTL methods in a similar manner.”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, background): “Recent theories [1, 23] show that meta-learner achieves a generalization guarantee, attracting tremendous studies to solve low-shot problems by meta-learning techniques.”
On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms (2019, background): “In addition, for strongly convex functions, this assumption is closely related to the one in (Khodak et al., 2019) which states the optimal point of loss functions of all tasks are within a ball where its radius quantifies the similarity.”
Provable and Efficient Continual Representation Learning (2022, background): “There is also a long line of works related to model agnostic meta-learning (Finn et al. (2017); Denevi et al. (2019); Balcan et al. (2019); Khodak et al. (2019)).”
Bootstrapped Meta-Learning (2021, methodology): “In online optimisation, the MG update can achieve strong convergence guarantees if the problem is well-behaved (van Erven & Koolen, 2016), with similar guarantees in the multi-task setting (Balcan et al., 2019; Khodak et al., 2019; Denevi et al., 2019).”
Provable Lifelong Learning of Representations (2021, background): “There is a line of work where all tasks approximately [Finn et al., 2017, Khodak et al., 2019, Balcan et al., 2019, Denevi et al., 2019] or conditionally [Wang et al., 2020, Denevi et al., 2020, Denevi et al., 2021] share a common representation.”
Variable-Shot Adaptation for Online Meta-Learning (2020, methodology): “Unlike prior online meta-learning works (Finn et al., 2019; Grant et al., 2019; Khodak et al., 2019; Zhuang et al., 2019; Ren et al., 2020; Antoniou et al., 2020; Wallingford et al., 2020; Yao et al., 2020), we develop an algorithm that is specifically tailored to leverage variable amounts of data, ”
Multi-Step Model-Agnostic Meta-Learning: Convergence and Improved Algorithms (2020, methodology): “Balcan et al. (2019) formalized a connection between meta-initialization and meta-regularization from an online learning perspective.”
Meta-Learning with Neural Tangent Kernels (2021, unknown): “…(MAML) (Finn et al., 2017) is perhaps one of the most popular and ﬂexible ones, with a number of follow-up works such as (Nichol et al., 2018; Finn et al., 2018; Yao et al., 2019; Khodak et al., 2019a;b; Denevi et al., 2019; Fallah et al., 2020; Lee et al., 2020; Tripuraneni et al., 2020).”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, background): “Let the task parameter θτ ∼ N(0, 1) be 2Others have studied the multi-dimensional version of this setup under different perspectives Balcan et al. (2019); Saunshi et al. (2020); Denevi et al. (2019) a normal distributed scalar and likewise, the covariate x ∼ N(0, 1).”
Bayesian Meta Sampling for Fast Uncertainty Adaptation (2020, unknown): “(2019) and to some theoretical analysis (Khodak et al., 2019).”
On Localized Discrepancy for Domain Adaptation (2020, background): “Lifelong learning or learning to learn maximizes the overall performance on the tasks drawn from underlying task distribution Pentina and Lampert (2014, 2015); Khodak et al. (2019).”