Multi-Step Model-Agnostic Meta-Learning: Convergence and Improved Algorithms
By Kaiyi Ji, Junjie Yang, Yingbin Liang. 2020
52 citations recorded by the source corpus.
As a popular meta-learning approach, the model-agnostic meta-learning (MAML) algorithm has been widely used due to its simplicity and effectiveness. However, the convergence of the general multi-step MAML still remains unexplored. In this paper, we develop a new theoretical framework, under which we characterize the convergence rate and the computational complexity of multi-step MAML. Our results indicate that N -step MAML attains the convergence with linearly increasing complexity with N under a properly chosen inner stepsize. We then take a further step to develop a more efficient Hessian-free MAML. We first show that the existing zeroth-order Hessian estimator contains a constant-level estimation error so that the MAML algorithm can perform unstably. To address this issue, we propose a novel Hessian estimator via a gradient-based Gaussian smoothing method, and show that it achieves a much smaller estimation bias and variance, and the resulting algorithm achieves the same performance guarantee as the original MAML under mild conditions. Our experiments validate our theory and demonstrate the effectiveness of the proposed Hessian estimator.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Junjie Yang
- Kaiyi Ji
- Yingbin Liang
- Mehdi Abbana Bennani
- Masashi Sugiyama
- Amos J. Storkey
- K. Müller
- Balaji Lakshminarayanan
Citations in the corpus
- Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, background): “…a statistical learning theory perspective, they define and provide regret bounds for lifelong learning (Alquier et al., 2017), or study the sample complexity and convergence of meta-learning algorithms (Denevi et al. 2018a, Du et al. 2020, Ji et al. 2020, Saunshi et al. 2020, Denevi et al. 2018b).”
- Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, methodology): “…framework in various machine learning applications including meta-learning (Franceschi et al., 2018; Bertinetto et al., 2018; Rajeswaran et al., 2019; Ji et al., 2020a optimization (Franceschi et al., 2018; Shaban et al., 2019; Feurer & Hutter, 2019), reinforcement learning (Konda & Tsitsiklis,…”