Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

By Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, Massimiliano Pontil. 2019

120 citations recorded by the source corpus.

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “• Adapting to the Task-Similarity: A major drawback of previous work is a reliance on knowing the task-similarity beforehand to set the learning rate [24] or regularization [19], or the use of a sub-optimal guess-and-tune approach using the doubling trick [34].”
Few-Shot Learning via Learning the Representation, Provably (2020, background): “Another recent line of theoretical work analyzed gradient-based meta-learning methods (Denevi et al., 2019, Finn et al., 2019, Khodak et al., 2019) and showed guarantees for convex losses by using tools from online convex optimization.”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, background): “Let the task parameter θτ ∼ N(0, 1) be 2Others have studied the multi-dimensional version of this setup under different perspectives Balcan et al. (2019); Saunshi et al. (2020); Denevi et al. (2019) a normal distributed scalar and likewise, the covariate x ∼ N(0, 1).”
On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms (2019, background): “[13] Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil.”
On the Theory of Transfer Learning: The Importance of Task Diversity (2020, methodology): “These methods have been analyzed using techniques from online convex optimization, using a (potentially data-dependent) notion of task similarity which assumes that tasks are close to a global task parameter [Finn et al., 2019, Khodak et al., 2019a, Denevi et al., 2019a,b, Khodak et al., 2019b].”
Efficient Meta Learning via Minibatch Proximal Update (2019, methodology): “Different from [22, 27] which focus on linear regression/classiﬁcation with convex loss functions, our approach is developed inside a broader context of convex and non-convex statistical learning and thus is of more practical interest especially in deep learning.”
Provable Representation Learning for Imitation Learning via Bi-level Optimization (2020, unknown): “Existing work on the sample complexity and regret of these methods (Denevi et al., 2019; Finn et al., 2019; Khodak et al., 2019) show guarantees for convex losses by leveraging tools from online convex optimization.”
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning (2020, methodology): “, 2015) and instead favoring the convex-case study of gradient-based algorithms that could potentially be applied to deep neural networks (Denevi et al., 2019; Khodak et al., 2019).”
Provable Meta-Learning of Linear Representations (2020, background): “Finn et al. (2019); Khodak et al. (2019a;b), and Denevi et al. (2019) work in the framework of online convex optimization (OCO) and use a notion of (a potentially data-dependent) task similarity that assumes closeness of all tasks to a single fixed point in parameter space to provide guarantees.”
Meta-Learning with Neural Tangent Kernels (2021, unknown): “…(MAML) (Finn et al., 2017) is perhaps one of the most popular and ﬂexible ones, with a number of follow-up works such as (Nichol et al., 2018; Finn et al., 2018; Yao et al., 2019; Khodak et al., 2019a;b; Denevi et al., 2019; Fallah et al., 2020; Lee et al., 2020; Tripuraneni et al., 2020).”