Online Learning Rate Adaptation with Hypergradient Descent

By Atilim Gunes Baydin, R. Cornish, David Martínez-Rubio, Mark W. Schmidt, Frank D. Wood. 2017

284 citations recorded by the source corpus.

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Other relevant threads of work include research on meta-learning to learn model hyper-parameters (Nichol et al., 2018; Baydin et al., 2017), model parameters (Ha et al., 2016a; Platanios et al., 2018) and models that learn new tasks with high sample ef-ﬁciency (Finn et al., 2017) without forgetting…”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “For example, learning rates, weight decays and weight initializations can all be learned via hypergradient descent (Maclaurin et al., 2015; Baydin et al., 2017; Drucker & Le Cun, 1992), Bayesian optimization (Snoek et al., 2012) and reinforcement learning (Daniel et al., 2016; Xu et al., 2017;…”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Table 3: We compare four AutoML algorithms [4, 13, 2, 33] on four search spaces.”
Automatic differentiation in machine learning: a survey (2015, background): “Nested AD is highly relevant in hyperparameter optimization as it can effortlessly provide exact hypergradients, that is, derivatives of a training objective with respect to the hyperparameters of an optimization routine (Maclaurin et al., 2015; Baydin et al., 2018).”
Meta Pseudo Labels (2020, background): “Similar bi-level optimization problems have been proposed to optimize a model’s learning process, such as learning the learning rate schedule [3], designing architectures [40], correcting wrong training labels [88], generating training examples [59], and re-weighting training data [73, 74, 54, 53].”
Online Meta-Learning (2019, neutral): “One exception is work that adapts hyperparameters online (Elfwing et al., 2017; Meier et al., 2017; Baydin et al., 2017).”
Meta-Learning in Neural Networks: A Survey (2020, background): “While the classic meta-learning pipeline deﬁnes the meta-optimization as an outer-loop of the inner base learner [19], [78], some studies have attempted to preform meta-optimization online within a single base learning episode [44], [167], [185], [186].”
La-MAML: Look-ahead Meta Learning for Continual Learning (2020, methodology): “While we adapt the gain based on the correlation between old and new task gradients to make shared progress on all tasks, [4, 25] use the correlation between two successive stochastic gradients to converge faster.”
Meta-Curvature (2019, background): “In addition, the meta-curvature matrices can be updated online to speed up training deep neural networks (Baydin et al., 2018; Almeida et al., 1999).”
A Generalizable Approach to Learning Optimizers (2021, background): “Another similar branch of work is on using hypergradients [27, 6, 16, 35], direct gradients from loss to hyperparameters, in order to optimize those hyperparameters on-the-ﬂy.”
Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives (2018, methodology): “In future work, we intend to apply ideas from (Baydin et al., 2017) to automatically adapt the weighting based on the data.”
Alpha MAML: Adaptive Model-Agnostic Meta-Learning (2019, methodology): “The solution we propose is based on the hypergradient descent (HD) algorithm (Baydin et al., 2018), which automatically updates a learning rate by performing gradient descent on the learning rate alongside original optimization steps.”
Using Wasserstein Generative Adversarial Networks for the design of Monte Carlo simulations (2019, unknown): “[2013]) for regularization purposes as well as a modification of the Adam algorithm that adapts the learning rate (Baydin et al. [2017]), thus reducing the time for hyperparameter search.”
Adaptive Scheduling for Multi-Task Learning (2019, background): “For single tasks, hypergradient descent [1] adjusts the global learning rate by considering the direction of the gradient and of the previous update.”
MetaInit: Initializing learning by learning to initialize (2019, methodology): “A form of meta-learning can also be used to improve optimization such as [6], which tunes the learning rate using hypergradient descent.”
Learning an Adaptive Learning Rate Schedule (2019, background): “Hypergradient takes the derivative of the learning rate and updates the learning rate based on its gradient [8].”