Gradient-based Hyperparameter Optimization through Reversible Learning

By Dougal Maclaurin, David Kristjanson Duvenaud, Ryan P. Adams. 2015

1,048 citations recorded by the source corpus.

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Structured Attention Networks (2017, methodology): “Many optimization algorithms (e.g. gradient descent, LBFGS) are also differentiable (Domke, 2012; Maclaurin et al., 2015), and have been used as output layers for structured prediction in energy-based models (Belanger & McCallum, 2016; Wang et al., 2016).”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, background): “Most of these works have considered good random initial-izations, though a number of papers have addressed data-dependent initializers (Kr¨ahenb¨uhl et al., 2016; Salimans & Kingma, 2016), including learned initializations (Husken & Goerick, 2000; Maclaurin et al., 2015).”
Learning to Optimize Neural Nets (2017, methodology): “…have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradient-based optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015).”
Unrolled Generative Adversarial Networks (2016, methodology): “Differentiating through unrolled optimization was ﬁrst scaled to deep networks in (Maclaurin et al., 2015), where it was used for hyperparameter optimization.”
The Reversible Residual Network: Backpropagation Without Storing Activations (2017, methodology): “[21] made use of the reversible nature of stochastic gradient descent to tune hyperparameters via gradient descent.”
A Closer Look at Memorization in Deep Networks (2017, methodology): “We compute g t x by unrolling t SGD steps and applying backpropagation over the unrolled computation graph, as done by Maclaurin et al. (2015).”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “[27, 51, 59] do not use any explicit meta-learner model, but instead unroll the base-learner gradient updates and optimize for model initializations which generalize well on novel tasks.”
Meta-learning with differentiable closed-form solvers (2018, methodology): “[37] and by the MAML algorithm of Finn et al.”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “For example, learning rates, weight decays and weight initializations can all be learned via hypergradient descent (Maclaurin et al., 2015; Baydin et al., 2017; Drucker & Le Cun, 1992), Bayesian optimization (Snoek et al., 2012) and reinforcement learning (Daniel et al., 2016; Xu et al., 2017;…”
DARTS: Differentiable Architecture Search (2018, background): “…α as the upper-level variable and w as the lower-level variable: The nested formulation also arises in gradient-based hyperparameter optimization (Maclaurin et al., 2015; Pedregosa, 2016), which is related in a sense that the continuous architecture α could be viewed as a special type of…”
Meta-Learning With Differentiable Convex Optimization (2019, background): “[15] where they studied low precision representations of the optimization trace of deep networks.”
Dataset Distillation (2018, methodology): “To our knowledge, this direction has only been slightly touched on previously (Maclaurin et al., 2015).”
Hyperparameter optimization with approximate gradient (2016, methodology): “However, this would change the experimental design while the goal of this ex-periment was to compare the methods in a similar setting to that of (Maclaurin et al., 2015).”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “These are indeed the elements required to compute the gradient of the outer objective with a backward pass (we refer the reader to [22] for more details).”
Differentiable Neural Input Search for Recommender Systems (2020, background): “The above problem formulation is actually consistent with hyperparameter optimization in a broader scope [13, 27, 33], since the mixed dimension schemeD can be considered as model hyperparameters to be determined according to model’s validation performance.”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, neutral): “[35] calculate the extract gradients w.”
AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, background): “This a bilevel optimization problem [19, 22, 23], where Θ is the the upper-level variable and W is the lower-level variable:”
Faster AutoAugment: Learning Augmentation Strategies using Backpropagation (2019, neutral): “s get complex. Therefore, efﬁcient methods are required to ﬁnd optimal strategies. If gradient information of these hyper-parameters is available, they can be efﬁciently optimized by gradient descent [20]. However, the gradient information is usually difﬁcult to obtain because some magnitude paramet”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “However, in most of the cases obtaining such a mapping is computationally expensive, therefore different heuristics have been proposed to approximate dF/dy (Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017; 2018).”