Forward and Reverse Gradient-Based Hyperparameter Optimization
By Luca Franceschi, Michele Donini, Paolo Frasconi, Massimiliano Pontil. 2017
477 citations recorded by the source corpus.
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. [2015] but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one large-scale experiment where the use of previous gradient-based methods would be prohibitive.
Subfields: Artificial Intelligence, Optimization & Training, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Dacheng Tao
- Pieter Abbeel
- Jianfeng Gao
- Percy Liang
In Optimization & Training
- Sergey Levine
- Qiang Yang
- Chelsea Finn
- Massimiliano Pontil
- Amos J. Storkey
- Hado van Hasselt
- David Silver
- Li Yang
In Neural Architecture Search & AutoML
- Frank Hutter
- Arber Zela
- Marius Thomas Lindauer
- Thomas Elsken
- Li Yang
- Abdallah Shami
- Thomas Brox
- Tonmoy Saikia
Citations in the corpus
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “However, in most of the cases obtaining such a mapping is computationally expensive, therefore different heuristics have been proposed to approximate dF/dy (Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017; 2018).”
- Meta-Gradient Reinforcement Learning (2018, methodology): “…works focusing on adapting learning rate through gradient-based methods [Sutton, 1992, Schraudolph, 1999, Maclaurin et al., 2015, Pedregosa, 2016, Franceschi et al., 2017], Bayesian optimisation methods [Snoek et al., 2012], or evolution based hyper-parameter tuning [Jaderberg et al., 2017a,…”
- Meta-Learning in Neural Networks: A Survey (2020, methodology): “include HO tasks that define a meta objective that is trained end-to-end with neural networks, such as gradient-based hyperparameter learning [67], [69] and neural architecture”
- Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, methodology): “Recent gradient-based techniques for HO, however, have significantly increased the number of hyperparameters that can be optimized (Domke, 2012; Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017) and it is now possible to tune as hyperparameters entire weight vectors associated with a”
- Online Hyper-Parameter Learning for Auto-Augmentation Strategy (2019, methodology): “To address this issue, [21, 9] explicitly utilized the parameter learning process to obtain such a gradient by proposing a Lagrangian formulation associated with the parameter optimization dynamics.”
- Discrete Graph Structure Learning for Forecasting Multiple Time Series (2021, background): “A prominent example is the recent work of Franceschi et al. (2019) (named LDS), which is a meta-learning approach that treats the graph as a hyperparameter in a bilevel optimization framework (Franceschi et al., 2017).”
- Learning What and Where to Transfer (2019, methodology): “To solve the above optimization problem, we use Reverse-HG (Franceschi et al., 2017) that can compute ∇ φ L org ( θ T +1 | x, y ) efficiently us-ing Hessian-vector products.”
- Meta Learning via Learned Loss (2019, methodology): “In this context, we can divide related work on meta-learning into learning model parameters θ that can be easily adapted to new tasks [7, 18, 10, 35], learning optimizer policies h that transform parameters updates with respect to known loss or reward functions [16, 2, 14, 8, 17, 5], or learning los”
- iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, background): “…the reverse model could be formulated as: Although the reverse-mode bi-level optimization is easy to implement, the memory requirement linearly increases with the number of steps T (Franceschi et al., 2017) as it needs to store all intermediate gradients, making it impractical for deep networks.”
- On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, methodology): “FAR-HO [120] is a hyper-parameter optimization package that employs gradient-based algorithms with TensorFlow.”
- Truncated Back-propagation for Bilevel Optimization (2018, methodology): “We also hypothesize thatK-RMD does almost as well as full RMD in identifying which samples were corrupted [9].”
- Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, methodology): “• reverse (Franceschi et al., 2017): an iterative differentiation based method that approximates the hypergradient via backpropagation.”
- Learning Discrete Structures for Graph Neural Networks (2019, methodology): “A standard approach involves replacing the minimization of L with the repeated application of an iterative optimization dynamics Φ such as (stochastic) gradient descent (Domke, 2012; Maclaurin et al., 2015; Franceschi et al., 2017).”
- Optimizing Millions of Hyperparameters by Implicit Differentiation (2019, methodology): “We use the same restricted dataset as in [5, 6] of 50 training and validation examples, which allows us to assess HO performance easily.”
- Learning to Adapt to Evolving Domains (2020, background): “Recent advances in meta-learning [4, 5, 27] have been gaining momentum.”
- Learning to Teach with Dynamic Loss Functions (2018, methodology): “We further introduce a new notation ˜ M ( f ω , D dev ) = ( x,y ) ∈ D dev ˜ m ( f ω ( x ) , y ) which approximates the objective of the teacher model We use Reverse-Mode Differentiation (RMD) [6, 7, 39] to fill in the gap between training data and development data.”
- Generalized Inner Loop Meta-Learning (2019, methodology): “…problem is that of learning to optimize hyperparameters through gradient-based methods (Bengio, 2000; Maclaurin et al., 2015; Luketina et al., 2016; Franceschi et al., 2017), as an alternative to grid/random search (Bergstra & Bengio, 2012) or Bayesian Optimization (Mockus et al., 1978; Pelikan et…”
- Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “More generally, nested automatic differentiation can be used to derive hypergradients, that is, derivatives of a training objective with respect to the hyperparameters (Franceschi et al., 2017; Maclaurin et al., 2015).”