Hyperparameter optimization with approximate gradient

By Fabian Pedregosa. 2016

532 citations recorded by the source corpus.

Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.

Subfields: Computational Mechanics, Computational Theory and Mathematics, Management Science and Operations Research, Meta & Continual Learning, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computational Theory and Mathematics

In Management Science and Operations Research

In Meta & Continual Learning

In Mathematical Optimization

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Hyperparameter optimization (HPO) has a long history under the framework of AutoML [339, 237, 241], but it mainly concerns how to improve the model accuracy [59, 364, 321, 159] rather than communication and computing efficacy for”
AutoML: A Survey of the State-of-the-Art (2019, background): “Many attempts ( [10], [109]–[113]) have been made to solve the problems encountered in the discrete samplebased approach.”
Dataset Distillation (2018, methodology): “Our work bears similarity with gradient-based hyperparameter optimization techniques, which compute the gradient of hyperparameter w.r.t. the ﬁnal validation loss by reversing the entire training procedure (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015; Pedregosa, 2016).”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “We then exploit a recent technique called back-gradient optimization, originally proposed for hyperparameter optimization [3, 14, 22, 31], to implement a much more computationally-efficient poisoning attack.”
Differentiable Neural Input Search for Recommender Systems (2020, background): “Equation 5 essentially defines a bi-level optimization problem [9], which has been studied in differentiable NAS [25] and gradient-based hyperparameter optimization [5, 13, 33].”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Following that, different approximation methods have been proposed [33, 37, 42].”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “However, in most of the cases obtaining such a mapping is computationally expensive, therefore different heuristics have been proposed to approximate dF/dy (Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017; 2018).”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “By assuming some properties of f—and therefore limiting the applicability of this approach to specific problem instance— gradient descent can still be used (Maclaurin et al., 2015; Pedregosa, 2016).”
Survey on Automated Machine Learning (2019, background): “Instead it can be approximated during the optimization (Pedregosa, 2016) or calculated explicitly through reversible learning (Bengio, 2000; Maclaurin et al., 2015).”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, methodology): “Both (Maclaurin et al., 2015) and (Pedregosa, 2016) managed to optimize a number of hyperparameters in the order of one thousand.”
Meta-Gradient Reinforcement Learning (2018, background): “There are many works focusing on adapting learning rate through gradient-based methods [Sutton, 1992, Schraudolph, 1999, Maclaurin et al., 2015, Pedregosa, 2016, Franceschi et al., 2017], Bayesian optimisation methods [Snoek et al., 2012], or evolution based hyper-parameter tuning [Jaderberg et…”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, methodology): “Modeling hierarchical decision making processes [54], bilevel optimization has witnessed an increasing popularity in machine learning recently, with applications ranging from meta-learning [20], to hyperparameter optimization [45, 21] and neural architecture search [37].”
Meta-Learning in Neural Networks: A Survey (2020, background): “Popular solutions include implicit differentiation of v [154], [163], [293], forward-mode differentiation of v [67], [69], [294], gradient preconditioning [105], evolutionary strategies [295], solving for a greedy version of v online by alternating inner and outer steps [21], [42], [198], truncation”
Automated Machine Learning: From Principles to Practices (2018, methodology): “As Bayesian optimization method do not add much assumption on the search space, it can be applied to various kinds of AutoML problems, such as neural architecture search [68, 131], hyper-parameter optimization [12, 41].”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, background): “While a possible strategy (implicit differentiation) is to apply the implicit function theorem to ∇ L λ = 0 (Pedregosa, 2016; Koh and Liang, 2017; Beirami et al., 2017), another compelling approach is to replace the inner problem with a dynamical system.”
Super-efficiency of automatic differentiation for functions defined as a minimum (2020, background): “Bi-level optimization appears in a variety of machine-learning problems, such as hyperparameter optimization [Pedregosa, 2016] or supervised dictionary learning [Mairal et al., 2012].”
Gradient Descent: The Ultimate Optimizer (2019, background): “Most existing work on gradient-based hyperparameter optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017) has focused on computing hyperparameter gradients after several iterations of training, which is computationally expensive because of the…”
Efficient and Modular Implicit Differentiation (2021, background): “Examples of applications in machine learning include hyper-parameter optimization [19, 63, 57, 30, 10, 11], neural networks [46], and meta-learning [31, 59].”
Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, methodology): “Recently, various bilevel optimization algorithms have been proposed for hyperparameter optimization, which include implicit differentiation based methods (Pedregosa, 2016), dynamical system based methods via reverse or forward gradient computation (Franceschi et al.”