Gradient-Based Optimization of Hyperparameters

By Yoshua Bengio. 2000

609 citations recorded by the source corpus.

Subfields: Computational Mechanics, Control and Systems Engineering, Numerical Analysis, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Control and Systems Engineering

In Numerical Analysis

In Optimization & Training

Citations in the corpus

Learning to Optimize Neural Nets (2017, methodology): “…have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradient-based optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015).”
Meta-learning with differentiable closed-form solvers (2018, neutral): “iin Figure 1). Compared to methods that adapt SGD, they exhibit an inherently fast rate of convergence, particularly in cases where a closed form solution exists. A similar idea has been discussed by Bengio (2000), where the analytic formulations of zero-gradient solutions are used to obtain meta-gr”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, methodology): “Larsen et al. (1998), Eigenmann & Nossek (1999), Chen & Hagan (1999), Bengio (2000), Abdel-Gawad & Ratner (2007), and Foo et al. (2008) showed that gradients of regularization parameters are available in closed form when training has converged exactly to a local minimum.”
Dataset Distillation (2018, methodology): “Our work bears similarity with gradient-based hyperparameter optimization techniques, which compute the gradient of hyperparameter w.r.t. the ﬁnal validation loss by reversing the entire training procedure (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015; Pedregosa, 2016).”
Hyperparameter optimization with approximate gradient (2016, methodology): “Originally motivated by the problem of setting the regularization parameter in the context of neural networks (Larsen et al., 1996; 1998; Bengio, 2000), has also been applied to the problem of selecting kernel parameters (Chapelle et al., 2002; Seeger, 2008) or multiple regularization parameters in…”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “We then exploit a recent technique called back-gradient optimization, originally proposed for hyperparameter optimization [3, 14, 22, 31], to implement a much more computationally-efficient poisoning attack.”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “Recent work (Keskar et al., 2016; Yao et al., 2018) focuses more on the settings of large/small batch size training, where observations show that small batch training tends to get attracted to flatter minima and generalizes better. Similarly, Nguyen et al. (2018) observed that this phenomenon manife”
Survey on Automated Machine Learning (2019, background): “Instead it can be approximated during the optimization (Pedregosa, 2016) or calculated explicitly through reversible learning (Bengio, 2000; Maclaurin et al., 2015).”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “, evolutionary search [38, 19], Bayesian optimization [42, 4, 18], and gradient-based approaches [2, 34].”
A genetic programming approach to designing convolutional neural network architectures (2017, methodology): “There are many hyperparameter tuning methods for themachine learning algorithm, such as grid search, gradient search [2], random search [3], and Bayesian optimization-based methods [13, 28].”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, methodology): “In this paper, we follow an alternative direction, where gradient-based algorithms are used to optimize the performance on a validation set with respect to the hyperparameters (Bengio, 2000; Larsen et al., 1996).”
Learning to Optimize (2016, methodology): “Most methods [13, 4, 24, 25, 9] rely on sequential model-based Bayesian optimization [22, 6], while others adopt a random search approach [3] or use gradient-based optimization [2, 8, 21].”
Online Learning Rate Adaptation with Hypergradient Descent (2017, background): “Previous authors, most notably Bengio [3], have noted that the search for good hyperparameter values for gradient descent can be cast as an optimization problem itself, which can potentially be tackled via another level of gradient descent using backpropagation.”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “prior learner by considering inductive [13,70] and transductive learning methods [25], and each with either epoch-dependent (e.g., LSTM) or epoch-independent (e.g., epochwise FC layer) architectures. (3) Extensive experiments on three challenging few-shot benchmarks, miniImageNet [73], tieredImageNe”
Automated Machine Learning: From Principles to Practices (2018, methodology): “Likewise, within the realm of oncology predictive modeling, an AutoML-based tool, JADBio [137], has been introduced to generating predictive models with high-performance criteria.”
A Generalized Framework for Population Based Training (2019, methodology): “Over the past years, there have been a variety of algorithms proposed to automatically tune hyperparameters of a machine learning model including Grid search, Random search [3], Bayesian optimization [13], Gradient-based optimization [2], Bandits [9] and Evolutionary algorithms [7].”
Understanding and correcting pathologies in the training of learned optimizers (2018, methodology): “Computing these gradients is costly as we need to iteratively apply the learned update rule, and then backpropagate through these applications, a technique commonly referred to as “unrolled optimization” (Bengio, 2000; Maclaurin et al., 2015).”