Algorithms for Hyper-Parameter Optimization

By James Bergstra, Rémi Bardenet, Yoshua Bengio, Balázs Kégl. 2011

5,706 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Learning to Optimize Neural Nets (2017, methodology): “Various kinds of methods have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradientbased optimization (Bengio, 2000; Domke, 2012; Macla”
True Few-Shot Learning with Language Models (2021, background): “Selection criteria can be optimized automatically, e.g. with bayesian optimization [44–46], evolutionary methods [45, 47, 48], reinforcement learning [49], or gradient descent [50–53].”
Energy and Policy Considerations for Deep Learning in NLP (2019, methodology): “…the energy associated with model tuning is by providing easy-to-use APIs implementing more efﬁcient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “A large body of prior work has explored learning hyperparameters of standard optimizers (Bergstra et al., 2011; Feurer & Hutter, 2019).”
On Network Design Spaces for Visual Recognition (2019, methodology): “General hyperparameter search techniques [2, 32] address the laborious model tuning process in machine learning.”
Representation Learning: A Review and New Perspectives (2012, background): “Recent work on automating hyperparameter search [29], [30], [191] is also making it more convenient, efficient, and reproducible.”
Neural Architecture Search with Reinforcement Learning (2016, background): “Hyperparameter optimization is an important research topic in machine learning, and is widely used in practice (Bergstra et al., 2011; Bergstra & Bengio, 2012; Snoek et al., 2012; 2015).”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “The proposed method is related to previous work in hyperparameter optimization [44, 4, 5, 54, 55, 6, 40] – especially recent approaches in designing architectures such as Neural Fabrics [48], DiffRNN [41], MetaQNN [3] and DeepArchitect [43].”
Green AI (2019, background): “While many hyperparameter optimization algorithms exist which can reduce the computational expense required to reach a given level of performance [4, 10], simple improvements here can have a large impact.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, background): “Most work on hyperparameter optimization tunes a number of impactful hyperparameters, such as the learning rate, the width of the layers in the model, and the strength of the regularization (Li et al., 2018; Bergstra et al., 2011).”
Practical Bayesian Optimization of Machine Learning Algorithms (2012, neutral): “Recently, Bergstra et al. [5] have explored various strategies for optimizing the hyperparameters of machine learning algorithms.”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, methodology): “The current gold standard for hyperparameter selection is gradient-free model-based optimization (Snoek et al., 2012; Bergstra et al., 2011; 2013; Hutter et al., 2011).”
Advances and Open Problems in Federated Learning (2019, background): “Hyperparameter optimization (HPO) has a long history under the framework of AutoML [339, 237, 241], but it mainly concerns how to improve the model accuracy [59, 364, 321, 159] rather than communication and computing efficacy for”
Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, methodology): “Similar to the other deep networks, the hyperparameters are searched with MS, GS, RS and bayesian methods [55, 56].”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “Code for BOHB and our benchmarks is publicly available at https://github.com/automl/HpBandSter To compare against TPE, we used the Hyperopt package (Bergstra et al., 2011), and for all GP-BO methods we used the RoBO python package (Klein et al., 2017b).”
Hyperparameter optimization with approximate gradient (2016, methodology): “Random search (Bergstra et al., 2011) has been proven to yield a faster exploration of the hyperparameter space than grid search, specially in spaces with multiple hyperparameters.”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, background): “, 2011) and the tree Parzen estimator (TPE) (Bergstra et al., 2011).”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, background): “mizing the architecture of a deep neural network on a particular problem, including, for example, the type of activation functions, the number of hidden layers, and the number of units for each layer [4, 10, 22].”