Practical Bayesian Optimization of Machine Learning Algorithms
By Jasper Snoek, Hugo Larochelle, Ryan P. Adams. 2012
9,594 citations recorded by the source corpus.
The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.
Subfields: Artificial Intelligence, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Kyunghyun Cho
In Optimization & Training
- Frank Hutter
- Quoc V. Le
- Yoshua Bengio
- Ameet Talwalkar
- Aaron Klein
- Simon Lacoste-Julien
- Ruslan Salakhutdinov
- Roger Baker Grosse
Citations in the corpus
- Network In Network (2013, background): “Test Error Stochastic Pooling [11] 15.13% CNN + Spearmint [14] 14.98% Conv. maxout + Dropout [8] 11.68% NIN + Dropout 10.41% CNN + Spearmint + Data Augmentation [14] 9.50% Conv. maxout + Dropout + Data Augmentation [8] 9.38% DropConnect + 12 networks + Data Augmentation [15] 9.32% NIN + Dropout +…”
- Maxout Networks (2013, background): “CNN + Spearmint + data augmentation (Snoek et al., 2012) 9.”
- Learning to Optimize Neural Nets (2017, methodology): “Various kinds of methods have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradientbased optimization (Bengio, 2000; Domke, 2012; Macla”
- On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “Grid search is inefficient (Bergstra and Bengio, 2012) and random search and Bayesian optimization algorithms tend to use priors oblivious to the meanings of different metaparameters (Snoek et al., 2012).”
- Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Standard methods may be employed to efficiently perform hyperparameter optimization [38], however given the extremely small search space we find that naive grid search is quite effective (Section 2).”
- Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “While the sampling procedure can be optimized by bayesian optimization (Snoek et al. 2012) or multi-armed bandit algorithms (Vermorel and Mohri 2005), we choose to keep the method as simple as possible and use random sampling.”
- Generative Moment Matching Networks (2015, methodology): “…and GMMN training, dropout rate for the auto-encoder, and number of hidden units on each layer of both auto-encoder and GMMN, were tuned using Bayesian optimization (Snoek et al., 2012; 2014) 1 to optimize the validation set likelihood under the Gaussian Parzen window density estimation.”
- True Few-Shot Learning with Language Models (2021, background): “Selection criteria can be optimized automatically, e.g. with bayesian optimization [44–46], evolutionary methods [45, 47, 48], reinforcement learning [49], or gradient descent [50–53].”
- Energy and Policy Considerations for Deep Learning in NLP (2019, methodology): “…the energy associated with model tuning is by providing easy-to-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
- Learning Parameterized Skills (2012, background): “Standard acquisition functions include Lower Confidence Bounds, Maximum Probability of Improvement and Expected Improvement [64].”
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…weight decays and weight initializations can all be learned via hypergradient descent (Maclaurin et al., 2015; Baydin et al., 2017; Drucker & Le Cun, 1992), Bayesian optimization (Snoek et al., 2012) and reinforcement learning (Daniel et al., 2016; Xu et al., 2017; 2019; Almeida et al., 2021).”
- On Network Design Spaces for Visual Recognition (2019, methodology): “General hyperparameter search techniques [2, 32] address the laborious model tuning process in machine learning.”
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “In our experiments with soft attention, we also used Whet-lab 1 (Snoek et al., 2012; 2014) in our Flickr8k experiments.”
- Representation Learning: A Review and New Perspectives (2012, background): “Recent work on automating hyperparameter search [29], [30], [191] is also making it more convenient, efficient, and reproducible.”
- Neural Architecture Search with Reinforcement Learning (2016, methodology): “Hyperparameter optimization is an important research topic in machine learning, and is widely used in practice (Bergstra et al., 2011; Bergstra & Bengio, 2012; Snoek et al., 2012; 2015).”
- Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “rovide a viable search strategy, RL does significantly improve our ability to learn neural architectures. 4. Related Work The proposed method is related to previous work in hyperparameter optimization [43, 4, 5, 53, 54, 6, 39] – especially recent approaches in designing architectures such as Neural F”
- MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, background): “Recently there has been lots of progress in algorithmic architecture exploration included hyper-parameter optimization [9, 10, 11] as well as various a r X i v : 1801 .”
- RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “Also, methods for choosing hyperparameters more efficiently than grid search have been recently developed [31, 32].”
- Progressive Neural Architecture Search (2017, methodology): “, (Snoek et al., 2012; Shahriari et al., 2016)) in which we use an acquisition function, such as expected improvement or upper confidence bound, to rank the candidate models; this takes into account uncertainty in the prediction of a model’s performance, which encourages us to try models for which o”