Taking the Human Out of the Loop: A Review of Bayesian Optimization
By Bobak Shahriari, Kevin Swersky, Ziyun Wang, Ryan P. Adams, Nando de Freitas. 2016
5,660 citations recorded by the source corpus.
Big Data applications are typically associated with systems involving large numbers of users, massive complex software systems, and large-scale heterogeneous computing and storage architectures. The construction of such systems involves many distributed design choices. The end products (e.g., recommendation systems, medical analysis tools, real-time game engines, speech recognizers) thus involve many tunable configuration parameters. These parameters are often specified and hard-coded into the software by various developers or teams. If optimized jointly, these parameters can result in significant improvements. Bayesian optimization is a powerful tool for the joint optimization of design choices that is gaining great popularity in recent years. It promises greater automation so as to increase both product quality and human productivity. This review paper introduces Bayesian optimization, highlights some of its methodological aspects, and showcases a wide range of applications.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Ruslan Salakhutdinov
- Pieter Abbeel
- Percy Liang
In Computational Theory and Mathematics
- Frank Hutter
- Aaron Klein
- Yaochu Jin
- Stefan Falkner
- Nando de Freitas
- Zoubin Ghahramani
- Peter I. Frazier
- José Miguel Hernández-Lobato
In Management Science and Operations Research
- Frank Hutter
- Aaron Klein
- Stefan Falkner
- Sergey Levine
- Marc-André Zöller
- Marco F. Huber
- Peter I. Frazier
- Zoubin Ghahramani
In Probabilistic & Bayesian Methods
- Yoshua Bengio
- Zoubin Ghahramani
- Yee Whye Teh
- Frank Hutter
- David M. Blei
- José Miguel Hernández-Lobato
- Richard E. Turner
- Ruslan Salakhutdinov
In Neural Architecture Search & AutoML
- Frank Hutter
- Quoc V. Le
- Aaron Klein
- Xuanyi Dong
- Marc-André Zöller
- Marius Thomas Lindauer
- Marco F. Huber
- Martin Wistuba
Citations in the corpus
- BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “, 2011) of Bayesian optimization (Brochu et al., 2010; Shahriari et al., 2016)) to propose a practical HPO method that fulfills all of these desiderata.”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Bayesian methods can make the sampling procedure in random search more efficient [25, 43, 44].”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “A lack of space limits further discussion on this topic; please refer to the review (Shahriari et al., 2015) for more information.”
- Real-Time Federated Evolutionary Neural Architecture Search (2020, methodology): “Other techniques such as Bayesian optimization [41] is also a good approach reducing the computational complexity of NAS [42].”
- Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “Hidden layers (M) [1, 8] [1, 3] [1, 8] Look back (T) [2, 30] [2, 30] [2, 30] Neurons per layer (N) [10, 100] [1, 100] [1, 300]”
- Towards modular and programmable architecture search (2019, methodology): “Gaussian process Bayesian optimization [7] and sequential model based optimization [8] are two popular approaches.”
- Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion (2019, methodology): “Popular derivative-free optimization methods such as Bayesian optimization (Shahriari et al. 2015), evolutionary optimization (Hansen, Müller, and Koumoutsakos 2003), classification-based optimization (Yu, Qian, and Hu 2016; Hu, Qian, and Yu 2017), etc.”
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “To actually implement the surrogate model fitting and configuration selection, Bayesian optimization (Brochu et al., 2010; Shahriari et al., 2016; Frazier, 2018) is used.”
- AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, methodology): “…by considering the recent work in hyperpa-rameter tuning for machine learning, which identifies the up to k cleaning components, we can create a parameter Quality Function Structure: Hyperparameter tuning algorithms Data Cleaning Method Structure: Similarly, black-box Parzen Estimator (TPE) [43] .”
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “We do not include Gaussian process based Bayesian optimization meth-ods (Shahriari et al., 2016), such as, e.g., Spearmint (Snoek et al., 2012), since this tends to have problems in high-dimensional discrete optimization tasks (Eggensperger et al., 2013).”
- Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, methodology): “To efficiently optimize in the joint space of architectures and hyperparameters, we use BOHB (Falkner et al., 2018), a recent combination of Bayesian optimization (Shahriari et al., 2016) and Hyperband (Li et al., 2017).”
- Designing Neural Network Architectures using Reinforcement Learning (2016, methodology): “Bayesian optimization has also been used (Shahriari et al., 2016) for automatic selection of network architectures (Bergstra et al., 2013; Domhan et al., 2015) and hyperparameters (Snoek et al., 2012; Swersky et al., 2013).”
- Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “differentiable model, which is preferable to the Gaussian kernel popular elsewhere, because it makes less restrictive 1Comprehensive tutorials are presented by Brochu et al. [17] and Shahriari et al. [18]. 2 smoothness assumptions, which can be helpful in the optimization setting [5]: k 5=2 (x;x 0) ”
- Grammar Variational Autoencoder (2017, methodology): “For this, we use Bayesian optimization methods, which choose the next point to evaluate by maximizing an acquisition function that quantifies the benefit of evaluating the property at a particular location (Shahriari et al., 2016).”
- Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “The middle nodes of the tree represent the parameters that are shared by a subset of the tasks.”
- Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “While many tools have been proposed to attack these problems, Bayesian optimization techniques have gained a substantial amount of interest in recent years [Brochu et al., 2009, Shahriari et al., 2016].”