A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
By Eric Brochu, Vlad M. Cora, Nando de Freitas. 2010
2,681 citations recorded by the source corpus.
We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions. Bayesian optimization employs the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function. This permits a utility-based selection of the next observation to make on the objective function, which must take into account both exploration (sampling from areas of high uncertainty) and exploitation (sampling areas likely to offer improvement over the current best observation). We also present two detailed extensions of Bayesian optimization, with experiments---active user modelling with preferences, and hierarchical reinforcement learning---and a discussion of the pros and cons of Bayesian optimization based on our experiences.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Percy Liang
- Frank Hutter
In Electrical and Electronic Engineering
- Sergey Levine
- Frank Hutter
- Zoubin Ghahramani
- Aaron Klein
- Katharina Eggensperger
- Ziyun Wang
- Jost Tobias Springenberg
- Andreas Krause
In Management Science and Operations Research
- Frank Hutter
- Aaron Klein
- Nando de Freitas
- Peter I. Frazier
- Kirthevasan Kandasamy
- Stefan Falkner
- Jeff Schneider
- Ziyun Wang
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Jan Peters
- Shimon Whiteson
- Yuxi Li
- Shixiang Shane Gu
- Marc Peter Deisenroth
In Optimization & Training
- Frank Hutter
- Aaron Klein
- Stefan Falkner
- Quoc V. Le
- Marc-André Zöller
- Marco F. Huber
- Ameet Talwalkar
- Yoshua Bengio
Citations in the corpus
- BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “…therefore to combine the strengths of several methods (in particular, Hyperband (Li et al., 2017) and a robust & effective variant (Bergstra et al., 2011) of Bayesian optimization (Brochu et al., 2010; Shahriari et al., 2016)) to propose a practical HPO method that fulfills all of these desiderata.”
- Hyperparameter optimization with approximate gradient (2016, methodology): “In recent years, sequential model-based optimization (SMBO) techniques have emerged as a powerful tool for hyperparameter optimization (see e.g. (Brochu et al., 2010) for an review on current methodologies).”
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “(5) Bayesian optimization is very efficient concerning the number of objective function evaluations (Brochu et al., 2010) as the acquisition function handles the trade-off between exploration and exploitation automatically.”
- Survey on Automated Machine Learning (2019, methodology): “Bayesian optimization (Brochu et al., 2010) is an iterative optimization framework being well suited for expensive objective functions.”
- Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “We use EI in this work, but the ideas apply just as well to other acquisitions [3].”
- Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “…x n +1 ∈ arg max a p ( x ) by numerical optimization; (2) evaluate the expensive and often noisy function y n +1 ∼ f ( x n +1 )+ N (0 , σ 2 ) and add the resulting data point ( x n +1 , y n +1 ) to the set of observations D n = ( x j , y j ) j =1 ...n ; and (3) update p ( f | D n +1 ) and a p ( f”
- Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “This interactive Bayesian optimization strategy is particulary effective as humans can be very good at comparing examples, but unable to produce an objective function whose optimum is the example of interest.”
- Learning to Optimize (2016, methodology): “Most methods [13, 4, 24, 25, 9] rely on sequential model-based Bayesian optimization [22, 6], while others adopt a random search approach [3] or use gradient-based optimization [2, 8, 21].”
- Neural-net-induced Gaussian process regression for function approximation and PDE solution (2018, background): “On the other hand, NNGP is able to estimate uncertainty of predictions, which is crucial to noisy-data handling and active learning [19].”
- Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “While many tools have been proposed to attack these problems, Bayesian optimization techniques have gained a substantial amount of interest in recent years [Brochu et al., 2009, Shahriari et al., 2016].”
- Freeze-Thaw Bayesian Optimization (2014, methodology): “Bayesian optimization (see [13, 14] for an in-depth explanation) proceeds by fitting a probabilistic model to the data and then using this model as an inexpensive proxy in order to determine the most promising location to evaluate next.”
- Learning Where to Attend with Deep Architectures for Image Tracking (2011, methodology): “By assuming that the reward surface is smooth, we can draw on the tools of Bayesian optimization (Brochu et al., 2010) to search for the optimal gaze location using as few exploratory steps as possible.”
- Learning to select data for transfer learning with Bayesian Optimization (2017, methodology): “For this reason, we propose to learn the following linear domain similarity measure S as a linear combination of feature values: where φ ( X ) ∈ R n × l are the similarity and diversity features further described in §3.2 for each training example, with l being the number of features, while w ∈ R l a”
- Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization (2012, methodology): “The exploration-exploitation tradeoff has also been studied in Bayesian global optimization and response surface modeling, where Gaussian process models are often used due to their flexibility in incorporating prior assumptions about the payoff function (Brochu et al., 2009).”
- Gaussian Processes for Data-Efficient Learning in Robotics and Control (2015, neutral): “[10] and Bayesian optimization [9], [33], [41], [51] to avoid getting stuck in local minima.”
- Automated Machine Learning on Big Data using Stochastic Algorithm Tuning (2014, background): “There are a wide range of options for the choice of acquisition function [3].”