A Tutorial on Bayesian Optimization

By Peter I. Frazier. 2018

2,412 citations recorded by the source corpus.

Bayesian optimization is an approach to optimizing objective functions that take a long time (minutes or hours) to evaluate. It is best-suited for optimization over continuous domains of less than 20 dimensions, and tolerates stochastic noise in function evaluations. It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample. In this tutorial, we describe how Bayesian optimization works, including Gaussian process regression and three common acquisition functions: expected improvement, entropy search, and knowledge gradient. We then discuss more advanced techniques, including running multiple function evaluations in parallel, multi-fidelity and multi-information source optimization, expensive-to-evaluate constraints, random environmental conditions, multi-task Bayesian optimization, and the inclusion of derivative information. We conclude with a discussion of Bayesian optimization software and future research directions in the field. Within our tutorial material we provide a generalization of expected improvement to noisy evaluations, beyond the noise-free setting where it is more commonly applied. This generalization is justified by a formal decision-theoretic argument, standing in contrast to previous ad hoc modifications.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “Parameterization of architectures We used graph2vec [33] method to create the characteristic features for architectures, which can be useful for some NAS methods like Bayesian Optimization [13].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The most common choice is the expected improvement (EI) because of its good performance and ease of use (Frazier, 2018).”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive [43, 13], and it has seen great success in hyperparameter optimization for deep learning [43, 14, 31].”
Scalable Global Optimization via Local Bayesian Optimization (2019, background): “BO has recently become the premier technique for global optimization of expensive functions, with applications in hyperparameter tuning, aerospace design, chemical engineering, and materials discovery; see [14, 41] for an overview.”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “By default, we use multi-start optimization via L-BFGS-B in conjunction with an initialization heuristic that exploits fast batch evaluation of acquisition functions (see Appendix D for details).”
Bayesian Optimization for Adaptive Experimental Design: A Review (2020, background): “Several reviews have presented BO for a technical audience [20]–[22].”
AdaLead: A simple and robust adaptive greedy search algorithm for sequence design (2020, methodology): “Otherwise BO is known to perform poorly in higher dimensional space (Frazier, 2018), and to our knowledge, no general purpose sequence design algorithm using BO has performed better than the models considered below.”
Methods for comparing uncertainty quantifications for material property predictions (2019, background): “active learning [5], online active learning [6], Bayesian optimization [7], active search [8], or goal oriented design of experiments [9]).”
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 (2021, methodology): “These black-box optimization problems are often solved using Bayesian optimization (BO) methods [19].”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “After that, they proposes an effective NAS methods based on a high-dimensional multi-dimensional Bayesian optimisation [47].”
Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search (2020, methodology): “BO has been an emerging technique for black-box optimization when function evaluations are expensive (Rana et al., 2017; Frazier, 2018; Gopakumar et al., 2018; Wan et al., 2021), and it has seen great success in hyperparameter optimization for deep learning (Li & Jamieson, 2018; Nguyen et al.,…”
Searching Better Architectures for Neural Machine Translation (2020, methodology): “( iii) We do not directly hyperparameter optimization techniques like Bayesian Optimization [31], hyberband [32], evolution based method [33], because it is cost to train an NMT model until convergence, and the performance gap of two groups of hyperparameters usually appear after long training.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Bayesian Optimization: Bayesian optimization (BO) [248] is a sequential design strategy for the global optimization of black-box functions that does not assume any functional forms.”
Parameterized quantum circuits as machine learning models (2019, neutral): “Another popular method is Bayesian optimization [58].”
NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, methodology): “Bayesian optimization utilizes an acquisition function defined from a surrogate model to sample the potential optimal solution [33], [34].”
Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization (2021, methodology): “We employ Bayesian optimization (BO), a popular method for black-box optimization of computationally expensive functions that achieves high sampleefficiency (Frazier, 2018).”