Warm starting Bayesian optimization

By Matthias Poloczek, Jialei Wang, Peter I. Frazier. 2016

74 citations recorded by the source corpus.

We develop a framework for warm-starting Bayesian optimization, that reduces the solution time required to solve an optimization problem that is one in a sequence of related problems. This is useful when optimizing the output of a stochastic simulator that fails to provide derivative information, for which Bayesian optimization methods are well-suited. Solving sequences of related optimization problems arises when making several business decisions using one optimization model and input data collected over different time periods or markets. While many gradient-based methods can be warm started by initiating optimization at the solution to the previous problem, this warm start approach does not apply to Bayesian optimization methods, which carry a full metamodel of the objective function from iteration to iteration. Our approach builds a joint statistical model of the entire collection of related objective functions, and uses a value of information calculation to recommend points to evaluate.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “In this paper, we focus only on combining Hyperband and Bayesian optimization, but we would like to mention that methods improving BO are potentially applicable to BOHB as well, such as meta learning (Swersky et al., 2013; Feurer et al., 2015; Poloczek et al., 2016; Springenberg et al., 2016), activ”
Multi-Information Source Optimization (2016, methodology): “[25] demonstrated that a variant of the algorithm proposed in this article can reduce the optimization costs significantly by warm-starting Bayesian optimization, as does the algorithm of Swersky et al.”
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (2019, result): “If these problems further come in a speciﬁc ordering (e.g., because of successive releases of an ML model), the successive surrogate models can be ﬁt to the residuals relative to predictions of the previous learned surrogate model [15, 33].”