Bayesian Optimization with Unknown Constraints

By Michael A. Gelbart, Jasper Snoek, Ryan P. Adams. 2014

533 citations recorded by the source corpus.

Recent work on Bayesian optimization has shown its effectiveness in global optimization of difficult black-box objective functions. Many real-world optimization problems of interest also have constraints which are unknown a priori. In this paper, we study Bayesian optimization for constrained problems in the general case that noise may be present in the constraint functions, and the objective and constraints may be evaluated independently. We provide motivating practical examples, and present a general framework to solve such problems. We demonstrate the effectiveness of our approach on optimizing the performance of online latent Dirichlet allocation subject to topic sparsity constraints, tuning a neural network given test-time memory constraints, and optimizing Hamiltonian Monte Carlo to achieve maximal effectiveness in a fixed time, subject to passing standard convergence diagnostics.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “Recent work (Gelbart et al., 2014; Snoek, 2013; Gramacy & Lee, 2010) has developed approaches for modeling unknown constraints in GP-based Bayesian optimization by learning a constraint classiﬁer and then discounting expected improvement by the probability of constraint violation.”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “[56] proposed the weighted expected improvement criterion (wEI) that multiplies EI by the probability of satisfying the constraints”
Google Vizier: A Service for Black-Box Optimization (2017, background): “In the case of Bayesian Optimization, previous work either assigns them a particularly bad objective value, attempts to incorporate a probability of infeasibility into the acquisition function to penalize points that are likely to be infeasible [3], or tries to explicitly model the shape of the infe”
Freeze-Thaw Bayesian Optimization (2014, methodology): “In practice, we follow [18, 19] and use Monte Carlo simulation to estimate P min on a discrete grid containing the points with the highest EI.”
Coupling Adaptive Batch Sizes with Learning Rates (2016, background): “Since that cost scales linearly with m , the optimal batch size is the one that maximizes expected gain per cost, This idea of maximizing improvement per cost has also received attention in Bayesian optimization [10, 25].”
CMA-ES for Hyperparameter Optimization of Deep Neural Networks (2016, neutral): “However, we still need to carry out a much broader and more detailed comparison, involving more test problems and various modiﬁcations of the algorithms considered here, such as the addition of constraints (Gelbart et al., 2014; Hern´andez-Lobato et al., 2014).”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, background): “weights [8] or energy per operation assumptions [22, 25]), or simulation results [21].”
Constrained Efficient Global Optimization of Expensive Black-box Functions (2022, methodology): “Along this line of research, a group of popular methods (Gardner et al., 2014; Gelbart et al., 2014) encodes the constraint information into the acquisition function (e.g., Constrained Expected Improvement).”
Maximizing acquisition functions for Bayesian optimization (2018, background): “2: for t = 1 ,...,T do : Givenmodel ,acquisition , : Givenmodel M ,acquisition L”
ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language (2019, unknown): “We can observe if a query has a “pass” or “fail” state [12, 13, 27].”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “Spearmint also supports constrained BO (Gelbart et al., 2014) and parallel optimization (Snoek et al., 2012).”
Constrained Bayesian Optimization with Noisy Experiments (2017, methodology): “The third problem is a constrained Branin problem used by Gelbart et al. (2014) and the fourth is a problem given by Gardner et al. (2014); these both have two parameters and one constraint.”
Predictive Entropy Search for Bayesian Optimization with Unknown Constraints (2015, methodology): “Following the experimental set up in Gelbart et al. (2014), we optimize the number of effective samples produced by an HMC sampler limited to 5 minutes of computation time, subject to passing of the Geweke (Geweke, 1992) and Gelman-Rubin (Gelman & Rubin, 1992) convergence diagnostics, as well as the”
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates (2017, background): “5 An alternative to removing the crashed runs would be to model them explicitly as unknown constraints (Gelbart et al 2014).”
Constrained Bayesian optimization for automatic chemical design using variational autoencoders (2019, unknown): “39 The constraint is that a latent point must decode successfully a large fraction of the times decoding is attempted.”
The Parallel Knowledge Gradient Method for Batch Bayesian Optimization (2016, background): “…thatthe parallel knowledge gradient method finds global optima significantly faster than previous batch Bayesian optimization algorithms on both synthetic test functions and when tuning hyperparameters of practical machine learning algorithms, especially when function evaluations are noisy.”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “If a is differentiable with respect to ζ , then In B O T ORCH , these chains of derivatives are handled by auto-differentiation.”