Constrained Bayesian Optimization with Noisy Experiments

By Benjamin Letham, Brian Karrer, Guilherme Ottoni, Eytan Bakshy. Bayesian Analysis, 2017

348 citations recorded by the source corpus.

Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multiple continuous parameters, but existing approaches degrade in performance when the noise level is high, limiting its applicability to many randomized experiments. We derive an expression for expected improvement under greedy batch optimization with noisy observations and noisy constraints, and develop a quasi-Monte Carlo approximation that allows it to be efficiently optimized. Simulations with synthetic functions show that optimization performance on noisy, constrained problems outperforms existing methods. We further demonstrate the effectiveness of the method with two real-world experiments conducted at Facebook: optimizing a ranking system, and optimizing server compiler flags.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/9b768bd9271aa5f7833d48fee04aa0f628583b5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “That is, NEI ( Here, we propose a novel full MC formulation of NEI that extends Letham et al. (2019) to joint parallel optimization and generic objectives.”
Recent Advances in Bayesian Optimization (2022, background): “[115] directly extends cEI to noisy observations with greedy batch optimization.”
Prescriptive analytics: Literature review and research challenges (2020, positive): “Typically, this process begins by evaluating a small number of randomly selected function values, and fitting a Gaussian process regression model to the results (Letham, Karrer, Ottoni, & Bakshy, 2019).”
Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization (2020, methodology): “1 Additional Synethetic Benchmarks We include two additional synthetic benchmarks: a Constrained Branin-Currin problem [6] and the additional commonly used disk constraint (d = 2,M = 2, V = 1) [28, 40], and the DTLZ2 problem from the MO literature [14] (d = 6,M = 2).”
HHVM JIT: a profile-guided, region-based compiler for PHP and Hack (2018, methodology): “We note that HHVM’s flags controlling these JIT optimizations, and well as various GCC optimization flags used to compile HHVM, have been auto-tuned using the method described in [29].”
Maximizing acquisition functions for Bayesian optimization (2018, methodology): “For q-EI, we recover the parallel asynchronous strategy proposed by [21, 52], further justifying its use as a synchronous one [39, 50].”
Closing the loop between brain and electrical stimulation: towards precision neuromodulation treatments (2023, background): “Measurement errors related to small effect sizes can affect optimization algorithms [114].”