Batch Bayesian Optimization via Local Penalization

By Javier I. González, Zhenwen Dai, Philipp Hennig, Neil D. Lawrence. 2015

413 citations recorded by the source corpus.

The popularity of Bayesian optimization methods for efficient exploration of parameter spaces has lead to a series of papers applying Gaussian processes as surrogates in the optimization of functions. However, most proposed approaches only allow the exploration of the parameter space to occur sequentially. Often, it is desirable to simultaneously propose batches of parameter values to explore. This is particularly the case when large parallel processing facilities are available. These facilities could be computational or physical facets of the process being optimized. E.g. in biological experiments many experimental set ups allow several samples to be simultaneously processed. Batch methods, however, require modeling of the interaction between the evaluations in the batch, which can be expensive in complex scenarios. We investigate a simple heuristic based on an estimate of the Lipschitz constant that captures the most important aspect of this interaction (i.e. local repulsion) at negligible computational overhead. The resulting algorithm compares well, in running time, with much more elaborate alternatives. The approach assumes that the function of interest, $f$, is a Lipschitz continuous function. A wrap-loop around the acquisition function is used to collect batches of points of certain size minimizing the non-parallelizable computational effort. The speed-up of our method with respect to previous approaches is significant in a set of computationally expensive experiments.

Subfields: Artificial Intelligence, Biophysics, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computational Theory and Mathematics

Citations in the corpus

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “e previous work on using neural network models in Bayesian optimization [39, 40], applied to the task of hyperparameter optimization. There are also techniques for parallelizing Bayesian optimization [10,16,33]. Predicting neural network accuracy. There are several approaches for predicting the vali”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, methodology): “Other methods incorporate parallelization or use non-bayesian techniques (Li et al., 2016; Golovin et al., 2017; Shah & Ghahramani, 2015; Springenberg et al., 2016; Gonz´alez et al., 2016) but still either require multiple rounds of optimization or large amounts of compute.”
A System for Massively Parallel Hyperparameter Tuning (2018, background): “Moreover, we note that Fabolas, along with most other Bayesian optimization approaches, can be parallelized using a constant liar (CL) type heuristic (Ginsbourger et al., 2010; González et al., 2016).”
Parallelised Bayesian Optimisation via Thompson Sampling (2018, background): “[13] Javier González, Zhenwen Dai, Philipp Hennig, and Neil D Lawrence.”
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 (2021, methodology): “GPyOpt uses EI with local penalization (González et al., 2016), which is the only option in GPyOpt that supports batch evaluations.”
BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning (2019, background): “In real-world settings, there are usually many such workers available and making optimal use of them is an open problem [14, 2] with some work exploring mutual information for optimising a multi-objective problem [17].”
Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search (2020, methodology): “This discrete property of NAS makes k-DPP ideal in sampling the choices overcoming the greedy selection used in the existing batch Bayesian optimization (Desautels et al., 2014; González et al., 2016; Wang et al., 2018).”
Bayesian Optimization for Categorical and Category-Specific Continuous Inputs (2019, neutral): “Several batch BO methods have been developed (see (González et al. 2016) for a review).”
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, methodology): “…literature of BO, surrogate models (step 1) (Springenberg et al., 2016), acquisition functions (step 2) (Wang and Jegelka, 2017) and batch sampling methods (step 3) (González et al., 2016) are well studied, but the problem of hyperparameter configuration’s evaluation (step 4) remains largely open.”
Massively Parallel Hyperparameter Tuning (2018, methodology): “While Fabolas, along with most other Bayesian optimization approaches, can be parallelized using a constant liar (CL) type heuristic (Ginsbourger et al., 2010; González et al., 2016), the parallel version will underperform the sequential version, since the latter uses a more accurate posterior to pr”
Asynchronous Parallel Bayesian Optimisation via Thompson Sampling (2017, methodology): “Hence, there has been a ﬂurry of recent activity in this area [7, 8, 10, 12, 17, 24, 34, 38–40].”
Scalable Global Optimization via Local Bayesian Optimization (2019, background): “While several batch acquisition functions have recently been proposed [9, 40, 50, 54, 55, 28, 18], these approaches do not scale to large batch sizes in practice.”
Recent Advances in Bayesian Optimization (2022, background): “Given that the sequentially selected inputs are suiciently diferent from each other, a maximization-penalization strategy is introduced by adding a local penalty to the AF [59].”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “Figure 11 illustrates this behavior for EI (we consider the simple case of q = 1 for which we have an analytic solution available).”