Scalable Global Optimization via Local Bayesian Optimization

By David Eriksson, Michael Pearce, Jacob V. Gardner, Ryan D. Turner, Matthias Poloczek. Neural Information Processing Systems, 2019

715 citations recorded by the source corpus.

Bayesian optimization has recently emerged as a popular method for the sample-efficient optimization of expensive black-box functions. However, the application to high-dimensional problems with several thousand observations remains challenging, and on difficult problems Bayesian optimization is often not competitive with other paradigms. In this paper we take the view that this is due to the implicit homogeneity of the global probabilistic models and an overemphasized exploration that results from global acquisition. This motivates the design of a local probabilistic approach for global optimization of large-scale high-dimensional problems. We propose the TuRBO algorithm that fits a collection of local models and performs a principled global allocation of samples across these models via an implicit bandit approach. A comprehensive evaluation demonstrates that TuRBO outperforms state-of-the-art methods from machine learning and operations research on problems spanning reinforcement learning, robotics, and the natural sciences.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/86d21de665d9e70882dd21d9923d04fbe6fc0a73

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance (2023, methodology): “, 2022): a random forest-based BO method widely used in practice, and • TurBO 7 (Eriksson et al., 2019): a recent strong baseline method used in the black-box optimization challenge 2020.”
Recent Advances in Bayesian Optimization (2022, background): “design a local probabilistic approach (namely TuRBO) for handling large-scale data in high-dimensional spaces [47].”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, methodology): “As a next step, we plan to integrate local BO approaches (Eriksson et al., 2019).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “The trust region shrinks or extends based on the performance of TurBO's suggested configuration.”
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 (2021, methodology): “The starter kit provided examples using the default conﬁgurations of several different optimization packages: Hyperopt [5], Nevergrad [54], OpenTuner [1], pySOT [14], Scikit-Optimize [30], and TuRBO [15], which serve as baselines.”