Bayesian Optimization for Categorical and Category-Specific Continuous Inputs
By Dang Nguyen, Sunil Gupta, Santu Rana, Alistair Shilton, Svetha Venkatesh. 2019
60 citations recorded by the source corpus.
Many real-world functions are defined over both categorical and category-specific continuous variables and thus cannot be optimized by traditional Bayesian optimization (BO) methods. To optimize such functions, we propose a new method that formulates the problem as a multi-armed bandit problem, wherein each category corresponds to an arm with its reward distribution centered around the optimum of the objective function in continuous variables. Our goal is to identify the best arm and the maximizer of the corresponding continuous function simultaneously. Our algorithm uses a Thompson sampling scheme that helps connecting both multi-arm bandit and BO in a unified framework. We extend our method to batch BO to allow parallel optimization when multiple resources are available. We theoretically analyze our method for convergence and prove sub-linear regret bounds. We perform a variety of experiments: optimization of several benchmark functions, hyper-parameter tuning of a neural network, and automatic selection of the best machine learning model along with its optimal hyper-parameters (a.k.a automated machine learning). Comparisons with other methods demonstrate the effectiveness of our proposed method.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yaochu Jin
- Markus Olhofer
- Andrés Camero
- Enrique Alba
- Thomas Bäck
- Sebastian Schmitt
- Xilu Wang
- Xing-yi Zhang
In Management Science and Operations Research
- Yaochu Jin
- Andrés Camero
- Markus Olhofer
- Enrique Alba
- Sebastian Schmitt
- Xilu Wang
- Thomas Bäck
- Ye Tian
Citations in the corpus
- Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, background): “, [31]), it is not our major aim herein to compare he performance of Bayesian optimization when coupled with ifferent surrogate models and hence we decided to choose the implest random forest model to validate the proposed algorithm.”
- Recent Advances in Bayesian Optimization (2022, methodology): “While replacing the GPs in the framework of Bayesian optimization is a possible approach in the mixed-variable setting [17], the bandit approaches have been integrated with BO by treating each variable as a bandit [130].”