Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes

By Eduardo C. Garrido-Merchán, Daniel Hernández-Lobato. 2017

289 citations recorded by the source corpus.

Abstract Some optimization problems are characterized by an objective that is very expensive, that lacks an analytical expression, and whose evaluations can be contaminated by noise. Bayesian Optimization (BO) methods can be used to solve these problems efficiently. BO relies on a probabilistic model of the objective, which is typically a Gaussian process (GP). This model is used to compute an acquisition function that estimates the expected utility (for solving the optimization problem) of evaluating the objective at each potential new point. A problem with GPs is, however, that they assume real-valued input variables and cannot easily deal with categorical or integer-valued values. Common methods to account for these variables, before evaluating the objective, include assuming they are real and then using a one-hot encoding, for categorical variables, or rounding to the closest integer, for integer-valued variables. We show that this leads to suboptimal results and introduce a novel approach to tackle categorical or integer-valued input variables within the context of BO with GPs. Several synthetic and real-world experiments support our hypotheses and show that our approach outperforms the results of standard BO using GPs on problems with categorical or integer-valued input variables.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Bayesian Optimization for Adaptive Experimental Design: A Review (2020, background): “function to be flat for the region where two continuous variables would be rounded to the same integer [121].”
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers (2019, background): “, real-valued, discrete, categorical, and hierarchically related variables [57, 29].”
Bayesian Optimization for Categorical and Category-Specific Continuous Inputs (2019, unknown): “We compare our method with four state-of-the-art base-lines that use different ways to deal with categorical variables: One-hot-Encoding (Golovin et al. 2017), Merchan-Lobato (Garrido-Merch´an and Hern´andez-Lobato 2018), SMAC (Hutter, Hoos, and Leyton-Brown 2011), and TPE (Bergstra et al. 2011).”
Recent Advances in Bayesian Optimization (2022, methodology): “A straightforward way is to construct GPs and optimize AFs by treating discrete variables as continuous, and then the closest integer for the identified next sample point with real values is obtained via a one-hot encoding strategy [52].”