Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning

By Valerio Perrone, Huibin Shen, Matthias W. Seeger, Cédric Archambeau, Rodolphe Jenatton. 2019

118 citations recorded by the source corpus.

Bayesian optimization (BO) is a successful methodology to optimize black-box functions that are expensive to evaluate. While traditional methods optimize each black-box function in isolation, there has been recent interest in speeding up BO by transferring knowledge across multiple related black-box functions. In this work, we introduce a method to automatically design the BO search space by relying on evaluations of previous black-box functions. We depart from the common practice of defining a set of arbitrary search ranges a priori by considering search space geometries that are learnt from historical data. This simple, yet effective strategy can be used to endow many existing BO methods with transfer learning properties. Despite its simplicity, we show that our approach considerably boosts BO by reducing the size of the search space, thus accelerating the optimization of a variety of black-box optimization problems. In particular, the proposed approach combined with random search results in a parameter-free, easy-to-implement, robust hyperparameter optimization strategy. We hope it will constitute a natural baseline for further research attempting to warm-start BO.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Perrone et al. (2019) use transfer learning to automatically construct a minimal search space from the best configurations on related ML tasks.”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Recent approaches have studied how to empirically find optimal “default” values that work well under many conditions, by tuning ranges and HPC prior distributions based on extensive meta-data (Feurer, Klein, et al., 2015a; Gijsbers et al., 2021; Perrone et al., 2019; Pfisterer et al., 2021; Probst e”
Amazon SageMaker Autopilot: a white box AutoML solution at scale (2020, background): “Several ML applications also involve optimizing multiple metrics simultaneously, such as inference latency, training time or memory consumption (Perrone et al., 2019a; 2020; Lee et al., 2020; Guinet et al., 2020), which could extend the current HPO procedure.”
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, background): “Future work to improve BOSS may further learn the search space (Perrone et al., 2019) or hyperparameter importance (Hoos and Leyton-Brown, 2014).”