Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

By Ryan D. Turner, David Eriksson, M. McCourt, J. Kiili, Eero Laaksonen, Zhen Xu, Isabelle M Guyon. 2021

404 citations recorded by the source corpus.

This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “Utilizing much more sample-efficient NAS and HPO methods (Turner et al., 2021; Ren et al., 2021) can translate directly into carbon footprint improvement.”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, methodology): “To address this, ﬁrst systems have been introduced that schedule several optimizers sequentially to make use of their respective strengths (Awad et al., 2020; Lan et al., 2020; Turner et al., 2020).”
Revisiting Deep Learning Models for Tabular Data (2021, methodology): “, 2019) to run Bayesian optimization (the Tree-Structured Parzen Estimator algorithm), which is reported to be superior to random search (Turner et al., 2021).”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “It should be noted that recent NeurIPS competitions found that Bayesian optimization is superior to random search in many settings [155].”
Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization (2021, methodology): “BO has been successfully used for tuning machine learning hyperparameters for some time (Snoek et al., 2012; Turner et al., 2021), but only recently emerged as a promising approach for NAS (White et al., 2019; Liu et al., 2018b; Falkner et al., 2018; Kandasamy et al., 2018; Shi et al., 2019; Li and…”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Turner et al. (2021) have conducted an optimization challenge for different HPO-implementations.”
Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance (2023, methodology): “, 2022): the winner solution of the black-box optimization challenge 2020 (Turner et al., 2021), • Random search (Bergstra & Bengio, 2012): the most basic baseline method in HPO, • SMAC 6 (Hutter et al.”