A Stratified Analysis of Bayesian Optimization Methods
By Ian Dewancker, M. McCourt, Scott C. Clark, Patrick Hayes, Alexandra Johnson, George Ke. 2016
39 citations recorded by the source corpus.
Empirical analysis serves as an important complement to theoretical analysis for studying practical Bayesian optimization. Often empirical insights expose strengths and weaknesses inaccessible to theoretical analysis. We define two metrics for comparing the performance of Bayesian optimization methods and propose a ranking mechanism for summarizing performance within various genres or strata of test functions. These test functions serve to mimic the complexity of hyperparameter optimization problems, the most prominent application of Bayesian optimization, but with a closed form which allows for rapid evaluation and more predictable behavior. This offers a flexible and efficient way to investigate functions with specific properties of interest, such as oscillatory behavior or an optimum on the domain boundary.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Afshin Rostamizadeh
- Ameet Talwalkar
- Lisha Li
- Giulia DeSalvo
- Frank Hutter
- Kevin G. Jamieson
- Sakorn Mekruksavanich
- Anuchit Jitpattanakul
In Computational Theory and Mathematics
- Ameet Talwalkar
- Lisha Li
- Afshin Rostamizadeh
- Giulia DeSalvo
- Kevin G. Jamieson
- Frank Hutter
- Xuanyi Dong
- Quoc V. Le
Citations in the corpus
- Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “Figure 2(a) presents a rank plot, which is standard in this field (Feurer et al., 2015a; Dewancker et al., 2016; Feurer et al., 2015b), in which the accuracy of each method is ranked for each dataset, and the average rank across 117 datasets is reported (lower is better).”
- LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes (2021, methodology): “Furthermore, the hyperparameters of LSTM networks were optimized using the Bayesian hyperparameter optimization on the SigOpt platform [53].”