BOHB: Robust and Efficient Hyperparameter Optimization at Scale
By Stefan Falkner, Aaron Klein, Frank Hutter. 2018
1,347 citations recorded by the source corpus.
Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit-based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and bandit-based methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Management Science and Operations Research, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Jianfeng Gao
- Frank Hutter
- Mohit Bansal
- Sergey Levine
- Percy Liang
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Cordelia Schmid
- Jitendra Malik
In Management Science and Operations Research
- Frank Hutter
- Aaron Klein
- Stefan Falkner
- Ameet Talwalkar
- Matthias Feurer
- Liam Li
- Afshin Rostamizadeh
- Marc-André Zöller
In Neural Architecture Search & AutoML
- Frank Hutter
- Colin White
- Yash Savani
- Aaron Klein
- Xuanyi Dong
- Marius Thomas Lindauer
- Stefan Falkner
- Arber Zela
Citations in the corpus
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, methodology): “We follow CLIP [66] to implement a simple grid-search style tuning pipeline, and leave more sophisticated methods like BOHB [20] and DEHB [3] as future work.”
- Quantifying the Carbon Emissions of Machine Learning (2019, background): “However, it has been shown that random search (and others) not only is a straightforward replacement but also has potential to significantly accelerate hyperparameter search [20, 21, 22], consequently reducing carbon emissions.”
- Advances and Open Problems in Federated Learning (2019, background): “Hyperparameter optimization (HPO) has a long history under the framework of AutoML [339, 237, 241], but it mainly concerns how to improve the model accuracy [59, 364, 321, 159] rather than communication and computing efficacy for”
- AutoML: A Survey of the State-of-the-Art (2019, neutral): “[108] propose Bayesian Optimization-based Hyperband (BOHB),”
- VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, background): “Neural architecture search (NAS) (Tan & Le, 2019; Jiang et al., 2020; Shi et al., 2020), hyperparameter optimization (HPO) (Falkner et al., 2018) and data augmentation (Cubuk et al., 2018) have made great progress in many tasks.”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “As mentioned previously, Bayesian OptimizationHyperBand (BOHB) (Falkner et al., 2018) is a combination of BO and HB, and introduces a guided sampling method instead of simple random search (Table 2).”
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “We make the following observations: (1) NAS methods without parameter sharing (REA, RS, REINFORCE, and BOHB) outperform others.”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “…algorithms including random search (Li and Talwalkar 2019), DARTS (Liu, Simonyan, and Yang 2018), regularized evolution (Real et al. 2019), BOHB (Falkner, Klein, and Hutter 2018), NASBOT (Kandasamy et al. 2018b), local search (White, Nolen, and Savani 2020), TPE (Bergstra et al. 2011), BONAS…”
- Towards modular and programmable architecture search (2019, methodology): “Hyperopt [2], Spearmint [3], SMAC [5, 8] and BOHB [4]) are general-purpose and domain-independent.”
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “As suspected, also the architectural parameters stay and end up in flatter areas of the architecture landscape, as shown in Figure 8”
- Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “com/Microsoft/nni celerates and simplifies the huge search space using builtin super-parameter selection algorithms including random search, naive evolutionary algorithms, simulated annealing, network morphism, grid search, hyper-band, and a bunch of Bayesian optimizations like SMAC [51], and BOHB [”
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, methodology): “…Figure 5): random search (Bergstra and Bengio, 2012), SMAC (Hutter et al., 2011) 2 , Tree Parzen Estimator (TPE) (Bergstra et al., 2011) 3 , Bohamiann (Springenberg et al., 2016) 4 , Regularized Evolution (Real et al., 2019), Hy-perband (HB) (Li et al., 2017) and BOHB (Falkner et al., 2018) 5 .”
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “For each function evaluation, BOHB passes the current budget and a configuration instance to the objective function.”
- Survey on Automated Machine Learning (2019, neutral): “Yet, surprisingly SMAC also has a constant overhead. hyperopt and BOHB should in theory have a similar overhead ofO(n) as BOHB internally uses an algorithm closely related to hyperopt.”
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “…for algorithms with open-sourced implementations and found the following: random search (RS) (Bergstra & Bengio, 2012), regularized evolution (RE) (Real et al., 2018), SMAC (Hut-ter et al., 2011), TPE (Bergstra et al., 2011), Hyperband (HB) (Li et al., 2018), and BOHB (Falkner et al., 2018).”
- Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “Although certain hyperparameter optimization methods [42, 34, 25] require non-trivial modification in order to work with NAS search spaces, others are easily applicable to NAS problems [18, 4, 10, 13, 27, 28].”
- Evolutionary-Neural Hybrid Agents for Architecture Search (2018, neutral): “…than deep RL and evolution, different approaches have been applied to architecture search and hyperparameter optimization: bayesian optimization [15], cascade-correlation [14], boosting [9], deep-learning based tree search [30], hill-climbing [13], model-based approaches [21], and random search…”
- Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “Especially in such cases, we advocate running extensive evaluations on tabular benchmarks, such as NAS-Bench-101 (Ying et al., 2019) and the NAS-HPO benchmarks (Klein & Hutter, 2019), or on surrogate benchmarks as proposed by Eggensperger et al. (2015; 2018) and used by Falkner et al. (2018).”
- A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “We benchmark Hyperband and SHA against PBT, BOHB (synchronous SHA with Bayesian optimization as introduced by Falkner et al. (2018)), and Fabolas, and examine the relative performance of SHA versus ASHA and Hy-perband versus asynchronous Hyperband.”