Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

By Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar. 2016

197 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

1. Frank Hutter
2. Aaron Klein
3. Arber Zela
4. Stefan Falkner
5. Xia Hu
6. Qingquan Song
7. Ameet Talwalkar
8. Liam Li

Citations in the corpus

• BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “Hyperband (Li et al., 2017) is a bandit strategy that dynamically allocates resources to a set of random configurations and uses successive halving (Jamieson & Talwalkar, 2016) to stop poorly performing configurations.”
• A Comprehensive Survey of Neural Architecture Search (2020, methodology): “In addition, SMASH also uses the early training performance of different networks according to the research in Hyperband [89] to provide meaningful guidance suggestions for the ranking of candidate network architectures.”
• Understanding and Robustifying Differentiable Architecture Search (2019, background): “…one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or multi-fidelity optimization [Baker et al., 2017b, Falkner et al., 2018, Li et al., 2017, Zela et al., 2018], however their applicability still often remains restricted to rather simple tasks and small datasets.”
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, methodology): “…Figure 5): random search (Bergstra and Bengio, 2012), SMAC (Hutter et al., 2011) 2 , Tree Parzen Estimator (TPE) (Bergstra et al., 2011) 3 , Bohamiann (Springenberg et al., 2016) 4 , Regularized Evolution (Real et al., 2019), Hy-perband (HB) (Li et al., 2017) and BOHB (Falkner et al., 2018) 5 .”
• Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “[6] provide a thorough comparison to random search, they use random search with full training even though partial training methods have been shown to be orders-of-magnitude faster than standard random search [27, 28].”
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, methodology): “Due to space constraints, we refer the reader to Falkner et al. (2018) and Li et al. (2017) for full details on these methods and only provide the basics here.”
• SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “Relatedly, bandit-based methods (Li et al., 2017) provide a framework for efficiently exploring the hyperparameter space by employing an adaptive early-stopping strategy, allocating more resources to models which show promise early in training.”
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “We also evaluated MTBO with 3 different auxiliary tasks ( N / 4 , N / 32 , and N / 512 ), but found this to perform worse than the variant with only one auxiliary tasks at N / 4 .”
• A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “This last observation (i) corroborates the results in Li et al. (2017), which found that the brackets with the most aggressive early-stopping rates performed the best; and (ii) follows from the discussion in Section 2 noting that PBT is more similar in spirit to SHA than Hyperband, as PBT / SHA…”
• Private selection from private candidates (2018, background): “While random search is a surprisingly effective way to do hyperparameter optimization in machine learning [24], there are more complex adaptive algorithms that often do better.”
• Transferable AutoML by Model Sharing Over Grouped Datasets (2019, methodology): “In particular, the method Hyperband [22] speeds up random search using an early-stopping strategy to allocate resources adaptively.”
• Show Your Work: Improved Reporting of Experimental Results (2019, background): “While these strategies may find better results quickly, they are generally less reproducible and harder to parallelize (Li et al., 2017).”
• RLlib: Abstractions for Distributed Reinforcement Learning (2017, methodology): “HyperBand and Population Based Training: Ray includes distributed implementations of hyperparameter search algorithms such as HyperBand and PBT (Li et al., 2016; Jaderberg et al., 2017).”
• Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning (2018, background): “A recent variation on random search called Hyperband focuses on speeding up random search by terminating ill-performing hyperparameter configurations (Li et al. [15]).”
• Towards Reproducible Neural Architecture and Hyperparameter Search (2018, methodology): “Hyperband achieved a reasonable performance relatively quickly but it converged to simple random search.”
• Practical Automated Machine Learning for the AutoML Challenge 2018 (2018, unknown): “Recently, SH and its extension Hyberband have been successfully applied to hyperparameter optimization problems (Jamieson and Talwalkar, 2016; Li et al., 2017), demonstrating strong performance with small budgets.”
• SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach (2018, background): “Hyperband is allowed at most 30 epochs per model and a total of 1000 epochs.”