Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

By Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar. 2016

2,847 citations recorded by the source corpus.

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, methodology): “To tune all model hyperparameters, we used HYPERBAND [22], a bandit-based algorithm that speeds up random search via an aggressive form of early-stopping called SUCCESSIVEHALVING [17].”
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets (2017, methodology): “This could be exploited by multi-ﬁdelity methods for architecture and hyperpa-rameter search (Li et al., 2016; Klein et al., 2016).”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “This strategy is not unlike bandit algorithms such as Successive Halving[53] and Hyperband[54], however we do not use a static population of individuals created a priori, but integrate our halving with the changing evolutionary population.”
Quantifying the Carbon Emissions of Machine Learning (2019, background): “However, it has been shown that random search (and others) not only is a straightforward replacement but also has potential to significantly accelerate hyperparameter search [20, 21, 22], consequently reducing carbon emissions.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, methodology): “Most work on hyperparameter optimization tunes a number of impactful hyperparameters, such as the learning rate, the width of the layers in the model, and the strength of the regularization (Li et al., 2018; Bergstra et al., 2011).”
The Evolved Transformer (2019, methodology): “al’s (2018) utilization of Hyperband (Li et al., 2017) and PNAS’s (Liu et al.”
AutoML: A Survey of the State-of-the-Art (2019, background): “[112], who introduce a novel search algorithm, Hyperband, which allocates limited resources (such as time or CPUs) to only the most”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “For benchmarking NAS algorithms on our search space, we tested the following methods within the environment: • Random Search in two modes: low-fidelity mode (RS 10E), which trains 5X networks for 10 epochs, and high-fidelity mode, which trains 1X networks for 50 epochs (RS 50E); • Hyperbands (HB) is”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “, using less training epochs or a subset of data [9, 23, 27, 31].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The paper on HB (Li et al., 2017) concludes by discussing potential improvements, including distributed implements, adjustment of the convergence rate, and non-random sampling.”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Other specific architecture aspects have also been targeted, such as the hyperparameters [77, 50, 32, 44], activation functions [66], a specific layer [35], the full forward pass [23], the data augmentation [17, 62, 18], etc.”
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “In this way, we could provide bandit-based algorithms (Falkner et al., 2018; Li et al., 2018) more options for the usage of short training budget (see more details in appendix).”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive [43, 13], and it has seen great success in hyperparameter optimization for deep learning [43, 14, 31].”
Towards modular and programmable architecture search (2019, background): “Random search has been found to be competitive for hyperparameter optimization [9, 10].”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “HyperBand is another bandit-based powerful multi-fidelity hedging technique that optimizes the search space when selecting from randomly sampled configurations [71].”
A Survey on Neural Architecture Search (2019, background): “(2017) global random n/a elitist Liu et al. (2018b) cell-based random tournament all Real et al.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “A limitation of Hyperband is the random generation of the tested configurations.”
AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, methodology): “Since Bayesian optimization is inherently sequential, for parallel and distributed settings, the community is increasingly studying randomized and grid search schemes [20, 32, 33].”
Survey on Automated Machine Learning (2019, methodology): “candidate configurations is sampled at random to comply with the theoretical guarantees of hyperband (Li et al., 2018).”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “…for algorithms with open-sourced implementations and found the following: random search (RS) (Bergstra & Bengio, 2012), regularized evolution (RE) (Real et al., 2018), SMAC (Hut-ter et al., 2011), TPE (Bergstra et al., 2011), Hyperband (HB) (Li et al., 2018), and BOHB (Falkner et al., 2018).”