Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits

By Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar. 2016

148 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

Citations in the corpus

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, background): “, 2012], bandit-based approaches [Li et al., 2016], and evolutionary strategies [Loshchilov and Hutter, 2016].”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, methodology): “Other methods incorporate parallelization or use non-bayesian techniques (Li et al., 2016; Golovin et al., 2017; Shah & Ghahramani, 2015; Springenberg et al., 2016; González et al., 2016) but still either require multiple rounds of optimization or large amounts of compute.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “With these ingredients, the following three steps are iterated [17]: (1) ﬁnd the most promising x n +1 ∈ arg max a p ( x ) by numerical optimization; (2) evaluate the expensive and often noisy function y n +1 ∼ f ( x n +1 )+ N (0 , σ 2 ) and add the resulting data point ( x n +1 , y n +1 ) to the…”
Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, methodology): “Our algorithm is easily extendable, e.g., by other network morphisms, evolutionary approaches for generating new models, other methods for cheap performance evaluation (such as, e.g., learning curve prediction (Klein et al., 2017) or hypernetworks (Ha et al., 2017; Brock et al., 2017)), or better re”
Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning (2018, methodology): “Hyperband algorithm is presented in Algorithm 1 (see [7] for details).”
LEAPSANDBOUNDS: A Method for Approximately Optimal Algorithm Configuration (2018, background): “…different problems are considered, e.g., by Luby et al. (1993); Adam (2001); Mnih et al. (2008); Audibert & Bubeck (2010); György & Kocsis (2011); Li et al. (2016). main technical contributions are as follows: We present an (arguably simpler) algorithm (LEAPSANDBOUNDS) that finds an…”
Probabilistic Matrix Factorization for Automated Machine Learning (2017, methodology): “One drawback of these techniques is that they are known to suffer in high-dimensional hyperparameter spaces, where they often perform comparably to random search (Li et al., 2016b).”
Hyperparameter optimization of deep neural networks: combining Hperband with Bayesian model selection (2017, background): “This has given rise recently to Hyperband [5], see Section 3.”
λOpt: Learn to Regularize Recommender Models in Finer Levels (2019, background): “Previous work [11, 21, 28, 30, 31] have dedicated to lower down the nontrivial search cost, along with developing some enhanced toolboxes [4, 7, 23].”
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization (2017, methodology): “There are, for example, some population-based approaches, such as Successive Halving [29] and Hyperband [39], which choose a set of hyperparameter configurations at random and incrementally train the learning algorithms in parallel.”
Interactive Machine Learning via a GPU-accelerated Toolkit (2017, methodology): “Recently, methods us-ing bandit optimization such as [24] began to explore early stopping strategy which saves a lot of computation.”
Learning to Optimize Tensor Programs (2018, methodology): “This result is particularly interesting compared to prior results in hyper-parameter tuning [25], where model-based approaches were shown to work only as well as random searching.”
Critical Hyper-Parameters: No Random, No Cry (2017, methodology): “We do not consider the possibility of saving computational resources by stopping some runs early based on an estimate of their final performance, as is done by methods such as Sequential Halving [11] and Hyperband [14] or validation curves modeling as in [5].”