Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

By Tobias Domhan, Jost Tobias Springenberg, Frank Hutter. 2015

642 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, background): “For example, [127] propose a learning curve model which is a weighted combination of a set of parametric curve models selected from the literature, which makes it possible to predict the performance of the network.”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, background): “One way of speeding up evaluation is to predict performance of a (partially) trained model (Domhan et al., 2015; Baker et al., 2017b; Klein et al., 2017; Liu et al., 2017).”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “, using less training epochs or a subset of data [9, 23, 27, 31].”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “arly, and for more promising architectures, more adequate training can be obtained. Early termination of training is not a new idea, many researchers have done a lot of related research. For example: [106] uses the probabilistic method to simulate the learning curve of the deep neural network, and t”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “In order to check the effectiveness of E2EPP, the FBO [30] and Peephole [32] algorithms, which are the representatives of existing performance predictors as detailed in Section II-B, are selected to do the quantitative and qualitative comparisons against E2EPP.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “This idea can be combined with search algorithms other than BO (Domhan et al., 2015).”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “, early stopping criteria based on the learning curve [13] or MRS [7]) to evolving the architecture of the network (neuroevolution).”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “encoding and dataset features to predict the accuracy for neural network and dataset pairs [13]. There is also work in predicting the learning curve of neural networks for hyperparameter optimization [19, 5] or NAS [2] using Bayesian techniques. None of 4 these methods have predicted the accuracy of”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “In addition, such technique resulted in speeding-up the optimization by a factor of 2 and achieved the state-of-the-art neural network on CIFAR-10 dataset [30].”
A Survey on Neural Architecture Search (2019, methodology): “Domhan et al. (2015) Typically learning curves are increasing and saturating functions.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “By observing the improvement in each iteration, the learning curve of an ML model can be predicted (Swersky et al., 2014; Domhan et al., 2015; Klein et al., 2017b).”
Survey on Automated Machine Learning (2019, methodology): “of an ML model is often an iterative procedure converging to a local minimum. By observing the improvement in each iteration, the learning curve of an ML model can be predicted (Swersky et al., 2014; Domhan et al., 2015; Klein et al., 2017b). By simultaneously considering multiple congurations in an”
Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “Although certain hyperparameter optimization methods [42, 34, 25] require non-trivial modification in order to work with NAS search spaces, others are easily applicable to NAS problems [18, 4, 10, 13, 27, 28].”
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, background): “On some benchmarks, these NAS methods have indeed led to new state-of-the-art performance (Zoph and Le, 2017; Real et al., 2018), although only at extreme computational costs on the order of 800 GPUs for two weeks.”
Path-Level Network Transformation for Efficient Architecture Search (2018, methodology): “…has been studied using various approaches which can be categorized as neuro-evolution (Real et al., 2017; Liu et al., 2018), Bayesian optimization (Domhan et al., 2015; Men-doza et al., 2016), Monte Carlo Tree Search (Negrinho & Gordon, 2017) and reinforcement learning (RL) (Zoph & Le, 2017;…”
TAPAS: Train-less Accuracy Predictor for Architecture Search (2018, methodology): “This approach yields superior performance compared to LCE, particularly at stages where the initial observed learning curve is not sufficient for the parametric algorithm to converge.”
Peephole: Predicting Network Performance Before Training (2017, background): “Note that this formulation differs fundamentally from previous works [5, 11, 2].”
Efficient Architecture Search by Network Transformation (2017, methodology): “In parallel, automatic architecture designing has also been studied in the context of Bayesian optimization (Bergstra and Bengio 2012; Domhan, Springenberg, and Hutter 2015; Mendoza et al. 2016).”
Designing Neural Network Architectures using Reinforcement Learning (2016, methodology): “Bayesian optimization has also been used (Shahriari et al., 2016) for automatic selection of network architectures (Bergstra et al., 2013; Domhan et al., 2015) and hyperparameters (Snoek et al., 2012; Swersky et al., 2013).”