Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures

By James Bergstra, Daniel L. K. Yamins, David D. Cox. 2013

2,576 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “BO algorithms can be divided into three categories, according to the type of probability model they contain: Gaussian Processes (GPs) [120, 121], Tree Parzen Estimators (TPE) [37], and Random Forests [35].”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “…works: Snoek et al. (2012) obtained state-of-the-art performance on CIFAR-10 by optimizing the hyperparameters of convolutional neural networks; Bergstra et al. (2014) used TPE (Bergstra et al., 2011) to optimize a highly parameterized three layer convolutional neural network; and Mendoza et…”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “4, parameterization of architectures) and a bagged [6] XGBoost regressor to estimate uncertainty of predictions; • Regularized Evolution (RE) [38] using graph2vec features; • Hyperopt with the Tree-structured Parzen Estimator (TPE); [5]; • SMAC [20] using 10-dimensional graph2vec features.”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, background): “Other non-architecture modifications involve hyperparameter tuning, including tuning libraries from the pre-deep-learning era [1].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “, 2017); and to make the choice of hyper-parameter set more convincing and the training results more reproducible (Bergstra et al., 2013).”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “Tree-Structured Parzen estimator (TPE) has shown great performance for hyper-parameter optimization tasks [13, 12, 35, 37, 115].”
Transfer Learning with Neural AutoML (2018, methodology): “Many AutoML methods exist, including random search [1], performance modelling [2, 3], Bayesian optimization [4], genetic algorithms [5, 6] and RL [7, 8].”
Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “While there has been some work on BO for architecture search [2, 15, 25, 37, 41], they have only been applied to optimise feed forward structures, e.”
DeepArchitect: Automatically Designing and Training Deep Architectures (2017, methodology): “efore requires considerable effort to use—for any ﬁxed model search space, using TPE requires the human expert to distill the hyperparameters of the search space, express the search space in Hyperopt [30] (an implementation of TPE), and write the code describing how values of the hyperparameters in ”
A genetic programming approach to designing convolutional neural network architectures (2017, methodology): “They have also proposed a meta-modeling approach [4] based on the TPE for supporting automatic hyperparameter optimization.”
Designing Neural Network Architectures using Reinforcement Learning (2016, methodology): “Notably, Bergstra et al. (2013) proposed a meta-modeling approach based on Tree of Parzen Estimators (TPE) (Bergstra et al., 2011) to choose both the type of layers and hyperparameters of feed-forward networks; however, they fail to match the performance of hand-crafted networks.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “For example, in several studies, it found better instantiations of convolutional network hyperparameters than domain experts, repeatedly improving the top score on the CIFAR-10 [12] benchmark without data augmentation [5, 13, 11].”
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, methodology): “…optimization benchmarks, such as deep belief networks (Bergstra et al. 2011), convolutional neural networks (Snoek, Larochelle, and Adams 2012; Bergstra, Yamins, and Cox 2013), and combined model selection and hyperparameter optimization in machine learning frameworks (Thornton et al. 2013).”
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces (2014, background): “This is a general problem in algorithm conﬁguration [6] that occurs in many machine learning contexts, such as, e.g., in deep neural networks [7]; ﬂexible computer vision architectures [8]; and the combined selection and hyperparameter optimization of machine learning algorithms [9].”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “This greatly extends the capabilities of the Bayesian optimization framework, allowing it to chain together individual algorithms to form sophisticated pipelines that can be jointly optimized [19], [151].”