Random Search for Hyper-Parameter Optimization

By James Bergstra, Yoshua Bengio. 2012

11,272 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, methodology): “We select a learning rate (scalar multiplier in RMSProp) to maximize the validation performance, out of 10 randomly chosen log-uniform candidates sampled from U(−12,−6) [Bergstra and Bengio, 2012].”
Learning to Optimize Neural Nets (2017, methodology): “…have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradient-based optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015).”
On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Grid search is inefﬁcient (Bergstra and Bengio, 2012) and random search and Bayesian optimization algorithms tend to use priors oblivious to the meanings of different metaparameters (Snoek et al., 2012).”
Gradient-Based Neural DAG Learning (2019, methodology): “Experiments: We explored multiple hyperparameter combinations using random search (Bergstra & Bengio, 2012).”
Predicting Tactical Solutions to Operational Planning Problems under Imperfect Information (2018, methodology): “Sets of hyperparameter values were generated randomly and the preferred set was determined according to validation results (Bergstra and Bengio, 2012).”
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, methodology): “…we validate that the CNN architecture used in DCGAN is not the key contributing factor of the model’s performance by training a purely supervised CNN with the same architecture on the same data and optimizing this model via random search over 64 hyperparameter trials (Bergstra & Bengio, 2012).”
Energy and Policy Considerations for Deep Learning in NLP (2019, methodology): “…the energy associated with model tuning is by providing easy-to-use APIs implementing more efﬁcient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques (Bergstra et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012).”
Quantifying the Carbon Emissions of Machine Learning (2019, background): “…been proven both empirically and theoretically that random hyperparameter search is more efﬁcient than grid search for hyperparameter optimization [10], and there is much research being done on ways to improve the efﬁciency of hyperparameter optimization [11, 12], which makes it possible to…”
On Network Design Spaces for Visual Recognition (2019, methodology): “To simulate random search experiments of varying size m we follow the procedure described in [1].”
Squeeze-and-Excitation Networks (2017, methodology): “By formulating architecture search as hyperparameter optimisation, random search [34] and other more sophisticated model-based optimisation techniques [35], [36] can also be used to tackle the problem.”
Representation Learning: A Review and New Perspectives (2012, background): “Recent work on automating hyperparameter search [29], [30], [191] is also making it more convenient, efficient, and reproducible.”
Neural Architecture Search with Reinforcement Learning (2016, background): “Although this baseline seems simple, it is often very hard to surpass (Bergstra & Bengio, 2012).”
Learning Transferable Architectures for Scalable Image Recognition (2017, positive): “…we demonstrate the effectiveness of reinforcement learning on architecture search on the CIFAR-10 image classiﬁcation problem as compared to brute-force random search (considered to be a very strong baseline for black-box optimization [2]) given an equivalent amount of computational resources.”
MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, background): “Recently there has been lots of progress in algorithmic architecture exploration included hyperparameter optimization [9, 10, 11] as well as various”
RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “Also, methods for choosing hyperparameters more efficiently than grid search have been recently developed [31, 32].”
Practical Bayesian Optimization of Machine Learning Algorithms (2012, neutral): “They demonstrated that grid search strategies are inferior to random search [9], and suggested the use of Gaussian process Bayesian optimization, optimizing the hyperparameters of a squared-exponential covariance, and proposed the Tree Parzen Algorithm.”
Ex Machina: Personal Attacks Seen at Scale (2016, methodology): “For each model, we performed 15 iterations of random search over a grid of relevant hyper-parameters[1].”
Show Your Work: Improved Reporting of Experimental Results (2019, background): “Grid search and uniform sampling are popular systematic methods; the latter has been shown to be superior for most search spaces (Bergstra and Bengio, 2012).”
Recurrent Models of Visual Attention (2014, methodology): “Hyperparameters such as the initial learning rate and the variance of the location policy were selected using random search [3].”