Critical Hyper-Parameters: No Random, No Cry

By Olivier Bousquet, Sylvain Gelly, Karol Kurach, Olivier Teytaud, Damien Vincent. 2017

45 citations recorded by the source corpus.

The selection of hyper-parameters is critical in Deep Learning. Because of the long training time of complex models and the availability of compute resources in the cloud, "one-shot" optimization schemes - where the sets of hyper-parameters are selected in advance (e.g. on a grid or in a random manner) and the training is executed in parallel - are commonly used. It is known that grid search is sub-optimal, especially when only a few critical parameters matter, and suggest to use random search instead. Yet, random search can be "unlucky" and produce sets of values that leave some part of the domain unexplored. Quasi-random methods, such as Low Discrepancy Sequences (LDS) avoid these issues. We show that such methods have theoretical properties that make them appealing for performing hyperparameter search, and demonstrate that, when applied to the selection of hyperparameters of complex Deep Learning models (such as state-of-the-art LSTM language models and image classification models), they yield suitable hyperparameters values with much fewer runs than random search. We propose a particularly simple LDS method which can be used as a drop-in replacement for grid or random search in any Deep Learning pipeline, both as a fully one-shot hyperparameter search or as an initializer in iterative batch optimization.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistics and Probability

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “We used quasi-random search (Bousquet et al., 2017) to tune the metaparameters with equal budgets of non-divergent(11) trials for different batch sizes.”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, methodology): “We used quasi-random search [Bousquet et al., 2017] to tune the metaparameters with fixed budgets of non-divergent4 trials (100 for Simple CNN, ResNet8, and Transformer, and 200 for ResNet32 and VGG11).”
On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “We used quasi-random uniform search (Bousquet et al., 2017), and continued the search until we obtained a ﬁxed number of feasible trials.”
Online Hyper-Parameter Learning for Auto-Augmentation Strategy (2019, methodology): “[3] later refined this random search process by using quasi random search.”