The Effects of Hyperparameters on SGD Training of Neural Networks

By Thomas Breuel. 2015

64 citations recorded by the source corpus.

The performance of neural network classifiers is determined by a number of hyperparameters, including learning rate, batch size, and depth. A number of attempts have been made to explore these parameters in the literature, and at times, to develop methods for optimizing them. However, exploration of parameter spaces has often been limited. In this note, I report the results of large scale experiments exploring these different parameters and their interactions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “…size, the region in metaparameter space corresponding to rapid training in terms of epochs becomes smaller (Figure 9a, consistent with the ﬁndings of Breuel, 2015b), while the region in metaparameter space corresponding to rapid training in terms of step-count grows larger (Figure 9b, although…”
A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, methodology): “(2011) and Breuel (Breuel, 2015), which investigated factors that effect performance of unsupervised feature learning and the effects of Stochastic Gradient Descent”
Cyclical Learning Rates for Training Neural Networks (2015, background): “A technical report by Breuel [3] provides guidance on a variety of hyper-parameters.”
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (2015, background): “When modifying the batch size, Breuel [6] and Krizhevsky [28] found that the choice of learning rate is crucial in order to preserve high accuracy.”
Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study (2015, background): “Hyperparameter optimization plays a central role in obtaining good accuracy from neural network models [3].”
Paleo: A Performance Model for Deep Neural Networks (2016, background): “However, it is important to note that we are increasing the effective batch size as we increase the number of workers, and it is well-known that training with large effective batch-sizes can yield models with substandard accuracy (Breuel, 2015).”
Optimizing deep learning hyper-parameters through an evolutionary algorithm (2015, background): “Studies of the effects of hyper-parameters on different deep learning architectures have shown complex relationships, where hyper-parameters that give great performance improvements in simple networks do not have the same effect in more complex architectures [5].”
A Comparative Study on Regularization Strategies for Embedding-based Neural Networks (2015, background): “may not play an important role in training neural networks, if the effect of parameter symmetry is ruled out (Breuel, 2015).”