A Bayesian Perspective on Generalization and Stochastic Gradient Descent

By Samuel L. Smith, Quoc V. Le. 2017

309 citations recorded by the source corpus.

We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? Our work responds to Zhang et al. (2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. We show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy. We propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the "noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$ the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, $B_{opt} \propto \epsilon N$. We verify these predictions empirically.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “[SL17] connected this temperature to the critical batch size, though they predict a dependence on dataset size which we do not observe.”
Deep Learning Scaling is Predictable, Empirically (2017, neutral): “While these theoretical and empirical results offer signiﬁcant insight about required model sizing, recent work has noted the need for more practical guidance (Zhang et al. (2017); Arpit et al. (2017); Smith and Le (2017); Kawaguchi et al. (2017)).”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “…wider range of batch sizes in experiments that either (i) used a step budget without changing the learning rate for each batch size (Smith and Le, 2018, Figures 4 and 6), or (ii) varied the learning rate and used a step budget that was a function of the learning rate (Smith and Le, 2018, Figure 5).”
Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Batch size can be set arbitrarily, but particular batch sizes result in best model accuracy depending on data set size [35].”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “According to this suggestion, training with a large learning rate (or with a small batch size) can improve performance because it leads to more stochasticity during training (Mandt et al., 2017; Smith et al., 2017; Smith & Le, 2018; Smith et al., 2018).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “The importance of the scale of noise for non-convex optimization has also been studied in [5, 49].”
An Alternative View: When Does SGD Escape Local Minima? (2018, background): “Under a Bayesian perspective, it is shown that the noise in stochastic gradient could drive SGD away from sharp minima, which decides the optimal batch size [27].”
signSGD: compressed optimisation for non-convex problems (2018, background): “There is some evidence (Smith & Le, 2018) that a certain level of noise can be good for generalisation, biasing the optimiser towards wider valleys in the objective function.”
Three Factors Influencing Minima in SGD (2017, background): “Concurrent with this work, Smith & Le (2017) derive an analytical expression for the stochastic noise scale and – based on the trade-oﬀ between depth and width in the Bayesian evidence – ﬁnd an optimal noise scale for optimizing the test accuracy.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “…related to the scale and distribution of SGD noise, e.g., the relationship between batch size, learning rate, and generalization (Keskar et al., 2017; Smith & Le, 2018; Smith et al., 2018) and the efﬁcacy of alternative learning rate schedules (Smith, 2017; Smith & Topin, 2018; Li & Arora, 2019).”
Understanding Batch Normalization (2018, background): “When normalization is turned off, smaller learning rates and longer training is necessary however we consistently ﬁnd that the highest learning rates that don’t diverge give the best results, see Table 2.”
Revisiting Small Batch Training for Deep Neural Networks (2018, methodology): “This linear scaling rule has been widely adopted, e.g., in Krizhevsky (2014), Chen et al. (2016), Bottou et al. (2016), Smith et al. (2017) and Jastrzebski et al. (2017).”
A Study on Overfitting in Deep Reinforcement Learning (2018, neutral): “Similar phenomenons are reported in supervised learning (Zhang et al., 2017), and recent work shows various approaches to explain this (Bartlett et al., 2017; Neyshabur et al., 2017; Advani & Saxe, 2017; Smith & Le, 2017; Dziugaite & Roy, 2017; Liang et al., 2017).”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “…we consider some optimization based measures based on the speed of the optimization algorithm as motivated by the work of (Hardt et al., 2015) and (Wilson et al., 2017a), and the magnitude of the gradient noise as motivated by the work of (Chaudhari and Soatto, 2018) and (Smith and Le, 2017).”
Training Tips for the Transformer Model (2018, background): “For example, Smith et al. (2017) suggest to increase the effective batch size (and number of GPUs) during training, instead of decaying the learning rate.”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “While there has been considerable excitement around heuristics that have been shown to make large batch training practical for certain problems [9, 29], we demonstrate that these techniques still suffer from the same convergence trends we observe, and they often decrease stability of the training…”
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries (2018, background): “Moreover, multiple studies have found that large batch size training suffers from poor generalization capability [16, 31].”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, neutral): “The latter comparison to SGD is suggestive that this optimization method mimics Bayesian inference – an idea that has been receiving increasing attention (Welling and Teh, 2011; Mandt et al., 2017; Smith and Le, 2018).”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, background): “The question of training with larger batch sizes is closely tied with learning rate, and many papers have empirically studied large batch/small LR phenomena [22, 18, 35, 34, 11, 41, 16, 38], particularly focusing on vision tasks using SGD as the optimizer.”