Three Factors Influencing Minima in SGD

By Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos J. Storkey. 2017

546 citations recorded by the source corpus.

We study the statistical properties of the endpoint of stochastic gradient descent (SGD). We approximate SGD as a stochastic differential equation (SDE) and consider its Boltzmann Gibbs equilibrium distribution under the assumption of isotropic variance in loss gradients.. Through this analysis, we find that three factors – learning rate, batch size and the variance of the loss gradients – control the trade-off between the depth and width of the minima found by SGD, with wider minima favoured by a higher ratio of learning rate to batch size. In the equilibrium distribution only the ratio of learning rate to batch size appears, implying that it’s invariant under a simultaneous rescaling of each by the same amount. We experimentally show how learning rate and batch size affect SGD from two perspectives: the endpoint of SGD and the dynamics that lead up to it. For the endpoint, the experiments suggest the endpoint of SGD is similar under simultaneous rescaling of batch size and learning rate, and also that a higher ratio leads to flatter minima, both findings are consistent with our theoretical analysis. We note experimentally that the dynamics also seem to be similar under the same rescaling of learning rate and batch size, which we explore showing that one can exchange batch size and learning rate in a cyclical learning rate schedule. Next, we illustrate how noise affects memorization, showing that high noise levels lead to better generalization. Finally, we find experimentally that the similarity under simultaneous rescaling of learning rate and batch size breaks down if the learning rate gets too large or the batch size gets too small.

Subfields: Molecular Biology, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Optimization & Training

Citations in the corpus

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “…the batch size initially achieves perfect scaling (i.e. doubling the batch size halves the number of steps needed) up to a problem-dependent critical batch size, beyond which it yields diminishing returns [Balles et al., 2017, Goyal et al., 2017, Jastrzębski et al., 2018, McCandlish et al., 2018].”
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], quali”
Why Should We Add Early Exits to Neural Networks? (2020, background): “1 Department of Information Engineering, Electronics and Telecommunications (DIET), “Sapienza” University of Rome, Via Eudossiana 18, 00184 Rome, Italy Thirdly, overfitting and vanishing gradient phenomena can still happen even with strong regularization, due to the possibility of raw memorization i”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “We argue that the main reason behind this is the reduced noise magnitude, which comes from the break down of γ Bglob scaling for large batch size and smaller dataset size when Bglob 6 N (Jastrzębski et al., 2017).”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “We likewise evaluate the effect of LiDER both in terms of its resilience to weight perturbations [57, 39] and the eigenvalues of the Hessian of the loss function [39, 34] in Fig.”
Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “One common technique of analysis in this direction is to measure the largest eigenvalues of the Hessian of the loss function, which quantifies the local curvature around minima [39, 16, 33, 32, 33].”
Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “Furthermore, [6, 18, 19] propose measuring flatness by evaluating the eigenvalues of ∇(2)θL: sharper minima correspond to larger Hessian eigenvalues.”
Understanding Batch Normalization (2018, background): “This has indeed been observed in practice in the context of parallelizing neural networks [14, 49] and derived in other theoretical models [24].”
Three Mechanisms of Weight Decay Regularization (2018, background): “Since higher learning rates lead to larger gradient noise, which has been shown to act as a stochastic regularizer (Neelakantan et al., 2015; Keskar et al., 2016; Jastrzębski et al., 2017; Hoffer et al., 2017), this means weight decay can indirectly exert a regularizing effect through the effective ”
Training Tips for the Transformer Model (2018, background): “Jastrzebski et al. (2017) shows that “ the invariance under simultaneous rescaling of learning rate and batch size breaks down if the learning rate gets too large or the batch size gets too small ”.”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “Recent work studying large batch training has looked primarily at image classification [14, 31], especially on the ImageNet dataset [6].”
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2018, background): “In particular, it is a popular view that models corresponding to wide minima of the loss in the parameter space generalize better than those corresponding to sharp minima (Hochreiter and Schmidhuber, 1997; Keskar et al., 2016; Jastrzębski et al., 2017; Wang et al., 2018).”
Optimal SGD Hyperparameters for Fully Connected Networks (2018, background): “As discussed in [1, 2, 9, 5, 11], the continuum limit of stochastic gradient descent is described by the stochastic differential equation dwi = −∂iLτdt+ dξi , E[dξidξj ]|w,t = 2gFij(w)dt , (4) where wi is used to denote the parameters of the neural network, and dξi is random noise.”
On the Intrinsic Privacy of Stochastic Gradient Descent (2019, background): “The properties of SGD have since become an object of study [6], [39], [45], with particular focus on the generalisation performance of the solutions it finds [14], [22], [25].”
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019, background): “As observed in (Mandt et al., 2017; Chaudhari & Soatto, 2017; Jastrzebski et al., 2017; Smith & Le, 2017), and reviewed in section 2.2, when the learning rate is sufficiently small, the behaviour of SGD is determined by the noise scale g, where for SGD, g = Ntrain B , (2) and for SGD with…”