Entropy-SGD: biasing gradient descent into wide valleys

By Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Tour Chayes, Levent Sagun, Riccardo Zecchina. 2016

882 citations recorded by the source corpus.

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “It has been proposed that noise helps SGD escape “sharp minima” which generalize poorly (Hochreiter & Schmidhuber, 1997; Chaudhari et al., 2016; Keskar et al., 2016; Smith & Le, 2017).”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “…loss landscape E(X,Y )∼P[`(f(X), Y )], and of the geometry of non-unique risk minimizers, including discussions of wide or narrow optima (see, e.g. Chaudhari et al., 2019), and connectivity between global modes in the context of model averaging (Izmailov et al., 2018; Fort et al., 2019; Wilson…”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “…algorithms that speciﬁcally seek out ﬂatter minima and furthermore effectively improve generalization on a range of state-of-the-art models have thus far been elusive (e.g., see (Chaudhari et al., 2016; Izmailov et al., 2018); we include a more detailed discussion of prior work in Section 5).”
A Closer Look at Memorization in Deep Networks (2017, background): “, 2013), or the shape of the loss minima found by SGD (Im et al., 2016; Chaudhari et al., 2016; Keskar et al., 2016).”
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “Generalization properties of stochastic gradient descent is extensively studied empirically [34, 28, 50, 16, 29, 24, 25].”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “Alternative explanations are proffered through the Bayesian view of learning (MacKay, 1992), and through the lens of free Gibbs energy; see e.g. Chaudhari et al. (2016).”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, methodology): “Based on exactly this observation combined with the K -replica method of the previous paragraph, a recent work produced promising practical results [Chaudhari et al., 2016].”
Wireless Network Intelligence at the Edge (2018, methodology): “Solving this PDE using a homogenization technique [97], [173] yields the loss dynamics that is identical to the elastic SGD’s loss dynamics.”
Visualizing the Loss Landscape of Neural Nets (2017, methodology): “[3] used local entropy as a measure of sharpness, which is invariant to the simple transformation in [7], but difficult to accurately compute.”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “We hypothesized that this might be related to the phenomenon of sharp local minima [Hochreiter and Schmidhuber, 1997, Keskar et al., 2016, Chaudhari et al., 2017, Yao et al., 2018], which have also been observed in the hyperparameter space [Nguyen et al., 2018].”
Critical Learning Periods in Deep Neural Networks (2017, background): “It is also worth noticing how there is evidence that convergence to flat minima (minima with low curvature) in a DNN correlates with a good generalization performance (Hochreiter & Schmidhuber, 1997; Li et al., 2018; Chaudhari et al., 2017; Keskar et al., 2017).”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, background): “Solutions that reside in wide valleys instead of narrow crevices generalize better (Chaudhari et al., 2019; Hochreiter & Schmidhuber, 1997; Keskar et al., 2016) as the predictions do not change drastically with small perturbations.”
SWAD: Domain Generalization by Seeking Flat Minima (2021, background): “Since the importance of flatness in loss landscapes has emerged [23–28], several methods have been proposed to find flat minima [25, 26, 39].”