On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length

By Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos J. Storkey. 2018

145 citations recorded by the source corpus.

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly understood. An empirical investigation shows that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. When studying the SGD dynamics in relation to the sharpest directions in this initial phase, we find that the SGD step is large compared to the curvature and commonly fails to minimize the loss along the sharpest directions. Furthermore, using a reduced learning rate along these directions can improve training speed while leading to both sharper and better generalizing solutions compared to vanilla SGD. In summary, our analysis of the dynamics of SGD in the subspace of the sharpest directions shows that they influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the overall training speed, and the generalization ability of the final model.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Consistently with prior work (Keskar et al., 2017; Jastrzebski et al., 2018), we observe that using a smaller η or using a larger S coincides with a larger maximum value of λ 1 H .”
Understanding Regularisation Methods for Continual Learning (2020, result): “This is consistent with previous empirical studies showing that the loss has positive curvature [18, 23].”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “(2020) aimed to explain the observation (Jastrzębski et al., 2019) that the SGD learning rate and batch size hyperparameters strongly influence the local geometry along the optimization trajectory, with large learning rates and small batch sizes guiding SGD into regions of the loss landscape charact”
Adaptive Gradient Methods at the Edge of Stability (2022, background): “In the more general case of SGD, similar effects seem to occur [20, 21], as we review in §5.”
Large Scale Structure of Neural Network Loss Landscapes (2019, background): “On the other end of the spectrum, Jastrz˛ebski et al. [2018] and Xing et al. [2018] characterize the shape of the sharpest directions in which optimization happens.”
Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin (2019, background): “Keskar et al. [2016], Chaudhari et al. [2016], Yao et al. [2018], Jastrzebski et al. [2018] study the relationship between generalization and “sharpness” of local minima in the context of small v.s. large batch SGD.”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “There has been a large body of prior work studying the relationship between gradient noise and generalization (Keskar et al., 2016; Keskar & Socher, 2017; Smith & Le, 2017; Jastrzebski et al., 2018; Xing et al., 2018; Li et al., 2019; Chaudhari & Soatto, 2018).”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, background): “It has been extensively studied that the convergence to a ﬂat region whose curvature is small beneﬁts the generalization of neural networks (Keskar et al., 2017; Kleinberg et al., 2018; Jastrz˛ebski et al., 2019; Chen & Hsieh, 2020; Smith & Le, 2018; Zela et al., 2020; Chaudhari et al., 2017).”
Pruning neural networks: is it time to nip it in the bud? (2018, unknown): “These observations are consistent with those made in previous works (Veit et al., 2016; Huang et al., 2016; Jastrze¸bski et al., 2018).”
On Learning Rates and Schrödinger Operators (2020, background): “Moreover, [JKA + 17, JKB + 18] related the learning rate to the generalization performance of neural networks in the early phase of training.”
Emergent properties of the local geometry of neural loss landscapes (2019, background): “Interestingly, a recent study [13] found that the spectral norm of the Hessian, as measured by the top Hessian eigenvalue, displays a non-monotonic dependence on training time.”