A Walk with SGD

By Chen Xing, Devansh Arpit, Christos Tsirigotis, Yoshua Bengio. 2018

145 citations recorded by the source corpus.

Exploring why stochastic gradient descent (SGD) based optimization methods train deep neural networks (DNNs) that generalize well has become an active area of research. Towards this end, we empirically study the dynamics of SGD when training over-parametrized DNNs. Specifically we study the DNN loss surface along the trajectory of SGD by interpolating the loss surface between parameters from consecutive \textit{iterations} and tracking various metrics during training. We find that the loss interpolation between parameters before and after a training update is roughly convex with a minimum (\textit{valley floor}) in between for most of the training. Based on this and other metrics, we deduce that during most of the training, SGD explores regions in a valley by bouncing off valley walls at a height above the valley floor. This 'bouncing off walls at a height' mechanism helps SGD traverse larger distance for small batch sizes and large learning rates which we find play qualitatively different roles in the dynamics. While a large learning rate maintains a large height from the valley floor, a small batch size injects noise facilitating exploration. We find this mechanism is crucial for generalization because the valley floor has barriers and this exploration above the valley floor allows SGD to quickly travel far away from the initialization point (without being affected by barriers) and find flatter regions, corresponding to better generalization.

Subfields: Genetics, Molecular Biology, Physiology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

In Physiology

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “In the non-convex setting the noise effects how the optimization trajectory navigates the loss surface (Xing et al., 2018; Jastrzębski et al., 2017).”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “There are two motivations for this rule: the first assumes that one large-batch gradient step should resemble a series of small-batch gradient steps in order for convergence rates to improve linearly [9]; the other regards the SGD update equation as the Euler-Maruyama discretization of a stochastic ”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Loss landscapes (Xing et al., 2018) are similar to training dynamics, but also consider variables from the stochastic optimization algorithm.”
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, methodology): “As a side product of our experiments, we also examine the distance that θ t has traversed in the parameter space, i.e., θ t − θ 0 2 , as shown in many other studies (Hoﬀer et al., 2017; Xing et al., 2018) discussing the generalization performance of various optimization algorithms for deep learning.”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, background): “A number of papers have proposed explanations for this phenomenon, such as sharpness of the local minima [22, 20, 24], the time it takes to move from initialization [18, 40], and the scale of SGD noise [38].”
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (2019, background): “Besides, during the dynamics of SGD it has been observed that the algorithm does not cross barriers except perhaps at the very initial phase (Xing et al., 2018; Baity-Jesi et al., 2018).”
Large Scale Structure of Neural Network Loss Landscapes (2019, background): “On the other end of the spectrum, Jastrz˛ebski et al. [2018] and Xing et al. [2018] characterize the shape of the sharpest directions in which optimization happens.”
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2018, methodology): “Our work is related to studies on SGD dynamics such as Goodfellow et al. (2014); Chaudhari and Soatto (2017); Xing et al. (2018); Zhu et al. (2018).”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “There has been a large body of prior work studying the relationship between gradient noise and generalization (Keskar et al., 2016; Keskar & Socher, 2017; Smith & Le, 2017; Jastrzebski et al., 2018; Xing et al., 2018; Li et al., 2019; Chaudhari & Soatto, 2018).”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Our theoretical model is motivated by recent work on learning dynamics of neural networks (Goodfellow et al., 2014; Masters & Luschi, 2018; Wu et al., 2018; Yao et al., 2018; Xing et al., 2018; Jastrzebski et al., 2018; Lan et al., 2019).”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “Certain aspects of the Edge of Stability have been observed in previous empirical studies of fullbatch gradient descent (Xing et al., 2018; Wu et al., 2018); our paper provides a unified explanation for these observations.”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “More direct approaches of studying the covariance structure of mini-batch gradients have also been explored (Jastrzebski et al., 2017; Xing et al., 2018; Zhu et al., 2018; Li et al., 2015).”
A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay (2018, background): “Several recent papers discuss the use of large learning rate and small batch size, such as Jastrzebski et al. (2017a;b); Xing et al. (2018).”
Interplay Between Optimization and Generalization of Stochastic Gradient Descent with Covariance Noise (2019, background): “…and generalization is not completely explicit, numerous works have suggested that the maximum eigenvalue of the Hessian is possibly correlated with generalization performance (Keskar et al., 2016; Chaudhari et al., 2016; Chaudhari & Soatto, 2017; Yoshida & Miyato, 2017; Xing et al., 2018).”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “Hy-perparameters details are given in Appendix A. Learning Rate Schedule During pretraining, we use a modified trapezoidal Learning Rate (LR) schedule (Xing et al., 2018), also known as Warmup-Stable-Decay (WSD) (Zhai et al., 2022; Hu et al., 2024).”