Step Size Matters in Deep Learning

By Kamil Nar, S. Sastry. 2018

54 citations recorded by the source corpus.

Training a neural network with the gradient descent algorithm gives rise to a discrete-time nonlinear dynamical system. Consequently, behaviors that are typically observed in these systems emerge during training, such as convergence to an orbit but not to a fixed point or dependence of convergence on the initialization. Step size of the algorithm plays a critical role in these behaviors: it determines the subset of the local optima that the algorithm can converge to, and it specifies the magnitude of the oscillations if the algorithm converges to an orbit. To elucidate the effects of the step size on training of neural networks, we study the gradient descent algorithm as a discrete-time dynamical system, and by analyzing the Lyapunov stability of different solutions, we show the relationship between the step size of the algorithm and the solutions that can be obtained with this algorithm. The results provide an explanation for several phenomena observed in practice, including the deterioration in the training error with increased depth, the hardness of estimating linear mappings with large singular values, and the distinct performance of deep residual networks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tal Ben-Nun
2. Torsten Hoefler
3. Daniel Soudry
4. Itay Hubara
5. Elad Hoffer
6. J. Kolter
7. Quoc V. Le
8. Niv Giladi

In Statistical and Nonlinear Physics

1. Itay Hubara
2. Daniel Soudry
3. Elad Hoffer
4. Tal Ben-Nun
5. J. Kolter
6. Ameet Talwalkar
7. Torsten Hoefler
8. Shankar Krishnan

Citations in the corpus

• Adaptive Gradient Methods at the Edge of Stability (2022, background): “Meanwhile, a line of recent work [34, 43, 20, 21, 8, 15] has shed light on the neural network training dynamics of non-adaptive gradient descent (both with and without momentum) in, at least, the full-batch and (sufficiently) large-batch regimes.”
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, methodology): “Intuitively, gradient descent with fixed step sizes acts like a constrained optimization algorithm: the use of step size η imposes an implicit 2/η constraint on the sharpness (Nar & Sastry, 2018), and at the Edge of Stability this constraint is “active.””
• Augment your batch: better training with larger batches (2019, background): “Previous works (Keskar et al., 2017; Nar & Sastry, 2018; Wu et al., 2018) suggested these issues may result from an implicit bias in the SGD training process: with large batch sizes, SGD selects minima with worse generalization.”