The Break-Even Point on Optimization Trajectories of Deep Neural Networks

By Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, Krzysztof J. Geras. 2020

203 citations recorded by the source corpus.

The early phase of training of deep neural networks is critical for their final performance. In this work, we study how the hyperparameters of stochastic gradient descent (SGD) used in the early phase of training affect the rest of the optimization trajectory. We argue for the existence of the "break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD. In particular, we demonstrate on multiple classification tasks that using a large learning rate in the initial phase of training reduces the variance of the gradient, and improves the conditioning of the covariance of gradients. These effects are beneficial from the optimization perspective and become visible after the break-even point. Complementing prior work, we also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers. In short, our work shows that key properties of the loss surface are strongly influenced by SGD in the early phase of training. We argue that studying the impact of the identified effects on generalization is a promising future direction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “…the overall distribution of eigenvalues, the maximum eigenvalue ( λ max ) at convergence (approximately 24 without SAM, 1.0 with SAM), and the bulk of the spectrum (the ratio λ max /λ 5 , commonly used as a proxy for sharpness (Jastrzebski et al., 2020); up to 11.4 without SAM, and 2.6 with SAM).”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “SGD training with large initial learning rates often leads to improved performance over training with small initial learning rates (see Li et al. (2019); Leclerc & Madry (2020); Xie et al. (2020); Frankle et al. (2020); Jastrzebski et al. (2020) for recent discussions).”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “Although having a higher learning rate tends to be helpful since it increases the probability of converging to a wider minima [34], considering a continual optimization problem, we can see it has another consequence: it contributes to the rate of change (i.”
How Do Vision Transformers Work? (2022, background): “, 2014), especially in the early phase of training (Jastrzebski et al., 2020; 2021).”
Adaptive Gradient Methods at the Edge of Stability (2022, background): “For one, provided that training is successful, the sharpness never ventures more than a bit above the stability threshold of the corresponding full-batch algorithm [21].”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, methodology): “As we discuss in §6, several recent papers have proposed stability analyses for SGD (Wu et al., 2018; Jastrzębski et al., 2020).”