Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks

By Yuanzhi Li, Colin Wei, Tengyu Ma. 2019

341 citations recorded by the source corpus.

Stochastic gradient descent with a large initial learning rate is a widely adopted method for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is annealed. Towards explaining this phenomenon, we devise a setting in which we can prove that a two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start. The key insight in our analysis is that the order of learning different types of patterns is crucial: because the small learning rate model first memorizes low noise, hard-to-fit patterns, it generalizes worse on higher noise, easier-to-fit patterns than its large learning rate counterpart. This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing. Our experiments show that this causes the small learning rate model's accuracy on unmodified images to suffer, as it relies too much on the patch early on.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “The learning rate influences which patterns networks focus on: Large learning rates lead to learning simple patterns that are shared across examples, while small learning rates facilitate complex pattern learning and memorisation [116, 117].”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “SGD training with large initial learning rates often leads to improved performance over training with small initial learning rates (see Li et al. (2019); Leclerc & Madry (2020); Xie et al. (2020); Frankle et al. (2020); Jastrzebski et al. (2020) for recent discussions).”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, neutral): “[4, 7, 8, 13, 14, 26–29, 38, 45, 50, 57, 61, 104, 115]).”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, background): “Other training techniques can also be used to regularize the network, such as early-stopping [Ulyanov et al. 2018; Williams et al. 2019], dropout [Srivastava et al. 2014], and learning rate decay [Li et al. 2019b].”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, background): “A natural direction for the future is connecting our observations to recent studies on the relation of measures, such as gradient variance, to the generalization of deep networks (Li et al., 2019; Jiang et al., 2020; Fort et al., 2019).”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, background): “In addition, prior work has not proposed that the beneﬁts of augmentation multiplicity are connected to the implicit regularization beneﬁt of SGD [Keskar et al., 2017, Li et al., 2019, Smith et al., 2021].”
The Two Regimes of Deep Network Training (2020, methodology): “The predominant hypothesis is that is acts as regularizer (Li et al., 2019; Hoffer et al., 2017).”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Interpretation: It is a major open question in deep learning theory to understand the regularization eﬀects of noise (Li et al., 2019).”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, neutral): “Our observations seem to coincide with the recent findings by (Achille et al., 2019; Li et al., 2019) about the two-stage optimization trajectory in training.”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, background): “More recent works study the implicit biases induced on MLPs by gradient descent, for both the “NTK” and “adaptive” regimes (Bietti & Mairal, 2019; Chizat & Bach, 2018; Li et al., 2019; Song et al., 2018).”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “, ensemble) in deep learning works very differently from model averaging in random feature mappings — in particular, different from the neural tangent kernel (NTK) approach [3, 6, 7, 9, 10, 20, 25, 27, 28, 40, 41, 45, 55, 59, 86, 92].”
Making Method of Moments Great Again? -- How can GANs learn the target distribution (2020, background): “On the other hand, unlike in supervised training where the training objective involving a (over-parameterized) deep neural network can be approximately convex [9, 48, 4, 10, 31, 15, 14, 80, 30, 27, 42, 36, 52, 40, 74, 26, 81, 26], to the best of our knowledge, there are also no theoretical result sh”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, background): “2 With huge non-convexity obstacles, existing theoretical works, to the best of our knowledge, have only been able to demonstrate that multi-layer neural networks can efficiently perform learning tasks that are already known solvable by non-hierarchical methods, such as kernel methods [3, 6, 7, 9, 1”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “…and Gu (2019); Du et al. (2018c); Arora et al. (2019a); Allen-Zhu and Li (2019b); Allen-Zhu et al. (2019c,b); Li and Liang (2018); Zou et al. (2018); Du et al. (2018a); Daniely et al. (2016); Ghorbani et al. (2019); Li et al. (2019a); Hanin and Nica (2019); Yang (2019) and the references therein.”