The Two Regimes of Deep Network Training

By Guillaume Leclerc, Aleksander Madry. 2020

52 citations recorded by the source corpus.

Learning rate schedule has a major impact on the performance of deep learning models. Still, the choice of a schedule is often heuristical. We aim to develop a precise understanding of the effects of different learning rate schedules and the appropriate way to select them. To this end, we isolate two distinct phases of training, the first, which we refer to as the "large-step" regime, exhibits a rather poor performance from an optimization point of view but is the primary contributor to model generalization; the latter, "small-step" regime exhibits much more "convex-like" optimization behavior but used in isolation produces models that generalize poorly. We find that by treating these regimes separately-and em specializing our training algorithm to each one of them, we can significantly simplify learning rate schedules.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

The large learning rate phase of deep learning: the catapult mechanism (2020, background): “SGD training with large initial learning rates often leads to improved performance over training with small initial learning rates (see Li et al. (2019); Leclerc & Madry (2020); Xie et al. (2020); Frankle et al. (2020); Jastrzebski et al. (2020) for recent discussions).”
Learning to Optimize: A Primer and A Benchmark (2021, background): “For example, it is well known that training over-parameterized networks will exhibit sharply distinct behaviors at small and large learning rates, and the two regimes usually co-exist in practical training and are separated by a phase transition [114; 115; 230].”
TRAK: Attributing Model Behavior at Scale (2023, unknown): “Prior works (Leclerc & Madry, 2020; Lewkowycz et al., 2020) suggest that neural network training can exhibit two stages or regimes: in the first stage, the features learned by the network evolve rapidly; in the second stage, the features remain approximately invariant and the overall optimization…”