A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

By Akhilesh Deepak Gotmare, Nitish Shirish Keskar, Caiming Xiong, Richard Socher. 2018

319 citations recorded by the source corpus.

The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz., mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.

Subfields: Artificial Intelligence, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

iShape: A First Step Towards Irregular Shape Instance Segmentation (2021, positive): “The learning rate is initially set to 0.01 and adopt warm-up [5] strategy that the learning rate gradually rises during the ﬁrst 625 iterations.”
On the Variance of the Adaptive Learning Rate and Beyond (2019, methodology): “For SGD, we set the momentum factor as 0 .”
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “Following several recent works [20, 25, 18, 16, 21, 8, 3] we measure this by applying Canonical Correlation Analysis (CCA) to the latent representations of the network.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, background): “Recent work analyzing the loss landscape suggests the local minima reached in the loss landscape of vanilla training can be connected by a simple path (Garipov et al., 2018; Gotmare et al., 2018), e.g., a Bézier curve, with low task loss (or high task accuracy) along the path.”
Transfusion: Understanding Transfer Learning for Medical Imaging (2019, methodology): “A recent tool that effectively overcomes these challenges is (Singular Vector) Canonical Correlation Analysis, (SV)CCA [22, 19], which has been used to study latent representations through training, across different models, alternate training objectives, and other properties [22, 19, 25, 18, 8, 17, ”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, methodology): “These techniques have been very successful in analyzing deep learning, from properties of neural network training (Gotmare et al., 2018; Neyshabur et al., 2020), objectives (Resnick et al., 2019; Thompson et al., 2019; Hermann & Lampinen, 2020), and dynamics (Maheswaranathan et al., 2019) to…”
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, background): “Previous schemes suggested that larger batches require larger learning rates or a dedicated optimizer [43, 10].”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, result): “This improvement has been attributed to the ease of predicting the output distribution of the teacher compared to the original ‘hard’ labels (Mobahi et al., 2020; Gotmare et al., 2019).”
Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints (2019, background): “They gave the intuition for this heuristics as escaping spurious local minima, which has been effectively questioned by [21].”
R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “Different from existing method that exploits dark knowledge from the model itself [20, 19] or distills knowledge between different layers [75], our strategy can be regarded as an instance-wise self-knowledge distillation, i.e., each pair of sampled sub models perform distillation between each other…”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, background): “(2020), or (b) yield trivial network connectivity results, such as non-linear low-loss paths, which can be found for any two network minimizers (Draxler et al., 2018; Garipov et al., 2018a; Gotmare et al., 2019; Fort & Jastrzebski, 2019).”