Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

By Aditya Golatkar, Alessandro Achille, Stefano Soatto. 2019

110 citations recorded by the source corpus.

Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, whose generalization gap is as bad as if regularization never happened. This suggests that what matters for training deep networks is not just whether or how, but when to regularize. The phenomena we observe are manifest in different datasets (CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), different regularization methods (weight decay, data augmentation), different learning rate schedules (exponential, piece-wise constant). They collectively suggest that there is a ``critical period'' for regularizing deep networks that is decisive of the final performance. More analysis should, therefore, focus on the transient rather than asymptotic behavior of learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, background): “Forgetting without the pretraining assumption is an interesting challenge, as it has been observed that slight perturbation of the initial critical learning period can lead to large difference in the final solution [2, 12].”
LQF: Linear Quadratic Fine-Tuning (2020, background): “For example, simply changing the multiplier for weight decay requires retraining from scratch, as optimizing the new loss starting from the previous solution gives suboptimal results [11].”
Affinity and Diversity: Quantifying Mechanisms of Data Augmentation (2020, background): “Turning off `2 regularization at the right time in training does not hurt performance [55].”
Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, result): “Recent works [1, 16] found that regularization does not need to exist in the whole training process, which is compatible with our work.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, background): “Instead of completely removing weight decay regularization as studied in [18], we find that annealing the loss incurred by weight decay regularization over the course of training is more effective.”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “Interestingly, very similar observations were made in a concurrent paper [20] which studies the effect of early relaxing of regularization procedures such as weight decay and data augmentation.”
TransGAN: Two Transformers Can Make One Strong GAN (2021, background): “We consider this locality-aware initialization as a regularizer that comes for the early training dynamics and then gradually fades away (Golatkar et al., 2019).”
The Early Phase of Neural Network Training (2020, methodology): “However, architecture, learning rate schedule, and regularization all modify the timing of the critical period, and follow-up work found that critical periods were also present for regularization, in particular weight decay and data augmentation (Golatkar et al., 2019).”
Tradeoffs in Data Augmentation: An Empirical Study (2021, background): “Turning off (cid:96) 2 at the right time does not hurt performance (Golatkar et al., 2019).”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, background): “Achille et al. (2017); Golatkar et al. (2019); Sagun et al. (2017); Keskar et al. (2017) demonstrate that adding regularization in the beginning of training affects the ﬁnal generalization disproportionately more compared to doing so later.”