Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity

By Jingzhao Zhang, Tianxing He, Suvrit Sra, Ali Jadbabaie. 2019

627 citations recorded by the source corpus.

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, \emph{gradient clipping} and \emph{normalized gradient}, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

Citations in the corpus

High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Gradient clipping is often used in language modeling to stabilize training (Merity et al., 2018), and recent work shows that it allows training with larger learning rates compared to gradient descent, accelerating convergence (Zhang et al., 2020).”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “Further, algorithmic innovations such as momentum [30, 4], adaptivity [17, 41, 42], and clipping [38, 39, 43] are critical for success in deep learning applications and need to be incorporated into the client updates, replacing the SGD update of FEDAVG.”
Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, background): “Indeed, the gradient clipping of DP-SGD does not bring the nice properties of gradient clipping commonly used to regularize deep learning (Zhang et al. 2019) because DP-SGD clips gradients at the granularity of individual training examples rather than at the level of a batch.”
Understanding Gradient Clipping in Private SGD: A Geometric Perspective (2020, background): “In Zhang et al. [2019], the divergence is characterized by a bias depending on the clipping probability.”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, positive): “Zhang et al. (2019); Mai & Johansson (2021) showed that global gradient clipping is faster than standard SGD when global smoothness does not hold.”
A novel time–frequency Transformer based on self–attention mechanism and its application in fault diagnosis of rolling bearings (2021, background): “Adam wields an adaptive gradient strategy to accelerate training error convergence and allow the training trajectory to cross non-smooth regions of the loss landscape [49].”
Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout (2020, methodology): “Apart from the traditional optimization methods [8, 19, 32, 33, 49], there has been a research thrust on developing different ways to apply gradients to deep networks [2, 7, 10, 15, 36, 45, 50].”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “For example, Zhang et al. (2020) proposed a “relaxedL-smoothness” condition that is less restrictive than standard L-smoothness, and proved a convergence guarantee for gradient descent under this condition which asserted that the training objective will decrease monotonically.”