On the momentum term in gradient descent learning algorithms

By Ning Qian. 1999

A momentum term is usually included in the simulations of connectionist learning algorithms. Although it is well known that such a term greatly improves the speed of learning, there have been few rigorous studies of its mechanisms. In this paper, I show that in the limit of continuous time, the momentum parameter is analogous to the mass of Newtonian particles that move through a viscous medium in a conservative force field. The behavior of the system near a local minimum is equivalent to a set of coupled and damped harmonic oscillators. The momentum term improves the speed of convergence by bringing some eigen components of the system closer to critical damping. Similar results can be obtained for the discrete time case used in computer simulations. In particular, I derive the bounds for convergence on learning-rate and momentum parameters, and demonstrate that the momentum term can increase the range of learning rate over which the system converges. The optimal condition for convergence is also analyzed.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, methodology): “Following the existing treatment of the deterministic version of the algorithm [Chiang, 1974, Qian, 1999, Yang et al., 2018, Goh, 2017], we divide our analysis two cases: overdamping and underdamping.”
Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “One can take the continuous time limit as in Qian [44], Su et al.”
Negative Momentum for Improved Game Dynamics (2018, background): “Optimization From an optimization point of view, a lot of work has been done in the context of understanding momentum and its variants (Polyak, 1964; Qian, 1999; Nesterov, 2013; Sutskever et al., 2013).”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “These benchmarks either use Adam (Kingma and Ba, 2014), AdamW (Loshchilov and Hutter, 2018), or Momentum (Qian, 1999).”
The Efficiency Misnomer (2021, background): “For instance, different optimizers use different amounts of memory on the device, for instance, SGD-Momentum (Qian, 1999) vs SAM (Foret et al., 2020).”
The Open Images Dataset V4 (2018, methodology): “For the Faster-RCNN architecture we use momentum optimizer (Qian, 1999), while for the SSD architecture we used RMS-prop All hyperparameters are kept ﬁxed across all training sets.”
Recurrent Latent Variable Networks for Session-Based Recommendation (2017, methodology): “To facilitate convergence, Nesterov momentum [23] is additionally applied during training.”
Federated Learning: Collaborative Machine Learning without Centralized Training Data (2022, positive): “The main idea of Momentum [20] is to accelerate GD in the relevant direction towards the minimum by dampening oscillating movements.”
DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “Optimizer: We use SGD optimizer with momentum [61] for all of the networks with a learning rate of 0.”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Distributed training with vanilla momentum SGD on N training nodes follows [16],”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, neutral): “For IID, Adam optimizer [51] works better than SGD with momentum [50], while for non-IID, SGD with momentum works better.”
Recent advances in convolutional neural networks (2015, background): “To solve this problem, they propose a double margin loss function which adds another margin parameter to affect the matching pairs:”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Currently, the most adopted optimizer for training a deep neural network is stochastic gradient descent (Robbins and Monro, 1951) with momentum (Qian, 1999) as well as its variants such as AdaGrad (Duchi et al., 2011), RMSprop (Hinton et al., 2012a), and Adam (Kingma and Ba, 2014).”
Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “The network parameters tuned by the old data will be overridden by the SGD optimizer using new data [11, 33], causing a drastic performance drop — catastrophic forgetting [36, 10, 28] — on the old classes.”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, background): “Examples include momentum [38], stochastic variance reduced gradient (SVRG) [21], and proximal stochastic variance reduced gradient (Prox-SVRG) [49].”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, methodology): “Our experiments cover the scope of SGD optimizers over vanilla SGD, SGD with momentum [36] and Adam [37].”
Adversarial Removal of Demographic Attributes from Text Data (2018, methodology): “We use randomly-initialized 300-dimensional embeddings, and train using SGD with Momentum (Qian, 1999) and a learning rate of 0.01 for 100 epochs.”
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks (2018, methodology): “For optimization, we used SGD with momentum (Qian, 1999; Nesterov, 1983).”