Aggregated Momentum: Stability Through Passive Damping

By James Lucas, Richard S. Zemel, Roger Baker Grosse. 2018

75 citations recorded by the source corpus.

Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient $\beta$. Large $\beta$ values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different $\beta$ parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive $\beta$ values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Sorting out Lipschitz function approximation (2018, methodology): “We used the Aggretated Momentum optimizer (AggMo) (Lucas et al., 2018) with its default parameters.”
Lookahead Optimizer: k steps forward, 1 step back (2019, background): “[23], to better understand the sensitivity of Lookahead to misspecified conditioning we fix the momentum coefficient of classical momentum and explore the convergence rate over varying condition number under the optimal learning rate.”
Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, background): “We note that recently it has been shown by [26], that there are problems where NAG generalizes better than ADAM even after tuning β 1 .”
Quasi-hyperbolic momentum and Adam for deep learning (2018, methodology): “• Appendix H provides discussion and an empirical comparison of QHM and AggMo (Lucas et al., 2018).”
Understanding and correcting pathologies in the training of learned optimizers (2018, background): “The MLP for each weight takes as input: the gradient with respect to that weight, the parameter value, exponentially weighted moving averages of gradients at multiple time scales (Lucas et al., 2018), as well as a representation of the current iteration number.”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, methodology): “We test optimizers using only a single momentum accumulator with (5)This is similar to what is done in the AggMo (Lucas et al., 2018) hand-designed optimizer.”