Optimal Stochastic Search and Adaptive Momentum

By Todd K. Leen, Genevieve B. Orr. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “…notions of critical batch size, but to our knowledge, all for SGD. Additionally, previous studies have shown that SGD and momentum SGD are equivalent for small learning rates (after appropriate rescaling), both for the continuous limit [Leen and Orr, 1994] and discrete settings Yuan et al. [2016].”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “ibutions of many stochastic gradients cancel each other out. Theoretical guarantees about the inclusion of momentum in stochastic settings are elusive, and although practical gains have been reported [92,150], more experimentation is needed. 69 7.2 Accelerated Gradient Methods A method with an itera”
YellowFin and the Art of Momentum Tuning (2017, background): “Existing adaptive momentum approach either consider the deterministic setting [33, 34, 35, 36, 37, 38] or only analyze the stochastic setting with O(1/t) learning rate [39].”
Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent (2002, background): “The investigation of asymptotically optimal adaptivemomentumforerst-orderstochasticgradientdescent( Leen&Orr, 1994 ) led Orr (1995) to propose the following weight update:”