Stochastic Modified Equations and Adaptive Stochastic Gradient Algorithms

By Qianxiao Li, Cheng Tai, E. Weinan. 2015

337 citations recorded by the source corpus.

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Further efforts to explain the generalization behavior leverage the connection of SGD to stoachstic differential equations [19, 25, 31, 33].”
Where is the Information in a Deep Neural Network? (2019, background): “(1) is a Langevin diffusion process with (non-isotropic) noise Tξk (Li et al., 2017).”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, methodology): “(This strategy was also used by Li et al. (2015) in a more restricted setting).”
Three Factors Influencing Minima in SGD (2017, methodology): “In contrast to previous work (see related work, in Section 4, e.g. Mandt et al. (2017); Li et al. (2017)), we draw attention to the fact that SGD processes with diﬀerent learning rates and batch sizes but the same ratio of learning rate to batch size are diﬀerent discretizations of the same…”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets.”
A Variational Analysis of Stochastic Gradient Algorithms (2016, neutral): “Li et al. (2015) discuss several one-dimensional cases and momentum.”
Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints (2017, background): “Besides, several recent works also studied the connection between SGD and stochastic diﬀeren-tial equations, such as SME (Li et al., 2015, 2017).”
Comparing dynamics: deep neural networks versus glassy systems (2018, background): “…the variance of the loss function’s gradient 8 , which reads (details on the deﬁnition of the noise can be found in several resources, see, for example, Li et al. (2015)): where L = 1 | train set | s ∈ train set L s is the empirical average and L s is the loss of the s -th image in the train set.”
On the adequacy of untuned warmup for adaptive optimization (2019, methodology): “This direction comes dangerously close to seeking the “holy grail” of an automatic learning rate tuner; existing attempts to devise such a method have achieved limited adoption as of yet (Li et al., 2017; Zhang et al., 2017; Baydin et al., 2018).”
On the diffusion approximation of nonconvex stochastic gradient descent (2017, methodology): “Throwing away this term reduces the weak error from O(η2) to O(η) as mentioned in [27, 13].”
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects (2018, background): “(7) can be treated as the numerical discretization of the following stochastic differential equation (SDE) (Li et al., 2017; Jastrzkebski et al., 2017; Chaudhari & Soatto, 2017),”