The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects

By Zhanxing Zhu, Jingfeng Wu, Bin Yu, Lei Wu, Jinwen Ma. 2018

292 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Recent works (Jastrzębski et al., 2018; Li et al., 2017; Mandt et al., 2017; Zhu et al., 2019) interpret eq.”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, unknown): “The gradient noise of SGD is considered to have an essential correlation with the generalization ability of the deep models [6, 13, 42, 44, 48].”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, background): “In terms of escaping from sharp minima, there are previous works (Zhu et al., 2019; Daneshmand et al., 2018) that study the inheret noise in SGD, showing that the noise covariance containing curvature information performs better.”
Neural networks with late-phase weights (2020, background): “Zhu et al. (2018) highlighted this trace quantitiy in equation 23 as a measurement of the escaping efficiency out of poor minima.”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “A similar observation has been made in Zhu et al. (2018) where the authors studied the effects of how using Fisher covariance noise relates to the efficiency of escaping from local minima and compare it to isotropic noise.”
A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast (2020, background): “(Zhu et al., 2019) shows that anisotropic noise helps escape from sharp minima.”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “Finally, a number of papers have attempted to mathematically model the propensity of SGD to “escape from sharp minima,” though sometimes using a different definition of “sharpness” than the one employed in this paper (Zhu et al., 2019; Xie et al., 2020b).”
Limitations of the Empirical Fisher Approximation (2019, methodology): “It has also been employed to explain certain characteristics of SGD [Zhu et al., 2018, Jastrzebski et al., 2018] or as a diagnostic tool during training [Liao et al., 2018].”