Interplay Between Optimization and Generalization of Stochastic Gradient Descent with Covariance Noise

By Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, Jimmy Ba. 2019

27 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Wen et al. (2019) propose to inject noise (sampled from the expensive empirical Fisher matrix) to large-batch SGD.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “The random noise from sampling mini-batches of data in SGD-like algorithms and random initialization of the deep neural networks, combined with the fact that there is a wide variety of local minima solutions in high dimensional optimization problem (Ge et al., 2015; Kawaguchi, 2016; Wen et al., 2019”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, background): “The question of training with larger batch sizes is closely tied with learning rate, and many papers have empirically studied large batch/small LR phenomena [22, 18, 35, 34, 11, 41, 16, 38], particularly focusing on vision tasks using SGD as the optimizer.”
Limitations of the Empirical Fisher Approximation (2019, background): “The ubiquity of the EF approximation reaches the extent that the EF is sometimes just called the Fisher [e.g., Chaudhari et al., 2017, Wen et al., 2019].”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Jastrzębski et al. (2017); Zhu et al. (2018); Wen et al. (2019) study how the noise distribution in SGD aﬀects generalization.”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, background): “The noise in SGD (Wu et al., 2020; Wen et al., 2019; Keskar et al., 2016) has long been studied in optimization.”