A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast

By Zeke Xie, Issei Sato, Masashi Sugiyama. 2020

18 citations recorded by the source corpus.

Stochastic optimization algorithms, such as Stochastic Gradient Descent (SGD) and its variants, are mainstream methods for training deep networks in practice. However, the theoretical mechanism behind gradient noise still remains to be further investigated. Deep learning is known to find flat minima with a large neighboring region in parameter space from which each weight vector has similar small error. In this paper, we focus on a fundamental problem in deep learning,"How can deep learning usually find flat minima among so many minima?"To answer the question, we develop a density diffusion theory (DDT) for revealing the fundamental dynamical mechanism of SGD and deep learning. More specifically, we study how escape time from loss valleys to the outside of valleys depends on minima sharpness, gradient noise and hyperparameters. One of the most interesting findings is that stochastic gradient noise from SGD can help escape from sharp minima exponentially faster than flat minima, while white noise can only help escape from sharp minima polynomially faster than flat minima. We also find large-batch training requires exponentially many iterations to pass through sharp minima and find flat minima. We present direct empirical evidence supporting the proposed theoretical results.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Stefano Soatto
2. Mehrdad Farajtabar
3. Alessandro Achille
4. Razvan Pascanu
5. Seyed Iman Mirzadeh
6. Yoshua Bengio
7. Sergey Levine
8. Jascha Narain Sohl-Dickstein

In Statistics and Probability

1. Alessandro Achille
2. Yasaman Bahri
3. Jascha Narain Sohl-Dickstein
4. Ethan Dyer
5. Giovanni Paolini
6. Mehrdad Farajtabar
7. Stefano Soatto
8. Razvan Pascanu

In Optimization & Training

1. Alessandro Achille
2. Stefano Soatto
3. Razvan Pascanu
4. Seyed Iman Mirzadeh
5. Jascha Narain Sohl-Dickstein
6. Hassan Ghasemzadeh
7. Mehrdad Farajtabar
8. Giovanni Paolini

In Scaling & Learning Theory

1. Stefano Soatto
2. Jascha Narain Sohl-Dickstein
3. Yasaman Bahri
4. Ethan Dyer
5. Alessandro Achille
6. Guy Gur-Ari
7. Hassan Ghasemzadeh
8. Seyed Iman Mirzadeh

Citations in the corpus

• The large learning rate phase of deep learning: the catapult mechanism (2020, background): “SGD training with large initial learning rates often leads to improved performance over training with small initial learning rates (see Li et al. (2019); Leclerc & Madry (2020); Xie et al. (2020); Frankle et al. (2020); Jastrzebski et al. (2020) for recent discussions).”
• Understanding the Role of Training Regimes in Continual Learning (2020, background): “The learning rate and batch size influence both the endpoint curvature and the whole trajectory [77, 17, 45].”
• Where is the Information in a Deep Neural Network? (2019, background): “However, recent work extends this result to anisotropic noise (Xie et al., 2020; Cheng et al., 2019).”