Sharp Minima Can Generalize For Deep Nets

By Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio. 2017

918 citations recorded by the source corpus.

Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice. However, explaining why this is the case is still an open area of research. One standing hypothesis that is gaining popularity, e.g. Hochreiter & Schmidhuber (1997); Keskar et al. (2017), is that the flatness of minima of the loss function found by stochastic gradient based methods results in good generalization. This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Quoc V. Le
  3. Mohit Bansal
  4. Percy Liang
  5. Sergey Levine
  6. Dacheng Tao
  7. Ruslan Salakhutdinov
  8. Jianfeng Gao

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Jitendra Malik
  6. Xiaogang Wang
  7. Bernt Schiele
  8. Yi Yang

In Scaling & Learning Theory

  1. Yuanzhi Li
  2. Tengyu Ma
  3. Jason D. Lee
  4. Greg Yang
  5. Zhi-Qin John Xu
  6. Behnam Neyshabur
  7. Michael W. Mahoney
  8. Yoshua Bengio

Citations in the corpus