Understanding deep learning (still) requires rethinking generalization

By Chiyuan Zhang, Samy Bengio, Moritz Hardt, B. Recht, Oriol Vinyals. 2021

2,784 citations recorded by the source corpus.

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Percy Liang
  3. Mohit Bansal
  4. Philip S. Yu
  5. Yoshua Bengio
  6. Dacheng Tao
  7. Luke S. Zettlemoyer
  8. Yonatan Belinkov

In Scaling & Learning Theory

  1. Tal Linzen
  2. R. Thomas McCoy
  3. Tongliang Liu
  4. Simon Lacoste-Julien
  5. Prateek Mittal
  6. Philip S. Yu
  7. Shuang Song
  8. James Y. Zou

Citations in the corpus