Deep learning generalizes because the parameter-function map is biased towards simple functions

By Guillermo Valle Pérez, Chico Q. Camargo, Ard A. Louis. 2018

293 citations recorded by the source corpus.

Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime where classical learning theory would instead predict that they would severely overfit. While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit. In this paper, we provide a new explanation. By applying a very general probability-complexity bound recently derived from algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions. We then provide clear evidence for this strong simplicity bias in a model DNN for Boolean functions, as well as in much larger fully connected and convolutional networks applied to CIFAR10 and MNIST. As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems. This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than the more conventional prior over parameter space. If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets. By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood, we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10 and for architectures including convolutional and fully connected networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “For example, it has been shown that neural networks are more likely to fit a simple function to a set of training data, even when no regularization is used.[15]”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “, 2018), deep networks’ structure biases learning towards simple functions (Neyshabur et al., 2014; Perez et al., 2018).”
Machine learning and deep learning—A review for ecologists (2022, background): “This would explain why deep neural networks display a bias towards simpler functions (De Palma et al., 2019; Valle-Pérez et al., 2019 ) that increases with the depth of the networks (Huh et al., 2021 ).”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “For example, there exist many more parameter choices which yield simple than complex functions [41].”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Recent work aims at identifying the reasons behind this phenomenon [97, 70, 22, 73, 51, 76, 100, 92, 83, 105, 42, 79, 4], while complementary work quantifies resulting shortcomings, including poor generalization to out-of-distribution (OOD) test data, reliance upon spurious correlations, and lack of”
A Fine-Grained Spectral Perspective on Neural Networks (2019, methodology): “[57] actually did their experiments on the {0, 1} boolean cube, whereas here, we have focused on the {±1} boolean cube.”
The Platonic Representation Hypothesis (2024, background): “…in the absence of external influences, deep networks naturally adhere to Occam’s razor, implicitly favoring simple solutions that fit the data (Solomonoff, 1964; Gunasekar et al., 2018; Arora et al., 2019a; Valle-Perez et al., 2019; Huh et al., 2023; Din-gle et al., 2018; Goldblum et al., 2023).”
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019, background): “Additionally, the NN-GP correspondence has led to new understanding of neural network training and generalization [42, 53, 61].”
Research Progress on Few-Shot Learning for Remote Sensing Image Interpretation (2021, background): “3052869 to be optimized [4], [5], the performance of the deep learning model heavily relies on plenty of samples.”