In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

By Behnam Neyshabur, Ryota Tomioka, Nati Srebro. 2014

729 citations recorded by the source corpus.

We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks (2019, background): “The implicit regularization provided by the choice of the optimization algorithm has recently become an active area of research in machine learning, putting lot of interest on the behavior of gradient descent on deep over-parametrized models [Neyshabur et al., 2015b, 2017, Zhang et al., 2017].”
A Closer Look at Memorization in Deep Networks (2017, background): “Neyshabur et al. (2014) note that the number of parameters does not con-trol the effective capacity of a DNN, and that the reason for DNNs’ generalization is unknown.”
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “On the other hand, the empirical observations [24, 31] suggest that practical networks are highly overparameterized, so our intuition may still be helpful there.”
Understanding deep learning requires rethinking generalization (2016, background): “This question was raised in a thought-provoking work by Neyshabur et al. (2014), who argued through experiments that network size is not the main form of capacity control for neural networks.”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Even with sufficient capacity to memorize training data, networks naturally learn simpler functions (Zhang et al., 2016; Neyshabur et al., 2014; Arpit et al., 2017).”
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data (2017, positive): “This is the key insight of Neyshabur, Tomioka, and Srebro.”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “…gradient descent converges towards them (Hochreiter & Schmidhuber, 1997; Kleinberg et al., 2018), gradient descent protects against overﬁtting (Advani & Saxe, 2017; Tachet et al., 2018), deep networks’ structure biases learning towards simple functions (Neyshabur et al., 2014; Perez et al., 2018).”
Exploring Generalization in Deep Learning (2017, background): “Moreover, measuring complexity in terms of number of parameters cannot explain the reduction in generalization error as the number of hidden units increase [20] (see also Figure 4).”
The Implicit Bias of Gradient Descent on Separable Data (2017, background): “…increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models (Neyshabur et al., 2014, 2015; Zhang et al., 2017; Keskar et al., 2017; Neyshabur et al., 2017; Wilson et al., 2017).”
Stronger generalization bounds for deep nets via a compression approach (2018, background): “But Zhang et al. [2017] questioned this received wisdom and fueled research in this area by showing experimentally that standard architectures using SGD and regularization can still reach low training error on randomly labeled examples (which clearly won’t generalize).”
Path-SGD: Path-Normalized Optimization in Deep Neural Networks (2015, neutral): “This view is similar to observations in [11] on the role of implicit regularization in deep learning.”
Fantastic Generalization Measures and Where to Find Them (2019, background): “Deep neural networks have seen tremendous success in a number of applications, but why (and how well) these models generalize is still a mystery (Neyshabur et al., 2014; Zhang et al., 2016; Recht et al., 2019).”
Does learning require memorization? a short tale about a long tail (2019, background): “Examples of this approach in the context of DNNs can be found in [NTS15; NBMS17; BFT17; NTSS17] (and references therein).”
Size-Independent Sample Complexity of Neural Networks (2017, background): “One of the major challenges involving neural networks is explaining their ability to generalize well, even if they are very large and have the potential to overﬁt the training data [Neyshabur et al., 2014, Zhang et al., 2016].”
Just Interpolate: Kernel "Ridgeless" Regression Can Generalize (2018, methodology): “Our analysis can then be viewed in the light of recent work on implicit regularization of optimization procedures [30, 23, 17, 21].”
Implicit Regularization in Matrix Factorization (2017, background): “In particular, implicit regularization has been shown to play a crucial role in training deep models [14, 13, 18, 11]: deep models often generalize well even when trained purely by minimizing the training error without any explicit regularization, and when there are more parameters than samples and ”
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018, methodology): “Surprisingly, in the case of neural networks we notice that increasing the model size only helps in improving the generalization error, even when the networks are trained without any explicit regularization - weight decay or early stopping [13, 26, 21].”