Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks

By Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, Nati Srebro. 2018

403 citations recorded by the source corpus.

Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes, and could potentially explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, neutral): “is correctly classified) [24, 25, 26, 27].”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Even though the overparametrization of NNs is pervasive (Neyshabur et al., 2018), many current CL approaches promote the expansion of parameter space.”
Continual Learning via Neural Pruning (2019, methodology): “lance to [3], a forbear of clustering based methods put forward in the 90s. 2 Methodology The core idea of our method is to take advantage of the fact that neural networks are vastly overparametrized [14]. A manifestation of this over-parametrization is through the practice of sparsiﬁcation, i.e. th”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “…measures that we consider in this work belong to a few different families: PAC-Bayes (McAllester, 1999; Dziugaite and Roy, 2017; Neyshabur et al., 2017); VC-dimension (Vapnik and Chervonenkis, 1971); and norm-based bounds (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a).”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Beyond the jamming point, the accuracy keeps steadily improving as the number of parameters increases [25, 26, 27], although it does so quite slowly.”
Uniform convergence may be unable to explain generalization in deep learning (2019, methodology): “Finally, we defer experiments conducted for other varied settings, and the neural network bound from Neyshabur et al. (2019), to Appendix A. Notably, the m-dependencies observed in this discussion are more pronounced with larger depth, and less pronounced with larger batch sizes.”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “4 suggests an n “ r ΩpLm2 ́2q sample complexity, which matches existing Rademacher complexity-based generalization error bounds (Bartlett et al., 2017; Neyshabur et al., 2017a,b; Golowich et al., 2017; Li et al., 2018a; Neyshabur et al., 2018).”
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, positive): “Our approach and [18] differ from classic learning theory for generalization bound of neural network [17, 4, 1].”
On the Margin Theory of Feedforward Neural Networks (2018, background): “We are interested in normalized margin because its inverse bounds the generalization error (see recent work (Bartlett et al., 2017; Neyshabur et al., 2017a, 2018) and our Theorem 3.1).”
A Survey on Vision Transformer (2020, background): “over-parameterization [151], [163] may be a potential point to the interpretability of vision transformers.”
Generalization bounds for deep convolutional neural networks (2019, background): “) Even more recently, authors have proved generalization bounds in terms of the distance from the initial setting of the weights instead of the size of the weights (Dziugaite and Roy, 2017; Bartlett et al., 2017; Neyshabur et al., 2019; Nagarajan and Kolter, 2019).”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “2B-C), and beyond this point the accuracy keeps improving as the number of parameters increases [32, 31, 4], although it does so quite slowly.”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “On the one hand, several recent works studying the role of optimization in this tradeoff argue that model size is not always a good predictor for overfitting [29, 37, 28, 17, 6], and consider instead other complexity measures of the function class, which favor CNNs due to their smaller complexity [1”
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee (2019, background): “r at ICLR 2020 These two choices of regularization are well motivated with clear intuitions. First, distance to initialization has been observed to be very related to generalization in deep learning (Neyshabur et al., 2019;Nagarajan and Kolter,2019), so regularizing by distance to initialization can”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “In the lazy regime, generalization guarantees can be derived from the distance traveled by the parameters (Neyshabur et al., 2018; Nagarajan and Kolter, 2019), which may decrease as the model capacity increases (Arora et al., 2019b; Li and Liang, 2018; Allen-Zhu et al., 2018a; Oymak et al., 2019).”