The role of over-parametrization in generalization of neural networks

By Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, Nati Srebro. 2019

214 citations recorded by the source corpus.

Subfields: Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, background): “Researchers have proposed norm-based generalization bounds (Bartlett & Mendelson, 2002; Bartlett et al., 2017a; Neyshabur et al., 2015, 2017, 2019; Konstantinos et al., 2017; Golowich et al., 2017; Li et al., 2018a) and compression-based bounds (Arora et al., 2018).”
Invariant Causal Prediction for Block MDPs (2020, background): “…class complexity exist: the Vapnik-Chernovenkis (VC) dimension (Vapnik and Cher-vonenkis, 1971), the Lipschitz constant, and classiﬁcation margin of a neural network (Bartlett et al., 2017b), and second-order properties of the loss landscape (Neyshabur et al., 2019) are just a few of many.”
Predicting Inductive Biases of Pre-Trained Models (2021, background): “Finally, ongoing theoretical work characterizes the ability of over-parameterized networks to generalize in terms of complexity (Neyshabur et al., 2019) and implicit regularization (Blanc et al., 2020).”
Training Graph Neural Networks with 1000 Layers (2021, background): “While there is evidence that over-parameterized models generalize better (Neyshabur et al., 2019; Belkin et al., 2019), the relationship between performance and parameters is best illustrated in the context ar X iv :2 10 6.”
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers (2019, background): “ng of extremely deep networks with hundreds of layers [13, 27, 29, 33, 35]. Second, it suggests that over-parameterization is not necessary during the course of training as has been argued previously [1, 2, 5, 6, 24, 25], but rather over-parameterization is merely necessary to ﬁnd a “good” initializ”
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, background): “The expected risk over the data distribution D and the empirical risk over a training set S = { ( x i , y i ) } =1 are deﬁned as It has been shown [Neyshabur et al., 2019] that the Redemacher complexity [Bartlett and Mendel-son, 2002] for two-layer ReLU networks depends on (cid:107) w − w (0) 2 .”
Winning the Lottery with Continuous Sparsification (2019, background): “Recent theoretical work suggest that overparameterization plays a key role in network training dynamics [1] and generalization [2].”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “Others suggest new measures of capacity (Liang et al., 2017; Neyshabur et al., 2019).”
Training Over-parameterized Deep ResNet Is almost as Easy as Training a Two-layer Network (2019, background): “Neyshabur et al. [2019] use unit-wise capacity and obtain a bound on the empirical Rademacher complexity, which can provide an explanation (not rigorous argument) of the generalization for over-parameterized two-layer ReLU networks.”
Time2Vec: Learning a Vector Representation of Time (2019, background): “We hypothesize that the main reasons include combining sine functions with a powerful model ( e.g. , LSTM) and using many sine functions which reduces the distance to the goal (see, e.g. , [45]).”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, background): “Deep neural networks exhibit good optimization and generalization performance in the overparameterized regime (Zhang et al., 2016; Neyshabur et al., 2019; Arora et al., 2019; Zhang et al., 2019b), but both training and inference for large networks are computationally expensive.”
Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP (2019, background): “This phenomenon suggests that over-parameterization in DNN training is beneficial primarily due to proper initialization rather than regularization during the training process itself [1, 2, 6, 7, 20, 21].”
Understanding Generalization in Recurrent Neural Networks (2020, background): “Neyshabur et al. (2019) proved generalization bounds for two layer ReLU feedforward networks, which decreased with the increasing number of hidden unit in the network.”