Exploring Generalization in Deep Learning
By Behnam Neyshabur, Srinadh Bhojanapalli, David A. McAllester, Nati Srebro. 2017
1,437 citations recorded by the source corpus.
With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Sergey Levine
- Ruslan Salakhutdinov
- Mohit Bansal
- Dacheng Tao
- Jianfeng Gao
In Scaling & Learning Theory
- Tengyu Ma
- Yuanzhi Li
- Jason D. Lee
- Samet Oymak
- Colin Wei
- Behnam Neyshabur
- Quanquan Gu
- Simon Shaolei Du
Citations in the corpus
- A Constructive Prediction of the Generalization Error Across Scales (2019, background): “Theoretical error bounds: Much attention has been given to theoretical explanations of the generalization capabilities of deep neural networks (Neyshabur et al., 2017a;b; Allen-Zhu et al., 2018a;b; Arora et al., 2018).”
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Across different model families, it has also been observed that bigger models with larger model capacities not only fit the training data better but also generalize better on test time (Zhang et al., 2017; Neyshabur et al., 2017; Huang et al., 2019).”
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “The idea of searching for “flat” minima can be traced back to Hochreiter & Schmidhuber (1995), and its connection to generalization has seen significant study (Shirish Keskar et al., 2016; Dziugaite & Roy, 2017; Neyshabur et al., 2017; Dinh et al., 2017).”
- The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, methodology): “…a crude approximation of a model’s capacity, more sophisticated metrics such as those based on PAC-Bayes [McAllester, 1999, Dziugaite and Roy, 2017, Neyshabur et al., 2017b], VC dimension [Vapnik and Chervonenkis, 1971], and parameter norms [Bartlett et al., 2017, Neyshabur et al., 2017a] also…”
- Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “With the success of deep networks, there is a renewed interest in understanding capacity of the neural networks under different norm constraints or network architectures [22, 5, 44, 26].”
- Visualizing the Loss Landscape of Neural Nets (2017, background): “[30] show that these quantitative measure of sharpness are not invariant to symmetries in the network, and are thus not sufficient to determine generalization ability.”
- Certifying Some Distributional Robustness with Principled Adversarial Training (2017, background): “This problem currently persists for most learning-theoretic guarantees in deep learning, and the recent works of Bartlett et al. [2], Dziugaite and Roy [22], and Neyshabur et al. [49] attempt to mitigate this shortcoming.”
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “Naturally trained DNNs typically suffer from their overparametrization [90, 57], leading to the tendency to overfit the training data by producing jagged decision boundaries that closely fit the seen examples.”
- Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, positive): “…how weight perturbations affect the expected loss L CiCL w.r.t. to the training set [64], [65]: specifically, we follow the hints of [64], [66] and weigh the perturbation according to the magnitude of parameters ( σ i = α θ i | ), thus preventing degenerate solutions [64]. to With reference Fig.”
- Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “The overfitting problem of DNNs is often attributed to the complex loss landscape containing multiple local optima, and the sharpness of the loss landscape has been widely used to characterize the generalization gap in standard training scenarios from both theoretical and empirical perspectives [27,”
- Sorting out Lipschitz function approximation (2018, background): “The Lipschitz constant of a neural network has been connected theoretically and empirically to its generalization performance [5, 4, 35, 36, 44].”
- Generalisation in humans and deep neural networks (2018, neutral): “[30–32]).”
- The Implicit Bias of Gradient Descent on Separable Data (2017, background): “…increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models (Neyshabur et al., 2014, 2015; Zhang et al., 2017; Keskar et al., 2017; Neyshabur et al., 2017; Wilson et al., 2017).”
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks (2017, methodology): “Langford and Caruana [6], and more recently Dziugaite and Roy [4] and Neyshabur et al. [12], used PAC-Bayes bounds to analyze generalization behavior in neural networks, evaluating the KL-divergence, “perturbation error” L [ f + ] − L [ f ] , or the entire bound numerically.”
- Stronger generalization bounds for deep nets via a compression approach (2018, methodology): “Experiments were performed by training a VGG-19 architecture Simonyan and Zisserman [2014] and a AlexNet Krizhevsky et al.”
- Understanding Batch Normalization (2018, background): “We see that the normalization has much more dramatic effects on accuracy and enables larger learning rate and faster convergence, but that both the mean and variance normalization are important.”
- A Study on Overfitting in Deep Reinforcement Learning (2018, background): “Similar phenomenons are reported in supervised learning (Zhang et al., 2017), and recent work shows various approaches to explain this (Bartlett et al., 2017; Neyshabur et al., 2017; Advani & Saxe, 2017; Smith & Le, 2017; Dziugaite & Roy, 2017; Liang et al., 2017).”
- What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation (2020, background): “These results have lead to tighter theoretical bounds and, in some cases, bounds that show correlation with the actual generalization gap (see [23, 22, 17] for analyses of numerous measures).”
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks (2018, methodology): “Neyshabur et al. (2017a,b) also derived similar spectrally-normalized margin bounds for deep neural networks with ReLU activation function using PAC-Bayes approach.”
- Generalization in Deep Learning (2017, background): “This observation has been called an “apparent paradox” and led to active discussions by many researchers (Arpit et al., 2017; Zhang et al., 2017b; Krueger et al., 2017; Hoffer et al., 2017; Wu et al., 2017; Neyshabur et al., 2017; Dziugaite and Roy, 2017; Dinh et al., 2017).”