Generalization in Deep Learning

By Kenji Kawaguchi, Leslie Pack Kaelbling, Yoshua Bengio. 2017

502 citations recorded by the source corpus.

This paper provides non-vacuous and numerically-tight generalization guarantees for deep learning, as well as theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also propose new open problems and discuss the limitations of our results.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Scaling & Learning Theory

Citations in the corpus

Deep Learning Scaling is Predictable, Empirically (2017, neutral): “While these theoretical and empirical results offer signiﬁcant insight about required model sizing, recent work has noted the need for more practical guidance (Zhang et al. (2017); Arpit et al. (2017); Smith and Le (2017); Kawaguchi et al. (2017)).”
Fixing Weight Decay Regularization in Adam (2017, background): “Kawaguchi et al. (2017) proposed a family of regularization techniques which are speciﬁc to the current batch and its size.”
Do CIFAR-10 Classifiers Generalize to CIFAR-10? (2018, neutral): “…• alexnet_tf [13] : Alexnet model without data augmentation • cudaconvnet [13]: Alexnet model with data augmentation • darc [11]: A ResNeXt [20] with depth 29 and an extra regularizer for promoting generalization • densenet_BC_100_12 : DenseNet [10] with batch size 32 ,…”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “This ability of deep networks to generalize well on real data, while simultaneously being able to ﬁt arbitrary data has recently received a lot of attention (e.g. Soudry et al. [2018], Dinh et al. [2017], Zhang et al. [2018], Arpit et al. [2017], Kawaguchi et al. [2017]).”
Visualizing the Loss Landscape of Neural Nets (2017, background): “[23] arguing that generalization is not directly related to the curvature of loss surfaces, and some authors proposing specialized training methods that achieve good performance with large batch sizes [19, 14, 6].”
Generalisation in humans and deep neural networks (2018, background): “…able to learn sufﬁciently general features on the training distribution to achieve a high accuracy on the i.i.d. test distribution despite having sufﬁcient capacity to completely memorise the training data [29], and considerable effort has been devoted to understand this phenomenon (e.g. [30–32]).”
Stronger generalization bounds for deep nets via a compression approach (2018, neutral): “Recently Kawaguchi et al. [2017] connects Path-Norm Neyshabur et al. [2015a] Figure 1: Attenuation of injected noise on a VGG-19 net trained on CIFAR-10.”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite & Roy, 2017).”
SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data (2017, methodology): “In parallel to our work, Kawaguchi et al. (2017) give generalization bounds for neural networks that are based on Rademacher complexity.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “As such, we need to use suitable inductive bias [1, 24] to guide the translation model to preserve the related content according to the specific requirements of different I2I applications.”
Model complexity of deep learning: a survey (2021, background): “However, it is often found that large over-parameterized neural networks exhibit good generalization capability [49, 76, 100].”
Are All Layers Created Equal? (2019, background): “This line of work resulted in improved generalization bounds for deep neural networks, see for instance Dziugaite and Roy (2016); Kawaguchi et al. (2017); Bartlett et al. (2017); Neyshabur et al. (2018, 2017); Liang et al. (2017); Arora et al. (2018); Zhou et al. (2019) and the references therein.”
Learning Compact Neural Networks with Regularization (2018, methodology): “…algorithms (Zhong et al., 2017b; Tian, 2017; Panigrahy et al., 2018; Soltanolkotabi et al., 2017; Ge et al., 2017; Janzamin et al., 2015) and the generalization properties of DNNs (Zhang et al., 2016; Hardt et al., 2015; Bartlett et al., 2017; Kawaguchi et al., 2017; Neyshabur et al., 2017).”
Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “Discussions on the generalization bound of neural networks can be found in (Bartlett et al., 2017; Neyshabur et al., 2018; 2017b; Kawaguchi et al., 2017; Neyshabur et al., 2017a; Arora et al., 2018; Zhou et al., 2019).”
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks (2019, background): “…of neural networks, many theoretical breakthroughs have been made progressively, including studying the properties of stochastic gradient descent [31], different complexity measures [46], generalization gaps [50], and many more from different model or algorithm perspectives [30, 43, 7, 51].”
Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks (2020, methodology): “(Kawaguchi, Kaelbling, and Bengio 2017) obtained models with better generalization by decreasing a regularization term related to the ERC.”