Generalization bounds for deep convolutional neural networks
By Philip M. Long, Hanie Sedghi. 2019
102 citations recorded by the source corpus.
We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments using CIFAR-10 with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hanie Sedghi
- Niladri S. Chatterji
- Behnam Neyshabur
- Marc Niethammer
- Roland Kwitt
- Florian Graf
- Sebastian Zeng
- Robert J. N. Baldock
In Computer Vision and Pattern Recognition
- Hanie Sedghi
- Niladri S. Chatterji
- Behnam Neyshabur
- Marc Niethammer
- Roland Kwitt
- Florian Graf
- Sebastian Zeng
- Robert J. N. Baldock
In Model Architectures
- Behnam Neyshabur
- Niladri S. Chatterji
- Marc Niethammer
- Florian Graf
- Sebastian Zeng
- Roland Kwitt
- Hanie Sedghi
- Robert J. N. Baldock
In Statistical Learning Theory
- Behnam Neyshabur
- Niladri S. Chatterji
- Hanie Sedghi
- Marc Niethammer
- Florian Graf
- Sebastian Zeng
- Roland Kwitt
Citations in the corpus
- Deep Learning Through the Lens of Example Difficulty (2021, background): “…2020; Sagun et al., 2016; Li et al., 2018; Pennington and Bahri, 2017; Sagun et al., 2018), studies of margin and effective generalization measures (Long and Sedghi, 2019; Unterthiner et al., 2020; Jiang et al., 2020, 2018; Kawaguchi et al., 2017) and mean-field studies of stochastic optimization…”
- The intriguing role of module criticality in the generalization of deep networks (2019, methodology): “Recently authors have emphasized on the role of distance to initialization rather than norm of the weights in generalization (Dziugaite and Roy, 2017; Nagarajan and Kolter, 2019b; Neyshabur et al., 2019; Long and Sedghi, 2019).”
- On Measuring Excess Capacity in Neural Networks (2022, background): “Alternatively, Long & Sedghi [31] derive a generalization bound which does not depend on l p norms or group norms, but only on the distance to initialization with respect to the spectral norm.”