Norm-Based Capacity Control in Neural Networks

By Behnam Neyshabur, Ryota Tomioka, Nati Srebro. 2015

685 citations recorded by the source corpus.

We investigate the capacity, convexity and characterization of a general family of norm-constrained feed-forward networks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

What Can ResNet Learn Efficiently, Going Beyond Kernels? (2019, neutral): “(14)This can found for instance in [16, 27].”
Understanding deep learning requires rethinking generalization (2016, background): “2015), but even these do not seem to explain the generalization behavior that we observe.”
Robust Large Margin Deep Neural Networks (2016, methodology): “Remark 1: To put results into perspective we compare our GE bounds to the GE bounds based on the Rademacher complexity in [33], which hold for DNNs with ReLUs.”
Exploring Generalization in Deep Learning (2017, background): “Without further conditions on weights, exponential dependence on depth is tight but the 4 dependence might be loose [19].”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, background): “Researchers have proposed norm-based generalization bounds (Bartlett & Mendelson, 2002; Bartlett et al., 2017a; Neyshabur et al., 2015; 2017; 2019; Konstantinos et al., 2017; Golowich et al., 2017; Li et al., 2018a) and compression-based bounds (Arora et al.”
Spectrally-normalized margin bounds for neural networks (2017, methodology): “This analysis can be adapted to give a Rademacher complexity analysis (Bartlett and Mendelson, 2002), and has been adapted to other norms (Neyshabur et al., 2015), although the preceding (cid:107) · (cid:107) ∞ setting is still needed to avoid extra combinatorial factors.”
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks (2017, background): “…norms (such as the Frobenius norm and elementwise ℓ 1 norms of layers), without a spectral norm dependence, all have a multiplicative dependence across layers or exponential dependence on depth Bartlett and Mendelson [3], Neyshabur et al. [11], or are for constant depth networks Bartlett [2].”
Stronger generalization bounds for deep nets via a compression approach (2018, neutral): “(Same holds for the earlier Bartlett and Mendelson [2002], Neyshabur et al. [2015b], Bartlett et al. [2017], Neyshabur et al. [2017a], Golowich et al. [2017]; see Figure 4).”
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks (2018, background): “…Anthony and Bartlett, 2009), a vast literature have recently studied the connection between the generalization performance of deep neural networks and their architectures (Neyshabur et al., 2015, 2017a,b; Bartlett et al., 2017; Golowich et al., 2017; Arora et al., 2018c; Allen-Zhu et al., 2018a).”
Generalization in Deep Learning (2017, background): “Previous theories (Sun et al., 2016; Neyshabur et al., 2015; Xie et al., 2015) characterize the generalization gap by the upper bounds on the model complexities via the norms of the weight matrices and exponential dependence on the depth 2 .”
Path-SGD: Path-Normalized Optimization in Deep Neural Networks (2015, methodology): “The (cid:96) p -path regularizer is then deﬁned as the p norm of π ( w ) [12]: The following Lemma establishes that the (cid:96) p -path regularizer corresponds to the minimum over all equivalent networks of the per-unit (cid:96) p norm: The deﬁnition (5) of the (cid:96) p -path regularizer…”
Fantastic Generalization Measures and Where to Find Them (2019, background): “…measures that we consider in this work belong to a few different families: PAC-Bayes (McAllester, 1999; Dziugaite and Roy, 2017; Neyshabur et al., 2017); VC-dimension (Vapnik and Chervonenkis, 1971); and norm-based bounds (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a).”
An analytic theory of generalization dynamics and transfer learning in deep linear networks (2018, background): “…an important generalization puzzle (Zhang et al., 2016): many existing theories that upper bound generalization error (Bartlett & Mendelson, 2002; Neyshabur et al., 2015; Dziugaite & Roy, 2017; Golowich et al., 2017; Neyshabur et al., 2017; Bartlett et al., 2017; Arora et al., 2018, e.g) in…”
Size-Independent Sample Complexity of Neural Networks (2017, background): “Neyshabur et al. [2015] also showed that this dependence can sometimes be avoided for anti-symmetric activations, but unfortunately this is a non-trivial assumption, which is not satisﬁed for common activations such as the ReLU.”
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018, methodology): “In fact the proof technique of Neyshabur et al. [20] does not allow for getting a bound with norms measured from initialization, and our new decomposition approach is the key for the tighter bound.”
Benign overfitting in linear regression (2019, background): “…have suggested viewing neural networks as ﬁnite-dimensional approximations to inﬁnite dimensional objects [28, 16, 2], and there are generalization bounds— although not for the overﬁtting regime—that are applicable to inﬁnite width deep networks with parameter norm constraints [3, 4, 32, 5, 22].”
SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data (2017, methodology): “More recent works, study alternative notions of complexity, such as Rademacher compexity (Bartlett & Mendelson, 2002; Neyshabur et al., 2015; Bartlett et al., 2017; Kawaguchi et al., 2017), Robustness (Xu & Mannor, 2012) and PAC-Bayes (Neyshabur et al., 2017b).”