SPECTRALLY-NORMALIZED MARGIN BOUNDS FOR NEURAL NETWORKS

By Srinadh Bhojanapalli, Nati Srebro. 2018

Subfields: Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Sorting out Lipschitz function approximation (2018, methodology): “The Lipschitz constant of neural networks has been connected theoretically and empirically to generalization performance (Bartlett, 1998; Bartlett et al., 2017; Neyshabur et al., 2017; 2018; Sokolić et al., 2017).”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, background): “A line of research proposed norm-based (e.g. Bartlett et al. (2017a)) and compression-based bounds (Arora et al., 2018).”
Spectrally-normalized margin bounds for neural networks (2017, background): “Since spectrally-normalized margin bounds were first proposed in the preprint [Bartlett et al., 2017], subsequent works [Neyshabur et al.”
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks (2017, background): “See [6] for an interesting result in this direction under margin assumptions.”
A Study on Overfitting in Deep Reinforcement Learning (2018, background): “Similar phenomenons are reported in supervised learning (Zhang et al., 2017), and recent work shows various approaches to explain this (Bartlett et al., 2017; Neyshabur et al., 2017; Advani & Saxe, 2017; Smith & Le, 2017; Dziugaite & Roy, 2017; Liang et al., 2017).”
To understand deep learning we need to understand kernel learning (2018, background): “, 2016) which fit the data exactly (only the classification error is reported there, other references also report MSE (Chaudhari et al., 2016; Huang et al., 2016; Sagun et al., 2017; Bartlett et al., 2017)).”
Rademacher Complexity for Adversarially Robust Generalization (2018, background): “, 2016a), when properly normalized by the margin, one can still derive generalization bound that matches experimental observations via Rademacher complexity (Bartlett et al., 2017).”
A Kernel Perspective for Regularizing Deep Neural Networks (2018, unknown): “Other generalization bounds relying on similar quantities have been proposed for controlling complexity (Bartlett et al., 2017; Neyshabur et al., 2018), suggesting that using them for regularization is relevant even beyond our kernel perspective, as observed by Cisse et al.”
Generalization and Representational Limits of Graph Neural Networks (2020, background): “Several works have established generalization guarantees for FFNs (Bartlett et al., 2017; Golowich et al., 2018; Neyshabur et al., 2018; Zhang et al., 2018a) and RNNs (Chen et al., 2019a; Allen-Zhu and Li, 2019).”
Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path? (2018, background): “An equally important question to understanding the convergence behavior of optimization algorithms for overparameterized models is understanding their generalization capabilities this is the subject of a few interesting recent papers (Arora et al., 2018b; Bartlett et al., 2017; Belkin et al., 2018a;”
Orthogonal Deep Neural Networks (2019, background): “The fact that our bound is scale-sensitive in the sense of [6] implies that for difficult learning tasks, e.”
Uniform convergence may be unable to explain generalization in deep learning (2019, methodology): “In the left figure, we plot the bounds from (Neyshabur et al., 2018; Bartlett et al., 2017) taking into account the random initialization.”
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019, background): “Theoretical generalization bounds are usually too loose for practical tasks (Harvey et al., 2017; Neyshabur et al., 2017; Bartlett et al., 2017; Dziugaite & Roy, 2017; Zhou et al., 2018; Nagarajan & Kolter, 2018), and practical approaches to hyper-parameter optimization are often developed in an ad-”
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks (2019, background): “Since smaller norms of weight matrices often correspond to better capacity control [2, 3, 4, 8], we would like to relate the empirical PL exponent α to the empirical Frobenius norm ‖W‖F .”
Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “Discussions on the generalization bound of neural networks can be found in (Bartlett et al., 2017; Neyshabur et al., 2018; 2017b; Kawaguchi et al., 2017; Neyshabur et al., 2017a; Arora et al., 2018; Zhou et al., 2019).”
Information-Theoretic Local Minima Characterization and Regularization (2019, methodology): “…DNNs have been proposed based on norm, margin, Lipschitz constant, compression and robustness (Bartlett & Mendelson, 2002; Neyshabur et al., 2015b; Sokoli ´ c et al., 2017; Xu & Mannor, 2012; Bartlett et al., 2017; Zhou et al., 2019; Dziugaite & Roy, 2017; Arora et al., 2018; Jiang et al., 2019).”