On the Margin Theory of Feedforward Neural Networks

By Colin Wei, Jason D. Lee, Qiang Liu, Tengyu Ma. 2018

96 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, methodology): “…ﬁeld analysis to establish that for inﬁnitely wide neural networks, the empirical distribution of the neural network parameters can be described as a Wasserstein gradient ﬂow (Mei et al., 2018; Chizat & Bach, 2018a; Sirignano & Spiliopoulos, 2018; Rotskoﬀ & Vanden-Eijnden, 2018; Wei et al., 2018).”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “Compared to standard uniform convergence based generalization error bounds (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2017; Golowich et al., 2017; Dziugaite and Roy, 2017; Arora et al., 2018c; Li et al., 2018a; Wei et al., 2018), the major advantage of our result is that, our ”
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks (2019, background): “Existing generalization bounds for deep neural networks [29, 6, 27, 15, 13, 5, 24, 35, 28] based on uniform convergence usually cannot provide non-vacuous bounds [21, 13] in the over-parameterized regime.”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, unknown): “To find label errors, we choose the top ⇢ ·n examples ordered by the normalized margin: p̂(ỹ=i;x) maxj 6=i p̂(ỹ=j;x) [47].”
Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation (2019, background): “With the exception of [Wei et al., 2018], these papers analyze simplified settings and do not apply to larger neural networks.”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, unknown): “Once label errors are found, we observe ordering label errors by the normalized margin: p̂(ỹ=i;x,θ)−maxj 6=i p̂(ỹ=j;x,θ) (Wei et al., 2018) works well.”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, methodology): “In theory, the expressive power of the linearized model is roughly the same as, and thus limited to, that of the corresponding random feature space (Allen-Zhu et al., 2018a; Wei et al., 2019) or the Reproducing Kernel Hilbert Space (RKHS) (Bietti and Mairal, 2019).”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “Following Wei et al. (2018), an infinitewidth neural network specifies a distribution of neuron weights.”
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss (2019, background): “, [4, 59]) depend on the inverse of the minimum margin among all the examples.”