Diversity Leads to Generalization in Neural Networks

By Bo Xie, Yingyu Liang, Le Song. 2016

36 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “Most of existing works (1) only focus on the case when the training data is spherical Gaussian [28–43], and some of them require heavy initialization using tensor decomposition, which might fail to capture the specific structure of the input and the property of a random initialization; or (2) only c”
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “There also exists a large body of work that analyzes the optimization landscape of learning neural networks [13, 26, 30, 10, 25, 29, 6, 32, 17, 5].”
Convergence Analysis of Two-layer Neural Networks with ReLU Activation (2017, background): “The assumption that the input follows a Gaussian distribution is not necessarily true in practice (Although this is a common assumption appeared in the previous papers [5, 41, 42], and also considered plausible in [6]).”
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks (2017, background): “Under the same setting, [54] showed that critical points with large “diversity” are near global optimality.”
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers (2018, background): “1 What Can Neural Networks Provably Learn? Most existing works analyzing the learnability of neural networks [9, 12, 13, 20, 21, 28, 33, 34, 42, 43, 47, 49, 50, 56] make unrealistic assumptions about the data distribution (such as being random Gaussian), and/or make strong assumptions about the netw”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, methodology): “In this paper, we consider a large range of activation functions, but for one-hidden-layer fully-connected NNs. 3) Improper Learning.”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, methodology): “Other closely related results (Soudry & Carmon, 2016; Xie et al., 2016) also used unrealistic assumptions, are discussed in section 6, in regards to the details of our main results.”
Making Method of Moments Great Again? -- How can GANs learn the target distribution (2020, background): “The recent advance in deep learning [5, 6] that extends the study of two-layer neural networks [44, 68, 73, 35, 67, 70, 25, 79, 53, 24, 50, 72, 22, 78, 47, 49] might make this direction a not-so-remote goal.”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, background): “For such reason, it is not surprising that most existing theoretical works on the efficient learning regime of neural networks either focus on (1) two-layer networks [9, 10, 12, 22, 23, 36, 41, 41, 42, 46, 50, 51, 53, 54, 56, 58, 61] which do not have any hierarchical structure, or (2) a multi-layer”
Can SGD Learn Recurrent Neural Networks with Provable Generalization? (2019, background): “Recently there are many interesting results on analyzing the learning process of feed-forward neural networks [7, 8, 11, 16, 18–20, 26, 27, 29, 30, 32].”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “Gaussian-like — see for instance (Kawaguchi, 2016; Soudry & Carmon, 2016; Xie et al., 2016) and many others.”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “There is a large body of work on learning two-layer neural networks over the last few years, such as Kawaguchi (2016); Soudry and Carmon (2016); Xie et al. (2016); Soltanolkotabi et al. (2017); Tian (2017); Brutzkus and Globerson (2017); Boob and Lan (2017); Vempala and Wilmes (2018); Oymak and…”
Theoretical properties of the global optimizer of two layer neural network (2017, background): “Under the same setting, [25] showed that critical points with large “diversity” are near global optimal.”
Improved OOD Generalization via Adversarial Training and Pre-training (2021, background): “This is observed or proved recently for over-parameterized neural networks (Xie et al., 2017; Du et al., 2019; Allen-Zhu et al., 2019; Liu et al., 2020).”