Neural Networks with Finite Intrinsic Dimension have no Spurious Valleys

By Luca Venturi, Afonso S. Bandeira, Joan Bruna. 2018

60 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
Gradient Descent Provably Optimizes Over-parameterized Neural Networks (2018, background): “For other activation functions, some previous works showed the landscape does have the desired geometric properties (Du and Lee, 2018; Soltanolkotabi et al., 2018; Nguyen and Hein, 2017; Kawaguchi, 2016; Haeffele and Vidal, 2015; Andoni et al., 2014; Venturi et al., 2018; Yun et al., 2018a).”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, background): “…to characterize geometric landscapes of objective functions (Safran & Shamir, 2017; Zhou & Liang, 2017; Freeman & Bruna, 2016; Hardt & Ma, 2016; Nguyen & Hein, 2017; Kawaguchi, 2016; Venturi et al., 2018; Soudry & Carmon, 2016; Du & Lee, 2018; Soltanolkotabi et al., 2018; Haeﬀele & Vidal, 2015).”
Gradient Descent Finds Global Minima of Deep Neural Networks (2018, background): “Many previous works thus try to study the optimization landscape of neural networks with different activation functions (Soudry & Hoffer, 2017; Safran & Shamir, 2018; 2016; Zhou & Liang, 2017; Freeman & Bruna, 2016; Hardt & Ma, 2016; Nguyen & Hein, 2017; Kawaguchi, 2016; Venturi et al., 2018; Soudry”
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport (2018, background): “[31] Luca Venturi, Afonso Bandeira, and Joan Bruna.”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Yet, in the regime where the number of parameters is large (often considered in practice), several numerical and rigorous works [7, 8, 9, 10, 11] suggest a diﬀerent landscape geometry where the loss function is characterized by a connected level set.”
Width Provably Matters in Optimization for Deep Linear Neural Networks (2019, background): “Many papers tried to identify the two desired geometric landscape properties of objective functions for non-linear neural networks (Freeman & Bruna, 2016; Nguyen & Hein, 2017; Venturi et al., 2018; Soudry & Carmon, 2016; Du & Lee, 2018; Soltanolkotabi et al., 2018; Haeffele & Vidal, 2017).”
On the Margin Theory of Feedforward Neural Networks (2018, background): “Haeffele & Vidal (2015), Nguyen & Hein (2017), and Venturi et al. (2018) show that for sufﬁciently overparametrized networks, all local minima are global, but do not show how to ﬁnd these minima via gradient descent.”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “L G ] 1 5 Fe b 20 and Vidal (2015); Nguyen and Hein (2017); Venturi et al. (2018) showed that for sufficiently overparametrized networks, all local minima are global, but do not show how to find these minima via gradient descent.”
Shallow Learning For Deep Networks (2018, background): “Secondly, understanding the link between shallow Neural Networks (NNs) and deep NNs is difficult: while generalization, approximation, or optimization results (Barron, 1994; Bach, 2014; Venturi et al., 2018; Neyshabur et al., 2018; Pinkus, 1999) for 1-hidden layer NNs are available, the same studies”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “…shallow Neural Networks (NNs) and deep NNs is difﬁcult: while generalization, approximation, or optimization results (Barron, 1994; Bach, 2014; Venturi et al., 2018; Neyshabur et al., 2018; Pinkus, 1999) for 1-hidden layer NNs are available, the same studies conclude that…”
Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks (2018, background): “SGD has been analyzed in some cases—for example, when the problem is known to be convex, as in the over-parameterized limit or other idealized settings [8, 9, 10, 11], there are rigorous guarantees of convergence and estimates of convergence rates [12].”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “Yet, several numerical and rigorous works [16, 43, 20, 40, 9] suggest a different landscape geometry where the loss function is characterized by a connected level set.”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “Despite the important insights on the role of over-parametrization in optimization Du et al. [2017], Arora et al. [2018], Venturi et al. [2018], the architectural bias prevails as a major factor to explain good generalization in visual classiﬁcation tasks – over-parametrized CNN models generalize…”