The loss landscape of overparameterized neural networks

By Y. Cooper. 2018

83 citations recorded by the source corpus.

We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from $\mathbb{R}^n$ to $\mathbb{R}$ - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has $n$ parameters and is trained on $d$ data points, with $n>d$, we show that the locus $M$ of global minima of $L$ is usually not discrete, but rather an $n-d$ dimensional submanifold of $\mathbb{R}^n$. In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that $M$ is typically a very high-dimensional subset of $\mathbb{R}^n$.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[23] Yaim Cooper, “The loss landscape of overparameterized neural networks,” arXiv preprint arXiv:1804.”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Related arguments were recently made for a quadratic loss [11].”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, background): “Emerging evidence has attributed this need for over-pararameterization to the geometry of the high-dimensional loss landscapes of overparameterized deep neural networks (Dauphin et al., 2014; Choromanska et al., 2014; Goodfellow et al., 2014; Im et al., 2016; Wu et al., 2017; Liao & Poggio, 2017; Co”
On exponential convergence of SGD in non-convex over-parametrized learning (2018, background): “Many modern machine learning methods are over-parametrized and result in manifolds of global minima [4].”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “Yet, several numerical and rigorous works [16, 43, 20, 40, 9] suggest a different landscape geometry where the loss function is characterized by a connected level set.”
Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “The loss landscape of neural networks has attracted great research interests in the deep learning community [9, 10, 32, 12, 15, 43, 36].”
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks (2020, background): “In contrast, the set of solutions of over-parameterized systems is generically a manifold of positive dimension [10] (and indeed, systems large enough have no non-global minima [21, 27, 37]).”