Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

By Lei Wu, Zhanxing Zhu, E. Weinan. 2017

243 citations recorded by the source corpus.

It is widely observed that deep learning models with learned parameters generalize well, even with much more model parameters than the number of training samples. We systematically investigate the underlying reasons why deep neural networks often generalize well, and reveal the difference between the minima (with the same training error) that generalize well and those they don't. We show that it is the characteristics the landscape of the loss function that explains the good generalization capability. For the landscape of loss function for deep networks, the volume of basin of attraction of good minima dominates over that of poor minima, which guarantees optimization methods with random initialization to converge to good minima. We theoretically justify our findings through analyzing 2-layer neural networks; and show that the low-complexity solutions have a small norm of Hessian matrix with respect to model parameters. For deeper networks, extensive numerical evidence helps to support our arguments.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Examples of such work, where some surrogate is deﬁned for the curvature, include the debate on ﬂat vs sharp minima [16, 5, 29, 15], explanations of the efﬁcacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”
Shortcut learning in deep neural networks (2020, background): “Stochastic gradient descent and its variants bias DNNs towards learning simple functions [112, 113, 114, 115].”
Three Factors Influencing Minima in SGD (2017, methodology): “To access the loss curvature at the end of training, we computed the largest eigenvalue and we approximated the Frobenius norm of the Hessian 12 (higher values imply a sharper minimum) using the ﬁnite diﬀerence method Wu et al. (2017).”
Generalization in Deep Learning (2017, background): “This observation has been called an “apparent paradox” and has led to active discussion by many researchers (Arpit et al., 2017; Krueger et al., 2017; Hoffer et al., 2017; Wu et al., 2017; Dziugaite and Roy, 2017; Dinh et al., 2017).”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, background): “Emerging evidence has attributed this need for over-pararameterization to the geometry of the high-dimensional loss landscapes of overparameterized deep neural networks (Dauphin et al., 2014; Choromanska et al., 2014; Goodfellow et al., 2014; Im et al., 2016; Wu et al., 2017; Liao & Poggio, 2017; Co”
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects (2018, background): “Hence SGD tends to converge to flatter solutions, which typically generalize well (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Neyshabur et al., 2017; Wu & Zhu, 2017).”
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Minima and Regularization Effects (2018, background): “Moreover, these ﬂat minima typically generalize well according to various works [6, 11, 16, 21].”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “Wu et al. Wu et al. (2017) argue that full-batch gradient descent also generalizes well, suggesting that SGD is not the main cause behind generalization.”
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019, background): “…& Roy, 2017; Neyshabur et al., 2018; E et al., 2018), local properties (sharpness/ﬂatness) of loss functions at minima (Hochreiter & Schmidhuber, 1995; Keskar et al., 2016; Dinh et al., 2017; Wu et al., 2017), stability of optimization algorithms (Bousquet & Elisseeff, 2002; Xu & a r X i v : 1901 .”
Training behavior of deep neural network in frequency domain (2018, methodology): “In this section, we provide an explanation based on the F-Principle of why DNNs capable of over-fitting often generalize well in practice [7,11,28,24,10,19].”
A Walk with SGD (2018, background): “…discussed that model parameters that are in a region of ﬂatter minima generalize better Hochreiter and Schmidhuber (1997); Keskar et al. (2016); Wu et al. (2017), and that SGD ﬁnds such minima when used with small batch size and large learning rate Keskar et al. (2016); Jastrz˛ebski et al.…”
A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast (2020, background): “More work reports that learning ﬂat minima closely relate to generalization (Hardt et al., 2016; Zhang et al., 2017; Arpit et al., 2017; Hoffer et al., 2017; Dinh et al., 2017; Neyshabur et al., 2017; Wu et al., 2017; Dziugaite and Roy, 2017; Kleinberg et al., 2018).”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “[23] explored the link between sharpness of local minima and generalization through Hessian analysis [32], and [36] argued in terms of the volume of basins of attraction.”