Geometry of Optimization and Implicit Regularization in Deep Learning

By Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, Nati Srebro. 2017

142 citations recorded by the source corpus.

We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the empirical optimization procedure can improve generalization, even if actual optimization quality is not affected. We do so by studying the geometry of the parameter space of deep networks, and devising an optimization algorithm attuned to this geometry.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “the probability that a data point outside of the training set is correctly classified) [39, 40, 6, 1].”
Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks (2019, background): “The implicit regularization provided by the choice of the optimization algorithm has recently become an active area of research in machine learning, putting lot of interest on the behavior of gradient descent on deep over-parametrized models [Neyshabur et al., 2015b, 2017, Zhang et al., 2017].”
A theory of independent mechanisms for extrapolation in generative models (2020, background): “This is in line with observations by Neyshabur et al. (2017) in ReLU networks (incoming synaptic weights of a hidden unit can be downscaled while upscaling all outgoing weights).”
Does learning require memorization? a short tale about a long tail (2019, background): “Examples of this approach in the context of DNNs can be found in [NTS15; NBMS17; BFT17; NTSS17] (and references therein).”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Yet another body of works focus on the eﬀects of sheer size of a neural network [25, 26, 27].”
Implicit Regularization in Matrix Factorization (2017, background): “In particular, implicit regularization has been shown to play a crucial role in training deep models [14, 13, 18, 11]: deep models often generalize well even when trained purely by minimizing the training error without any explicit regularization, and when there are more parameters than samples and ”
Implicit Bias of Gradient Descent on Linear Convolutional Networks (2018, background): “…crucial role in learning deep neural networks [Neyshabur et al., 2015b,a, Hochreiter and Schmidhuber, 1997, Keskar et al., 2016, Chaudhari et al., 2016, Dinh et al., 2017, Andrychowicz et al., 2016, Neyshabur et al., 2017, Zhang et al., 2017, Wilson et al., 2017, Hoffer et al., 2017, Smith, 2018].”
Stochastic Mirror Descent on Overparameterized Nonlinear Models (2019, background): “There exist a number of results that characterize the implicit regularization properties of different algorithms in different contexts [20], [21], [37]–[42].”
Interplay Between Optimization and Generalization of Stochastic Gradient Descent with Covariance Noise (2019, unknown): “The choice of an optimization algorithm and the choice of optimization metaparameters induces an implicit bias (Neyshabur et al., 2017a;b; Gunasekar et al., 2018; Wilson et al., 2017) which can lead to solutions of varying generalization quality.”
Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization (2018, background): “In particular, it has been argued that the optimization algorithms perform an implicit regularization , while optimizing the loss function, which is why the solution generalizes well (Neyshabur et al., 2017; Ma et al., 2017; Gunasekar et al., 2017; 2018; Soudry et al., 2018).”
Step Size Matters in Deep Learning (2018, background): “For example, changing the geometry of optimization was proposed in (Neyshabur et al., 2017) and a regularized descent algorithm was proposed to prevent the gradients from exploding and vanishing during training.”
Geometric compression of invariant manifolds in neural nets (2020, background): “These are practically useful limits to consider as performance generally improves with width [13, 14, 15, 16, 17], which simply comes from the fact that convergence to these asymptotic algorithms removes noise stemming from the random initialization of the weights [18, 19, 20].”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “The same phenomenon is referred to as implicit bias, implicit regularization, simplicity bias and spectral bias in several works [83, 75, 36, 74, 70, 53, 94, 10, 13, 35, 82, 66].”
From Variational to Deterministic Autoencoders (2019, background): “This is a remarkable result that follows a long series of recent confirmations that neural networks are surprisingly smooth by design [37].”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “2B-C), and beyond this point the accuracy keeps improving as the number of parameters increases [32, 31, 4], although it does so quite slowly.”
Characterizing Implicit Bias in Terms of Optimization Geometry (2018, background): “…plays a crucial role in the generalization ability of learned models, especially in deep learning as it provides effective regularization not directly speciﬁed in the objective (Neyshabur et al., 2015b,a; Zhang et al., 2017; Keskar et al., 2016; Wilson et al., 2017; Neyshabur et al., 2017).”