Implicit Regularization in Deep Learning

By Behnam Neyshabur. 2017

168 citations recorded by the source corpus.

In an attempt to better understand generalization in deep learning, we study several possible explanations. We show that implicit regularization induced by the optimization method is playing a key role in generalization and success of deep learning models. Motivated by this view, we study how different complexity measures can ensure generalization and explain how optimization algorithms can implicitly regularize complexity measures. We empirically investigate the ability of these measures to explain different observed phenomena in deep learning. We further study the invariances in neural networks, suggest complexity measures and optimization algorithms that have similar invariances to those in neural networks and evaluate them on a number of learning tasks.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “…), implements an aﬃne transformation parameterised by a weight matrix, W , and a bias vector, ~b : Others have already established that, under the ℓ 2 norm, the Lipschitz constant of a fully connected layer is given by the spectral norm of the weight matrix (Miyato et al., 2017; Neyshabur, 2017).”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “Neyshabur et al. (2015); Neyshabur (2017) point out that because increasing network width does not lead to a U-shaped test error curve, there must be some form of implicit regularization controlling capacity.”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “The choice of the optimization method implicitly affects generalization performance of deep neural networks (Neyshabur, 2017).”
Implicit Geometric Regularization for Learning Shapes (2020, unknown): “The preferable local minima found by the optimization procedure could be seen as a geometric version of the well known implicit regularization phenomenon in neural network optimization (Neyshabur et al., 2014; Neyshabur, 2017; Soudry et al., 2018).”
The Optimal Ridge Penalty for Real-world High-dimensional Data Can Be Zero or Negative due to the Implicit Ridge Regularization (2020, result): “Moreover, increasing network size p can improve test-set performance even after p is large enough to ensure zero training error (Neyshabur, 2017; Nakkiran et al., 2020a), which is qualitatively similar to what we observed here.”
Bad Global Minima Exist and SGD Can Reach Them (2019, background): “In practice, SGD appears to avoid bad global minima, as different models produced by SGD from independent random initializations tend to all generalize equally well, a phenomenon attributed to an inherent bias of the algorithm to converge to models of low complexity [8, 9, 10, 11, 12, 13].”
Observational Overfitting in Reinforcement Learning (2019, background): “…modiﬁcations in SL such as overparametrization and residual connections is implicit regularization (Neyshabur et al., 2015a; Gunasekar et al., 2017; Neyshabur, 2017), as overparametrization through more layer depth and wider layers has proven to have no (cid:96) p -regularization equivalent…”
AUC Maximization in the Era of Big Data and AI: A Survey (2022, background): “Implicit regularization by gradient based methods [128] for DAM is an interesting question to be explored.”
Model complexity of deep learning: a survey (2021, background): “Some studies [75, 100] suggest that deep learning models are often overparameterized in practice and have significantly more parameters than samples.”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “[55, 57, 22]), or ones analyzing other sources of implicit regularization such as dropout (e.”