Generalization in Deep Networks: The Role of Distance from Initialization

By Vaishnavh Nagarajan, J. Kolter. 2019

97 citations recorded by the source corpus.

Why does training deep neural networks using stochastic gradient descent (SGD) result in a generalization error that does not worsen with the number of parameters in the network? To answer this question, we advocate a notion of effective model capacity that is dependent on {\em a given random initialization of the network} and not just the training algorithm and the data distribution. We provide empirical evidences that demonstrate that the model capacity of SGD-trained deep networks is in fact restricted through implicit regularization of {\em the $\ell_2$ distance from the initialization}. We also provide theoretical arguments that further highlight the need for initialization-dependent notions of model capacity. We leave as open questions how and why distance from initialization is regularized, and whether it is sufficient to explain generalization.

Subfields: Artificial Intelligence, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “In fact, empirical observations have been made and connected to generalization in [22, 1].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “This BC initialization in conjunction with gradient descent based optimization may be seen as implicitly biasing the optimized π θ to be close to the data-collection policy (Nagarajan & Kolter, 2019), and thus works as a remedy for the distribution shift problem (Ross et al., 2011).”
Fantastic Generalization Measures and Where to Find Them (2019, background): “The empirically motivated measures from prior literature that we consider are based on sharpness measure (Keskar et al., 2016); Fisher-Rao measure (Liang et al., 2017); distance of trained weights from initialization (Nagarajan and Kolter, 2019b) and path norm (Neyshabur et al., 2015a).”
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018, background): “Nagarajan and Kolter [18] also make a similar empirical observation on the signiﬁcant role of initialization, and in fact prove an initialization dependent generalization bound for linear networks.”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “In the lazy regime, generalization guarantees can be derived from the distance traveled by the parameters (Neyshabur et al., 2018; Nagarajan and Kolter, 2019), which may decrease as the model capacity increases (Arora et al., 2019b; Li and Liang, 2018; Allen-Zhu et al., 2018a; Oymak et al., 2019).”
The intriguing role of module criticality in the generalization of deep networks (2019, methodology): “Recently authors have emphasized on the role of distance to initialization rather than norm of the weights in generalization (Dziugaite and Roy, 2017; Nagarajan and Kolter, 2019b; Neyshabur et al., 2019; Long and Sedghi, 2019).”
On Measuring Excess Capacity in Neural Networks (2022, background): “Recent works range from relating generalization to weight norms [3, 14, 26, 37, 38], measures of the distance to initializations [34], implicit regularization induced by the optimization algorithm [8, 41], or model compression [2, 6, 42].”
On Warm-Starting Neural Network Training (2019, background): “Work relating initialization to generalization suggests that networks whose weights have moved far from their initialization are less likely to generalize well compared with ones that have remained relatively nearby [37].”
Physics-informed neural networks with hard constraints for inverse design (2021, background): “This could be explained by the analysis that neural networks trained with gradient descent have an implicit regularization and tend to converge to smooth solutions [35, 43, 19].”
Generalization bounds for deep convolutional neural networks (2019, background): “) Even more recently, authors have proved generalization bounds in terms of the distance from the initial setting of the weights instead of the size of the weights (Dziugaite and Roy, 2017; Bartlett et al., 2017; Neyshabur et al., 2019; Nagarajan and Kolter, 2019).”
Understanding training and generalization in deep learning by Fourier analysis (2018, result): “This discussion is consistent with other studies (Poggio et al. (2018), Nagarajan & Kolter (2017)).”
Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience (2019, background): “For example, one could modify our analysis to allow perturbations large enough to flip a small proportion of the activation states; one could potentially formulate such realistic conditions by drawing inspiration from the conditions in Neyshabur et al. (2017); Arora et al.”
Uniform convergence may be unable to explain generalization in deep learning (2019, background): “One might then wonder if such a bound could be extended to the original classifier, like it was explored in Nagarajan & Kolter (2019) for deep networks.”