Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks

By Gauthier Gidel, Francis R. Bach, Simon Lacoste-Julien. 2019

198 citations recorded by the source corpus.

When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequently, this choice can be considered as an implicit regularization for the training of over-parametrized models. In this work, we push this idea further by studying the discrete gradient dynamics of the training of a two-layer linear network with the least-squares loss. Using a time rescaling, we show that, with a vanishing initialization and a small enough step size, this dynamics sequentially learns the solutions of a reduced-rank regression with a gradually increasing rank.

Subfields: Artificial Intelligence, Computational Mechanics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Scaling & Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “Another line of work has identified that deep linear networks gradually increase the rank during training [Arora et al., 2019a, Saxe et al., 2014, Lampinen and Ganguli, 2018, Gidel et al., 2019].”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “Two-layer linear networks are widely studied in the literature on implicit regularization (Saxe et al., 2014; Gunasekar et al., 2017; Gidel et al., 2019; Arora et al., 2018).”
Implicit Regularization via Neural Feature Alignment (2020, background): “…this aspect of the non-linear dynamics as favoring a sequentialization of learning across patterns of diﬀerent complexity (‘easy patterns ﬁrst’), a phenomenon analogous to one pointed out in the context of deep linear networks (Saxe et al., 2014; Lampinen et al., 2018; Gidel et al., 2019).”
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning (2020, methodology): “[2014] and Gidel et al. [2019].”
Implicit Regularization in Deep Matrix Factorization (2019, background): “[19] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “[90, 2, 32] provide closed-form equations for the dynamics of linear networks in terms of the principal components of the input covariance matrix.”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “[25] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien.”
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks (2019, background): “Other related works include [Ma et al., 2017, Gidel et al., 2019, Arora et al., 2019a, Suggala et al., 2018, Blanc et al., 2019, Banburski et al., 2019, Neyshabur et al., 2014, 2015].”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, background): “While linear layers are often cited for implicit regularization to improve generalization of models, (Gidel et al., 2019; Saxe et al., 2019) in our settings, there is no overfitting.”