An analytic theory of generalization dynamics and transfer learning in deep linear networks

By Andrew Kyle Lampinen, Surya Ganguli. 2018

152 citations recorded by the source corpus.

Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge may transfer across tasks, so that multi-task learning can improve generalization on individual tasks. However we lack analytic theories that can quantitatively predict how the degree of knowledge transfer depends on the relationship between the tasks. We develop an analytic theory of the nonlinear dynamics of generalization in deep linear networks, both within and across tasks. In particular, our theory provides analytic solutions to the training and testing error of deep networks as a function of training time, number of examples, network size and initialization, and the task structure and SNR. Our theory reveals that deep networks progressively learn the most important task structure first, so that generalization error at the early stopping time primarily depends on task structure and is independent of network size. This suggests any tight bound on generalization error must take into account task structure, and explains observations about real data being learned faster than random data. Intriguingly our theory also reveals the existence of a learning algorithm that proveably out-performs neural network training through gradient descent. Finally, for transfer learning, our theory reveals that knowledge transfer depends sensitively, but computably, on the SNRs and input feature alignments of pairs of tasks.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “…level of explanation, such a question may ultimately be resolved by promising theoretical analyses of neural net generalization (Arora et al., 2018; Lampinen & Ganguli, 2018; Allen-Zhu et al., 2018; Arora et al., 2019), and some more general intuitions can be gleaned from developmental psychology…”
Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks (2019, neutral): “Prior work [Saxe et al., 2013, 2014, Advani and Saxe, 2017, Saxe et al., 2018, Lampinen and Ganguli, 2019] studied the gradient dynamics of two-layer linear networks and proved a result similar to our Thm. 2.”
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “Another line of work has identified that deep linear networks gradually increase the rank during training [Arora et al., 2019a, Saxe et al., 2014, Lampinen and Ganguli, 2018, Gidel et al., 2019].”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, methodology): “The teacher-student approach has recently been used to study transfer learning , both in linear networks (Lampinen & Ganguli, 2018) and in non-linear perceptron models (Dhi-fallah & Lu, 2021), which correspond to the K = M = 1 case of our setup.”
A Theory for Knowledge Transfer in Continual Learning (2022, background): “Other theoretical frameworks in transfer learning have studied how the degree of relatedness among tasks helps transfer (Lampinen & Ganguli, 2018), and how transfer helps curriculum learning (Weinshall et al., 2018).”
Integration of New Information in Memory: New Insights from a Complementary Learning Systems Perspective (2020, methodology): “For a derivation of this formula see Lampinen and Ganguli [1].”
Implicit Regularization in Deep Matrix Factorization (2019, background): “[44, 1, 31]) could show that singular vectors are stationary if they align with the singular vectors of the gradient.”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “This observation that models progressively differentiate task structure has been leveraged to explain the generalization benefits of optimal stopping: since stronger features have a higher signal-to-noise ratio, optimal stopping captures features to the extent that they are both important and not ov”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Several works including [90, 91, 1, 60] investigate the dynamics of deep linear networks trained with squared-error loss.”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “However, their dynamical behavior shares some similarities with that of deep nonlinear networks, and their analytical tractability has given them significant attention [6, 28, 29].”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “[75, 6, 53, 7]), we study gradient flow (gradient descent with infinitesimally small learning rate): Ẇl(t) := d dtWl(t) = − ∂ ∂Wl φ(W1(t),W2(t), .”
Understanding self-supervised Learning Dynamics without Contrastive Pairs (2021, background): “…of nonconvex loss landscapes (Kawaguchi, 2016; Du & Hu, 2019; Laurent & Brecht, 2018) and nonlinear learning dynamics (Saxe et al., 2013; 2019; Lampinen & Ganguli, 2018; Arora et al., 2018) in these landscapes, yielding insights like dynamical isome-try (Saxe et al., 2013; Pennington et al.,…”
Emergent Systematic Generalization in a Situated Agent (2019, background): “Depending on the desired level of explanation, such an explanation may ultimately come from theoretical analyses of neural network generalization (Arora et al., 2018; Lampinen & Ganguli, 2018; Allen-Zhu et al., 2018; Arora et al., 2019).”
Generalization in multitask deep neural classifiers: a statistical physics approach (2019, methodology): “We will attempt to closely follow the notational conventions in [16] with the hope of establishing a common language for analyzing these sorts of problems.”
Implicit Regularization via Neural Feature Alignment (2020, background): “(4)), we interpret this aspect of the non-linear dynamics as favoring a sequentialization of learning across patterns of different complexity (‘easy patterns first’), a phenomenon analogous to one pointed out in the context of deep linear networks (Saxe et al., 2014; Lampinen et al., 2018; Gidel et ”