Disentangling trainability and generalization in deep learning

By Lechao Xiao, Jeffrey Pennington, Sam Schoenholz. 2019

35 citations recorded by the source corpus.

A fundamental goal in deep learning is the characterization of trainability and generalization of neural networks as a function of their architecture and hyperparameters. In this paper, we discuss these challenging issues in the context of wide neural networks at large depths where we will see that the situation simplifies considerably. To do this, we leverage recent advances that have separately shown: (1) that in the wide network limit, random networks before training are Gaussian Processes governed by a kernel known as the Neural Network Gaussian Process (NNGP) kernel, (2) that at large depths the spectrum of the NNGP kernel simplifies considerably and becomes "weakly data-dependent" and (3) that gradient descent training of wide neural networks is described by a kernel called the Neural Tangent Kernel (NTK) that is related to the NNGP. Here we show that in the large depth limit the spectrum of the NTK simplifies in much the same way as that of the NNGP kernel. By analyzing this spectrum, we arrive at a precise characterization of trainability and a necessary condition for generalization across a range of architectures including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs). In particular, we find that there are large regions of hyperparameter space where networks can only memorize the training set in the sense they reach perfect training accuracy but completely fail to generalize outside the training set, in contrast with several recent results. By comparing CNNs with- and without-global average pooling, we show that CNNs without average pooling have very nearly identical learning dynamics to FCNs while CNNs with pooling contain a correction that alters its generalization performance. We perform a thorough empirical investigation of these theoretical results and finding excellent agreement on real datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

The large learning rate phase of deep learning: the catapult mechanism (2020, background): “For a discussion of trainability and the connection to the NTK in the lazy phase see Xiao et al. (2019).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, background): “2 6 2 7 2 8 2 9 2 10 2 11 2 12 2 13 2 14 2 15 Training Set Size (Anonymous, 2020) could be the cause of numerical issues when evaluating predictive NLL for this kernels.”
Auto-scaling Vision Transformers without Training (2022, background): “In addition to L E , we also include the NTK condition number κ Θ = λ max λ min to indicate the trainability of ViTs (Chen et al., 2021e; Xiao et al., 2019; Yang, 2020; Hron et al., 2020). λ max and λ min are the largest and smallest eigenvalue of NTK matrix Θ .”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “These correspondences underlie the transferability of our KRR results to neural networks , and have been utilized in understanding trainability [Xiao et al., 2020], generalizations [Adlam and Pennington, 2020], training dynamics [Lewkowycz et al., 2020, Lewkowycz and Gur-Ari, 2020], uncertainty…”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “This is further leveraged to decouple the trainability and generalization of networks (Xiao et al., 2019).”