The Emergence of Spectral Universality in Deep Networks

By Jeffrey Pennington, Sam Schoenholz, Surya Ganguli. 2018

185 citations recorded by the source corpus.

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network's Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Statistical Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

In Statistical Learning Theory

In Signal Processing

Citations in the corpus

ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “In recent years the dynamics of signal propagation in randomly initialized deep and wide neural networks have been formalized via mean ﬁeld theory [13, 8, 14].”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “We will then arrive at a theory for the singular value distribution of the Jacobian following Pennington et al. (2017; 2018).”
An analytic theory of generalization dynamics and transfer learning in deep linear networks (2018, background): “Moreover, theory developed for the learning dynamics of deep linear networks directly inspired better initialization schemes for nonlinear networks (Schoenholz et al., 2016; Pennington et al., 2017; 2018).”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, methodology): “and s1 is the first term in the Taylor expansion of the Stransform of the eigenvalue distribution of WWT (Pennington et al., 2018).”
Tensor Programs III: Neural Matrix Laws (2020, methodology): “A major application of FIP is in rigorously justifying a prevalent free independence assumption in the calculation of NN Jacobian singular value distribution [25, 36, 37, 43, 48], as we overview below.”
A Signal Propagation Perspective for Pruning Neural Networks at Initialization (2019, methodology): “The procedure for finding the rescaling σ(2) w for various nonlinearties are discussed in [8, 9].”
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs (2019, methodology): “…and the spectral properties of the Jacobian at initialization can be studied using mean ﬁeld theory and random matrix theory (Poole et al., 2016; Schoenholz et al., 2016; Yang & Schoenholz, 2017; 2018; Xiao et al., 2018; Pennington et al., 2017; 2018; Chen et al., 2018; Yang et al., 2019).”
Universal statistics of Fisher information in deep neural networks: mean field approach (2018, methodology): “One widely used approach for this purpose is to consider deep networks with random connectivity and a large width limit [2–14].”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, methodology): “Recent investigations are performed on the spectrum of the input-output Jacobian matrix [31], the Hessian matrix wrt the neural network weights [30], and the FIM [19, 32].”
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients? (2018, methodology): “To provide some context for our results, we contrast both our approach and contributions with the recent work [PSG17, PSG18].”
Batch normalization provably avoids ranks collapse for randomly initialised deep networks (2020, background): “A more recent line of research tries to explain this effect by the condition number of the input-output Jacobian (see, e.g., [32, 33, 23, 7]).”
Information Geometry of Orthogonal Initializations and Training (2018, methodology): “Although the random matrix predictions derived by [27, 28] closely match their experimental data, the mean ﬁeld assumption does not hold, as seen in Fig.”
Understanding self-supervised Learning Dynamics without Contrastive Pairs (2021, background): “…& Brecht, 2018) and nonlinear learning dynamics (Saxe et al., 2013; 2019; Lampinen & Ganguli, 2018; Arora et al., 2018) in these landscapes, yielding insights like dynamical isome-try (Saxe et al., 2013; Pennington et al., 2017; 2018) that lead to improved training of nonlinear deep networks.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “This signal propagation perspective can be reﬁned via random matrix theory [44, 45].”