Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

By Jeffrey Pennington, Sam Schoenholz, Surya Ganguli. 2017

307 citations recorded by the source corpus.

It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is $O(1)$ is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condition that all singular values of the Jacobian concentrate near $1$ is a property known as dynamical isometry. For deep linear networks, dynamical isometry can be achieved through orthogonal weight initialization and has been shown to dramatically speed up learning; however, it has remained unclear how to extend these results to the nonlinear setting. We address this question by employing powerful tools from free probability theory to compute analytically the entire singular value distribution of a deep network's input-output Jacobian. We explore the dependence of the singular value distribution on the depth of the network, the weight initialization, and the choice of nonlinearity. Intriguingly, we find that ReLU networks are incapable of dynamical isometry. On the other hand, sigmoidal networks can achieve isometry, but only with orthogonal weight initialization. Moreover, we demonstrate empirically that deep nonlinear networks achieving dynamical isometry learn orders of magnitude faster than networks that do not. Indeed, we show that properly-initialized deep sigmoidal networks consistently outperform deep ReLU networks. Overall, our analysis reveals that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “In recent years the dynamics of signal propagation in randomly initialized deep and wide neural networks have been formalized via mean ﬁeld theory [13, 8, 14].”
Fixup Initialization: Residual Learning Without Normalization (2019, background): “Recently, the study of dynamical isometry (Saxe et al., 2013) provides a more detailed characterization of the forward and backward signal propogation at initialization (Pennington et al., 2017; Hanin, 2018), enabling training 10,000-layer CNNs from scratch (Xiao et al., 2018).”
Sorting out Lipschitz function approximation (2018, methodology): “This property has been shown to speed up training by orders of magnitude when enforced during weight initialization [37, 43], and explored in the contexts of training RNNs [9] and very deep convolutional neural networks [54].”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “(2), they and others (Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Chen et al., 2018; Xiao et al., 2018; Pennington et al., 2017) relied on Assumption 2.”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
An analytic theory of generalization dynamics and transfer learning in deep linear networks (2018, background): “Moreover, theory developed for the learning dynamics of deep linear networks directly inspired better initialization schemes for nonlinear networks (Schoenholz et al., 2016; Pennington et al., 2017; 2018).”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “FM1 for fully connected networks has been previously studied [8, 21, 25].”
A Mean Field Theory of Batch Normalization (2019, background): “Recently, progress has been made (Poole et al., 2016; Schoenholz et al., 2016; Daniely et al., 2016; Pennington et al., 2017; Hanin & Rolnick, 2018; Yang, 2019) in this direction by considering neural networks at initialization, before any training has occurred.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, methodology): “The end-to-end Jacobian of a vanilla RNN with untied weights is in fact formally identical to the input-output Jacobian of a feedforward network, and thus the results from (Pennington et al., 2017) regarding conditions for dynamical isometry apply directly.”
The Singular Values of Convolutional Layers (2018, background): “Orthogonal initialization (Saxe et al., 2013; Pennington et al., 2017) and Parseval networks (Cisse et al., 2017) are motivated by similar considerations.”
ournal of Statistical Mechanics : J Theory and Experiment (2019, unknown): “Recent work (Pennington et al., 2017) found that inducing dynamical isometry, i.e. equilibrating the singular value distribution of the input-output Jacobian, can greatly speed up training.”
Understanding self-supervised Learning Dynamics without Contrastive Pairs (2021, background): “, 2018) in these landscapes, yielding insights like dynamical isometry (Saxe et al., 2013; Pennington et al., 2017; 2018) that lead to improved training of nonlinear deep networks.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “This signal propagation perspective can be reﬁned via random matrix theory [44, 45].”
Tensor Programs III: Neural Matrix Laws (2020, methodology): “This idea is known as dynamical isometry [36] and has been successfully applied in practice to train ultra-deep networks (as deep as 10,000 layers! [48]).”