Deep Information Propagation

By Sam Schoenholz, Justin Gilmer, Surya Ganguli, Jascha Narain Sohl-Dickstein. 2016

429 citations recorded by the source corpus.

We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these depth scales diverges. Thus arbitrarily deep networks may be trained only sufficiently close to criticality. We show that the presence of dropout destroys the order-to-chaos critical point and therefore strongly limits the maximum trainable depth for random networks. Finally, we develop a mean field theory for backpropagation and we show that the ordered and chaotic phases correspond to regions of vanishing and exploding gradient respectively.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “In particular, a wealth of recent literature suggests that the mean field theory governing the wide network limit of fully-connected models [32, 33] extends naturally to residual networks [35], CNNs [34], RNNs [39], batch normalization [40], and to broad architectures [37].”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “This situation will occur for a poor initialization of the weights as discussed in [66], or for example if all biases are too negative on the neurons of one layer for ReLU activation function.”
ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “For some activation functions and network architectures, elaborate initialization schemes allow the network to satisfy dynamical isometry at initialization, which signiﬁcantly improves training dynamics [17, 5, 18, 19].”
Deep Neural Networks as Gaussian Processes (2017, methodology): “They are exactly the relations derived in the mean ﬁeld theory of signal propagation in fully-connected random neural networks (Poole et al. (2016); Schoenholz et al. (2017)) and also appear in the literature on compositional kernels (Cho & Saul (2009); Daniely et al. (2016)).”
Understanding Batch Normalization (2018, methodology): “Due to space limitations, we defer discussion regarding variants of batch normalization, random matrix theory, generalization as well as further related work to Appendix A in the online version [4].”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, background): “We hope Cor 2.3 will allow future works to optimize initialization hyperparameters and prevent gradient explosion/vanishing problems for modern architectures in the way of Schoenholz et al. (2017); Yang & Schoenholz (2017; 2018); Chen et al. (2018); Xiao et al. (2018); Yang et al. (2018).”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “We will follow the general methodology established in Poole et al. (2016); Schoenholz et al. (2017); Yang & Schoen-holz (2017).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, background): “The recurrence relationship defining the NNGP kernel has additionally been studied separately from the GP correspondence in (Cho and Saul, 2009; Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2016; Yang and Schoenholz, 2017; Xiao et al., 2018).”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “This situation will occur for a poor initialization of the weights, for example if all biases are too negative on the neurons of one layer for ReLU activation function (see for instance [34]).”
Mean Field Residual Networks: On the Edge of Chaos (2017, methodology): “2(a) was first made in [11] for computing the mean field theory of gradients for feedforward tanh networks.”
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice (2017, background): “The critical line ( w, b) = 1 determines the boundary between two phases [12, 13]: (a) a chaotic phase when > 1, where forward signal propagation expands and folds space in a chaotic manner and back-propagated gradients exponentially explode, and (b) an ordered phase when < 1, where forward signal p”
An analytic theory of generalization dynamics and transfer learning in deep linear networks (2018, background): “Moreover, theory developed for the learning dynamics of deep linear networks directly inspired better initialization schemes for nonlinear networks (Schoenholz et al., 2016; Pennington et al., 2017; 2018).”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “An interesting systematic approach to predicting the correct multiplicative constant in the variance of weights as a function of the non-linearity is proposed in [22, 25] (e.”
A Mean Field Theory of Batch Normalization (2019, methodology): “Neural networks that are deep enough so that their pre-activation statistics lie in this regime have been shown to be untrainable (Schoenholz et al., 2016).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “This has proved important in the study of initialisation and learning dynamics (Schoenholz et al., 2017) and expressivity (Poole et al.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “In this setting it was shown in Schoenholz et al. (2017) that in the vicinity of a ﬁxed point, the off-diagonal term in eq.”
Information Geometry of Orthogonal Initializations and Training (2018, methodology): “As pointed out in [27, 32], the conditioning of the Jacobian matrix affects the conditioning of the back-propagated gradients for all layers.”