Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

By Minmin Chen, Jeffrey Pennington, Sam Schoenholz. 2018

126 citations recorded by the source corpus.

Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent networks after random initialization using a combination of mean field theory and random matrix theory. To simplify our discussion, we introduce a new RNN cell with a simple gating mechanism that we call the minimalRNN and compare it with vanilla RNNs. Our theory allows us to define a maximum timescale over which RNNs can remember an input. We show that this theory predicts trainability for both recurrent architectures. We show that gated recurrent networks feature a much broader, more robust, trainable region than vanilla RNNs, which corroborates recent experimental findings. Finally, we develop a closed-form critical initialization scheme that achieves dynamical isometry in both vanilla RNNs and minimalRNNs. We show that this results in significantly improvement in training dynamics. Finally, we demonstrate that the minimalRNN achieves comparable performance to its more complex counterparts, such as LSTMs or GRUs, on a language modeling task.

Subfields: Artificial Intelligence, Atomic and Molecular Physics, and Optics, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Atomic and Molecular Physics, and Optics

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “In particular, a wealth of recent literature suggests that the mean ﬁeld theory governing the wide network limit of fully-connected models (Poole et al., 2016; Schoenholz et al., 2017) extends naturally to residual networks (Yang & Schoenholz, 2017), CNNs (Xiao et al., 2018), RNNs (Chen et al., 2018”
Sorting out Lipschitz function approximation (2018, methodology): “This property has been shown to speed up training by orders of magnitude when enforced during weight initialization [37, 43], and explored in the contexts of training RNNs [9] and very deep convolutional neural networks [54].”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “, 2017; Yang & Schoenholz, 2017; 2018; Chen et al., 2018; Xiao et al., 2018; Pennington et al., 2017) relied on Assumption 2.2 (Gradient Independence Assumption). In backpropagation, whenever we multiply by W> for some weight matrix W , we multiply by an iid copy instead. that was first discovered b”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
A Mean Field Theory of Batch Normalization (2019, background): “, 2018), and gated recurrent networks (Chen et al., 2018) have been developed.”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “The line of work examining signal propagation in random deep networks (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Hanin & Rolnick, 2018; Chen et al., 2018; Yang et al., 2018) is related to the construction of the GPs we consider.”
Universal statistics of Fisher information in deep neural networks: mean field approach (2018, methodology): “One widely used approach for this purpose is to consider deep networks with random connectivity and a large width limit [2–14].”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Closely related to the discussion of CK and NTK is the signal propagation literature, which tries to understand how to prevent pathological behaviors in randomly initialized neural networks when they are deep [47, 51, 69, 68, 22, 23, 13, 70, 43, 24, 46].”
Tensor Programs III: Neural Matrix Laws (2020, background): “[7] Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz.”
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification (2019, methodology): “For such models, we have theoretical and empirical justiﬁcation that deep and non-linear architectures can enhance representation power (Telgarsky, 2016; Chen et al., 2018b; Zhou & Feng, 2018).”
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019, methodology): “[9] Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz.”
MetaInit: Initializing learning by learning to initialize (2019, background): “More recent contributions have included initialization schemes for fully-connected networks [52, 46], residual networks [23, 58, 63], convolutional networks [57], recurrent networks with gating [9, 19], and batch normalization [59].”
Information Geometry of Orthogonal Initializations and Training (2018, background): “It was shown experimentally in [10, 27, 36] that networks with these critical initial conditions train orders of magnitude faster than networks with arbitrary initializa-tions.”