Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs

By Dar Gilboa, Bo-Hao Chang, Minmin Chen, Greg Yang, Sam Schoenholz, Ed H. Chi, Jeffrey Pennington. 2019

44 citations recorded by the source corpus.

Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed to ameliorate these issues, including various algorithmic and architectural modifications. Two of the most successful RNN architectures, the LSTM and the GRU, do exhibit modest improvements over vanilla RNN cells, but they still suffer from instabilities when trained on very long sequences. In this work, we develop a mean field theory of signal propagation in LSTMs and GRUs that enables us to calculate the time scales for signal propagation as well as the spectral properties of the state-to-state Jacobians. By optimizing these quantities in terms of the initialization hyperparameters, we derive a novel initialization scheme that eliminates or reduces training instabilities. We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower. We also observe a beneficial effect on generalization performance using this new initialization.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “For some activation functions and network architectures, elaborate initialization schemes allow the network to satisfy dynamical isometry at initialization, which significantly improves training dynamics [19, 5, 20, 21].”
A Mean Field Theory of Batch Normalization (2019, background): “N E ] 5 M ar 2 01 9 recurrent networks (Chen et al., 2018; Gilboa et al., 2019) have been developed.”
Disentangling trainability and generalization in deep learning (2019, methodology): “Jacot et al. (2019) demonstrate that batch normalization helps removes the “ordered phase” (as in Yang et al. (2019)) and a layer-dependent learning rate allows every layer in a netwrok to contribute to learning.”
MetaInit: Initializing learning by learning to initialize (2019, background): “More recent contributions have included initialization schemes for fully-connected networks [52, 46], residual networks [23, 58, 63], convolutional networks [57], recurrent networks with gating [9, 19], and batch normalization [59].”