Full-Capacity Unitary Recurrent Neural Networks

By Scott Wisdom, Thomas Powers, John R. Hershey, Jonathan Le Roux, Les E. Atlas. 2016

332 citations recorded by the source corpus.

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second, we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery (2016, methodology): “A single recurrent layer of the SISTA-RNN is equivalent to another recently proposed architecture, the unitary RNN (uRNN) [7, 8], except that the SISTA-RNN neither uses a unitary constraint nor complex-valued hidden states.”
Concept whitening for interpretable image recognition (2020, background): “In the ﬁeld of deep learning, many initial works about incorporating orthogonality constraints are targeted for RNNs (Vorontsov et al., 2017; Mhammedi et al., 2017; Wisdom et al., 2016), since orthogonality could help avoid vanishing gradients or exploding gradients in RNNs.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “In the more challenging P-MNIST setting, the order of the sequence is permuted at random (Le et al., 2015; Arjovsky et al., 2016; Wisdom et al., 2016; Krueger et al., 2017).”
Dilated Recurrent Neural Networks (2017, methodology): “We follow a similar setup in Different from the settings in [2, 24], the average cross-entropy loss is only measured at the last 10 timestamps.”
Continual Learning in Low-rank Orthogonal Subspaces (2020, methodology): “Orthogonal matrices are used in RNNs for avoiding exploding/ vanishing gradient problem [Arjovsky et al., 2016, Wisdom et al., 2016, Jing et al., 2017].”
Sorting out Lipschitz function approximation (2018, methodology): “Other researchers [2, 53, 46] have explicitly parameterized square orthogonal weight matrices using, for example, Householder transformations [25].”
Spectral Norm Regularization for Improving the Generalizability of Deep Learning (2017, background): “In [1, 30], it is shown that by restricting the weight matrices in RNN to be unitary or orthogonal, that is, matrices with all singular values equal to one, the problem of diminishing and exploding gradients can be prevented and better performance can be obtained.”
Improving Variational Auto-Encoders using Householder Flow (2016, methodology): “lder matrix. Nevertheless, the motivation of the linIAF differs from ours completely. The Householder transformations (reﬂections) were also exploited in the context of learning recurrent neural nets [1, 15, 26]. They were used for modelling unitary weights instead of more ﬂexible variational poster”
On orthogonality and learning recurrent networks with long term dependencies (2017, background): “Recent work (Wisdom et al., 2016) has proposed the optimization of unitary matrices along the Stiefel manifold using geodesic gradient descent.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “(2017) advocate going beyond initialization to constrain the transition matrix to be orthogonal throughout the entire learning process either through re-parametrisation or by constraining the optimization to the Stiefel manifold (Wisdom et al., 2016).”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, background): “In [1, 43], a unitary evolution RNN was proposed where the unitary recurrent weights are defined empirically.”
Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs (2016, methodology): “In this section, we compare the performance of our Efﬁ-cient Unitary Recurrent Neural Network (EURNN) with 1. an LSTM RNN (Hochreiter & Schmidhuber, 1997), 2. a Partial Space URNN (Arjovsky et al., 2015), and 3. a Projective full-space URNN (Wisdom et al., 2016).”
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (2019, methodology): “The task accounts for predicting the log-magnitude of in-coming frames of a short-time Fourier transform (STFT) as it was ﬁrst proposed in (Wisdom et al., 2016).”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, background): “Taking advantage of sequence modeling, it may also be interesting to find further extensions with complex-valued LSTMs (Arjovsky et al., 2016; Wisdom et al., 2016).”
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs (2019, background): “…to alleviate the vanishing gradient problem by constraining the weight matrices to be exactly or approximately orthogonal (Pascanu et al., 2013; Wisdom et al., 2016; Vorontsov et al., 2017; Jose et al., 2017), or more recently by modifying some terms in the gradient (Arpit et al., 2018), while…”
Regularizing and Optimizing LSTM Language Models (2017, background): “This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).”
Convolutional Sequence Modeling Revisited (2018, methodology): “…on the toy baseline tasks that have been frequently used to evaluate sequential modeling (Hochreiter & Schmidhuber, 1997; Martens & Sutskever, 2011; Pascanu et al., 2013; Le et al., 2015; Cooijmans et al., 2016; Zhang et al., 2016; Krueger et al., 2017; Wisdom et al., 2016; Arjovsky et al., 2016).”
Revisiting Activation Regularization for Language RNNs (2017, background): “, 2016), removing the recurrent matrix entirely, while others act to restrict the capacity by parameterizing the recurrent matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016).”
Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (2020, methodology): “Training: Wisdom et al. (2016) restricted the transition unitary matrix on the Stiefel manifold via a closed-form Cayley transform.”