Unitary Evolution Recurrent Neural Networks

By Martín Arjovsky, Amar Shah, Yoshua Bengio. 2015

909 citations recorded by the source corpus.

Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture that learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an expressive unitary weight matrix by composing several structured matrices that act as building blocks with parameters to be learned. Optimization with this parameterization becomes feasible only when considering hidden states in the complex domain. We demonstrate the potential of this architecture by achieving state of the art results in several hard tasks involving very long-term dependencies.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Much of the progress in this area has been made by mitigating the vanishing gradients problem [14] by architectures such as LSTMs [15], GRU [3], Recurrent Highway Networks [51], Uni-tary RNNs [1] and others.”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
Regularizing and Optimizing LSTM Language Models (2017, background): “This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).”
Recurrent Batch Normalization (2016, background): “Although learning long-term dependencies is fundamentally difﬁcult (Bengio et al., 1994), its effects can be mitigated through architectural variations such as LSTM (Hochreiter & Schmidhuber, 1997), GRU (Cho et al., 2014) and i RNN/ u RNN (Le et al., 2015; Arjovsky et al., 2015).”
Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery (2016, methodology): “A single recurrent layer of the SISTA-RNN is equivalent to another recently proposed architecture, the unitary RNN (uRNN) [7, 8], except that the SISTA-RNN neither uses a unitary constraint nor complex-valued hidden states.”
Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “Others have introduced methods for recurrent neural networks encouraging the hidden activations to remain stable in norm, or constraining the recurrent weight matrix to have eigenvalues close to one (Krueger & Memisevic, 2015; Arjovsky et al., 2015; Le et al., 2015).”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “In the more challenging P-MNIST setting, the order of the sequence is permuted at random (Le et al., 2015; Arjovsky et al., 2016; Wisdom et al., 2016; Krueger et al., 2017).”
Dilated Recurrent Neural Networks (2017, methodology): “We follow a similar setup in Different from the settings in [2, 24], the average cross-entropy loss is only measured at the last 10 timestamps.”
Continual Learning in Low-rank Orthogonal Subspaces (2020, positive): “Orthogonal matrices are used in RNNs for avoiding exploding/ vanishing gradient problem [Arjovsky et al., 2016, Wisdom et al., 2016, Jing et al., 2017].”
Sorting out Lipschitz function approximation (2018, methodology): “Other researchers [2, 53, 46] have explicitly parameterized square orthogonal weight matrices using, for example, Householder transformations [25].”
Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU) (2016, background): “It is a synthetic problem for testing the ability of the neural network to capture the long-term dependencies in data [19], [12], [21].”
Spectral Norm Regularization for Improving the Generalizability of Deep Learning (2017, background): “In [1, 30], it is shown that by restricting the weight matrices in RNN to be unitary or orthogonal, that is, matrices with all singular values equal to one, the problem of diminishing and exploding gradients can be prevented and better performance can be obtained.”
Multiplicative LSTM for sequence modelling (2016, background): “, 2015), and unitary RNNs (Arjovsky et al., 2015), the learnable weights are essentially multiplicative biases, combined with special matrices that can be multiplied efficiently.”
Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs (2017, methodology): “Enforcing the Lipschitz constraint on an RNN is a topic for further research, but may be aided by use of unitary RNNs [4, 13].”
Architectural Complexity Measures of Recurrent Neural Networks (2016, methodology): “6% on p MNIST, and achieves almost the same performance as LSTM on the MNIST dataset with only 25% number of parameters (Arjovsky et al., 2015).”
Full-Capacity Unitary Recurrent Neural Networks (2016, methodology): “Thus, we can show that a previously proposed parameterization [10] cannot represent all unitary matrices larger than 7 × 7 .”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, background): “…algebraic properties to the parameters for purposes other than reducing model sizes, e.g. to make training more stable as in unitary evolution RNNs (Arjovsky et al., 2015) and in weight normalization (Salimans & Kingma, 2016), to inject inductive biases (Thomas et al., 2018), and to alter (Dinh et…”