Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections

By Zakaria Mhammedi, Andrew D. Hellicar, Ashfaqur Rahman, James Bailey. 2016

146 citations recorded by the source corpus.

The problem of learning long-term dependencies in sequences using Recurrent Neural Networks (RNNs) is still a major challenge. Recent methods have been suggested to solve this problem by constraining the transition matrix to be unitary during training which ensures that its norm is equal to one and prevents exploding gradients. These methods either have limited expressiveness or scale poorly with the size of the network when compared with the simple RNN case, especially when using stochastic gradient descent with a small mini-batch size. Our contributions are as follows; we first show that constraining the transition matrix to be unitary is a special case of an orthogonal constraint. Then we present a new parametrisation of the transition matrix which allows efficient training of an RNN while ensuring that the matrix is always orthogonal. Our results show that the orthogonal constraint on the transition matrix applied through our parametrisation gives similar benefits to the unitary constraint, without the time complexity limitations.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Concept whitening for interpretable image recognition (2020, background): “In the ﬁeld of deep learning, many initial works about incorporating orthogonality constraints are targeted for RNNs (Vorontsov et al., 2017; Mhammedi et al., 2017; Wisdom et al., 2016), since orthogonality could help avoid vanishing gradients or exploding gradients in RNNs.”
Improving Variational Auto-Encoders using Householder Flow (2016, background): “The Householder transformations (reflections) were also exploited in the context of learning recurrent neural nets [1, 15, 26].”
On orthogonality and learning recurrent networks with long term dependencies (2017, background): “Hyland & Rätsch (2017) and Mhammedi et al. (2016) introduced more computationally efficient full capacity parameterizations. Harandi & Fernando (2016) also find that the use of fully connected “Stiefel layers” improves the performance of some convolutional neural networks.”
Orthogonal Over-Parameterized Training (2020, background): “Unrolling orthogonalization has been considered in different scenarios [27, 69, 56].”
FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising (2017, background): “In addition, recent studies have demonstrated the advantage of orthogonal regularization in enhancing the network generalization ability in applications of deep hashing and image classification [41], [42], [43], [44], [45].”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “…of the transition matrix as e.g., with the product of simpler matrices (Arjovsky et al., 2016), Givens rotations (Jing et al., 2017), Householder reﬂections (Mhammedi et al., 2017), or as exponentials of skew-symmetric matrices (Hyland and Rätsch, 2017; Lezcano-Casado and Martınez-Rubio, 2019).”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, background): “Many new RNN architectures, with unitary or orthogonal weight matrices to avoid gradient explosion or vanishing, have been proposed and tested on this task (Le et al., 2015; Arjovsky et al., 2016; Wisdom et al., 2016; Mhammedi et al., 2017; Trinh et al., 2018).”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “Various techniques have been proposed to stabilize RNN training (Wisdom et al., 2016; Arjovsky et al., 2015; Jing et al., 2016; Mhammedi et al., 2016; Zhang et al., 2018; Jose et al., 2017; Kusupati et al., 2019) .”
Task-Oriented Feature Distillation (2020, background): “improve the efficiency of training the RNN model while ensuring its orthogonality through a new parameterization of the transition matrix [44].”
Continual Learning with Gated Incremental Memories for sequential data processing (2020, methodology): “rely on different mechanisms to solve the vanishing gradient problem, like parameterizing recurrent connections with an orthogonal matrix [27].”
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (2017, background): “Orthogonal matrix has been actively explored in Recurrent Neural Networks (RNNs) [7, 3, 35, 33, 21].”
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (2019, methodology): “A third approach has been presented in (Mhammedi et al., 2017) via the use of Householder reflections.”