On orthogonality and learning recurrent networks with long term dependencies

By Eugene Vorontsov, C. Trabelsi, Samuel Kadoury, C. Pal. 2017

265 citations recorded by the source corpus.

It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property. This paper explores issues with optimization convergence, speed and gradient stability when encouraging or enforcing orthogonality. To perform this analysis, we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation. We find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Concept whitening for interpretable image recognition (2020, methodology): “In the ﬁeld of deep learning, many initial works about incorporating orthogonality constraints are targeted for RNNs (Vorontsov et al., 2017; Mhammedi et al., 2017; Wisdom et al., 2016), since orthogonality could help avoid vanishing gradients or exploding gradients in RNNs.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “However, as was pointed out in Vorontsov et al. (2017), strictly enforcing orthogonality during training may hinder training speed and generalization performance.”
HUBERT Untangles BERT to Improve Transfer across NLP Tasks (2019, background): “Training recurrent models is usually difﬁ-cult, due to exploding or vanishing gradients, and has been studied for many years (Le et al., 2015; Vorontsov et al., 2017).”
Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (2020, methodology): “We specify an efficient iterative algorithm for estimating the Cayley transform that consists of only a few matrix multiplications, while the closed-form Cayley transform requires costly matrix inverse operations (Vorontsov et al., 2017; Wisdom et al., 2016).”
Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (2016, methodology): “…if B is the mini-batch size and T is the average length of the input sequences, then a network with n hidden units trained using other methods (Vorontsov et al., 2017; Wisdom et al., 2016; Hyland & Rätsch, 2016) that enforce orthogonality (see Section 2), would have time complexity O(BTn2 +…”
On the Learning Dynamics of Deep Neural Networks (2018, background): “Their paper is part of a larger trend focusing on the spectra of neural network weight matrices and their evolution during learning (Vorontsov et al., 2017; Odena et al., 2018; Pennington et al., 2017).”
Task-Oriented Feature Distillation (2020, background): “2 Orthogonal Loss As a desirable property in both convolutional neural networks and recurrent neural networks, the orthogonality of weights can solve the vanishing and exploding gradients problem by stabilizing the norm of gradients [61, 4, 19].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “This regularization technique is mainly used for learning the weight matrices with orthogonality constraints, for its efﬁciency in computation [4], [16], [146], [154], [155].”
Non-Hermitian physics (2020, methodology): “To remedy this, normal matrices with unit-norm eigenvalues are used as the connectivity matrix W to efficiently train the RNN [394, 395]:”