Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

By Mario Lezcano Casado, David Martínez-Rubio. 2019

251 citations recorded by the source corpus.

We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Concept whitening for interpretable image recognition (2020, background): “Other work explores ways to learn orthogonal weights or representations for all types of neural network (not just RNNs)38–41.”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, neutral): “[4] 99.20 98.13 73.42 TCN [3] 99.0 97.2 - LSTM [17, 21] 98.9 95.11 63.01 r-LSTM [42] 98.4 95.2 72.2 Dilated GRU [5] 99.0 94.6 - Dilated RNN [5] 98.0 96.1 - IndRNN [25] 99.0 96.0 - expRNN [24] 98 Baselines.”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, neutral): “We also compare to several RNN baselines designed for long-term dependencies, including the LSTM [34], GRU [17], expRNN [48], and LMU [71].”
Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (2020, background): “Also, Vorontsov et al. (2017); Wisdom et al. (2016); Lezcano-Casado & Martı́nez-Rubio (2019); Helfrich et al. (2017) perform Riemannian optimization on the group of unitary matrices to stablize RNNs training, but their technique cannot be applied to non-square parameter matrices.”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “…of the transition matrix as e.g., with the product of simpler matrices (Arjovsky et al., 2016), Givens rotations (Jing et al., 2017), Householder reﬂections (Mhammedi et al., 2017), or as exponentials of skew-symmetric matrices (Hyland and Rätsch, 2017; Lezcano-Casado and Martınez-Rubio, 2019).”
Orthogonal Over-Parameterized Training (2020, methodology): “[27, 42, 3, 75, 58, 31] use principled orthogonalization methods to guarantee the neurons are orthogonal to each other.”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, methodology): “To impose the orthogonality constraint on Σ , we use the Padé approximation with a scale-squaring trick of Lezcano-Casado & Martínez-Rubio (2019).”
SpinQuant: LLM quantization with learned rotations (2024, background): “These outliers dominate the quantization range, leaving only a few effective bits for the majority of values.”
Simplified State Space Layers for Sequence Modeling (2022, unknown): “[35] Mario Lezcano-Casado and David Martınez-Rubio.”